Intelligent Retrieval Method and System for Multimedia Materials Based on Digital Multimedia

Abstract

This application provides a method and system for intelligent retrieval of multimedia materials based on digital multimedia, relating to the field of digital multimedia information processing technology. First, multimedia materials are aggregated according to the semantic similarity of their scenes to form scene semantic clusters, with each material carrying a scene semantic tag. Then, based on the dynamic changes in scene semantics between the user's current search request text and historical search request texts, a scene semantic flow for the user's search request is constructed. Next, a mapping between the core scene semantics of the material cluster and the scene semantics of the flow unit is established and adapted for filtering, yielding initial adaptation results. The search priority is adjusted by combining scene semantic association strength and time influence weighting. Finally, the enhanced adaptation results are pushed to the user's search interface in priority order. This invention improves retrieval efficiency and accuracy, and optimizes the user experience.

Description

Technical Field

[0001] This application relates to the field of digital multimedia information processing technology, and more specifically, to a method and system for intelligent retrieval of multimedia materials based on digital multimedia. Background Technology

[0002] In today's era of digital information explosion, the quantity of digital multimedia materials is growing exponentially, encompassing various forms such as images, videos, and audio. These materials are widely used in various fields, such as film and television production, advertising, and online education. However, with the rapid increase in the number of materials, how to efficiently and accurately retrieve materials that meet user needs from massive multimedia material libraries has become an urgent problem to be solved.

[0003] Traditional multimedia content retrieval methods primarily rely on keyword matching, requiring users to input precise keywords to find relevant materials. However, this method has significant limitations. Firstly, keyword descriptions are often incomplete and inaccurate, failing to encompass all the features and semantic information of the content. For example, for an image containing a specific scene, a user may struggle to accurately describe its content using just a few simple keywords. Secondly, traditional methods neglect the semantic connections between multimedia materials and the dynamic changes in user search needs. User search needs are typically not isolated but evolve with time and context, and different materials may share similar scene semantics or thematic connections. Existing retrieval methods fail to fully utilize this information, resulting in low accuracy and relevance of search results, failing to meet the increasingly diverse and personalized search needs of users. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method and system for intelligent retrieval of multimedia materials based on digital multimedia.

[0005] According to a first aspect of this application, a method for intelligent retrieval of multimedia materials based on digital multimedia is provided, the method comprising:

[0006] Based on the set of multimedia materials to be retrieved, multiple material clusters are formed by aggregating them according to the similarity of the scene semantics to which the materials belong, resulting in multimedia material scene semantic clusters. Each material cluster corresponds to a single core scene semantics, and each material in the material cluster carries a scene semantic tag.

[0007] Based on the dynamic changes in the contextual semantics of the user's current search request text and historical search request text, a contextual semantic flow of the user's search request is constructed. The flow units are arranged in chronological order and each flow unit corresponds to the contextual semantic description of the search request, thus obtaining the contextual semantic flow of the user's search request.

[0008] Establish a mapping between the core scene semantics of the material cluster and the scene semantics of the streaming unit, perform scene semantic adaptation processing between the multimedia material scene semantic cluster and the user retrieval requirement scene semantic stream, filter the material clusters and materials within the multimedia material scene semantic clusters that are adapted to the streaming semantics, and obtain the initial adaptation results.

[0009] By combining the scene semantic association strength between materials within the material cluster and the time influence weight of scene semantics in the streaming unit, the initial adaptation results are subjected to scene semantic association enhancement processing, the retrieval priority of materials is adjusted, and enhanced adaptation results are obtained.

[0010] The multimedia materials in the enhanced adaptation results will be sorted by search priority and pushed to the user's search interface for the user to select.

[0011] According to a second aspect of this application, a multimedia material intelligent retrieval system based on digital multimedia is provided, including a machine-readable storage medium and a processor. The machine-readable storage medium stores machine-executable instructions. When the processor executes the machine-executable instructions, the multimedia material intelligent retrieval system based on digital multimedia implements the aforementioned multimedia material intelligent retrieval method based on digital multimedia.

[0012] According to a third aspect of this application, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when the computer-executable instructions are executed, the aforementioned intelligent retrieval method for multimedia materials based on digital multimedia is implemented.

[0013] Based on any of the above aspects, the technical effect of this application is as follows:

[0014] First, based on the set of multimedia materials to be retrieved, the materials are aggregated according to the similarity of their respective scene semantics, forming multiple material clusters with a single core scene semantics. This constructs a multimedia material scene semantic cluster, and the materials are tagged with scene semantics. Next, based on the dynamic changes in scene semantics of the user's current and historical search request text, a user search request scene semantic stream is constructed. The stream units are arranged chronologically and correspond to scene semantic descriptions, capturing the dynamic evolution of the user's search requests. By establishing a mapping between the core scene semantics of the material clusters and the scene semantics of the stream units, scene semantic adaptation processing is performed to filter out suitable material clusters and materials, obtaining initial adaptation results, effectively improving the targeting and accuracy of the search. Then, combining the scene semantic association strength between materials within the material clusters and the time influence weight of the scene semantics of the stream units, scene semantic association enhancement processing is performed on the initial adaptation results, adjusting the material search priority and further optimizing the search results, prioritizing materials that better meet user needs and have close semantic associations. Finally, the enhanced adaptation results are pushed to the user search interface in order of search priority for easy selection. The overall process improves the efficiency, accuracy, and user experience of multimedia material retrieval. Attached Figure Description

[0015] Figure 1 A flowchart illustrating the intelligent retrieval method for multimedia materials based on digital multimedia provided in an embodiment of this application is shown.

[0016] Figure 2 This illustration shows a schematic diagram of the component structure of the intelligent multimedia material retrieval system based on digital multimedia provided in an embodiment of this application. Detailed Implementation

[0017] Figure 1 This paper illustrates a flowchart of a multimedia material intelligent retrieval method and system based on digital multimedia provided in an embodiment of this application. The detailed steps include:

[0018] Step S110: Based on the set of multimedia materials to be retrieved, multiple material clusters are formed by aggregating them according to the similarity of the scene semantics to which the materials belong, resulting in multimedia material scene semantic clusters. Each material cluster corresponds to a single core scene semantics, and each material in the material cluster carries a scene semantic tag.

[0019] In this embodiment, the processing is based on a collection of catering-related multimedia materials to be retrieved. This collection contains a large number of images, videos, audio, and textual materials related to catering, such as pictures of dishes from different cuisines, videos of restaurant environments, audio recordings of cooking processes, and textual reviews of food. By aggregating the similarity of the semantics of the scenes to which the above materials belong, multiple material clusters are formed. Each cluster revolves around a core scene semantic, such as "Chinese formal dining scene," "Western fast food scene," and "baking and dessert scene," and each material within a cluster carries a corresponding scene semantic tag for subsequent retrieval and management.

[0020] Step S111: Divide each multimedia material in the multimedia material collection to be retrieved into scene semantic dimensions. Scene semantic dimensions include visual scene dimension, auditory scene dimension, and text scene dimension. The visual scene dimension corresponds to the scene type of the visual content of the material, the auditory scene dimension corresponds to the scene type of the auditory content of the material, and the text scene dimension corresponds to the scene type of the text description of the material. Determine the specific scene type of each multimedia material in the visual scene dimension, auditory scene dimension, and text scene dimension one by one, and obtain the scene dimension type record of each multimedia material.

[0021] For each catering-related multimedia material in the collection, a scene semantic dimension is defined. Taking a video clip of "Chinese formal meal cooking" as an example, in the visual scene dimension, the scene type presented in the visual content is analyzed, which may include "kitchen cooking scene," "table plating scene," "ingredient processing scene," etc.; in the auditory scene dimension, the scene type is determined based on the audio content, such as "vegetable chopping sound scene," "stir-frying sound scene," "kitchen utensil clattering sound scene," etc.; in the text scene dimension, if the video is accompanied by relevant text descriptions, such as "traditional braised pork cooking tutorial," then its scene type is "Chinese cooking text description scene." The above analysis is performed on each material to determine the specific scene type under the three dimensions, forming a scene dimension type record.

[0022] Step S112: Extract the scene semantic elements of each multimedia material in each scene semantic dimension. The scene semantic elements of the visual scene dimension include the types of objects, environmental features and lighting style in the visual scene. The scene semantic elements of the auditory scene dimension include the sound source, sound effect style and rhythm characteristics in the auditory scene. The scene semantic elements of the text scene dimension include the theme direction, emotional tendency and expression style in the text scene. Organize and collect the scene semantic elements of the visual scene dimension, auditory scene dimension, and text scene dimension of each material to obtain the scene semantic element set of each multimedia material.

[0023] Next, elements under each scene's semantic dimension are extracted for each piece of material. Taking the video material on "Chinese formal cooking" as an example, in the visual scene dimension, object types might include "wok," "knife," "cutting board," "pork belly," and "scallions, ginger, and garlic," while environmental features include "kitchen stove," "tile wall," and "range hood," with a lighting style of "bright natural light." In the auditory scene dimension, sound sources include "sound of knives cutting ingredients," "sizzling sound of ingredients entering the wok," and "sound of stirring and colliding," with a sound effect style of "realistic cooking environment sound effects," and a rhythmic characteristic of "slower preparation in the early stages and faster stirring in the middle." In terms of the textual scene dimension, the theme is "traditional Chinese cuisine cooking," the emotional inclination is "professional and instructional," and the expression style is "concise, clear, and step-by-step." These elements are then organized and grouped to form the scene semantic element set for this video material.

[0024] Step S113: Compare the scene semantic element sets of any two multimedia materials, count the number of overlapping elements and calculate the semantic similarity of overlapping elements, obtain the proportion of overlapping elements and the average semantic similarity of overlapping elements, and according to the preset correlation determination rules, combine the proportion of overlapping elements and the average semantic similarity of overlapping elements to obtain the scene semantic correlation degree between the two materials, complete the calculation of the scene semantic correlation degree between all materials, and obtain the scene semantic correlation degree matrix between materials.

[0025] Step S1131: Select two multimedia materials to be calculated, and extract the scene semantic element set of the first multimedia material and the scene semantic element set of the second multimedia material respectively.

[0026] In a collection of multimedia materials related to the catering industry, materials X and Y were selected for scene semantic relevance calculation. Material X is a video showcasing Chinese wedding banquet dishes, and material Y is a video showcasing Chinese birthday banquet dishes. The scene semantic elements of material X were extracted. Visual scene elements include "dragon and phoenix decorations," "round table setting," "red tablecloth," and "carved tableware." Auditory scene elements include "festive music," "guest applause," and "master of ceremonies' speech." Textual scene elements include "wedding banquet," "auspicious meaning," and "multiple hot dishes." Similarly, the scene semantic elements of material Y include visual scene elements such as "longevity peach decorations," "round dining table," "red tableware," and "Fu character ornaments." Auditory scene elements include "birthday songs," "family blessings," and "clinking glasses." Textual scene elements include "birthday banquet," "longevity noodles," and "traditional dishes."

[0027] Step S1132: Check one by one whether each element in the first material scene semantic element set has the same or similar elements in the second material scene semantic element set. Same elements refer to elements with completely identical semantics, and similar elements refer to elements with semantic similarity exceeding a preset similarity threshold. Count the total number of the same and similar elements in the two sets as the number of overlapping elements.

[0028] The semantic elements of Material X and Material Y were compared one by one. In the visual scene dimension, both Material X's "round table setting" and Material Y's "round dining table" reflect the characteristics of a round dining table setting, are semantically similar, and exceed the preset similarity threshold, thus belonging to similar elements. While both "red tablecloth" and Material Y's "red tableware" contain red elements, the former is a tablecloth and the latter is tableware, resulting in a significant semantic difference, but not exceeding the similarity threshold. "Carved tableware" and Material Y's "red tableware" have the same function but different decorations and colors, exceeding the preset threshold in semantic similarity, thus belonging to similar elements. In the auditory scene dimension, both Material X's "festive music" and Material Y's "birthday song" belong to the banquet / celebration music category, are semantically similar, and exceed the threshold, thus belonging to similar elements. "Guest applause" and Material Y's "clinking glasses" both belong to the banquet scene sounds, are semantically similar, and exceed the threshold, thus belonging to similar elements. "Master of Ceremonies' speech" and Material Y's "family blessings" both belong to the human blessing category, are semantically similar, and exceed the threshold, thus belonging to similar elements. In terms of textual context, both Material X's "multiple hot dishes" and Material Y's "traditional dishes" refer to dishes served at Chinese banquets, exhibiting semantic similarity exceeding the threshold, thus classifying them as similar elements. While "wedding banquet" and "birthday banquet" both belong to the category of Chinese banquets, their themes differ, and their semantic similarity does not exceed the threshold. The number of overlapping elements is calculated by counting all identical elements (no two elements are completely identical) and similar elements.

[0029] Step S1133: Calculate the ratio of the number of overlapping elements to the total number of elements in the first set of materials and the ratio of the number of overlapping elements to the total number of elements in the second set of materials. Take the average of the two ratios as the final percentage of overlapping elements.

[0030] Suppose that the total number of scene semantic elements in material X is A, of which A1 are visual scene elements, A2 are auditory scene elements, and A3 are text scene elements, where A = A1 + A2 + A3; and the total number of scene semantic elements in material Y is B, of which B1 are visual scene elements, B2 are auditory scene elements, and B3 are text scene elements, where B = B1 + B2 + B3. The number of overlapping elements is C. Therefore, the proportion of overlapping elements to the total number of elements in the first material's element set is C / A, and the proportion to the total number of elements in the second material's element set is C / B. The final percentage of overlapping elements is (C / A + C / B) / 2.

[0031] Step S1134: Calculate the semantic similarity for each overlapping element. The semantic similarity of the same element is set to a fixed maximum value. The semantic similarity of similar elements is obtained by comparing the semantic connotation, expression method and scene relevance of the elements. The semantic similarity of all overlapping elements is counted and the average value is calculated as the average semantic similarity of overlapping elements.

[0032] For identical elements among overlapping elements (if any), their semantic similarity is set to a fixed maximum value. For similar elements, such as "round table setting" in material X and "round dining table" in material Y, both semantically relate to a round dining table setting; one emphasizes the plating method, while the other emphasizes the table shape; and both belong to a Chinese banquet scene. These factors are considered to arrive at their semantic similarity value. Similarly, semantic similarity values ​​are calculated for other similar elements such as "carved tableware" and "red tableware," "festive music" and "birthday songs," etc. The semantic similarity values ​​of all overlapping elements are summed and then divided by the total number of overlapping elements to obtain the average semantic similarity of the overlapping elements.

[0033] Step S1135: Based on the core requirements of scene semantic retrieval, allocate the weight of element overlap ratio and the weight of average similarity. The weight of element overlap ratio is greater than the weight of average similarity, and the specific difference is set according to the preset standard.

[0034] In scene semantic retrieval of multimedia materials related to the catering industry, the core requirement is to prioritize ensuring the overlap of scene semantic elements between materials. The amount of element overlap directly reflects the degree of fit between two materials in terms of scene composition, which is the basis for judging scene association. The semantic similarity of overlapping elements then supplements the accuracy on top of this basic fit. Therefore, the weight of the element overlap percentage should be set higher than the weight of the average similarity. The specific difference should be implemented according to the pre-set standards of catering scene retrieval practices. For example, the pre-set element overlap percentage weight should be a fixed value within the range of 0.1-0.3 greater than the average similarity weight. If the element overlap percentage weight is set to 0.6, then the average similarity weight would be 0.4, with a difference of 0.2. This highlights the core impact of the amount of element overlap on scene association and can also correct the association results through the average semantic similarity, avoiding misjudgments of association caused by superficial element overlap but large semantic differences.

[0035] Step S1136: Record the percentage of overlapping elements and the average semantic similarity of overlapping elements respectively; based on the preset association determination rules, combine the percentage of overlapping elements and the average semantic similarity of overlapping elements to obtain the scene semantic association between the two materials.

[0036] In the preset relevance determination rules, the weight of the percentage of overlapping elements is W1, and the weight of the average semantic similarity of overlapping elements is W2, where W1 + W2 = 1. Therefore, the scene semantic relevance of two materials = percentage of overlapping elements × W1 + average semantic similarity of overlapping elements × W2. For example, if W1 and W2 are both 0.5, the percentage of overlapping elements is R, and the average semantic similarity of overlapping elements is S, then the scene semantic relevance = R × 0.5 + S × 0.5. For example, in the actual calculation of catering-related materials, taking material X, a video showcasing Chinese wedding banquet dishes, and material Y, a video showcasing Chinese birthday banquet dishes, as examples, assuming that the statistically overlapping element ratio R is 0.7 and the average semantic similarity S of the overlapping elements is 0.65, if weighted according to W1=0.6 and W2=0.4 (meeting the requirement in S1135 that the weight of the overlapping element ratio is greater than the weight of the average similarity), then the scene semantic relevance of the two is 0.68. This result not only reflects the high degree of overlap in the composition of scene elements, but also corrects the relevance deviation caused by the difference in the themes of "wedding banquet" and "birthday banquet" through semantic similarity, making the relevance calculation result more in line with the actual retrieval needs of catering scenarios.

[0037] Step S1137: Repeat the above steps to calculate the scene semantic correlation degree between any two materials in the multimedia material set to be retrieved. Arrange the calculation results in matrix form, with matrix rows corresponding to different materials, matrix columns corresponding to different materials, and matrix elements representing the scene semantic correlation degree between the two materials, to obtain the scene semantic correlation degree matrix between materials.

[0038] For each pair of multimedia materials in the catering category, steps S1131 to S1136 are performed to calculate the scene semantic correlation between them. The correlation values ​​are then arranged in matrix form, where the rows and columns of the matrix correspond to the materials in the material set. The element in the i-th row and j-th column of the matrix represents the scene semantic correlation between the i-th material and the j-th material, thus obtaining the scene semantic correlation matrix between the materials.

[0039] Step S1138: Check if there are any errors in the correlation calculation in the scene semantic correlation matrix between materials. If there are any values ​​that exceed the preset reasonable range, recalculate the correlation of the corresponding material pair.

[0040] The preset reasonable range for the correlation degree is between 0 and 1. Iterate through each element in the scene semantic correlation degree matrix between materials. If an element's value is found to be less than 0 or greater than 1, the correlation degree is determined to be calculated incorrectly. For the incorrect correlation degree values, reselect the corresponding two materials and recalculate their scene semantic correlation degree according to steps S1131 to S1135, and update the incorrect values ​​in the matrix.

[0041] Step S114: Starting with a single multimedia material, other materials whose semantic relevance to the scene of the starting material is not lower than the scene semantic aggregation threshold are grouped into the same temporary material cluster. Repeat the above operation until all materials are grouped into the corresponding temporary material cluster. If the relevance of any material to multiple starting materials meets the standard, then the material is grouped into the temporary material cluster corresponding to the starting material with the highest relevance value.

[0042] Starting from the set of multimedia materials related to food and beverage, a single material is selected as the starting point. The semantic relevance matrix between these materials is then examined. Other materials with a semantic relevance of at least the threshold of the semantic aggregation of the starting material are grouped into the same temporary material cluster. For example, a video of "Sichuan cuisine cooking" is selected as the starting material, and materials with the required relevance, such as videos of "spicy hot pot making" and images of "boiled fish cooking," are grouped into the same temporary cluster. This process is repeated, selecting materials not yet grouped into a cluster as new starting points each time, until all materials are grouped into their corresponding temporary material clusters. If a material has a required relevance with multiple starting materials, such as a video of "steamed fish" having a required relevance with both the starting materials in the "Cantonese cuisine cooking" cluster and the starting materials in the "seafood cuisine" cluster, the relevance values ​​of this material with each starting material are compared, and it is grouped into the temporary material cluster corresponding to the starting material with the highest relevance value.

[0043] Step S115: Statistically analyze the set of scene semantic elements of all materials in each temporary material cluster, extract the combination of scene semantic elements with the highest frequency of occurrence as the core scene semantics of the material cluster. The core scene semantics covers the key elements of the visual scene dimension, auditory scene dimension, and text scene dimension, and obtain the core scene semantic description of each temporary material cluster.

[0044] For all materials within each temporary material cluster, the frequency of each element in its scene semantic element set is statistically analyzed. Taking a temporary cluster containing multiple "Western baking" materials as an example, the statistics show that elements such as "oven," "baking pan," "flour," and "butter" appear frequently in the visual scene dimension; "stirring sound" and "oven beep" appear frequently in the auditory scene dimension; and "baking steps" and "dessert making" appear frequently in the text scene dimension. These combinations of the most frequently occurring scene semantic elements are extracted as the core scene semantics of this material cluster. Its description covers key elements in three dimensions, such as "Western baking scene: includes tools such as ovens and baking pans, sounds such as stirring and baking, and textual descriptions of dessert making steps."

[0045] Step S116: Add scene semantic tags to the materials in each temporary material cluster to obtain the scene semantic tags of each material. The tag content of the scene semantic tags includes the core scene semantics of the material cluster and the scene type of the material itself in the visual scene dimension, auditory scene dimension, and text scene dimension.

[0046] After determining the core scene semantics of each temporary material cluster, a scene semantic tag is added to each material within the cluster. The tag content includes the core scene semantics of the cluster to which the material belongs and the scene type of the material itself in three scene dimensions. For example, a "chocolate cake making" video material in the "Western baking" cluster has the following scene semantic tags: "Core scene semantics: Western baking scene; Visual scene type: oven baking scene, ingredient mixing scene; Auditory scene type: mixer working sound scene, oven heating sound scene; Text scene type: baking tutorial text description scene."

[0047] Step S117: Check if there are any temporary material clusters whose core scene semantic similarity exceeds the preset integration threshold. If the core scene semantic similarity of two temporary material clusters exceeds the preset integration threshold, merge the two temporary material clusters into a new material cluster, recalculate the core scene semantics of the new material cluster and update the scene semantic labels of the materials in the new material cluster, and delete the two temporary material clusters before merging.

[0048] Examine the core scene semantics of all temporary material clusters to determine if any clusters have similarity exceeding a preset integration threshold. For example, consider two temporary clusters: "Western-style cake baking" and "Western-style bread baking." By comparing their core scene semantics, it's found that both contain elements like "oven" and "dough" in the visual dimension, "baking-related sounds" in the auditory dimension, and "Western-style baking tutorials" in the text dimension. Their core scene semantic similarity is high, exceeding the preset integration threshold. In this case, merge these two temporary clusters into a new "Western-style baking comprehensive" material cluster. Re-analyze the scene semantic element set of all materials within the new cluster, extract the most frequently occurring element combination as the new core scene semantics, update the scene semantic tags of all materials within the new cluster, and delete the original two temporary clusters.

[0049] Step S118: Repeat the checking and merging steps until there are no material clusters that meet the integration conditions. The multiple material clusters that are finally formed together constitute a multimedia material scene semantic cluster.

[0050] Following the above-described checking and merging steps, all temporary material clusters are processed repeatedly. After each merge, the semantic similarity of the newly formed cluster to the core scenes of other clusters is checked again to see if there are any clusters that meet the integration criteria. This process is repeated multiple times until no material clusters have a core scene semantic similarity exceeding the preset integration threshold, ultimately forming multiple stable material clusters. These clusters together constitute the semantic cluster of catering-related multimedia material scenes.

[0051] Step S120: Construct a user search request scenario semantic flow based on the dynamic changes in scenario semantics between the user's current search request text and historical search request text. The flow units are arranged in chronological order and each flow unit corresponds to the scenario semantic description of the search request, thus obtaining the user search request scenario semantic flow.

[0052] By combining users' current and historical search requests for restaurant-related content, a semantic flow of user search request scenarios is constructed. For example, a user might currently search for "recommendations for Chinese restaurants suitable for family gatherings," while historical search requests might include "good places for weekend brunch" or "custom birthday cakes." By analyzing the dynamic changes in the semantic context of these search request texts, flow units are arranged chronologically, with each flow unit corresponding to a semantic description of a search request scenario, thus obtaining the semantic flow of user search request scenarios.

[0053] Step S121: Collect the user's current input search request text, record the time information of the input search request text, clean the current search request text, parse the scene semantics of the cleaned current search request text, extract descriptive words related to visual scene, auditory scene, and text scene in the current search request text, determine the scene type under each scene dimension, integrate to form the scene semantic description of the current search request, and obtain the current scene semantic flow unit by associating the input time information.

[0054] The system collects the user's current search query text related to dining, such as "creative Japanese restaurants suitable for friend gatherings," and records the time information of the text input. The search query text is then cleaned to remove irrelevant symbols and duplicate words. Next, the cleaned text is analyzed to extract descriptive terms related to the visual scene, such as "creative Japanese restaurants" and "restaurant environment," determining the scene type under the visual scene dimension as "Japanese restaurant environment scene." Descriptive terms related to the auditory scene might include "friends' chat atmosphere," classifying the scene type as "party chat sound scene." Descriptive terms related to the text scene include "suitable for friend gatherings" and "creative Japanese cuisine," classifying the scene type as "restaurant recommendation text description scene." This information is integrated to form a scene semantic description of the current search query, and linked to the input time information to obtain the current scene semantic flow unit.

[0055] Step S122: Retrieve the user's historical search request text set. The user's historical search request text set contains the request text entered by the user in each search operation within a preset time range. At the same time, obtain the search time information corresponding to each historical text and sort the historical request texts in order from earliest to latest search time to obtain an ordered historical search request text sequence.

[0056] The system retrieves a set of historical search request texts from the user within a preset time frame (e.g., the past three months). This set includes the texts the user entered each time they searched for restaurant-related content, such as "nearby high-value Western restaurants," "recommendations for a romantic Valentine's Day dinner," and "hot pot ingredient shopping list." It also obtains the search time information for each historical text and sorts these texts in ascending order of search time to form an ordered sequence of historical search request texts.

[0057] Step S123: For each historical demand text in the ordered historical retrieval demand text sequence, perform text cleaning, scene semantic dimension extraction and scene semantic element integration in sequence to form a scene semantic description corresponding to each historical demand text, and obtain multiple historical scene semantic flow units by associating their respective retrieval time information.

[0058] For each historical request text in the ordered historical retrieval request text sequence, processing is performed sequentially. Taking one of the requests, "nearby high-value Western restaurants," as an example, text cleaning is first performed to remove invalid characters. Then, the scene semantic dimension is extracted. The visual scene dimension may involve "Western restaurant environment" and "food plating," while the auditory scene dimension includes "Western restaurant background music" and "clinking sounds of cutlery." The text scene dimension includes "restaurant recommendations" and "value-for-money descriptions." The scene semantic elements are then integrated to form a scene semantic description of the historical request text, which is then linked to its retrieval time information to obtain a historical scene semantic flow unit. The above operations are performed on all historical request texts in the sequence, resulting in multiple historical scene semantic flow units.

[0059] Step S124: Compare the scene semantic descriptions of two adjacent scene semantic flow units. For each scene dimension, count the number of overlapping semantic elements and the semantic similarity of overlapping elements to obtain the element overlap ratio and the average semantic similarity of overlapping elements in each dimension. Then, based on the preset rules, synthesize the scene semantic similarity of adjacent flow units.

[0060] Step S1241: Select two adjacent scene semantic flow units in the user retrieval demand scene semantic flow, extract the scene semantic description of the preceding scene semantic flow unit and the scene semantic description of the following scene semantic flow unit respectively, and decompose the semantic elements of the visual scene dimension, auditory scene dimension and text scene dimension.

[0061] In the semantic flow of the user's search request scenario, adjacent flow units P and Q are selected. Flow unit P corresponds to the user's previous search request "high-end Chinese restaurants suitable for business banquets". After decomposing its scenario semantic description, the semantic elements of the visual scenario dimension include "luxurious decoration", "private room environment", "exquisite tableware", "lighting atmosphere", etc.; the semantic elements of the auditory scenario dimension include "light music", "quiet conversation", "the clinking of cups and plates", etc.; and the semantic elements of the text scenario dimension include "business banquet", "high-end dishes", "attentive service", "good privacy", etc. Flow unit Q corresponds to the user's current search request "high-end Western restaurants suitable for business banquets". After decomposing its scenario semantic description, the semantic elements of the visual scenario dimension include "elegant decoration", "Western-style dining table", "silver tableware", "soft lighting", etc.; the semantic elements of the auditory scenario dimension include "classical music", "quiet conversation", "the clinking of cutlery", etc.; and the semantic elements of the text scenario dimension include "business banquet", "Western cuisine", "professional service", "quiet environment", etc.

[0062] Step S1242: In the visual scene dimension, count the number of overlapping visual semantic elements between the preceding scene semantic flow unit and the following scene semantic flow unit. The number of overlapping elements includes identical elements and similar elements. Use the same method as comparing the semantic elements of the material to determine whether the elements overlap. Similarly, complete the counting of overlapping elements in the auditory scene dimension and the text scene dimension to obtain the number of overlapping elements in the visual scene dimension, auditory scene dimension and text scene dimension.

[0063] In the visual scene dimension, the semantic elements of flow unit P and flow unit Q are compared. Flow unit P's "luxurious decoration" and flow unit Q's "elegant decoration" both indicate a high-end style of decoration, are semantically similar and exceed the preset similarity threshold, thus belonging to similar elements; "exquisite tableware" and "silver tableware" both point to high-quality tableware, are semantically similar, and belong to similar elements; "lighting atmosphere" and "soft lighting" both involve lighting environment, are semantically similar, and belong to similar elements. Counting these similar elements yields the number of overlaps in the visual scene dimension. In the auditory scene dimension, flow unit P's "light music" and flow unit Q's "classical music" both belong to background music and have a relatively soothing style, and are semantically similar; "whispering" and "talking softly" have essentially the same meaning, belonging to the same element; "the soft clinking of cups and plates" and "the soft clinking of knives and forks" both belong to the sounds of tableware colliding, and are semantically similar. Counting these identical and similar elements yields the number of overlaps in the auditory scene dimension. In terms of text scene dimension, the phrase "business banquet" in flow unit P is exactly the same as "business banquet" in flow unit Q, belonging to the same element; "attentive service" and "professional service" both reflect high service quality and are semantically similar; "good privacy" and "quiet environment" both involve the quietness of the environment and are semantically similar. By counting these identical and similar elements, the number of overlaps in the text scene dimension is obtained.

[0064] Step S1243: Calculate the overlap ratio of elements in each scene dimension. The overlap ratio of the visual scene dimension is obtained by dividing the number of overlapping elements in the visual scene dimension by the sum of the number of visual scene dimension elements in the preceding and following flow units, and then multiplying by a preset ratio calculation coefficient. The overlap ratio of the auditory scene dimension is obtained by dividing the number of overlapping elements in the auditory scene dimension by the sum of the number of auditory scene dimension elements in the preceding and following flow units, and then multiplying by a preset ratio calculation coefficient. The overlap ratio of the text scene dimension is obtained by dividing the number of overlapping elements in the text scene dimension by the sum of the number of text scene dimension elements in the preceding and following flow units, and then multiplying by a preset ratio calculation coefficient.

[0065] Let CV be the number of overlapping elements in the visual scene dimension, VP be the number of visual scene elements in the pre-flow unit, and VQ be the number of visual scene elements in the post-flow unit. Let KV be the preset percentage calculation coefficient. Then, the percentage of overlapping elements in the visual scene dimension = (CV / (VP+VQ)) × KV. Similarly, let CA be the number of overlapping elements in the auditory scene dimension, AP be the number of auditory scene elements in the pre-flow unit, and AQ be the number of auditory scene elements in the post-flow unit. Let KA be the preset percentage calculation coefficient. Then, the percentage of overlapping elements in the auditory scene dimension = (CA / (AP+AQ)) × KA. And CT be the number of overlapping elements in the text scene dimension, TP be the number of text scene elements in the pre-flow unit, TQ be the number of text scene elements in the post-flow unit, and KT be the preset percentage calculation coefficient. Then, the percentage of overlapping elements in the text scene dimension = (CT / (TP+TQ)) × KT.

[0066] Step S1244: Calculate the semantic similarity of each overlapping element in each scene dimension. The similarity of the same element is a fixed maximum value. The similarity of similar elements is calculated according to the degree of semantic connotation matching. Take the average of the similarity of all overlapping elements in the visual scene dimension as the average semantic similarity of overlapping elements in the visual scene dimension. Take the average of the similarity of all overlapping elements in the auditory scene dimension as the average semantic similarity of overlapping elements in the auditory scene dimension. Take the average of the similarity of all overlapping elements in the text scene dimension as the average semantic similarity of overlapping elements in the text scene dimension.

[0067] For overlapping elements in the visual scene dimension, such as "luxury decoration" and "elegant decoration," similarity is calculated based on the similarity of their semantic connotations regarding decoration styles; "exquisite tableware" and "silver tableware" are similarized based on the matching degree of tableware quality and presentation; "lighting atmosphere" and "soft lighting" are similarized based on the similarity of the atmosphere created by the lighting. The similarity of identical elements (if any) is set to a fixed maximum value. The similarity of all overlapping elements in the visual scene dimension is summed and then divided by the number of overlapping elements to obtain the average semantic similarity of overlapping elements in the visual scene dimension. The same method is used to calculate the average semantic similarity of overlapping elements in the auditory scene dimension and the text scene dimension.

[0068] Step S1245: Assign scene dimension weights according to the importance of each scene dimension in the user's search needs, and determine the specific weight values ​​of the visual scene dimension, auditory scene dimension, and text scene dimension.

[0069] Based on users' attention to each scenario dimension in the restaurant search scenario, the weight of the visual scenario dimension is set as WV, the weight of the auditory scenario dimension is set as WA, and the weight of the textual scenario dimension is set as WT, and WV+WA+WT=1. For example, considering that users pay more attention to visual presentation and textual description when choosing a business banquet venue, WV can be set to be greater than WT, and WT greater than WA.

[0070] Step S1246: For each scene dimension, record the percentage of overlapping elements and the average semantic similarity of overlapping elements.

[0071] The overlap ratios of visual scene dimension elements (RV), auditory scene dimension elements (RA), and text scene dimension elements (RT) calculated in step S1243, as well as the average semantic similarity of overlapping visual scene dimension elements (SV), auditory scene dimension elements (SA), and text scene dimension elements (ST) calculated in step S12444, are recorded respectively.

[0072] Step S1247: Based on the preset dimensional similarity judgment rules, combine the element overlap ratio of each scene dimension with the average semantic similarity of overlapping elements to obtain the semantic matching level of each scene dimension.

[0073] In the preset dimensional similarity determination rules, the dimensional semantic matching level is jointly determined by the percentage of overlapping elements and the average semantic similarity of overlapping elements. For example, the dimensional comprehensive value is obtained by adding the percentage of overlapping elements to the average semantic similarity of overlapping elements, and the semantic matching level is determined based on the interval in which the dimensional comprehensive value falls. If the comprehensive value of the visual scene dimension is RV+SV, the higher the value, the higher the corresponding semantic matching level, which may be divided into three levels: high, medium, and low, each corresponding to a different level value. According to this rule, the semantic matching levels LV, LA, and LT for the visual scene dimension, auditory scene dimension, and text scene dimension are obtained respectively.

[0074] Step S1248: Combining the semantic matching levels and corresponding dimension weights of the visual scene dimension, auditory scene dimension, and text scene dimension, and according to the overall similarity synthesis rules, the overall scene semantic similarity of two adjacent stream units is obtained.

[0075] The overall similarity synthesis rule is: Overall scene semantic similarity of adjacent flow units = Visual scene dimension semantic matching level × WV + Auditory scene dimension semantic matching level × WA + Text scene dimension semantic matching level × WT. Substituting LV, LA, LT and WV, WA, WT into the formula, the overall scene semantic similarity of flow unit P and flow unit Q is calculated.

[0076] Step S1249: Calculate the scene semantic similarity between all adjacent flow units in the semantic flow of the user's retrieval demand scenario in turn, record the similarity value of each pair of adjacent units, and form a list of semantic similarity between adjacent flow units.

[0077] For all adjacent flow units in the semantic flow of the user's retrieval demand scenario, such as flow unit 1 and flow unit 2, flow unit 2 and flow unit 3, etc., the scenario semantic similarity between them is calculated according to the process of steps S1241 to S1248. The above similarity values ​​are arranged in order to form a list of semantic similarity between adjacent flow units.

[0078] Step S125: Arrange the current scene semantic flow unit and all historical scene semantic flow units in chronological order to form an initial scene semantic flow framework. Mark the corresponding scene semantic similarity between adjacent flow units to reflect the degree of change in scene semantics between flow units.

[0079] For example, the current scene semantic flow unit is the semantic description corresponding to "creative Japanese restaurants suitable for friends gatherings". The historical scene semantic flow units, ordered from morning to night by retrieval time, are the semantic descriptions corresponding to "good places for weekend brunch", "custom birthday cakes", "nearby cost-effective Western restaurants", and "recommendations for romantic Valentine's Day dinners". Arranging these flow units in chronological order forms the initial scene semantic flow framework. Subsequently, the previously calculated scene semantic similarity is labeled between adjacent flow units. For example, the semantic similarity between "good places for weekend brunch" and "custom birthday cakes" is 0.3, the semantic similarity between "custom birthday cakes" and "nearby cost-effective Western restaurants" is 0.25, the semantic similarity between "nearby cost-effective Western restaurants" and "recommendations for romantic Valentine's Day dinners" is 0.7, and the semantic similarity between "recommendations for romantic Valentine's Day dinners" and "creative Japanese restaurants suitable for friends gatherings" is 0.45. These similarity scores clearly reflect the semantic shift in user search needs, from "casual dining" to "customized desserts," then to "Western cuisine," "romantic banquets," and finally "creative Japanese food parties." The highest semantic similarity is found between "nearby high-value Western restaurants" and "recommendations for romantic Valentine's Day dinners," indicating that these two search needs are most closely related in terms of scenario semantics, both revolving around the "high-end Western cuisine" scenario. Conversely, the lowest semantic similarity is found between "customized birthday cakes" and "nearby high-value Western restaurants," suggesting a significant shift in user search needs at this stage.

[0080] Step S126: Check whether there are adjacent flow units in the initial scene semantic flow framework with scene semantic similarity lower than the preset optimization threshold. If the semantic similarity between any two adjacent flow units is lower than the preset optimization threshold, analyze whether there is a missing intermediate semantic transition. If there is a missing transition, supplement with transition scene semantic flow units inferred from historical retrieval behavior.

[0081] Examine the semantic similarity of all adjacent flow units in the initial scene semantic flow framework to see if any fall below a preset optimization threshold. For example, two adjacent flow units might correspond to the search queries "hot pot ingredient purchase list" and "creative Japanese restaurants suitable for friend gatherings," respectively. Their scene semantic similarity is low, below the preset optimization threshold. In this case, analyze whether there is a missing intermediate semantic transition between these two flow units. Combining this with the user's historical search behavior, such as the user browsing content related to "Japanese hot pot introduction" between these two searches, infer that there might be a transitional semantic related to "Japanese hot pot." Then, add a transitional scene semantic flow unit based on this inference to the initial framework.

[0082] Step S127: Delete duplicate scene semantic flow units in the initial scene semantic flow framework to finally form a user retrieval demand scene semantic flow arranged in chronological order and with coherent semantic changes.

[0083] After supplementing the transitional scenario semantic flow units, check whether there are duplicate scenario semantic flow units in the initial scenario semantic flow framework. Duplicate flow units refer to units with basically the same scenario semantic description and a short time interval; these duplicate units are deleted. After the above optimization process, a user retrieval requirement scenario semantic flow is finally formed, arranged in chronological order and with coherent semantic changes.

[0084] Step S130: Establish the association mapping between the core scene semantics of the material cluster and the scene semantics of the stream unit, perform scene semantic adaptation processing between the multimedia material scene semantic cluster and the user retrieval requirement scene semantic stream, filter the material clusters and materials within the multimedia material scene semantic clusters that are adapted to the stream semantics, and obtain the initial adaptation results.

[0085] Establish a mapping relationship between the core scene semantics of each material cluster in the multimedia material scene semantic cluster of catering category and the scene semantics of each flow unit in the user retrieval demand scene semantic flow. Then, perform scene semantic adaptation processing to select the material clusters and materials in them that are adapted to the flow semantics as the initial adaptation results.

[0086] Step S131: Extract the core scene semantic description of each material cluster in the multimedia material scene semantic cluster, and at the same time extract the scene semantic description of each scene semantic flow unit in the user retrieval demand scene semantic flow, and organize them into a cluster core semantic list and a flow unit semantic list respectively.

[0087] The core scene semantic descriptions of each multimedia material cluster are extracted from the scene semantic clusters, such as "Chinese formal dining scene," "Western fast food scene," and "baking and dessert scene," and these descriptions are organized into a cluster core semantic list. Simultaneously, scene semantic descriptions of each scene semantic flow unit in the user search demand scene semantic flow are extracted, such as "semantic description of recommended Chinese restaurants suitable for family gatherings" and "semantic description of good places for weekend brunch," and these are organized into a flow unit semantic list.

[0088] Step S132: Associate the visual scene dimension, auditory scene dimension, and text scene dimension of the core semantics of the material cluster with the corresponding scene dimension of the stream unit semantics, and establish a dimensional mapping relationship between the core scene semantics of the material cluster and the scene semantics of the stream unit.

[0089] The three scene dimensions (visual, auditory, and text) of the core semantics of the material cluster are associated with the corresponding scene dimensions of the streaming unit semantics. For example, the visual scene dimension of the core semantics of the "Chinese formal meal scene" cluster is associated with the visual scene dimension of the streaming unit semantics, the auditory scene dimension is associated with the auditory scene dimension of the streaming unit semantics, and the text scene dimension is associated with the text scene dimension of the streaming unit semantics. This establishes a dimensional mapping relationship between the core scene semantics of the material cluster and the scene semantics of the streaming unit.

[0090] Step S133: Compare the semantic elements of the core scene semantics of the material cluster with the semantic elements of the scene semantics of the streaming unit in the corresponding dimensions. For each scene dimension, count the number of overlapping elements and the semantic similarity of overlapping elements to obtain the element overlap rate and the average element similarity of each dimension. According to the preset rules, combine the element overlap rate, the average element similarity and the dimension weight of each dimension to calculate the overall scene semantic fit between the core scene semantics of the material cluster and the scene semantics of the streaming unit.

[0091] Step S1331: Select the core scene semantic description of a material cluster and the scene semantic description of a scene semantic flow unit, and decompose them into semantic elements of visual scene dimension, auditory scene dimension, and text scene dimension, respectively, to form a cluster dimension semantic element list and a flow unit dimension semantic element list.

[0092] We selected a material cluster M (with the core scene semantics of "high-end business Western dining scene") and a scene semantic flow unit N (corresponding to the search requirement "high-end Western restaurants suitable for business banquets"). The core scene semantic description of material cluster M was broken down into visual scene dimension semantic elements, including "elegant Western-style decor," "white tablecloth," "metal cutlery," and "crystal chandelier," etc.; auditory scene dimension semantic elements, including "piano music," "soft conversation," and "the soft clinking of cutlery," etc.; and text scene dimension semantic elements, including "business banquet," "Western formal dining," "high-end service," and "exquisite plating," forming a list of cluster dimension semantic elements. Similarly, the scene semantic description of scene semantic flow unit N was broken down into visual scene dimension semantic elements, including "elegant decor," "Western-style dining table," "silver cutlery," and "soft lighting," etc.; auditory scene dimension semantic elements, including "classical music," "soft conversation," and "the soft clinking of cutlery," etc.; and text scene dimension semantic elements, including "business banquet," "Western cuisine," "professional service," and "quiet environment," forming a list of flow unit dimension semantic elements.

[0093] Step S1332: In the visual scene dimension, compare the cluster visual semantic elements with the stream unit visual semantic elements, count the number of overlapping elements, the number of overlapping elements includes identical elements and similar elements, calculate the ratio of the number of overlapping elements to the total number of cluster visual elements as the visual element overlap rate, and at the same time calculate the average semantic similarity of overlapping elements as the average visual element similarity.

[0094] In the visual scene dimension, elements are compared between the semantic element lists of the cluster dimension and the semantic element lists of the stream unit dimension. "Western-style elegant decoration" in cluster M and "elegant decoration" in stream unit N are semantically similar and belong to similar elements; "metal tableware" and "silver tableware" both refer to high-quality tableware and are semantically similar and belong to similar elements; "crystal chandelier" and "soft lighting" both involve lighting and are semantically similar and belong to similar elements. The number of these similar elements is counted to obtain the number of overlapping elements (CVM) in the visual scene dimension. The total number of visual elements in cluster M is VM, so the visual element overlap rate (RVM) = CVM / VM. The semantic similarity of each overlapping element is calculated, such as the semantic similarity between "Western-style elegant decoration" and "elegant decoration," and the semantic similarity between "metal tableware" and "silver tableware." These similarities are summed and divided by the number of overlapping elements to obtain the mean visual element similarity (SVM).

[0095] Step S1333: In the auditory scene dimension, compare the auditory semantic elements of the cluster with the auditory semantic elements of the stream unit, count the number of overlapping elements, the number of overlapping elements includes identical elements and similar elements, calculate the proportion of the number of overlapping elements to the total number of auditory elements in the cluster as the auditory element overlap rate, and at the same time calculate the average semantic similarity of overlapping elements as the auditory element average.

[0096] In the auditory scene dimension, "piano performance" in cluster M and "classical music" in stream unit N both belong to elegant background music and are semantically similar, thus belonging to similar elements; "whispering" and "whispering" are semantically essentially the same, belonging to the same element; "the soft clinking of cutlery" and "the soft clinking of knives and forks" are semantically similar, belonging to similar elements. The number of overlapping elements (CAM) is calculated, the total number of auditory elements in cluster M is AM, and the auditory element overlap rate (RAM) is calculated as CAM / AM. The semantic similarity of overlapping elements is calculated, such as the semantic similarity between "piano performance" and "classical music," and the semantic similarity between "whispering" and "whispering" (with a fixed maximum value for the same element), and the average value is taken to obtain the mean auditory element similarity (SAM).

[0097] Step S1334: In the text scene dimension, compare the semantic elements of the clustered text with the semantic elements of the stream unit text, count the number of overlapping elements, the number of overlapping elements includes identical elements and similar elements, calculate the proportion of the number of overlapping elements to the total number of clustered text elements as the text element overlap rate, and at the same time calculate the average semantic similarity of overlapping elements as the average text element similarity.

[0098] In terms of text context, "business banquet" in cluster M is identical to "business banquet" in stream unit N, belonging to the same element; "high-end service" and "professional service" are semantically similar, belonging to similar elements; "exquisite plating" is semantically similar to "exquisite dishes" or other text elements that may exist in stream unit N, belonging to similar elements. The number of overlapping elements (CTM) is calculated, the total number of text elements in cluster M is TM, and the text element overlap rate (RTM) is calculated as CTM / TM. The semantic similarity of overlapping elements is calculated, with identical elements taking a fixed maximum value, and similar elements calculated based on semantic matching degree, with the average value taken to obtain the mean text element similarity (STM).

[0099] Step S1335: For each scene dimension (visual, auditory, and text), record the element overlap rate and the average element similarity.

[0100] The visual element overlap rate (RVM) and visual element similarity mean (SVM) obtained in step S1332, the auditory element overlap rate (RAM) and auditory element similarity mean (SAM) obtained in step S1333, and the text element overlap rate (RTM) and text element similarity mean (STM) obtained in step S1334 are recorded respectively.

[0101] Step S1336: Based on the preset dimensional adaptation judgment rules, combine the element overlap rate and the average element similarity of each scene dimension to obtain the adaptation level of each scene dimension.

[0102] The pre-defined rules for determining dimensional fit stipulate that the dimensional fit level is jointly determined by the element overlap rate and the mean element similarity. For example, the element overlap rate and the mean element similarity are added together to obtain the dimensional fit comprehensive value. The fit level is then divided into three levels: high, medium, and low, each corresponding to a numerical value. For the visual scene dimension, the comprehensive fit value is calculated as RVM + SVM, and the visual dimension fit level (LV) is determined based on the range of this value. Similarly, the auditory scene dimension's comprehensive fit value is calculated as RAM + SAM, determining the auditory dimension fit level (LA); and the text scene dimension's comprehensive fit value is calculated as RTM + STM, determining the text dimension fit level (LT).

[0103] Step S1337: Combining the adaptation levels and corresponding dimension weights of the visual scene dimension, auditory scene dimension, and text scene dimension, and according to the overall adaptation degree synthesis rules, the overall scene semantic adaptation degree of the core scene semantics of the material cluster and the scene semantics of the streaming unit is obtained.

[0104] The overall scene semantic fit rule is: Overall Scene Semantic Fit = LV × WV + LA × WA + LT × WT, where WV, WA, and WT are the weights of the visual, auditory, and textual scene dimensions, respectively, and WV + WA + WT = 1. Substituting the visual dimension fit level LV, auditory dimension fit level LA, and textual dimension fit level LT, along with their corresponding weights, into the formula, the overall scene semantic fit between the core scene semantics of the material cluster M and the scene semantics of the streaming unit N is calculated.

[0105] Step S1338: Record the overall scene semantic fit between the core semantics of the current material cluster and the semantics of the current stream unit. Repeat the above steps to calculate the fit between the core semantics of the material cluster and the semantics of all stream units in the semantic stream of the user's retrieval needs scene.

[0106] Record the calculated overall scene semantic fit between the material cluster M and the stream unit N. Then, select the next stream unit in the user retrieval requirement scene semantic stream, and calculate and record the overall scene semantic fit between the material cluster M and the stream unit according to steps S1331 to S1337. Repeat this process until the semantic fit between the material cluster M and the semantic fit between all stream units in the user retrieval requirement scene semantic stream is calculated.

[0107] Step S1339: Replace the material cluster, and calculate the compatibility between the core semantics of the newly selected material cluster and the semantics of all streaming units according to the same process, until the compatibility calculation between all material clusters and all streaming units in the multimedia material scene semantic cluster is completed.

[0108] Select the next material cluster P in the multimedia material scene semantic cluster, and calculate the fit degree between material cluster P and the semantics of all stream units in the user retrieval requirement scene semantic stream, following steps S1331 to S1338. Repeat this process until all material clusters in the multimedia material scene semantic cluster have completed the fit degree calculation with the semantics of all stream units.

[0109] Step S13310: Organize all adaptation calculation results to form a scene semantic adaptation matrix with material clusters as rows and stream units as columns. The matrix elements are the overall scene semantic adaptation values ​​of the corresponding material cluster and the corresponding stream unit.

[0110] The calculation results of the overall scene semantic fit between all material clusters and all stream units are organized and constructed into a matrix. The rows of the matrix represent different material clusters, and the columns represent different stream units in the semantic flow of the user's retrieval needs. Each element in the matrix is ​​the overall scene semantic fit value between the corresponding material cluster and the corresponding stream unit, forming a scene semantic fit matrix.

[0111] Step S134: For each material cluster, select the stream unit with the highest adaptation value to the core semantics of the cluster as the optimal adaptation unit for that material cluster, and record the adaptation value between the material cluster and the optimal adaptation unit, as well as the time information of the corresponding stream unit.

[0112] For each content cluster, calculate the overall scene semantic fit between its core scene semantics and the scene semantics of all stream units in the user retrieval demand scene semantic stream. Then, select the stream unit with the highest fit score from these values; this stream unit is the optimal fit unit for that content cluster. Record the fit score between the content cluster and the optimal fit unit, as well as the corresponding time information for that stream unit. For example, the "Chinese formal dining scene" cluster has the highest fit score with the "Recommendations of Chinese Restaurants Suitable for Family Gatherings" stream unit; record this score and the retrieval time for that stream unit.

[0113] Step S135: Select material clusters with optimal adaptability not lower than the scene semantic adaptability threshold. The selected material clusters are used as material clusters that adapt to the semantic flow of the user's search needs scene, and a list of adapted material clusters is obtained.

[0114] Set a scene semantic adaptation threshold, and compare the optimal adaptation value of each material cluster with this threshold. Select material clusters whose optimal adaptation value is not lower than the threshold. These clusters are the material clusters that are adapted to the semantic flow of the user's retrieval needs scene, and organize them into a list of adapted material clusters.

[0115] Step S136: Extract all multimedia materials in each material cluster in the adapted material cluster list, collect the scene semantic tags and basic material information of each material. The basic material information includes material name, material type, and material duration. The material duration is only included in video or audio materials.

[0116] Extract all multimedia materials from the list of adapted material clusters. For each material, collect its scene semantic tags and basic material information, such as material name "Chinese family dinner dish display", material type may be image, video, audio or text-based, and for video or audio materials, also include material duration information.

[0117] Step S137: Calculate the correlation between each material in the adaptation material cluster and the core scene semantics of the material cluster. After deleting materials with a correlation lower than the average correlation within the material cluster, integrate all materials retained in the adaptation material cluster, classify and organize them according to their respective material clusters, and form an initial adaptation result containing basic material information, scene semantic tags and information of the material cluster to which they belong.

[0118] The correlation between each material within the adaptation material cluster and the core scene semantics of that cluster is statistically analyzed. This correlation is derived from the degree of matching between the scene semantic elements of the material and the core scene semantic elements of the cluster. The average correlation of all materials within each cluster is calculated, and materials with a correlation below this average are removed from the cluster. Then, all materials retained within the adaptation material clusters are integrated and categorized according to their respective material clusters to form an initial adaptation result. This initial adaptation result includes basic information about the materials, scene semantic tags, and information about the material cluster to which they belong.

[0119] Step S140: Combining the scene semantic association strength between materials within the material cluster and the time influence weight of the scene semantics of the streaming unit, the initial adaptation results are subjected to scene semantic association enhancement processing, the material retrieval priority is adjusted, and the enhanced adaptation results are obtained.

[0120] Based on the initial adaptation results, the scene semantic association strength between materials within the material cluster and the time influence weight of the scene semantics of the streaming unit are combined to strengthen the scene semantic association of the materials, thereby adjusting the retrieval priority of the materials and obtaining the enhanced adaptation results.

[0121] Step S141: Extract the scene semantic association strength between all materials in each adapted material cluster in the initial adaptation result, and organize them into a list of material association strength within the cluster by material pairs. The scene semantic association strength comes from the scene semantic association degree matrix between materials calculated when constructing the multimedia material scene semantic cluster.

[0122] From the initial adaptation results, for each adapted material cluster, the scene semantic association strength between all materials within that cluster is extracted. This association strength data originates from the scene semantic association degree matrix between materials calculated during the previous construction of multimedia material scene semantic clusters. These association strengths are then organized by material pairs, such as the association strength between material A and material B, the association strength between material A and material C, etc., forming a list of material association strengths within the cluster.

[0123] Step S142: Determine the time influence weight of each stream unit based on the difference between the time corresponding to the stream unit and the current time. If the time difference decreases, the time influence weight increases. Calculate the specific weight value through the correspondence between the time difference and the preset weight coefficient to obtain the time influence weight of each scene semantic stream unit in the user retrieval demand scene semantic stream.

[0124] For each semantic flow unit in the user retrieval request scenario semantic flow, the weight of time influence is determined based on the difference between its corresponding time and the current time. The smaller the time difference, the closer the retrieval request corresponding to that flow unit is to the current time, and the greater the weight of time influence; conversely, the larger the time difference, the smaller the weight. The specific weight value is calculated by the correspondence between the time difference and the preset weight coefficient. For example, flow units with smaller differences correspond to larger preset weight coefficients. The time influence weight of each semantic flow unit is obtained after calculation.

[0125] Step S143: Calculate the average correlation between all materials in each adapted material cluster and the core scene semantics of the material cluster. Combine the adaptation degree values ​​of the material cluster and the optimal adapted stream unit, and sum them according to the preset weights to obtain the cluster enhancement coefficient of each adapted material cluster. The weight allocation is based on the importance settings of the semantic consistency of the material cluster and the adaptability of the stream unit.

[0126] The average relevance of all materials within each adapted material cluster to the core scene semantics of that cluster is calculated. This average reflects the overall relevance level between the materials within the cluster and the core semantics. The cluster enhancement coefficient is obtained by combining the adaptation scores of the material cluster and the optimal adapted stream unit with preset weights and performing a weighted summation. When allocating weights, the importance of semantic consistency of the material cluster (reflected by the average relevance score) and stream unit adaptability (reflected by the adaptation score) are considered. If both are considered equally important, a certain proportion of weight is allocated to each, such as half for each, and then a weighted summation is performed.

[0127] Step S144: Calculate the average scene semantic association strength between each material and all other materials in the material cluster. The larger the average association strength value, the larger the material association coefficient. At the same time, combine the frequency of the material being selected in the material cluster in history. As the frequency increases, the coefficient will increase by a fixed percentage. Determine the increase by the correspondence between frequency and preset percentage coefficient to obtain the material association coefficient of each material.

[0128] For each material, the average semantic association strength between it and all other materials within the same material cluster is calculated. A higher average association strength indicates a stronger association between the material and other materials in the cluster, resulting in a higher material association coefficient. Simultaneously, the frequency of the material's selection by users in historical searches within the cluster is examined. Higher frequency suggests greater user interest in the material, and a fixed percentage is added to the material association coefficient. The increment is determined by the correspondence between frequency and a preset percentage coefficient; for example, higher frequency corresponds to a larger preset percentage coefficient. This process is used to calculate the material association coefficient for each material.

[0129] Step S145: Based on the optimal adaptation stream unit of the material cluster to which the material belongs, obtain the time influence weight of the optimal adaptation stream unit, compare the semantic tags of the material scene with the semantic description of the optimal adaptation stream unit to obtain the adaptation degree between the material and the semantics of the optimal adaptation stream unit, and multiply the two to obtain the time adaptation coefficient of each material.

[0130] The optimal adaptation stream unit (DMU) of each content cluster is determined, and then the temporal influence weight of this DMU is obtained. The scene semantic tags of the content are compared with the semantic description of the DMU to analyze the degree of matching and determine the semantic fit between the content and the DMU. The temporal influence weight is multiplied by this fit to obtain the temporal fit coefficient for each content. This temporal fit coefficient comprehensively considers the impact of both temporal factors and semantic fit on the content.

[0131] Step S146: Based on the cluster enhancement coefficient, material correlation coefficient, and time adaptation coefficient, and combined with the preset priority comprehensive judgment rules, determine the retrieval priority level of each material. The higher the retrieval priority value, the greater the match between the material and the user's retrieval needs.

[0132] Step S1461: Extract the cluster enhancement coefficient of the material cluster to which the material to be calculated belongs, and at the same time extract the material correlation coefficient and the time adaptation coefficient of the material to be calculated, so that the cluster enhancement coefficient, material correlation coefficient and time adaptation coefficient are all within the same preset value range.

[0133] The material Z to be calculated belongs to material cluster M. The cluster enhancement coefficient K of material cluster M is extracted. The material correlation coefficient G and time adaptation coefficient T of material Z are extracted. The preset value range is [0, 1]. If the cluster enhancement coefficient K, material correlation coefficient G, or time adaptation coefficient T exceeds this range, they are adjusted to the [0, 1] range through normalization to ensure that the three coefficients are in the same preset value range.

[0134] Step S1462: Determine the retrieval priority level of the material to be calculated according to the preset priority comprehensive judgment rule. The priority comprehensive judgment rule defines the priority level corresponding to the cluster enhancement coefficient, material association coefficient and time adaptation coefficient of different value ranges.

[0135] The preset priority comprehensive judgment rule divides the value ranges of the cluster enhancement coefficient K, material association coefficient G, and time adaptation coefficient T into multiple intervals, with each interval corresponding to a sub-level. For example, the interval [0, 1] is divided into three sub-intervals: high, medium, and low, corresponding to sub-levels 3, 2, and 1, respectively. Based on the sub-intervals where K, G, and T of material Z are located, their respective sub-levels are obtained, such as K corresponding to sub-level Kz, G corresponding to sub-level Gz, and T corresponding to sub-level Tz. Then, according to the preset priority calculation formula, such as priority level = Kz × Wk + Gz × Wg + Tz × Wt (where Wk, Wg, and Wt are the weights of the cluster enhancement coefficient, material association coefficient, and time adaptation coefficient, respectively, and Wk + Wg + Wt = 1), the retrieval priority level of material Z is calculated.

[0136] Step S1463: Repeat the above steps to determine the search priority level for all materials in the enhanced adaptation results, so that each material has a unique corresponding search priority level.

[0137] For each material in the enhanced adaptation results, the cluster enhancement coefficient of its cluster, its own material correlation coefficient and time adaptation coefficient are extracted according to the process of steps S1461 to S1462. After normalization, its retrieval priority level is calculated and determined according to the priority comprehensive judgment rule to ensure that each material has a unique corresponding retrieval priority level.

[0138] Step S1464: Check whether the determined retrieval priority level is within the preset level range. If the priority level of any material is not within the preset level range, re-examine the calculation process of the cluster enhancement coefficient, material correlation coefficient, and time adaptation coefficient, correct the errors, and re-determine the priority level.

[0139] The preset search priority level range is [1, 5], with level 1 being the lowest and level 5 the highest. The search priority levels of all materials in the enhanced adaptation results are traversed. If a material's priority level is found to be less than 1 or greater than 5, it indicates that the calculation of the material's cluster enhancement coefficient, material association coefficient, or time adaptation coefficient may be incorrect. The calculation process of the cluster enhancement coefficient of the material's cluster is re-examined, including steps such as the average association degree and the weighted summation of adaptation values; the calculation process of the material association coefficient is checked, including the processing of the average association strength and historical search frequency; the calculation process of the time adaptation coefficient is checked, including the calculation of time influence weight and semantic adaptation. After identifying and correcting the errors, the search priority level of the material is recalculated.

[0140] Step S1465: Organize the retrieval priority levels of all materials, associate them with the basic information of the materials and the scene semantic tags, and store them to form a material priority association table.

[0141] The search priority level of all materials in the enhanced adaptation results will be enhanced and associated with their basic information (such as material name, material type, material storage path, etc.) and scene semantic tags. These will be organized and stored in a certain format to form a material priority association table, which will facilitate the subsequent sorting and push of materials.

[0142] Step S147: Sort all materials in the initial adaptation results in descending order of retrieval priority. For materials with the same retrieval priority, sort them in descending order of cluster enhancement coefficient of their respective material clusters. For materials with the same cluster enhancement coefficient, sort them in descending order of the frequency of the material being selected in historical retrievals.

[0143] Based on the determined search priority levels, all materials in the initial adaptation results are sorted from highest to lowest priority. For materials with the same search priority level, they are sorted from highest to lowest according to the cluster enhancement coefficient of their respective material clusters; if the cluster enhancement coefficients are also the same, they are sorted from highest to lowest according to the frequency of the materials being selected in historical searches, with materials selected more frequently appearing first.

[0144] Step S148: Compare the scene semantic tags and content of the sorted materials. If the scene semantic tags of two materials are completely identical and the similarity of the material content exceeds the preset duplication threshold, then retain the material with the highest retrieval priority level, integrate the sorted and deduplicated materials, and form an enhanced adaptation result that includes basic material information, scene semantic tags, and retrieval priority level.

[0145] After sorting, the scene semantic tags and content of the materials are compared. If two materials have completely identical scene semantic tags and their content similarity exceeds a preset duplication threshold (e.g., two images of "Chinese braised pork dish" with identical tags and highly similar content), the material with the highest search priority is retained, and the other is deleted. The sorted and deduplicated materials are then integrated to form an enhanced adaptation result, which includes the basic information of the materials, scene semantic tags, and search priority level.

[0146] Step S150: After sorting the multimedia materials in the enhanced adaptation results according to the search priority, push them to the user search interface for the user to perform the selection operation.

[0147] The multimedia materials related to catering in the enhanced adaptation results are sorted according to search priority and then pushed to the user's search interface, allowing the user to view and select them.

[0148] For example, step S151: Extract the retrieval priority level, basic information of the material and scene semantic tags of each multimedia material in the enhanced adaptation results. The basic information of the material includes the material name, material type, material format and material storage path.

[0149] From the enhanced adaptation results, the retrieval priority level, basic information of the material, and scene semantic tags are extracted for each multimedia material. The basic information of the material specifically includes the material name, such as "a compilation of Chinese dishes for a family dinner"; the material type, such as video, image, etc.; the material format, such as JPEG, MP4, etc.; and the material storage path, indicating the specific location of the material on the server.

[0150] Step S152: Perform a preliminary sorting of all materials in descending order of search priority. Materials with the same search priority are sorted in descending order of cluster enhancement coefficient of their respective material clusters. Materials with the same cluster enhancement coefficient are sorted in descending order of the frequency of being selected in historical searches.

[0151] All materials are initially sorted according to their search priority from highest to lowest. For materials with the same priority, the cluster enhancement coefficient of their respective material clusters is compared, with the one with the larger coefficient ranked first. If the cluster enhancement coefficients are also the same, the materials are sorted according to the frequency of their selection in historical searches, from most to least, with the most frequently selected materials given priority.

[0152] Step S153: Generate display information for each sorted material. The display information includes a thumbnail preview of the material, a summary of the material's basic information, and a simplified description of the scene's semantic tags. The thumbnail preview of image-type materials is a thumbnail image, the thumbnail preview of video-type materials is a keyframe thumbnail image, the thumbnail preview of audio-type materials is a waveform image, and the thumbnail preview of text-image-type materials is a comprehensive preview image, making the display information concise and clear and able to reflect the core attributes of the material.

[0153] For each sorted piece of content, display information is generated. This information includes thumbnail previews: image-type content is directly generated as a thumbnail; video-type content has keyframes selected to create thumbnails; audio-type content has waveforms generated as thumbnail previews; and text-and-image content has a comprehensive preview image. A summary of basic content information is also provided, briefly outlining the content name, type, etc. Scene semantic tags are described in a simplified manner, extracting the core scene semantic tag content. This concise and clear display information allows users to quickly understand the core attributes of the content.

[0154] Step S154: Organize the sorted material display information into structured data according to the display format requirements of the user search interface. The structured data includes the material sorting number, each field of the display information, and the material storage path.

[0155] Based on the display format requirements of the user search interface, the sorted material display information is organized into structured data. The structured data includes the material's sorting sequence number, numbered sequentially according to the previous sorting results; various fields of the display information, such as links to thumbnail previews, basic information summary text, and simplified descriptions of tags; and the material storage path for easy retrieval of the complete material content later.

[0156] Step S155: Establish a communication connection with the user search interaction interface, send structured data to the background of the user search interaction interface, monitor the material loading status of the user search interaction interface, and after the interface completes the material display, receive the user operation information fed back by the interface in real time. The user operation information includes the material identifier corresponding to the user's click to view operation, the material identifier corresponding to the user's favorite operation, and the material identifier corresponding to the user's download operation.

[0157] Establish a communication connection with the user search interface and send structured data to the interface's backend system via the network. After transmission, monitor the loading status of the materials on the user search interface to see if the materials have been successfully loaded and displayed. Once the interface has finished displaying all materials, receive real-time feedback on user operation information from the interface. This information includes the material identifier corresponding to when the user clicks to view a material, the material identifier corresponding to when the user performs a favorite operation, and the material identifier corresponding to when the user performs a download operation.

[0158] Step S156: When the operation information of the user clicking to view any material is received, the complete content data of the material is retrieved from the multimedia material storage server according to the storage path of the material, and sent to the user's search interface through streaming transmission.

[0159] When a user clicks to view a specific piece of content, the system retrieves the complete content data from the multimedia content storage server based on the content's storage path in the structured data. Using streaming, the complete content data is sent to the user's search interface, allowing for a smooth viewing experience.

[0160] Step S157: Record the user's operation behavior information on the material, and store the operation behavior information in the user behavior database along with the material identifier and retrieval request information. The operation behavior information includes information such as viewing duration, whether the collection operation was performed, and whether the download operation was performed.

[0161] Record various user actions related to content, such as viewing duration, whether the user added the content to their favorites, or whether they downloaded it. This information is then associated with the corresponding content identifier and the user's search request, and stored together in the user behavior database.

[0162] Figure 2This application illustrates a multimedia intelligent retrieval system 100 based on digital multimedia, comprising a processor 1001, a memory 1003, and program code stored in the memory 1003. The processor 1001 executes the program code to implement the steps of the multimedia intelligent retrieval method based on digital multimedia. The processor 1001 and the memory 1003 are connected, for example, via a bus 1002. Optionally, the multimedia intelligent retrieval system 100 may further include a transceiver 1004, which can be used for data interaction between this multimedia intelligent retrieval system and other multimedia intelligent retrieval systems, such as sending and / or receiving data. It should be noted that in actual scheduling, the transceiver 1004 is not limited to one, and the structure of this multimedia intelligent retrieval system 100 does not constitute a limitation on the embodiments of this application.

[0163] The memory 1003 is used to store program code for executing the embodiments of this application, and its execution is controlled by the processor 1001. The processor 1001 is used to execute the program code stored in the memory 1003 to implement the steps shown in the foregoing method embodiments.

[0164] This application provides a computer-readable storage medium storing program code, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0165] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application, without departing from the technical concept of this application, also fall within the protection scope of the embodiments of this application.

Claims

1. A method for intelligent retrieval of multimedia material based on digital multimedia, characterized in that, The method comprises: Based on the multimedia material set to be retrieved, a plurality of material clusters are aggregated according to the similarity of the scene semantics to which the materials belong, and a multimedia material scene semantic cluster is obtained, each material cluster corresponding to a single core scene semantic, and each material in the material cluster carrying a scene semantic label; Based on the dynamic change relationship of the scene semantics of the current retrieval demand text and the historical retrieval demand text of the user, a user retrieval demand scene semantic flow is constructed, the flow units are arranged in chronological order, and each flow unit corresponds to a scene semantic description of the retrieval demand, and a user retrieval demand scene semantic flow is obtained; An association mapping between the core scene semantics of the material cluster and the scene semantics of the flow unit is established, a scene semantic adaptation process of the multimedia material scene semantic cluster and the user retrieval demand scene semantic flow is performed, and material clusters and materials in the material clusters that are adapted to the flow semantics in the multimedia material scene semantic cluster are screened, and an initial adaptation result is obtained; In combination with the scene semantic association strength between the materials in the material cluster and the time influence weight of the scene semantic of the flow unit, a scene semantic association strengthening process is performed on the initial adaptation result, the retrieval priority of the materials is adjusted, and a strengthened adaptation result is obtained; The multimedia materials in the strengthened adaptation result are sorted according to the retrieval priority and pushed to the user retrieval interaction interface for the user to perform a selection operation.

2. The digital multimedia-based intelligent retrieval method of multimedia material according to claim 1, characterized in that, The method comprises: The scene semantic dimension of each multimedia material in the multimedia material set to be retrieved is divided, the scene semantic dimension covers a visual scene dimension, an auditory scene dimension and a text scene dimension, the visual scene dimension corresponds to a scene type of visual content presentation of the material, the auditory scene dimension corresponds to a scene type of auditory content presentation of the material, and the text scene dimension corresponds to a scene type of text description presentation of the material, the specific scene type of each multimedia material in the visual scene dimension, the auditory scene dimension and the text scene dimension is determined one by one, and a scene dimension type record of each multimedia material is obtained; The scene semantic elements of each multimedia material in each scene semantic dimension are extracted, the scene semantic elements of the visual scene dimension include object types, environment features and light styles in the visual scene, the scene semantic elements of the auditory scene dimension include sound sources, sound effect styles and rhythm characteristics in the auditory scene, and the scene semantic elements of the text scene dimension include theme directions, emotional tendencies and expression styles in the text scene, the visual scene semantic elements, the auditory scene semantic elements and the text scene semantic elements of each material are sorted and collected, and a scene semantic element set of each multimedia material is obtained; The scene semantic element set of any two multimedia materials is compared, the number of coincident elements is counted, and the semantic similarity of the coincident elements is calculated to obtain the coincidence number proportion and the average semantic similarity of the coincident elements. According to a preset correlation degree determination rule, the coincidence number proportion and the average semantic similarity of the coincident elements are comprehensively considered to obtain the scene semantic correlation degree of the two materials, the scene semantic correlation degrees of all materials are calculated, and a scene semantic correlation degree matrix of the materials is obtained. A single multimedia material is taken as a starting point, other materials with a scene semantic correlation degree not lower than a scene semantic aggregation threshold value are classified into the same temporary material cluster, and the above operation is repeated until all materials are classified into corresponding temporary material clusters. If the correlation degree of any material and multiple starting materials all meets the standard, the material is classified into the temporary material cluster corresponding to the starting material with the largest correlation degree value. The scene semantic element set of all materials in each temporary material cluster is counted, and the combination of the scene semantic elements with the highest occurrence frequency is extracted as the core scene semantic of the material cluster. The core scene semantic includes the key elements in the visual scene dimension, the auditory scene dimension and the text scene dimension, and the core scene semantic description of each temporary material cluster is obtained. Scene semantic labels are added to the materials in each temporary material cluster to obtain the scene semantic labels of each material. The label content of the scene semantic label includes the core scene semantic of the material cluster and the scene type of the material in the visual scene dimension, the auditory scene dimension and the text scene dimension. It is checked whether there are temporary material clusters with a core scene semantic similarity exceeding a preset integration threshold value. If the core scene semantic similarity of two temporary material clusters exceeds the preset integration threshold value, the two temporary material clusters are merged into a new material cluster, the core scene semantic of the new material cluster is recalculated, the scene semantic labels of the materials in the new material cluster are updated, and the two temporary material clusters before merging are deleted. The checking and merging steps are repeated until there is no material cluster meeting the integration condition, and finally a plurality of material clusters are formed, which together constitute a multimedia material scene semantic cluster.

3. The digital multimedia-based intelligent retrieval method of multimedia material according to claim 1, characterized in that, The scene semantic dynamic change relationship of the current search demand text and the historical search demand text of the user is used to construct a user search demand scene semantic flow. The flow units are arranged in chronological order, and each flow unit corresponds to a scene semantic description of the search demand. The user search demand scene semantic flow is obtained, including: The search demand text input by the user is collected, the time information of the search demand text input is recorded, the current search demand text is cleaned and processed, and the scene semantic of the cleaned current search demand text is analyzed. The description words related to the visual scene, the auditory scene and the text scene in the current search demand text are extracted, the scene type and the corresponding scene semantic element in each scene dimension are determined, the scene semantic description of the current search demand is integrated, and the current scene semantic flow unit is obtained by associating the input time information. The user historical search demand text set is called, the user historical search demand text set contains the demand text input by the user in each search operation within a preset time range, and the search time information corresponding to each historical text is obtained, the historical demand texts are sorted in order from early to late according to the search time, and an ordered historical search demand text sequence is obtained; For each historical demand text in the ordered historical search demand text sequence, text cleaning, scene semantic dimension extraction and scene semantic element integration are sequentially performed, a scene semantic description corresponding to each historical demand text is formed, and a plurality of historical scene semantic stream units are obtained by associating the respective search time information; The scene semantic descriptions of two adjacent scene semantic stream units are compared, the number of coinciding semantic elements and the semantic similarity of the coinciding elements are counted for each scene dimension, the element coincidence ratio and the average semantic similarity of the coinciding elements of each dimension are obtained, and then the scene semantic similarity of the adjacent stream units is obtained according to a preset rule; The current scene semantic stream unit and all historical scene semantic stream units are arranged in time sequence, an initial scene semantic stream framework is formed, and the corresponding scene semantic similarity between adjacent stream units is marked to reflect the degree of change of the scene semantics between the stream units; It is checked whether there are adjacent stream units with a scene semantic similarity lower than a preset optimization threshold in the initial scene semantic stream framework, if the semantic similarity of any two adjacent stream units is lower than the preset optimization threshold, it is analyzed whether there is a missing intermediate semantic transition, and if there is a missing intermediate semantic transition, a transition scene semantic stream unit inferred based on historical search behavior is supplemented; Repeated scene semantic stream units in the initial scene semantic stream framework are deleted, and finally a user search demand scene semantic stream arranged in time sequence and with coherent semantic changes is formed.

4. The digital multimedia-based intelligent retrieval method of multimedia material according to claim 1, characterized in that, The association mapping of the core scene semantics of the material cluster and the scene semantics of the stream unit is established, the scene semantic adaptation processing of the multimedia material scene semantic cluster and the user search demand scene semantic stream is performed, the material cluster and the materials in the material cluster that are adapted to the stream semantics are selected from the multimedia material scene semantic cluster, and an initial adaptation result is obtained, including: The core scene semantic description of each material cluster in the multimedia material scene semantic cluster is extracted, and the scene semantic description of each scene semantic stream unit in the user search demand scene semantic stream is extracted, and a cluster core semantic list and a stream unit semantic list are respectively formed; The visual scene dimension, the auditory scene dimension and the text scene dimension of the core scene semantics of the material cluster are respectively associated with the corresponding scene dimensions of the stream unit semantics, and the dimension mapping relationship between the core scene semantics of the material cluster and the scene semantics of the stream unit is established; The semantic elements under the corresponding dimensions of the core scene semantics of the material cluster and the scene semantics of the stream unit are compared, the number of coinciding elements and the semantic similarity of the coinciding elements are counted for each scene dimension, the element coincidence rate and the element similarity average of each dimension are obtained, and the overall scene semantic adaptation degree of the core scene semantics of the material cluster and the scene semantics of the stream unit is calculated according to a preset rule by comprehensively considering the element coincidence rate, the element similarity average and the dimension weight of each dimension. Filtering out the stream unit with the largest adaptation degree value as the optimal adaptation unit of the material cluster, and recording the adaptation degree value of the material cluster and the time information of the corresponding stream unit; Filtering out the material cluster with an optimal adaptation degree not lower than the scene semantic adaptation threshold value, and the filtered material cluster is used as the material cluster adapted to the scene semantic stream of the user search demand to obtain an adapted material cluster list; Extracting all multimedia materials in each material cluster in the adapted material cluster list, collecting the scene semantic label and material basic information of each material, and the material basic information includes the material name, material type, and material duration, wherein the material duration is only included in the video type material or the audio type material; Statistically analyzing the relevance of each material in the adapted material cluster to the core scene semantic of the material cluster, deleting the materials with a relevance lower than the average relevance in the material cluster, and integrating the remaining materials in all the adapted material clusters to form an initial adaptation result including the material basic information, the scene semantic label, and the information of the belonging material cluster.

5. The digital multimedia-based intelligent retrieval method of multimedia material according to claim 1, characterized in that, The initial adaptation result is subjected to a scene semantic correlation strengthening process by combining the scene semantic correlation strength between the materials in the material cluster and the time influence weight of the stream unit, and the search priority of the material is adjusted to obtain a strengthened adaptation result, including: Extracting the scene semantic correlation strength between all the materials in each adapted material cluster in the initial adaptation result, arranging the correlation strength between the materials in the cluster to form a cluster material correlation strength list, and the scene semantic correlation strength is derived from the scene semantic correlation degree matrix between the materials calculated when the scene semantic cluster of the multimedia material is constructed; Determining the time influence weight of each stream unit according to the difference between the time corresponding to the stream unit and the current time, and the time influence weight increases as the time difference decreases, and the specific weight value is calculated through the corresponding relationship between the time difference and the preset weight coefficient to obtain the time influence weight of each scene semantic stream unit in the scene semantic stream of the user search demand; Statistically analyzing the average relevance of all the materials in each adapted material cluster to the core scene semantic of the material cluster, combining the adaptation degree value of the material cluster and the optimal adaptation stream unit, and weighting and summing the preset weight to obtain the cluster strengthening coefficient of each adapted material cluster, and the weight distribution is based on the importance of the semantic consistency of the material cluster and the adaptation of the stream unit; Statistically analyzing the average scene semantic correlation strength of each material and all the other materials in the material cluster, and the correlation coefficient of the material increases as the average correlation strength value increases, and the coefficient is additionally increased by a fixed proportion as the frequency of the material being selected by the historical search in the material cluster increases, and the increase amplitude is determined through the corresponding relationship between the frequency and the preset proportion coefficient to obtain the material correlation coefficient of each material; According to the optimal adaptation stream unit of the material belonging to the material cluster, the time influence weight of the optimal adaptation stream unit is obtained, the adaptation degree of the material and the optimal adaptation stream unit is obtained by comparing the scene semantic label of the material and the semantic description of the optimal adaptation stream unit, and the time adaptation coefficient of each material is obtained by multiplying the two. According to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and in combination with a preset priority comprehensive judgment rule, a retrieval priority level of each material is determined, and the greater the retrieval priority value is, the greater the matching degree of the material and the user retrieval demand is; All materials in the initial adaptation result are sorted in descending order of retrieval priority level, and materials with the same retrieval priority level are sorted in descending order of the cluster reinforcement coefficient of the material cluster to which the materials belong; The scene semantic labels and the material contents of the sorted materials are compared, if the scene semantic labels of two materials are completely consistent and the material content similarity exceeds a preset repetition threshold, the material with the highest retrieval priority level is retained, the sorted and deduplicated materials are integrated, and a reinforced adaptation result including material basic information, scene semantic labels and retrieval priority level is formed.

6. The digital multimedia-based intelligent retrieval method of multimedia material according to claim 2, characterized in that, The scene semantic element sets of any two multimedia materials are compared, the number of coinciding elements is counted and the semantic similarity of the coinciding elements is calculated, the element coincidence proportion and the average semantic similarity of the coinciding elements are obtained, according to a preset correlation degree judgment rule, the element coincidence proportion and the average semantic similarity of the coinciding elements are comprehensively considered, the scene semantic correlation degree of the two materials is obtained, the scene semantic correlation degrees between all materials are calculated, and a scene semantic correlation degree matrix between materials is obtained, including: Two multimedia materials to be calculated are selected, and the scene semantic element set of the first multimedia material and the scene semantic element set of the second multimedia material are extracted; Whether each element in the scene semantic element set of the first material exists in the scene semantic element set of the second material is checked one by one, the same element refers to an element with completely consistent semantics, the similar element refers to an element with a semantic similarity exceeding a preset similarity threshold, the total number of the same elements and the similar elements in the two sets is counted as the number of coinciding elements; The proportion of the number of coinciding elements in the total number of elements of the first material and the proportion of the number of coinciding elements in the total number of elements of the second material are calculated respectively, and the average of the two proportions is taken as the final element coincidence proportion; The semantic similarity of each coinciding element is calculated, the semantic similarity of the same element is set as a fixed maximum value, the semantic similarity value of the similar element is obtained by comparing the semantic connotation, expression mode and scene correlation degree of the element, the semantic similarity of all coinciding elements is counted, and the average is calculated as the average semantic similarity of the coinciding elements; According to the core demand of scene semantic retrieval, the element coincidence proportion weight and the similarity average weight are allocated, the element coincidence proportion weight is greater than the similarity average weight, and the specific difference is set according to a preset standard; The element coincidence proportion and the average semantic similarity of the coinciding elements are recorded respectively, and according to a preset correlation degree judgment rule, the element coincidence proportion and the average semantic similarity of the coinciding elements are comprehensively considered to obtain the scene semantic correlation degree of the two materials. The scene semantic correlation degrees between any two multimedia materials in the set of multimedia materials to be searched are calculated in sequence by repeating the above steps, and the calculation results are arranged in a matrix form, with the matrix rows corresponding to different materials, the matrix columns corresponding to different materials, and the matrix elements being the scene semantic correlation degrees of the corresponding two materials, to obtain a scene semantic correlation degree matrix between the materials; It is checked whether there is a correlation degree calculation error in the scene semantic correlation degree matrix between the materials, and if there is a value beyond a preset reasonable range, the correlation degree of the corresponding material pair is recalculated.

7. The digital multimedia-based intelligent retrieval method of multimedia material according to claim 3, characterized in that, The scene semantic descriptions of the adjacent two scene semantic flow units are compared, the number of coinciding semantic elements and the semantic similarity of the coinciding elements in the visual scene dimension, the auditory scene dimension and the text scene dimension are counted, and the scene semantic similarity of the adjacent flow units is comprehensively obtained, including: Adjacent two scene semantic flow units in the scene semantic flow of the user search requirement are selected, and the scene semantic description of the previous scene semantic flow unit and the scene semantic description of the subsequent scene semantic flow unit are extracted, and the semantic elements in the visual scene dimension, the auditory scene dimension and the text scene dimension are split out; In the visual scene dimension, the number of coinciding visual semantic elements of the previous scene semantic flow unit and the subsequent scene semantic flow unit is counted, and the coinciding number includes the same elements and similar elements. Whether the elements coincide is judged in the same way as the comparison of the material semantic elements. Similarly, the number of coinciding elements in the auditory scene dimension and the text scene dimension is counted, and the number of coinciding elements in the visual scene dimension, the auditory scene dimension and the text scene dimension is obtained. The element coincidence proportion of each scene dimension is calculated. The coincidence proportion of the visual scene dimension is obtained by dividing the number of coinciding visual scene dimension elements by the sum of the number of visual scene dimension elements of the previous flow unit and the number of visual scene dimension elements of the subsequent flow unit, and then multiplying by a preset proportion calculation coefficient. The coincidence proportion of the auditory scene dimension is obtained by dividing the number of coinciding auditory scene dimension elements by the sum of the number of auditory scene dimension elements of the previous flow unit and the number of auditory scene dimension elements of the subsequent flow unit, and then multiplying by a preset proportion calculation coefficient. The coincidence proportion of the text scene dimension is obtained by dividing the number of coinciding text scene dimension elements by the sum of the number of text scene dimension elements of the previous flow unit and the number of text scene dimension elements of the subsequent flow unit, and then multiplying by a preset proportion calculation coefficient. The above calculation method reflects the semantic overlap degree of the two flow units in the corresponding dimension, and the element coincidence proportion of the visual scene dimension, the auditory scene dimension and the text scene dimension is obtained. The semantic similarity of each coinciding element in each scene dimension is calculated one by one. The similarity of the same element is a fixed maximum value, and the similarity of the similar element is calculated according to the semantic connotation matching degree. The average value of the similarity of all coinciding elements in the visual scene dimension is taken as the average value of the semantic similarity of the coinciding elements in the visual scene dimension, the average value of the similarity of all coinciding elements in the auditory scene dimension is taken as the average value of the semantic similarity of the coinciding elements in the auditory scene dimension, and the average value of the similarity of all coinciding elements in the text scene dimension is taken as the average value of the semantic similarity of the coinciding elements in the text scene dimension. According to the importance of each scene dimension in the user search demand, scene dimension weights are allocated to determine the specific weight values of the visual scene dimension, the auditory scene dimension and the text scene dimension; for each scene dimension, the element coincidence proportion and the average value of the semantic similarity of the coincident elements are recorded respectively; According to the preset dimension similarity judgment rule, the element coincidence proportion and the average value of the semantic similarity of the coincident elements of each scene dimension are comprehensively obtained to obtain the semantic matching level of each scene dimension; combined with the semantic matching levels of the visual scene dimension, the auditory scene dimension and the text scene dimension and the corresponding dimension weights, the overall scene semantic similarity between the two adjacent flow units is comprehensively obtained according to the overall similarity synthesis rule; The scene semantic similarity between all adjacent flow units in the user search demand scene semantic flow is calculated in turn, the similarity values of each pair of adjacent units are recorded to form a list of semantic similarity of adjacent flow units.

8. The digital multimedia-based intelligent retrieval method of multimedia material according to claim 4, characterized in that, The semantic elements of the core scene semantics of the comparison material cluster and the scene semantics of the flow unit under the corresponding dimension are compared, and for each scene dimension, the number of coincident elements and the semantic similarity of the coincident elements are counted respectively to obtain the element coincidence rate and the average value of the element similarity of each dimension; according to the preset rule, the element coincidence rate, the average value of the element similarity and the dimension weight of each dimension are comprehensively calculated to obtain the overall scene semantic adaptation degree of the core scene semantics of the material cluster and the scene semantics of the flow unit, including: The core scene semantic description of a material cluster and the scene semantic description of a scene semantic flow unit are selected and divided into visual scene dimension, auditory scene dimension and text scene dimension semantic elements respectively to form a cluster dimension semantic element list and a flow unit dimension semantic element list; Under the visual scene dimension, the cluster visual semantic elements and the flow unit visual semantic elements are compared, the number of coincident elements is counted, the number of coincident elements includes the same elements and similar elements, the proportion of the number of coincident elements to the total number of cluster visual elements is calculated as the visual element coincidence rate, and the average value of the semantic similarity of the coincident elements is calculated as the average value of the visual element similarity; For each scene dimension, the element coincidence rate and the average value of the element similarity are recorded respectively; According to the preset dimension adaptation degree judgment rule, the element coincidence rate and the average value of the element similarity of each scene dimension are comprehensively obtained to obtain the adaptation level of each scene dimension; Combined with the adaptation levels of the visual scene dimension, the auditory scene dimension and the text scene dimension and the corresponding dimension weights, the overall scene semantic adaptation degree of the core scene semantics of the material cluster and the scene semantics of the flow unit is comprehensively obtained according to the overall adaptation degree synthesis rule; The overall scene semantic adaptation degree of the current material cluster core semantics and the current flow unit semantics is recorded, and the above steps are repeated to calculate the adaptation degree of the core semantics of the material cluster and the semantics of all flow units in the user search demand scene semantic flow; The material cluster is replaced, and the adaptation degree of the core semantics of the newly selected material cluster and the semantics of all flow units is calculated according to the same process until the adaptation degree calculation of all material clusters and all flow units in the multimedia material scene semantic cluster is completed. All the adaptation degree calculation results are sorted to form a scene semantic adaptation degree matrix with material clusters as rows and flow units as columns, and the matrix elements are the overall scene semantic adaptation degree values of the corresponding material clusters and the corresponding flow units.

9. The digital multimedia-based intelligent retrieval method of multimedia material according to claim 5, characterized in that, The retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the material correlation coefficient and the time adaptation coefficient, and a preset priority comprehensive determination rule, and the retrieval priority level of each material is determined according to the cluster reinforcement coefficient, the ​ ​ ​ ​ 10. A digital multimedia-based intelligent retrieval system of multimedia material, characterized by, ​

Images