Video retrieval method and device

Abstract

The embodiment of the invention provides a video retrieval method. The video retrieval method comprises the following steps: acquiring an input query text; splitting the query text into a plurality of split texts by utilizing a large language model; the large language model is used for conducting split design on the basis of the split text, a plurality of retrieval conditions are formed on the basis of the split design, and the split design comprises video clip picture content expected to be matched and entity description expected to appear; and obtaining a video retrieval result from a video retrieval system based on the retrieval condition. According to the technical scheme provided by the embodiment of the invention, the understanding degree of the text input by the user can be improved, and the video material retrieval accuracy is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a video retrieval method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Video retrieval refers to the process of finding video content that meets user needs in massive video data through a specific method. At present, video retrieval generally searches based on query keywords entered by users and returns video content that matches the query keywords.

[0003] However, the current method of video retrieval based on keywords cannot accurately understand user intent, resulting in the retrieved video materials not matching the results the user wants, low retrieval accuracy, and poor user experience.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a video retrieval method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0006] One aspect of an embodiment of the present application provides a video retrieval method, the method comprising: Get the input query text; Using a large language model to split the query text into multiple storyboard texts; Using the large language model to perform storyboard design based on the storyboard text, and forming a plurality of search conditions based on the storyboard design, wherein the storyboard design includes the video clip content expected to be matched and the entity description expected to appear; A video retrieval result is obtained from the video retrieval system based on the retrieval condition.

[0007] Optionally, several search conditions are formed based on the storyboard design, including: Acquire the limiting conditions of the storyboard text, wherein the limiting conditions include the returned video duration limiting conditions and the matching material library type limiting conditions; A number of search conditions are formed based on the limiting conditions of the storyboard text and the storyboard design.

[0008] Optionally, the method further comprises: Repeat the steps of using the large language model to perform storyboard design based on the storyboard text, and forming a plurality of search conditions based on the storyboard design, so as to form at least two groups of search conditions for each storyboard design, wherein at least one group of search conditions is used for downgraded search.

[0009] Optionally, obtaining a video retrieval result from a video retrieval system based on the retrieval condition includes: Performing coarse-grained recall from the video retrieval system based on the retrieval condition to obtain a coarse-grained recall result; Scoring the coarse-grained recall results in multiple dimensions to obtain scoring results in multiple dimensions for each of the coarse-grained recall results; The coarse-grained recall results are sorted based on the multiple dimension scoring results, and the video retrieval results are obtained based on the sorting results.

[0010] Optionally, the coarse-grained recall results are scored in multiple dimensions to obtain multiple dimension scoring results for each of the coarse-grained recall results, including: Obtaining entity matching degree, picture matching degree, picture quality and retrieval condition priority of the coarse-grained recall result; The multiple dimension scoring results of each of the coarse-grained recall results are determined based on the entity matching degree, the picture matching degree, the picture quality and the retrieval condition priority.

[0011] Optionally, performing a coarse-grained recall from a video retrieval system based on the retrieval condition to obtain a coarse-grained recall result includes: Determine all entity constraints for retrieval based on the retrieval condition; Recall the first video retrieval result that meets all entity constraints; When the number of the first video retrieval results is less than a preset number, the second video retrieval results that meet the partial entity constraints are recalled.

[0012] Optionally, the method further comprises: Determine the total number of the first video search results and the second video search results; When the total number is less than the preset number, the search is switched to the next search condition for searching.

[0013] Optionally, the video retrieval system includes multiple video slices and vector features of each of the video slices, and the video retrieval system is used to recall retrieval results based on the vector features.

[0014] Optionally, the video retrieval system also includes entity information included in the video slice, and the retrieval system is further used to recall retrieval results based on the entity information.

[0015] Another aspect of the embodiments of the present application provides a video retrieval device, the device comprising: The acquisition module is used to obtain the input query text; A splitting module, used for splitting the query text into multiple storyboard texts using a large language model; A design module, configured to use the large language model to perform storyboard design based on the storyboard text, and form a plurality of search conditions based on the storyboard design, wherein the storyboard design includes the video clip content expected to be matched and the entity description expected to appear; The retrieval module is used to obtain video retrieval results from the video retrieval system based on the retrieval conditions.

[0016] Another aspect of an embodiment of the present application provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein: the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0017] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0018] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0019] The above technical solution adopted in the embodiment of the present application may have the following advantages: By obtaining the input query text, the query text is split into multiple storyboard texts using the large language model; storyboard design is performed based on the storyboard text using the large language model, and several search conditions are formed based on the storyboard design, wherein the storyboard design includes the expected matching video clip screen content and the expected entity description; and video search results are obtained from the video search system based on the search conditions. It can be seen that the embodiment of the present application can intelligently storyboard the query text input by the user based on the large language model, thereby improving the overall coherence and expressiveness of the video; at the same time, it can also improve the understanding of the text input by the user and improve the accuracy of video material retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0021] Figure 1 A flowchart of a video retrieval method according to Embodiment 1 of the present application is schematically shown; Figure 2 Schematically shows Figure 1 Flow chart of sub-steps of step S104; Figure 3 Schematically shows Figure 1 Flow chart of sub-steps of step S106; Figure 4 Schematically shows Figure 3 Flow chart of sub-steps of step S302; Figure 5 Schematically shows Figure 3 Flow chart of sub-steps of step S300; Figure 6 The newly added process of the video retrieval method according to the first embodiment of the present application is schematically shown; Figure 7 The schematic diagram shows a principle diagram of a video retrieval method according to the first embodiment of the present application; Figure 8 A block diagram schematically shows a video retrieval device according to the second embodiment of the present application; and Fig. 9 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0023] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0024] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0025] First, the following terms are explained: Large Language Model (LLM): A model with billions of parameters that is pre-trained to obtain extensive knowledge from a large unlabeled text corpus. It can understand and generate natural language and has achieved remarkable success in various natural language processing tasks. In addition, through methods such as multimodal instruction fine-tuning, the large language model can also incorporate information from other modalities such as vision to handle more complex multimodal tasks.

[0026] In the field of artificial intelligence, an agent refers to an intelligent system or program that can perceive the environment and take actions to achieve specific goals.

[0027] Prompt Engineering: Design, adjust, and optimize the text instructions (i.e., prompts) input into the AI ​​model to obtain the desired output.

[0028] Storyboarding: The breaking down of a video or film script into a series of sequential shots or scenes.

[0029] Entity: An identifiable, independent, and trackable object.

[0030] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: When editing a video, users usually need to obtain the video material first, and then edit it based on the video material. The acquisition of video material usually requires searching in the video material database. At present, the retrieval of video material generally relies on keyword matching and simple semantic analysis; in terms of video storyboards, related technologies mainly rely on manually preset rules or templates to divide scenes and design transitions; in terms of retrieval strategies, generally a single retrieval strategy is adopted, or only a simple fallback strategy is adopted.

[0031] The applicant understands that the related technology still has the following defects: (1) Keyword matching and simple semantic analysis methods are difficult to accurately understand the user's intention, resulting in the retrieved video material not matching the text content entered by the user; (2) Storyboard methods based on preset rules or templates lack flexibility and cannot adapt to complex content requirements, thus affecting the coherence and expressiveness of the video.

[0032] (3) A single retrieval method is difficult to handle complex multimodal information, which reduces the comprehensiveness and accuracy of the retrieval. At the same time, it lacks an effective downgrade and optimization mechanism, making it difficult to deal with situations where retrieval fails or the results are unsatisfactory.

[0033] To this end, the embodiment of the present application provides a video retrieval technology solution. In this technical solution: (1) by introducing a large language model, the understanding of the user input text can be improved, and the accuracy of video material retrieval can be improved; (2) by using a large language model to storyboard the user input text, intelligent storyboarding can be achieved, and the storyboarding strategy can be dynamically adjusted according to the specific content, thereby improving the overall coherence and expressiveness of the video; (3) by constructing a multi-modal multi-level multi-channel recall system, full use can be made of multi-dimensional information such as text and vision to improve the comprehensiveness and accuracy of retrieval; at the same time, by designing flexible retrieval strategies and degradation mechanisms, the robustness of the retrieval system and its adaptability to various complex scenarios can be improved. See below for details.

[0034] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0035] Embodiment 1 Figure 1 The flowchart of the video retrieval method according to the first embodiment of the present application is schematically shown.

[0036] like Figure 1 As shown, the video retrieval method may include steps S100 to S106, wherein: Step S100, obtaining the input query text.

[0037] Step S102: using a large language model to split the query text into multiple storyboard texts.

[0038] Step S104, using the large language model to perform storyboard design based on the storyboard text, and forming a plurality of search conditions based on the storyboard design, wherein the storyboard design includes the video clip content expected to be matched and the entity description expected to appear.

[0039] Step S106: acquiring video retrieval results from the video retrieval system based on the retrieval condition.

[0040] The video retrieval method provided in this embodiment obtains the input query text, and uses the large language model to split the query text into multiple storyboard texts; uses the large language model to perform storyboard design based on the storyboard text, and forms several retrieval conditions based on the storyboard design, wherein the storyboard design includes the expected matching video clip screen content and the expected entity description; and obtains the video retrieval result from the video retrieval system based on the retrieval conditions. It can be seen that the embodiment of the present application can intelligently storyboard the query text input by the user based on the large language model, thereby improving the overall coherence and expressiveness of the video; at the same time, it can also improve the understanding of the user input text and improve the accuracy of video material retrieval.

[0041] The following combination Figure 1 , each step in steps S100~S106 and other optional steps are explained in detail.

[0042] Step S100 , get the input query text.

[0043] The input query text may be a query text input by a user for video retrieval. Specifically, it may be the expected video screen content, entities contained in the video, video type, video length, video scene, and elements to be excluded. Specifically, it may be in the form of one sentence, multiple sentences, or multiple keywords.

[0044] Step S102 , using a large language model to split the query text into multiple storyboard texts.

[0045] The large language model can perform natural language processing on the query text and split the query text into finer-grained text segments that can be used to generate storyboards. Natural language processing can include word segmentation, part-of-speech tagging, named entity recognition, dependency analysis, and semantic analysis of the query text. Named entity recognition can identify entities such as names of people and places included in the query text; dependency analysis can analyze the grammatical relationship between words in the query text to better understand the meaning of the sentence. Based on natural language processing, the large language model can split the query text into multiple clauses according to the grammatical structure of the query text. Each clause can be further supplemented and expanded to form multiple storyboard texts.

[0046] When using a large language model to split a query text into multiple storyboards, prompt word engineering can be used. For example, the prompt word can be a clear instruction such as "Please split the input query text into multiple storyboards." In addition, the prompt word can include specific requirements for the storyboard text, such as output format, style, storyboard granularity, and splitting standards.

[0047] Step S104 , using the large language model to perform storyboard design based on the storyboard text, and forming a plurality of retrieval conditions based on the storyboard design, wherein the storyboard design includes the video clip content expected to be matched and the entity description expected to appear.

[0048] The semantic understanding ability of the large language model can be used to deeply analyze each storyboard text, extract the key information in each storyboard text, and then convert this key information into the expected matching video clip screen content and / or the expected entity description to perform storyboard design, and then form several retrieval conditions based on the storyboard design. Specifically, the prompt word engineering can be used to implement the corresponding storyboard design. For example, the prompt word can be a clear instruction such as "perform storyboard design for each storyboard text"; in addition, the prompt word can include specific requirements for storyboard design, such as determining the expected matching video clip screen content, determining the expected entity description, the focus of the storyboard design and the storyboard scene, etc.

[0049] In order to make the search results more accurate, more other factors can be considered when forming the search conditions, so that the search results returned are more in line with the actual needs of the user. Some exemplary solutions are provided below.

[0050] In an optional embodiment, if Figure 2 As shown, step S104 may include: Step S200, obtaining the limiting conditions of the storyboard text, wherein the limiting conditions include the returned video duration limiting conditions and the matching material library type limiting conditions.

[0051] Step S202, forming a plurality of search conditions based on the limiting conditions of the storyboard text and the storyboard design.

[0052] The limiting conditions of the storyboard text can be input by the user and can be included in the query text. Optionally, the large language model can also add according to the text length corresponding to each storyboard text and the video creation type corresponding to the storyboard text. For example, the large language model determines the returned video duration limiting conditions according to the text length corresponding to each storyboard text, and the large language model determines the limiting conditions of the matching material library type (such as promotional videos, film and television dramas, empty shots, etc.) according to the video creation type corresponding to the storyboard text. Among them, the video creation type can be determined by the large language model after the semantic understanding of the storyboard text. When adding limiting conditions, the intelligent agent technology of the large language model can be used to use different intelligent agents to add different limiting conditions. In addition, the above-mentioned different processes of splitting the query text into multiple storyboard texts and storyboard design can also be carried out by different intelligent agents, so that multiple intelligent agents can perform different divisions of labor and collaboration, improve the accuracy of processing, and improve processing efficiency at the same time.

[0053] After obtaining the limiting conditions of the storyboard text, the limiting conditions and the storyboard design can be combined to obtain several search conditions.

[0054] In this optional embodiment, by forming the search conditions together with the limiting conditions of the storyboard text and the storyboard design, the search can be made more accurate and better meet the search needs of the user.

[0055] Because the search may not be able to retrieve enough or the search results the user wants due to the search conditions, in this case, a downgraded search can be performed to expand the search scope and retrieve more possible results. The following are some exemplary solutions: In an optional embodiment, the video retrieval solution provided by the embodiment of the present application may also include: repeatedly executing step S104, that is, repeatedly executing the step of performing storyboard design based on storyboard text using a large language model, and forming a plurality of retrieval conditions based on the storyboard design, so as to form at least two sets of retrieval conditions for each storyboard design, wherein at least one set of retrieval conditions is used for downgraded retrieval.

[0056] The search conditions corresponding to the downgraded search can be to generalize the original search keywords to obtain more general keywords; or to obtain words with high semantic similarity based on the original search keywords; or to transform the more specific scenarios in the original search conditions into more general scenarios. Specific forms include synonym replacement, hypernym replacement or sentence conversion.

[0057] In this embodiment, by repeatedly executing the step of forming the search conditions, at least two groups of search conditions are formed for each storyboard, and at least one group of search conditions is used for downgraded search. A flexible downgrade strategy can be implemented according to actual needs to improve the robustness and adaptability of the search system.

[0058] Step S106 , obtaining video retrieval results from the video retrieval system based on the retrieval conditions.

[0059] Specifically, when the video retrieval system supports keyword retrieval, the keywords in the retrieval conditions can be matched with the keywords in the video retrieval system, and the video retrieval results can be obtained according to the matching results; and / or, when the video retrieval system maintains vectors related to videos, etc., at least part of the content in the retrieval conditions can be converted into vectors, and the similarity can be calculated with the vectors in the video retrieval system, and the video retrieval results can be obtained according to the calculation results of the similarity. The following provides some exemplary schemes for the video retrieval system.

[0060] In an optional embodiment, the video retrieval system includes multiple video slices and vector features of each video slice, and the video retrieval system is used to recall retrieval results based on the vector features of the video slices.

[0061] The video retrieval system can slice the video in advance to obtain multiple video slices of each video; then, each video slice is converted into a vector to obtain the vector feature of each video slice; after receiving the retrieval condition, at least part of the content in the retrieval condition is converted into a vector, and the similarity between the converted vector and the vector feature of the video slice is calculated, and the corresponding video slice is returned when the similarity reaches a preset threshold. For example, the screen content of the video clip expected to be matched in the retrieval condition can be converted into a vector, and then the converted vector is calculated with the vector feature of the video slice in the video retrieval system. Similarity.

[0062] In this embodiment, the video retrieval system includes multiple video slices and vector features of each video slice. The video retrieval system recalls retrieval results based on the vector features of the video slices, so that the video retrieval system can support the recall of video slices and achieve more fine-grained retrieval.

[0063] In an optional embodiment, the video retrieval system also includes entity information included in the video slice, and the retrieval system is further used to recall retrieval results based on the entity information.

[0064] The entity information may specifically be a keyword corresponding to the entity. After obtaining the search condition, the video search system may match the keyword in the search condition with the keyword corresponding to the entity, and obtain the video search result according to the matching result.

[0065] In this embodiment, the video retrieval system also includes entity information included in the video slice, and the retrieval system is also used to recall retrieval results based on the entity information, so that the video retrieval system can support recall based on entity information and improve the accuracy of recall.

[0066] When obtaining video retrieval results from a video retrieval system based on retrieval conditions, in order to improve the recall effect, a batch of results may be roughly recalled first, and then a portion of the results may be selected from the recalled results as the final recalled results. Some exemplary solutions are provided below.

[0067] In an optional embodiment, if Figure 3 As shown, step S106 may include: Step S300 , performing a coarse-grained recall from the video retrieval system based on the retrieval condition to obtain a coarse-grained recall result.

[0068] Step S302 , performing multiple-dimensional scoring on the coarse-grained recall results to obtain multiple-dimensional scoring results for each of the coarse-grained recall results.

[0069] Step S304, sorting the coarse-grained recall results based on the multiple dimension scoring results, and obtaining the video retrieval results based on the sorting results.

[0070] Specifically, keyword matching and / or similarity calculations can be performed in the video retrieval system based on the retrieval conditions, and all results whose keyword matching and / or similarity meet the preset conditions are used as the results of the coarse-grained recall. When all results are used as the results of the coarse-grained recall, if the number of recalls is too large, only part of the results can be used as the results of the coarse-grained recall to reduce the amount of subsequent calculations.

[0071] After obtaining the coarse-grained recall results, they can be scored based on multiple dimensions such as the picture quality of the coarse-grained recall results, the degree of matching with the retrieval conditions, and the popularity of the video, to obtain multiple dimensional scoring results for each coarse-grained recall result; then the multiple dimensional scoring results are sorted, and the top N coarse-grained recall results are returned as the final video retrieval results.

[0072] In this embodiment, by performing coarse-grained recall from the video retrieval system based on the retrieval conditions, scoring the coarse-grained recall results in multiple dimensions, and obtaining video retrieval results based on the scoring results in multiple dimensions, the accuracy of video retrieval can be effectively improved.

[0073] In an optional embodiment, if Figure 4 As shown, step S302 may include: Step S400, obtaining the entity matching degree, picture matching degree, picture quality and search condition priority of the coarse-grained recall result.

[0074] Step S402 : determining multiple dimension scoring results of each of the coarse-grained recall results based on the entity matching degree, the picture matching degree, the picture quality and the search condition priority.

[0075] The entity matching degree can be determined by first determining the entities included in the search conditions, and then comparing the entities included in the coarse-grained recall results with the entities included in the search conditions, and is determined based on the number of entity matches. The more the number of entity matches, the higher the entity matching degree.

[0076] The picture matching degree may be determined by the similarity between the vector obtained by converting the retrieval condition and the vector corresponding to the coarse-grained recall result (eg, the vector of the video slice), for example, the picture matching degree may be determined according to the cosine similarity between the two.

[0077] The picture quality can be determined by performing aesthetic scoring and optical flow scoring on the video corresponding to the coarse-grained recall result, and then using these scores to perform linear weighting. The aesthetic score can be scored based on the resolution, composition, color, light and shadow, and content of the video based on a machine learning algorithm; the optical flow score can be scored by calculating the optical flow field of adjacent frames in the video using an optical flow estimation algorithm.

[0078] The retrieval condition priority may be the priority of the retrieval condition corresponding to the coarse-grained recall result, which may be specifically determined according to different scenarios and is not limited here.

[0079] After obtaining the entity matching, image matching, image quality and retrieval condition priority of the coarse-grained recall result, the entity matching, image matching, image quality and retrieval condition priority can be integrated and scored. There are many ways to integrate the scoring, such as weighted summation, and obtaining multiple dimensional scoring results of each coarse-grained recall result based on the weighted summation result. For example, the following formula can be used to calculate the multiple dimensional scoring results: score=aX+bY+cZ+dW; Among them, X represents entity matching, Y represents picture matching, Z represents picture quality, W represents retrieval condition priority, a represents the weight of entity matching, b represents the weight of picture matching, c represents the weight of picture quality, and d represents the weight of retrieval condition priority. Each weight can be learned from the model or adjusted according to test data.

[0080] In this embodiment, by obtaining the entity matching degree, picture matching degree, picture quality and retrieval condition priority of the coarse-grained recall results, and determining the scoring results of multiple dimensions according to the entity matching degree, picture matching degree, picture quality and retrieval condition priority, the coarse-grained recall results can be effectively and accurately evaluated, thereby effectively improving the accuracy of the retrieval results.

[0081] In actual applications, searching based on search conditions may result in fewer search results. In this case, the search can be downgraded to expand the search scope and obtain more search results. Some exemplary solutions are provided below.

[0082] In an optional embodiment, if Figure 5 As shown, step S300 may include: Step S500: determining all entity constraints to be retrieved based on the retrieval condition.

[0083] Step S502: Recall the first video retrieval results that meet all entity constraints.

[0084] Step S504: when the number of the first video retrieval results is less than a preset number, recall the second video retrieval results that meet some entity constraints.

[0085] At the beginning of the search, all entity constraints to be retrieved can be determined based on the search conditions, and then the video retrieval results that meet all entity constraints can be tried to be recalled as the first video retrieval results; the number of first video retrieval results is obtained, and if the number is less than a preset number, the video retrieval results of one or several entity constraints among all entity constraints can be tried to be recalled as the second video retrieval results. Optionally, when the number of first video retrieval results is less than a preset number, the video retrieval results of any entity constraint can be tried to be recalled as the second video retrieval result. After the second video retrieval result is obtained, the first video retrieval result and the second video retrieval result can be integrated as a coarse-grained recall result.

[0086] In this embodiment, by recalling video retrieval results that meet some entity constraints when the number of retrieval results recalled based on all entity constraints is insufficient, it is possible to downgrade video retrieval, expand the retrieval scope, and return more possible retrieval results.

[0087] In an optional embodiment, if Figure 6 As shown, the video retrieval method provided in the embodiment of the present application may also include: Step S600: determine the total number of the first video retrieval results and the second video retrieval results.

[0088] Step S602: When the total number is less than the preset number, switch to the next search condition for searching.

[0089] The second video retrieval result may include all or part of the first video retrieval result. When determining the total number of the first video retrieval result and the second video retrieval result, the number of overlapping parts of the two can be calculated only once, and then the difference between the two parts and the result of the calculation of the overlapping part can be added to obtain the total number of the first video retrieval result and the second video retrieval result. Then, the total number is compared with the preset number. If the total number is less than the preset number, switch to the next retrieval condition for retrieval. In the case of grouping of retrieval conditions, it can be switched to the next group of retrieval conditions for retrieval. For example, if there are three groups of retrieval conditions, the first group is a normal retrieval, the second group is a downgraded retrieval, and the third group is a downgraded retrieval based on the second group, then if the first group of retrieval does not reach the preset number, switch to the second group for retrieval; if necessary, switch to the third group for retrieval.

[0090] In this embodiment, by determining the total number of the first video retrieval results and the second video retrieval results, when the total number is less than a preset number, switching to the next retrieval condition for retrieval can be performed, and effective downgraded retrieval can be performed based on multiple retrieval conditions, thereby expanding the retrieval scope and returning more retrieval results.

[0091] In order to make this application easier to understand, the following Figure 7 Provide an example application. Figure 7 As shown, video retrieval can include the following processes: 1. The user enters the complete text, which is then fed into the large language model; 2. The large language model splits the text input by the user and passes the split results to the search condition design module; 3. The search condition design module performs storyboard splitting, search condition addition, storyboard design, search condition merging and downgrade strategy design to form the search conditions used in the end; 4. The retrieval condition design module transmits the formed retrieval condition to the multi-modal multi-level multi-way recall module; 5. The multimodal multi-level multi-way recall module determines the capability of the retrieval system, converts the content that needs to be converted into vectors in the retrieval conditions into vectors, and then attempts to recall the retrieval results that meet all entity constraints from the video slice database; if the retrieval results are insufficient, then attempts to recall the retrieval results of any entity constraint to perform downgraded recall; if the retrieval results are still insufficient, then switches to the next set of retrieval conditions to continue the retrieval until all retrieval conditions have been searched or the retrieval results reach the preset number, and a coarse-grained recall result is obtained; 6. The multimodal multi-level multi-way recall module passes the coarse-grained recall results to the fusion scoring and re-ranking module; 7. The fusion scoring and re-ranking module performs multi-dimensional scoring and fusion scoring on the coarse-grained recall results, re-ranks the coarse-grained recall results according to the fusion scoring results, and performs tail truncation as required based on the re-ranking results to obtain the final recall results; 8. The fusion scoring and re-ranking module returns the final recall result to the user.

[0092] In this exemplary application, through storyboard splitting, retrieval condition design, multimodal multi-level multi-way recall, fusion scoring and re-ranking, the retrieval system can more accurately understand the user's retrieval intent, make the retrieved video materials more in line with the user's needs, and improve the accuracy of retrieval and user experience.

[0093] Embodiment 2 Figure 8 The block diagram of the video retrieval device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 8 As shown, the apparatus 700 may include: an acquisition module 710, a splitting module 720, a design module 730 and a retrieval module 740, wherein: The acquisition module 710 is used to acquire the input query text; A splitting module 720 is used to split the query text into multiple storyboard texts using a large language model; A design module 730 is used to perform a storyboard design based on the storyboard text using the large language model, and form a plurality of search conditions based on the storyboard design, wherein the storyboard design includes the video clip content expected to be matched and the entity description expected to appear; The retrieval module 740 is used to obtain video retrieval results from the video retrieval system based on the retrieval conditions.

[0094] As an optional embodiment, several search conditions are formed based on the storyboard design, including: Acquire the limiting conditions of the storyboard text, wherein the limiting conditions include the returned video duration limiting conditions and the matching material library type limiting conditions; A number of search conditions are formed based on the limiting conditions of the storyboard text and the storyboard design.

[0095] As an optional embodiment, the device 700 is further used for: Repeat the steps of using the large language model to perform storyboard design based on the storyboard text, and forming a plurality of search conditions based on the storyboard design, so as to form at least two groups of search conditions for each storyboard design, wherein at least one group of search conditions is used for downgraded search.

[0096] As an optional embodiment, obtaining a video retrieval result from a video retrieval system based on the retrieval condition includes: Performing coarse-grained recall from the video retrieval system based on the retrieval condition to obtain a coarse-grained recall result; Scoring the coarse-grained recall results in multiple dimensions to obtain scoring results in multiple dimensions for each of the coarse-grained recall results; The coarse-grained recall results are sorted based on the multiple dimension scoring results, and the video retrieval results are obtained based on the sorting results.

[0097] As an optional embodiment, the coarse-grained recall results are scored in multiple dimensions to obtain multiple dimension scoring results for each of the coarse-grained recall results, including: Obtaining entity matching degree, picture matching degree, picture quality and retrieval condition priority of the coarse-grained recall result; The multiple dimension scoring results of each of the coarse-grained recall results are determined based on the entity matching degree, the picture matching degree, the picture quality and the retrieval condition priority.

[0098] As an optional embodiment, performing a coarse-grained recall from the video retrieval system based on the retrieval condition to obtain a coarse-grained recall result includes: Determine all entity constraints for retrieval based on the retrieval condition; Recall the first video retrieval result that meets all entity constraints; When the number of the first video retrieval results is less than a preset number, the second video retrieval results that meet the partial entity constraints are recalled.

[0099] As an optional embodiment, the device 700 is further used for: Determine the total number of the first video search results and the second video search results; When the total number is less than the preset number, the search is switched to the next search condition for searching.

[0100] As an optional embodiment, the video retrieval system includes multiple video slices and vector features of each of the video slices, and the video retrieval system is used to recall retrieval results based on the vector features.

[0101] As an optional embodiment, the video retrieval system further includes entity information included in the video slice, and the retrieval system is further used to recall retrieval results based on the entity information.

[0102] Embodiment 3 Fig. 9 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the video retrieval method according to the third embodiment of the present application is shown schematically. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig. 9 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed in the computer device 10000, such as program codes of a video retrieval method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0103] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0104] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0105] It should be pointed out that Fig. 9 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0106] In this embodiment, the video retrieval method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0107] Embodiment 4 An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the video retrieval method in the embodiment are implemented.

[0108] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the video retrieval method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0109] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0110] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0111] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video retrieval method, characterized in that: The method comprises: Get the input query text; Using a large language model to split the query text into multiple storyboard texts; Using the large language model to perform storyboard design based on the storyboard text, and forming a plurality of search conditions based on the storyboard design, wherein the storyboard design includes the video clip content expected to be matched and the entity description expected to appear; A video retrieval result is obtained from the video retrieval system based on the retrieval condition.

2. The method according to claim 1, characterized in that Based on the storyboard design, several search conditions are formed, including: Acquire the limiting conditions of the storyboard text, wherein the limiting conditions include the returned video duration limiting conditions and the matching material library type limiting conditions; A number of search conditions are formed based on the limiting conditions of the storyboard text and the storyboard design.

3. The method according to claim 2, characterized in that The method further comprises: Repeat the steps of using the large language model to perform storyboard design based on the storyboard text, and forming a plurality of search conditions based on the storyboard design, so as to form at least two groups of search conditions for each storyboard design, wherein at least one group of search conditions is used for downgraded search.

4. The method according to claim 1, characterized in that: Obtaining video retrieval results from the video retrieval system based on the retrieval condition includes: Performing coarse-grained recall from the video retrieval system based on the retrieval condition to obtain a coarse-grained recall result; Scoring the coarse-grained recall results in multiple dimensions to obtain scoring results in multiple dimensions for each of the coarse-grained recall results; The coarse-grained recall results are sorted based on the multiple dimension scoring results, and the video retrieval results are obtained based on the sorting results.

5. The method according to claim 4, characterized in that The coarse-grained recall results are scored in multiple dimensions to obtain multiple dimension scoring results for each of the coarse-grained recall results, including: Obtaining entity matching degree, picture matching degree, picture quality and retrieval condition priority of the coarse-grained recall result; The multiple dimension scoring results of each of the coarse-grained recall results are determined based on the entity matching degree, the picture matching degree, the picture quality and the retrieval condition priority.

6. The method according to claim 4, characterized in that Based on the retrieval condition, a coarse-grained recall is performed from the video retrieval system to obtain a coarse-grained recall result, including: Determine all entity constraints for retrieval based on the retrieval condition; Recall the first video retrieval result that meets all entity constraints; When the number of the first video retrieval results is less than a preset number, the second video retrieval results that meet the partial entity constraints are recalled.

7. The method according to claim 6, characterized in that The method further comprises: Determine the total number of the first video search results and the second video search results; When the total number is less than the preset number, the search is switched to the next search condition for searching.

8. The method according to any one of claims 1 to 7, characterized in that: The video retrieval system includes multiple video slices and vector features of each of the video slices, and the video retrieval system is used to recall retrieval results based on the vector features.

9. The method according to claim 8, characterized in that The video retrieval system also includes entity information included in the video slice, and the retrieval system is also used to recall retrieval results based on the entity information.

10. A video retrieval device, characterized in that: The device comprises: The acquisition module is used to obtain the input query text; A splitting module, used for splitting the query text into multiple storyboard texts using a large language model; A design module, configured to use the large language model to perform storyboard design based on the storyboard text, and form a plurality of search conditions based on the storyboard design, wherein the storyboard design includes the video clip content expected to be matched and the entity description expected to appear; The retrieval module is used to obtain video retrieval results from the video retrieval system based on the retrieval conditions.

11. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images