Video retrieval method and device

Abstract

The embodiment of the invention provides a video retrieval method, and the method comprises the steps: obtaining N videos, segmenting the videos, and obtaining video slices of the videos, N being a positive integer; multiple pieces of modal information of the video slices are obtained, and the modal information comprises visual modal information, audio modal information and text modal information; constructing a video retrieval library based on the multiple pieces of modal information and the video slices; and obtaining video retrieval information, and determining a target video slice corresponding to the video retrieval information in the video retrieval library based on the multiple pieces of modal information. According to the technical scheme, multi-modal information can be fused for video retrieval, the accuracy of video retrieval is improved, meanwhile, fine-grained retrieval is achieved, and the retrieval fineness is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular, to a video retrieval method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Video retrieval refers to the process of finding video content that meets user needs in massive video data in a specific way. At present, the main methods of video retrieval include metadata-based retrieval, deep learning-based retrieval, or concept-based retrieval.

[0003] However, current video retrieval methods are unable to deeply explore video content, cannot use multimodal information for retrieval, have excessive computational complexity, and have too large a retrieval granularity, resulting in low retrieval accuracy and efficiency, and unsatisfactory results.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a video retrieval method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0006] One aspect of an embodiment of the present application provides a video retrieval method, the method comprising: Obtain N videos, and segment the videos to obtain video slices of the videos, where N is a positive integer; Acquire multiple modal information of the video slice, wherein the modal information includes visual modal information, audio modal information and text modal information; Building a video retrieval library based on the multiple modal information and the video slices; Video retrieval information is obtained, and a target video slice corresponding to the video retrieval information is determined in the video retrieval library based on the multiple modal information.

[0007] Optionally, segmenting the video to obtain video slices of the video includes: Performing shot boundary detection on the frame image of the video to determine M shot boundaries included in the frame image, where M is a natural number; The video is sliced ​​based on the M shot boundaries to obtain video slices of the video.

[0008] Optionally, slicing the video based on the M shot boundaries to obtain video slices of the video includes: When the duration of the current video slice is less than the minimum limited duration, the tail of the current video slice is extended so that the duration of the current video slice is greater than or equal to the minimum limited duration; When the duration of the current video slice is greater than the maximum limited duration, the current video slice is cropped twice so that the duration of the video slices obtained by the secondary cropping is less than or equal to the maximum limited duration.

[0009] Optionally, obtaining multiple modal information of the video slice includes: Performing target evaluation on each of the video slices, wherein the target evaluation includes image quality evaluation and optical flow evaluation; Screening the video slices according to the evaluation results to obtain screened video slices; Acquire multiple modal information of the filtered video slices; Correspondingly, constructing a video retrieval library based on the multiple modal information and the video slices includes: A video retrieval library is constructed based on the multiple modal information and the filtered video slices.

[0010] Optionally, obtaining multiple modal information of the video slice includes: Performing face recognition and face feature extraction on the video slice to obtain a face feature vector of the video slice; Performing speech recognition on the audio of the video slice to obtain a speech recognition text of the video slice; Performing optical character recognition on the frame image of the video slice to obtain optically recognized text of the video slice; The frame image of the video slice is converted into a low-dimensional vector to obtain a video vector of the video slice.

[0011] Optionally, constructing a video retrieval library based on the multiple modal information and the video slices includes: Acquire video dimension information of the video, where the video dimension information includes at least one of a video title, video description information, and video tag information; Using a plurality of expert models to identify the video slices to obtain model identification results, wherein each of the expert models is used for a classification task; A video retrieval library is constructed based on the video dimension information, the model recognition result and the video slice information, and an index is constructed according to the video retrieval library, wherein the video slice information includes the face feature vector, the speech recognition text, the optical recognition text and the video vector.

[0012] Optionally, determining a target video slice corresponding to the video retrieval information in the video retrieval library based on the multiple modal information includes: matching the video retrieval information with at least a portion of the modality information; At least part of the video slices are obtained from the matching results as target video slices corresponding to the video retrieval information.

[0013] Another aspect of the embodiments of the present application provides a video retrieval device, the device comprising: A segmentation module, used to obtain N videos, segment the videos to obtain video slices of the videos, where N is a positive integer; An acquisition module, used to acquire multiple modal information of the video slice, wherein the modal information includes visual modal information, audio modal information and text modal information; A construction module, used for constructing a video retrieval library based on the multiple modal information and the video slices; The retrieval module is used to obtain video retrieval information and determine a target video slice corresponding to the video retrieval information in the video retrieval library based on the multiple modal information.

[0014] Another aspect of an embodiment of the present application provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein: the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described above.

[0015] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0016] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0017] The above technical solution adopted in the embodiment of the present application may have the following advantages: By acquiring N videos, slicing the videos, obtaining video slices of the videos, acquiring multiple modal information of the video slices, the modal information including visual modal information, audio modal information and text modal information, constructing a video retrieval library based on the multiple modal information and the video slices, acquiring the video retrieval information, and determining the target video slices corresponding to the video retrieval information in the video retrieval library based on the multiple modal information. It can be seen that the embodiment of the present application can retrieve and return video slices by slicing the videos, realize more fine-grained retrieval, and improve the precision of the retrieval; at the same time, by constructing a video retrieval library through multi-modal information and determining the retrieval results based on the multi-modal information, the multi-modal information can be integrated to accurately capture the video content and improve the accuracy of the video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 A flowchart of a video retrieval method according to Embodiment 1 of the present application is schematically shown; Figure 2 Schematically shows Figure 1 Flow chart of sub-steps of step S100; Figure 3 Schematically shows Figure 2 Flow chart of sub-steps of step S202; Figure 4 Schematically shows Figure 1 Flow chart of sub-steps of step S102; Figure 5 Schematically shows Figure 1 Another sub-step flow chart of step S102; Figure 6 Schematically shows Figure 1 Flow chart of sub-steps of step S104; Figure 7 Schematically shows Figure 1 Flow chart of sub-steps of step S106; Figure 8 The schematic diagram shows a principle diagram of a video retrieval method according to the first embodiment of the present application; Fig. 9 A block diagram schematically shows a video retrieval device according to the second embodiment of the present application; and Fig.10 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0021] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0023] First, the following terms are explained: Shot boundary detection: also known as shot segmentation or shot switching detection, refers to automatically locating shot switching points in a video sequence and segmenting the video into a series of continuous shot segments. In other words, it is to find the dividing line between different scenes in the video.

[0024] Video slicing: split a complete video file into multiple smaller video clips. These clips can be of equal or unequal lengths.

[0025] Aesthetic scoring: It is a way to quantify the aesthetic level of videos and other works. It attempts to simulate human perception of beauty through computer algorithms and give an objective and quantitative aesthetic evaluation.

[0026] Optical flow: refers to the apparent motion of each pixel in an image sequence between consecutive image frames, that is, the instantaneous speed of the pixel motion of a moving object in space on the observation imaging plane.

[0027] K-Nearest Neighbors (KNN): is a basic classification and regression algorithm based on the assumption that the category of a sample is most similar to the categories of its nearest k neighbors.

[0028] BM25: A ranking function for text retrieval.

[0029] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: Video retrieval has a wide range of demands and applications in security monitoring, video editing, education and other scenarios. At present, the main methods of video retrieval include metadata-based retrieval, content-based retrieval, deep learning-based retrieval, keyframe-based retrieval and concept detection-based retrieval.

[0030] However, the applicant has learned that the related video retrieval methods still have the following defects: (1) Since the entire video often needs to be processed, the indexing and retrieval time is long and the retrieval efficiency is low; (2) Using a single modality or simple meta-information for retrieval fails to fully utilize the multimodal information of the video, making it difficult to accurately capture the semantic content of the video, thus affecting retrieval accuracy; (3) There is a lot of invalid data, and the retrieval system loses a lot of efficiency in processing invalid data; (4) The retrieval granularity is relatively coarse, making it difficult to achieve fine-grained retrieval of video content.

[0031] To this end, the embodiment of the present application provides a video retrieval technology solution. In this technical solution: (1) by processing the video slices, a more fine-grained retrieval of the video content can be achieved, and refined retrieval can be realized; (2) by integrating multimodal information to fully capture the video content, the accuracy of video retrieval can be improved; (3) by quality assessment and preliminary filtering, invalid data processing can be reduced and the efficiency of the retrieval system can be improved; (4) by extracting multi-dimensional features of video slices, parallel processing and fast retrieval can be achieved. See below for details.

[0032] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0033] Embodiment 1 Figure 1 The flowchart of the video retrieval method according to the first embodiment of the present application is schematically shown.

[0034] like Figure 1 As shown, the video retrieval method may include steps S100 to S106, wherein: Step S100, obtaining N videos, dividing the videos into segments to obtain video slices of the videos, where N is a positive integer.

[0035] Step S102, obtaining multiple modal information of the video slice, wherein the modal information includes visual modal information, audio modal information and text modal information.

[0036] Step S104: construct a video retrieval library based on the multiple modal information and the video slices.

[0037] Step S106, obtaining video retrieval information, and determining a target video slice corresponding to the video retrieval information in the video retrieval library based on the multiple modal information.

[0038] The video retrieval method provided in this embodiment obtains N videos, slices the videos, obtains video slices of the videos, obtains multiple modal information of the video slices, the modal information includes visual modal information, audio modal information and text modal information, builds a video retrieval library based on the multiple modal information and video slices, obtains video retrieval information, and determines the target video slice corresponding to the video retrieval information in the video retrieval library based on the multiple modal information. It can be seen that the embodiment of the present application can retrieve and return video slices by slicing the video, realize more fine-grained retrieval, and improve the precision of the retrieval; at the same time, by constructing a video retrieval library through multi-modal information and determining the retrieval results based on the multi-modal information, the multi-modal information can be integrated to accurately capture the video content and improve the accuracy of video retrieval.

[0039] The following combination Figure 1 , each step in steps S100~S106 and other optional steps are explained in detail.

[0040] Step S100 , obtain N videos, segment the videos, and obtain video slices of the videos, where N is a positive integer.

[0041] Specifically, a large number of videos can be collected through the Internet and / or from an internal data resource library, and then each video can be sliced ​​to obtain a video slice of each video. When slicing a video, the video can be cut into a video slice based on a fixed time interval (such as 5 seconds); or a certain target (such as a person) in the video can be tracked and sliced ​​when the target changes significantly. Some exemplary solutions are provided below.

[0042] In an optional embodiment, if Figure 2 As shown, step S100 may include: Step S200: performing shot boundary detection on the frame image of the video to determine M shot boundaries included in the frame image, where M is a natural number.

[0043] Step S202: Slice the video based on the M shot boundaries to obtain video slices of the video.

[0044] When performing shot boundary detection on the frame images of a video, the shot boundary can be determined by judging whether a shot switch occurs between adjacent frame images. The determination of shot switching can be performed by calculating the pixel difference between adjacent frame images of the video, and determining that a shot switch occurs when the pixel difference between adjacent frame images exceeds a preset threshold; or, the histograms between adjacent frame images of the video can be compared, and when the histogram difference is large, the shot switch occurs; or, the motion information between adjacent frame images can be analyzed, and when the motion information changes significantly, the shot switch occurs; or, a neural network model (such as a deep learning model) can be used to extract the deep features of the frame images of the video, and the extracted features can be used to determine whether a shot switch occurs.

[0045] In this embodiment, by performing shot boundary retrieval on the frame image of the video, M shot boundaries in the frame image are determined, and the video is sliced ​​at the M shot boundaries to obtain video slices of the video. The video can be effectively sliced ​​to generate video slices with independent semantics, which is convenient for subsequent retrieval to return video slices with finer granularity, thereby improving the accuracy of the retrieval. At the same time, it also makes it unnecessary to process the entire video, reducing the indexing and retrieval time and improving the efficiency of the retrieval.

[0046] It is understandable that when slicing a video based on shot boundary detection, the duration of each video slice is not equal, and the duration of some video slices is too short, while the duration of some video slices is too long. In order to make the duration of the video slices more uniform, the duration of the video slices that are too short can be extended, while the duration of the video slices that are too long can be shortened. Some exemplary solutions are provided below.

[0047] In an optional embodiment, if Figure 3 As shown, step S202 may include: Step S300: When the duration of the current video slice is less than the minimum limited duration, the tail of the current video slice is extended to make the duration of the current video slice greater than or equal to the minimum limited duration.

[0048] Step S302: When the duration of the current video slice is greater than the maximum duration, the current video slice is cropped for a second time, so that the duration of the video slices obtained by the second cropping is less than or equal to the maximum duration.

[0049] Specifically, after slicing the video based on M shot boundaries, the duration of each video slice can be determined. If the duration of the current video slice is less than the minimum duration, the tail of the current video slice is extended to make the duration of the current video slice greater than or equal to the minimum duration. For example, the last few frames of the current video slice can be copied continuously until the minimum duration is reached; or a silent video can be added to the end of the current video slice so that the duration after the addition reaches the minimum duration; or the front part of the next video slice can be incorporated so that the duration after the incorporation reaches the minimum duration.

[0050] Similarly, the duration of each video slice can be compared with the maximum duration. If the duration of the current video slice is greater than the maximum duration, the current video slice is cropped twice so that the duration of the video slices obtained by the secondary cropping is less than or equal to the maximum duration. For example, the current video slice can be equally divided and cropped into two or more video slices so that each cropped video slice is less than the maximum duration. Alternatively, the video slices are cropped using a pre-trained machine learning model so that each cropped video slice is less than the maximum duration. Optionally, the maximum duration can be twice or more than the minimum duration, so that the duration of the video slice obtained after the secondary cropping will not be less than the minimum duration. Alternatively, a judgment is made before the secondary cropping. If the duration of the video slice after the secondary cropping is less than the minimum duration, other cropping schemes are adopted.

[0051] In this embodiment, when the current video slice is less than the minimum limited duration, the tail of the current video slice is extended so that the duration of the current video slice is greater than or equal to the minimum limited duration; when the duration of the current video slice is greater than the maximum limited duration, the current video slice is cropped twice so that the duration of the video slices obtained by the secondary cropping is less than or equal to the maximum limited duration. This can achieve adaptive control of the duration of the video slices, thereby ensuring that the duration of the video slices is more suitable for actual application scenarios.

[0052] Step S102 , obtaining multiple modal information of the video slice, wherein the modal information includes visual modal information, audio modal information and text modal information.

[0053] It is understandable that, without processing, there may be many low-quality video slices, such as video slices with poor image quality. Returning these video slices as search results will affect the search quality. At the same time, processing these low-quality video slices will also cause efficiency loss. Therefore, the video slices can be filtered to improve the search quality and reduce the efficiency loss. Some exemplary solutions are provided below.

[0054] In an optional embodiment, if Figure 4 As shown, step S102 may include: Step S400: performing target evaluation on each of the video slices, wherein the target evaluation includes image quality evaluation and optical flow evaluation.

[0055] Step S402: Screen the video slices according to the evaluation results to obtain screened video slices.

[0056] Step S404, obtaining multiple modal information of the filtered video slices.

[0057] Correspondingly, the subsequent step S104 may include: constructing a video retrieval library based on multiple modal information and the filtered video slices.

[0058] Image quality assessment can be performed using methods such as structural similarity index, peak signal-to-noise ratio, information entropy, etc., or using a pre-trained image quality assessment model. The image quality assessment module can be implemented using a deep learning model. Optical flow assessment can use an optical flow estimation algorithm to calculate the optical flow field between adjacent frame images of a video slice, and perform assessment by calculating the amount of motion between frames.

[0059] Specifically, image quality evaluation and optical flow evaluation can be performed on each video slice separately to identify video slices that may be of low quality, such as distortion, overexposure, poor image quality, and small dynamic range. Then, these low-quality video slices are filtered to select high-quality video slices to obtain filtered video slices. Based on the filtered video slices, multiple modal information is obtained. Subsequently, a video retrieval library is constructed based on the filtered video slices. Low-quality video slices can be deleted or stored separately as alternative resources.

[0060] In this embodiment, image quality evaluation and optical flow evaluation are performed on each video slice, and the video slices are screened according to the evaluation results to obtain screened video slices, and multiple modal information of the screened video slices is obtained. A video retrieval library is constructed based on the multiple modal information and the screened video slices, and low-quality video slices can be filtered out, thereby improving the retrieval quality of the video, while reducing the processing of low-quality videos and reducing efficiency losses.

[0061] When acquiring multiple modal information of a video slice, at least two modal information of multiple modal information such as video frame images, audio, subtitles, etc. may be acquired respectively to reflect the content of the video slice from multiple dimensions. Some exemplary solutions are provided below.

[0062] In an optional embodiment, if Figure 5 As shown, step S102 may include: Step S500, performing face recognition and face feature extraction on the video slice to obtain a face feature vector of the video slice.

[0063] Step S502: performing speech recognition on the audio of the video slice to obtain a speech recognition text of the video slice.

[0064] Step S504: performing optical character recognition on the frame image of the video slice to obtain optically recognized text of the video slice.

[0065] Step S506: convert the frame image of the video slice into a low-dimensional vector to obtain a video vector of the video slice.

[0066] Specifically, a pre-trained machine learning model (such as a deep learning model) can be used to perform face recognition on the video slice. When a face is recognized, the features of the recognized face are extracted to obtain a face feature vector of the video slice. The audio of the video slice is obtained, and the audio of the video slice is processed using speech recognition technology (such as automatic speech recognition (ASR)) to convert it into a speech recognition text of the video slice. The frame image of the video slice is obtained, and the frame image is processed using optical character recognition (OCR) technology, and the optical recognition text of the video slice is obtained according to the recognition result. The frame image of the video slice is converted into a low-dimensional vector through a video vector model to obtain a video vector of the video slice. Among them, the face feature vector facilitates the subsequent video retrieval library to support the recall of retrieval results by name or photo; the speech recognition text and the optical recognition text facilitate the subsequent video retrieval library to support the recall of retrieval results by keywords; and the video vector facilitates the subsequent video retrieval library to support the recall of retrieval results by picture description.

[0067] It is worth mentioning that the face feature vector, speech recognition text, optical recognition text and video vector, as well as the model recognition results of the subsequent expert model can be processed in parallel to improve the processing efficiency.

[0068] In this embodiment, by performing face recognition and face feature extraction on video slices to obtain face feature vectors, performing speech recognition on the audio of the video slices to obtain speech recognition text, performing optical character recognition on the frame images of the video slices to obtain optical recognition text, and converting the frame images of the video slices into low-dimensional vectors to obtain video vectors, information of the video slices can be extracted from multiple dimensions, which is convenient for subsequent support of multiple-dimensional retrieval, and is also conducive to simultaneous retrieval from multiple dimensions, thereby improving the accuracy of video retrieval.

[0069] Step S104 , building a video retrieval library based on the multiple modal information and the video slices.

[0070] After obtaining multiple modal information of video slices, a video retrieval library can be directly constructed based on the obtained multiple modal information and video slices. It is also possible to add other scene or business required information based on the obtained multiple modal information, and jointly construct a video retrieval library with the added other information. Some exemplary solutions are provided below.

[0071] In an optional embodiment, if Figure 6 As shown, step S104 may include: Step S600, obtaining video dimension information of the video, wherein the video dimension information includes at least one of a video title, video description information, and video tag information.

[0072] Step S602: Use a plurality of expert models to identify the video slices to obtain model identification results, wherein each of the expert models is used for a classification task.

[0073] Step S604, constructing a video retrieval library based on the video dimension information, the model recognition result and the video slice information, and constructing an index according to the video retrieval library, wherein the video slice information includes the face feature vector, the speech recognition text, the optical recognition text and the video vector.

[0074] Among them, the expert model can be used to identify specific objects including landmarks, brands, products, etc. in the frame images of video slices. Specifically, several expert models can be added according to the scene and business needs, and the model recognition results can be obtained based on the expert model's recognition results of the frame images of the video slices.

[0075] Specifically, the video dimension information of the video can be obtained based on the information contained in the video itself or other data sources; several expert models are used to identify the video slices to obtain the model recognition results; then, the video dimension information, model recognition results, and the above-mentioned face feature vector, speech recognition text, optical recognition text, optical recognition text and video vector are used to construct a video retrieval library, and the index of the video retrieval library is constructed based on these elements to facilitate subsequent retrieval. Among them, the index structure can be, for example: (1) video dimension information: video title, video description information, video tag information; (2) text information of the slice dimension: speech recognition text, optical character recognition text; (3) vector information of the slice dimension: face feature vector, video vector; (4) other information of the slice dimension: model recognition results, etc. For the indexing of text information, sparse indexing technology (such as BM25) can be used for indexing; for vector information, similarity retrieval technology can be used for retrieval, such as KNN.

[0076] In this embodiment, by obtaining the video dimension information of the video, an expert model is used to identify the video slices to obtain a model recognition result, and a video retrieval library and an index are constructed based on the video dimension information, model recognition results, face feature vectors, speech recognition text, optical character recognition text and video vectors. This allows the video retrieval library to support retrieval in multiple dimensions, facilitate multi-way recall, improve the flexibility and applicability of the retrieval, and also improve the accuracy of the retrieval.

[0077] Step S106 , obtain video retrieval information, and determine a target video slice corresponding to the video retrieval information in the video retrieval library based on the multiple modal information.

[0078] The video retrieval information may be text and / or image, wherein the text may be a keyword or a screen description. When the video retrieval information is a keyword, the keyword may be matched with the text (such as speech recognition text) included in the video retrieval library, and the corresponding target video slice may be returned based on the matching situation. When the video retrieval information is a screen description, the screen description may be converted into a vector, and then the similarity between the converted vector and the vector included in the video retrieval library may be calculated, and the corresponding target video slice may be returned based on the calculation result of the similarity. When the video retrieval information is an image, the image may be converted into a vector, and then the similarity of the vector may be calculated to return the corresponding target video slice. Alternatively, if the video retrieval information is a face image, the face image may be identified and features extracted, and then the extracted features may be similarity calculated with the face feature vector in the video retrieval library, and the corresponding target video slice may be returned.

[0079] When searching, the video search information may be matched with one or more modal information to search; when the search results are obtained, the search results may be directly returned or returned after filtering. Some exemplary solutions are provided below.

[0080] In an optional embodiment, if Figure 7 As shown, step S106 may include: Step S700, matching the video retrieval information with at least part of the modality information.

[0081] Step S702: Acquire at least part of the video slices from the matching results as target video slices corresponding to the video retrieval information.

[0082] Specifically, when the video retrieval information is text, the search can be based on keywords; when the video retrieval information is an image, the image can be converted into a vector for vector search. When the video retrieval information includes both text and images, the text-based keyword search and the image-based vector search can be performed simultaneously.

[0083] After obtaining the matching results, the matching results can be directly returned as the target video slices. Alternatively, the matching results can be scored for picture quality and content relevance, sorted according to the scoring results, and then the top matching results can be returned as the target video slices.

[0084] In this embodiment, by matching the video retrieval information with at least part of the modal information, at least part of the video slices are obtained from the matching results as the target video slices corresponding to the video retrieval information. Different matches can be performed according to different retrieval information, thereby improving the flexibility and applicability of the retrieval; at the same time, the retrieval results can be further screened as needed to improve the quality of the returned retrieval results.

[0085] In order to make this application easier to understand, the following Figure 8 Here is an example application: 1. Perform shot boundary detection on the original video and preliminarily divide the video into multiple video slices; 2. Control the duration of each video slice: For video slices whose duration is less than the minimum limit, the tail is extended, and for video slices whose duration is greater than the maximum limit, secondary cropping is performed to make the cropped video slices less than the maximum limit; 3. Perform aesthetic scoring and optical flow scoring on each video slice, and perform preliminary filtering on the video slices based on the results of the aesthetic scoring and optical flow scoring to filter out some low-quality video slices; 4. Perform feature extraction on the filtered video slices, and perform face vector extraction, speech recognition, optical character recognition, video vector extraction and other expert model recognition processing on the video slices in parallel; 5. Perform feature indexing based on the extracted features. For text, sparse indexing techniques such as BM25 can be used, and for vectors, similarity indexing such as KNN can be used to form a multimodal retrieval system. 6. Perform search based on the search information entered by the user and return the video slice as the search result.

[0086] In this exemplary application, by slicing the video, a more fine-grained retrieval of the video content can be achieved; by fusing multiple modal information, the video content can be retrieved based on the multiple modal information, thereby improving the accuracy of video retrieval; at the same time, by performing quality assessment and filtering on the video slices, the processing of invalid data can be reduced, thereby improving the efficiency of the retrieval system; in addition, by performing multi-dimensional feature extraction on the video slices, parallel processing and fast retrieval can be achieved.

[0087] Embodiment 2 Fig. 9The block diagram of the video retrieval device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Fig. 9 As shown, the device 800 may include: a segmentation module 810, an acquisition module 820, a construction module 830 and a retrieval module 840, wherein: A segmentation module 810 is used to obtain N videos and segment the videos to obtain video slices of the videos, where N is a positive integer; An acquisition module 820, configured to acquire multiple modal information of the video slice, wherein the modal information includes visual modal information, audio modal information, and text modal information; A construction module 830, configured to construct a video retrieval library based on the multiple modal information and the video slices; The retrieval module 840 is used to obtain video retrieval information and determine a target video slice corresponding to the video retrieval information in the video retrieval library based on the multiple modal information.

[0088] As an optional embodiment, segmenting the video to obtain video slices of the video includes: Performing shot boundary detection on the frame image of the video to determine M shot boundaries included in the frame image, where M is a natural number; The video is sliced ​​based on the M shot boundaries to obtain video slices of the video.

[0089] As an optional embodiment, slicing the video based on the M shot boundaries to obtain video slices of the video includes: When the duration of the current video slice is less than the minimum limited duration, the tail of the current video slice is extended so that the duration of the current video slice is greater than or equal to the minimum limited duration; When the duration of the current video slice is greater than the maximum limited duration, the current video slice is cropped twice so that the duration of the video slices obtained by the secondary cropping is less than or equal to the maximum limited duration.

[0090] As an optional embodiment, obtaining multiple modality information of the video slice includes: Performing target evaluation on each of the video slices, wherein the target evaluation includes image quality evaluation and optical flow evaluation; Screening the video slices according to the evaluation results to obtain screened video slices; Acquire multiple modal information of the filtered video slices; Correspondingly, constructing a video retrieval library based on the multiple modal information and the video slices includes: A video retrieval library is constructed based on the multiple modal information and the filtered video slices.

[0091] As an optional embodiment, obtaining multiple modality information of the video slice includes: Performing face recognition and face feature extraction on the video slice to obtain a face feature vector of the video slice; Performing speech recognition on the audio of the video slice to obtain a speech recognition text of the video slice; Performing optical character recognition on the frame image of the video slice to obtain optically recognized text of the video slice; The frame image of the video slice is converted into a low-dimensional vector to obtain a video vector of the video slice.

[0092] As an optional embodiment, constructing a video retrieval library based on the multiple modal information and the video slices includes: Acquire video dimension information of the video, where the video dimension information includes at least one of a video title, video description information, and video tag information; Using a plurality of expert models to identify the video slices to obtain model identification results, wherein each of the expert models is used for a classification task; A video retrieval library is constructed based on the video dimension information, the model recognition result and the video slice information, and an index is constructed according to the video retrieval library, wherein the video slice information includes the face feature vector, the speech recognition text, the optical recognition text and the video vector.

[0093] As an optional embodiment, determining a target video slice corresponding to the video retrieval information in the video retrieval library based on the multiple modal information includes: matching the video retrieval information with at least a portion of the modality information; At least part of the video slices are obtained from the matching results as target video slices corresponding to the video retrieval information.

[0094] Embodiment 3 Fig.10The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the video retrieval method according to the third embodiment of the present application is shown schematically. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig.10 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed in the computer device 10000, such as program codes of a video retrieval method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0095] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0096] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0097] It should be pointed out that Fig.10 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0098] In this embodiment, the video retrieval method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0099] Embodiment 4 An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the video retrieval method in the embodiment are implemented.

[0100] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the video retrieval method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0101] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0102] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0103] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video retrieval method, characterized in that: The method comprises: Obtain N videos, and segment the videos to obtain video slices of the videos, where N is a positive integer; Acquire multiple modal information of the video slice, wherein the modal information includes visual modal information, audio modal information and text modal information; Building a video retrieval library based on the multiple modal information and the video slices; Video retrieval information is obtained, and a target video slice corresponding to the video retrieval information is determined in the video retrieval library based on the multiple modal information.

2. The method according to claim 1, characterized in that Slicing the video to obtain video slices of the video includes: Performing shot boundary detection on the frame image of the video to determine M shot boundaries included in the frame image, where M is a natural number; The video is sliced ​​based on the M shot boundaries to obtain video slices of the video.

3. The method according to claim 2, characterized in that Slicing the video based on the M shot boundaries to obtain video slices of the video includes: When the duration of the current video slice is less than the minimum limited duration, the tail of the current video slice is extended so that the duration of the current video slice is greater than or equal to the minimum limited duration; When the duration of the current video slice is greater than the maximum limited duration, the current video slice is cropped twice so that the duration of the video slices obtained by the secondary cropping is less than or equal to the maximum limited duration.

4. The method according to claim 1, characterized in that: Acquire multiple modal information of the video slice, including: Performing target evaluation on each of the video slices, wherein the target evaluation includes image quality evaluation and optical flow evaluation; Screening the video slices according to the evaluation results to obtain screened video slices; Acquire multiple modal information of the filtered video slices; Correspondingly, constructing a video retrieval library based on the multiple modal information and the video slices includes: A video retrieval library is constructed based on the multiple modal information and the filtered video slices.

5. The method according to claim 1, characterized in that: Acquire multiple modal information of the video slice, including: Performing face recognition and face feature extraction on the video slice to obtain a face feature vector of the video slice; Performing speech recognition on the audio of the video slice to obtain a speech recognition text of the video slice; Performing optical character recognition on the frame image of the video slice to obtain optically recognized text of the video slice; The frame image of the video slice is converted into a low-dimensional vector to obtain a video vector of the video slice.

6. The method according to claim 5, characterized in that Building a video retrieval library based on the multiple modal information and the video slices includes: Acquire video dimension information of the video, where the video dimension information includes at least one of a video title, video description information, and video tag information; Using a plurality of expert models to identify the video slices to obtain model identification results, wherein each of the expert models is used for a classification task; A video retrieval library is constructed based on the video dimension information, the model recognition result and the video slice information, and an index is constructed according to the video retrieval library, wherein the video slice information includes the face feature vector, the speech recognition text, the optical recognition text and the video vector.

7. The method according to any one of claims 1 to 6, characterized in that: Determining a target video slice corresponding to the video retrieval information in the video retrieval library based on the multiple modal information includes: matching the video retrieval information with at least a portion of the modality information; At least part of the video slices are obtained from the matching results as target video slices corresponding to the video retrieval information.

8. A video retrieval device, characterized in that: The device comprises: A segmentation module, used to obtain N videos, segment the videos to obtain video slices of the videos, where N is a positive integer; An acquisition module, used to acquire multiple modal information of the video slice, wherein the modal information includes visual modal information, audio modal information and text modal information; A construction module, used for constructing a video retrieval library based on the multiple modal information and the video slices; The retrieval module is used to obtain video retrieval information and determine a target video slice corresponding to the video retrieval information in the video retrieval library based on the multiple modal information.

9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images