Video search method, electronic equipment, storage medium and program

Abstract

The invention discloses a video searching method, electronic equipment, a storage medium and a program, and the method comprises the steps: obtaining a query statement of a to-be-retrieved video, and generating a text feature of the query statement; text description features of a feature database of the to-be-retrieved video are matched according to the text features, a target query object is obtained, and text description features of a retrieval object of the feature database are determined based on image content description texts of the to-be-retrieved video; and generating a query result according to the target image of the target query object in the to-be-retrieved video, wherein the text description feature of the target query object corresponds to the image content description text of the target image. According to the embodiment of the invention, automatic query of the video data is realized, the text features do not need to be completely matched with the text description features in the feature database, the data processing amount of feature matching can be reduced, the video query efficiency can be improved, and the performance overhead of a system can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a video search method, electronic equipment, storage medium and program. Background Art

[0002] With the development of monitoring technology, it has been widely used in urban security, traffic management, industrial production and other fields. With the increasing density of surveillance camera deployment and the continuous extension of recording time, the amount of video data generated is growing exponentially. The traffic monitoring system of a medium-sized city can accumulate PB of data every month. At present, the search of surveillance videos is mainly achieved through manual retrieval, which cannot locate video clips of specific time periods, specific areas or specific objects / people in such a large amount of information, resulting in poor accurate retrieval capabilities of surveillance videos. In addition, the video retrieval process mainly relies on manual retrieval, which not only consumes a lot of manpower and time costs, but also has the problem of errors and omissions in video retrieval results. Summary of the invention

[0003] The present invention provides a video search method, electronic equipment, storage medium and program to solve the problem that existing videos cannot be accurately searched, improve the semantic search efficiency of videos, and enhance the accuracy of video retrieval.

[0004] According to one aspect of the present invention, a video search method is provided, wherein the method comprises:

[0005] Acquire a query statement of the video to be retrieved, and generate text features of the query statement; match the text description features of the feature database of the video to be retrieved according to the text features to obtain a target query object, wherein the text description features of the retrieval object of the feature database are determined based on the image content description text of the video to be retrieved;

[0006] A query result is generated according to a target image of the target query object in the video to be retrieved, wherein a text description feature of the target query object corresponds to an image content description text of the target image.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0008] at least one processor; and

[0009] a memory communicatively connected to the at least one processor; wherein,

[0010] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video search method described in any embodiment of the present invention.

[0011] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video search method described in any embodiment of the present invention when executed.

[0012] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the video search method according to any embodiment of the present invention is implemented.

[0013] The technical solution of the embodiment of the present invention obtains a query statement corresponding to a video to be retrieved and generates text features of the query statement, searches for a target query object with matching text description features in a feature database of the video to be retrieved according to the text features, determines a target image of the target query object in the video to be retrieved, wherein the image content description text of the target image corresponds to the target query object, and generates a query result corresponding to the target image, thereby realizing automatic query of video data, improving data query efficiency, matching text description features in a feature database with text features of the query statement, quickly acquiring the target query object, without having to match all text features with text description features in the feature database, reducing the amount of data processing for feature matching, improving video query efficiency, and reducing system performance overhead.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 is a flowchart of a video search method provided according to Embodiment 1 of the present invention;

[0017] Figure 2 is a flowchart of a video search method provided according to Embodiment 2 of the present invention;

[0018] Figure 3 is a flowchart of a video search method provided according to Embodiment 3 of the present invention;

[0019] Figure 4 This is an example diagram of query library generation according to the fourth embodiment of the present invention;

[0020] Figure 5 is an exemplary diagram of a video search method provided according to Embodiment 4 of the present invention;

[0021] Figure 6 is a structural schematic diagram of a video search device provided according to Embodiment 5 of the present invention;

[0022] Figure 7 It is a structural schematic diagram of an electronic device for implementing the video search method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] Embodiment 1

[0026] Figure 1 1 is a flowchart of a video search method provided according to the first embodiment of the present invention. This embodiment is applicable to the case of semantic query of video data. The method can be executed by a video search device. The video search device can be implemented in the form of hardware and / or software. The device can be configured in a server or terminal device. Figure 1 As shown, the method includes:

[0027] Step 110: Obtain a query statement for the video to be retrieved, and generate text features of the query statement.

[0028] The video to be retrieved may be video data that needs to be queried. The video to be retrieved may include but is not limited to surveillance videos in the fields of urban security, traffic management, industrial production, etc. The video to be retrieved may include images of people, objects, scenes, etc. The query statement may be used to query the target person or object in the video to be retrieved. The query statement may exist in the form of voice or text. The query statement may include a description of the query object. For example, the query statement may be "a green truck" or "a person carrying a panda bag", etc. The text feature may be feature information extracted from the text data of the query statement. The text feature may include lexical features, semantic features, syntactic features, etc. The text feature may be extracted from the text data of the query statement by means of word frequency statistics, semantic analysis, etc.

[0029] In an embodiment of the present invention, query statements can be obtained for one or more videos to be retrieved. For example, a user can select one or more videos to be retrieved through a visual interface, and generate a query statement for searching the selected videos to be retrieved through language or text. The collected query statement can be converted into a text form, and feature extraction can be performed on the query statement in the text form to obtain text features of the query statement. The text features may include but are not limited to lexical features, semantic features, syntactic features, etc. of the query statement.

[0030] Further, based on the above-mentioned embodiments of the invention, the query statement at least includes description information of the object or person to be queried.

[0031] In the embodiment of the present invention, the query statement may exist in the form of language or text, and the query statement may include description information of the object or person to be queried. The description information may include partial features of the object or person, and the partial features may be input by the user.

[0032] Step 120: Match the text description features of the feature database of the video to be retrieved according to the text features to obtain the target query object, wherein the text description features of the retrieval object in the feature database are determined based on the image content description text of the video to be retrieved.

[0033] Among them, the search object can be an object and / or person identified in the video to be searched, and the search object can be generated by performing image recognition on the video to be searched, and the search object can be obtained by performing single target detection and tracking or multi-target detection and tracking on the video to be searched; the feature database can be a storage database including text description features of the search object, and the search object can be obtained by performing target detection and tracking on each frame of the video to be searched, and the text description features of the search object can be obtained by performing image content description on the image frame of the search object, and the image content description can be realized by multimodal model processing. The search object can have corresponding multiple frames of images in the video to be searched, and each frame of the image can generate an image content description text respectively, and the text description features of the search object can be obtained by performing feature extraction on the image content description text. The target query object can be a search object whose text description features in the feature database match the text features of the query statement, and the search object can be realized by performing single target detection and tracking on the video to be searched or multi-target detection and tracking. The matching of text description features and text features can include the cosine similarity between the text description features and the text features being less than a threshold, the Euclidean distance being less than a threshold, the Manhattan distance being less than a threshold, and the dot product result being greater than a threshold.

[0034] In an embodiment of the present invention, a feature database can be pre-constructed for a video to be retrieved, and the feature database includes at least feature information of a retrieval object in the video to be retrieved. Each retrieval object can be obtained by performing target detection and tracking on the video to be retrieved, and the image content description text in the video to be retrieved can be extracted for each retrieval object, thereby constructing a feature database according to the image content description text of each retrieval object. For matching the text features of the query statement with the text description features stored in the feature database, the retrieval object with the matched text description features can be used as the target query object. It can be understood that in the process of matching the text features of the query statement with the text description features stored in the feature database, a text description feature matching the text features is obtained, and the corresponding target query object can be determined according to the text description feature. For example, a tracking ID corresponding to the above-mentioned matching text description feature can be extracted from the feature database, and the retrieval object with the tracking ID can be used as the target query object.

[0035] Step 130: Generate a query result according to the target image of the target query object in the video to be retrieved, wherein the text description feature of the target query object corresponds to the image content description text of the target image.

[0036] Among them, the target image may be the image information corresponding to the target query object, and the target image may be an image frame in the video to be retrieved. It can be understood that the image frame may be the original image frame of the video to be retrieved or an image frame generated after processing, and the processing may include video compression or image enhancement of the video to be retrieved. The text description feature of the target query object may correspond to the image content description text of the target image. For example, the text description feature may be obtained by extracting the feature of the image content description text of the target image, or the feature matching the text description feature may be obtained. The image content description text of the target image may be text information generated by describing the image content of the target image, and the image content description text may be obtained by multimodal model processing. The query result may be an image query result generated based on the target image, and the image query result may be based on the text or picture generated by the target image. The image query result may include the motion trajectory, position, color feature description, contour feature description and other information of the target query object. In some embodiments, the type of information output by the image query result may match the query statement, and the query statement may include parameters indicating the output type of the image query result. One or more target image generation information may be selected according to the above parameters in the query statement as the generated query result. For example, if the query statement is "find the driving track of a certain truck", when the query result may include the target query object as the truck, then the motion track corresponding to the target query object in the video to be retrieved may be used as the query result.

[0037] In the embodiment of the present invention, an image frame or a partial image of an image frame including or associated with the target query object can be determined in the video to be retrieved according to the target query object, the image frame or partial image can be used as the target image, and the target image or the processing result generated by the target image can be used as the query result. The processing can be performed by performing image recognition algorithm processing, target tracking algorithm processing, multimodal model processing, etc. on the target image.

[0038] The embodiment of the present invention obtains a query statement corresponding to a video to be retrieved and generates text features of the query statement, searches for a target query object with matching text description features in a feature database of the video to be retrieved according to the text features, determines a target image of the target query object in the video to be retrieved, wherein the image content description text of the target image corresponds to the target query object, and generates a query result corresponding to the target image, thereby realizing automatic query of video data, improving data query efficiency, matching text description features in a feature database with text features of the query statement, quickly acquiring the target query object, without having to match all text features with text description features in the feature database, reducing the amount of data processing for feature matching, improving video query efficiency, and reducing system performance overhead.

[0039] Embodiment 2

[0040] Figure 2 is a flowchart of a video search method provided according to the second embodiment of the present invention. The embodiment of the present invention is a specific embodiment of the above embodiment, and describes the process of constructing a feature database. Figure 2 The method provided in the embodiment of the present invention specifically includes the following steps:

[0041] Step 210: Perform multi-target detection and tracking on the video to be retrieved to obtain a tracking result of at least one retrieval object.

[0042] Among them, multi-target detection and tracking can be a technology for locating and tracking the trajectories of multiple targets in a video or image sequence. Multi-target detection and tracking can be implemented by detection-based tracking, feature matching-based tracking, and deep learning-based tracking. The implementation algorithm of multi-target detection and tracking can include but is not limited to SORT multi-target tracking algorithm, YOLO8 multi-target tracking algorithm, motrv2 algorithm, or custom algorithm. The retrieval object can be a tracking target identified in the video to be retrieved, and the retrieval object can include a person or object in the video to be retrieved. The tracking result can be the processing result of multi-target detection and tracking of the video to be retrieved. The tracking result includes at least relevant information of a retrieval object. The tracking result can be composed of parameters such as identification information of the retrieval object, tracking frame information, classification category, image frame number, confidence, timestamp, etc. In some embodiments, the tracking result can also include acceleration, trajectory, appearance characteristics, occlusion status, etc. of the retrieval object.

[0043] In the embodiment of the present invention, after the video to be retrieved is acquired, multi-target detection and tracking may be performed on the video to be retrieved to determine the tracking result corresponding to each retrieval object in the video to be retrieved.

[0044] For example, taking the surveillance video in the security scenario as an example, the surveillance video within 24 hours can be scheduled as the video to be retrieved, and the ORT multi-target tracking algorithm or the YOLO8 multi-target tracking algorithm can be called to process the video to be retrieved. The processing result output by the above multi-target tracking algorithm can be obtained. The processing result may include the tracking identification, tracking box information, classification category, confidence, timestamp, acceleration, trajectory, appearance characteristics, occlusion status and other information of one or more retrieval objects as the tracking result.

[0045] Step 220: Determine the target image data of the search object in the video to be searched according to the tracking result, and generate a text description of the target image data.

[0046] The target image data may be image information including the retrieval object, the target image data may be an image frame of the video to be retrieved or a partial image within the image frame, and the target image data may be extracted from the video to be retrieved through the tracking result. The text description may be text describing the target image data, obtained by performing multimodal model processing on the target image data.

[0047] In an embodiment of the present invention, target image data including the search object can be extracted from the video to be searched according to the tracking result. For example, the image frame number in the tracking result can be extracted for each search object, and the image frame corresponding to the image frame number in the video to be searched can be used as the target image data. Alternatively, the tracking frame position can be extracted from the tracking result, and image information can be extracted from the image frame including the search object according to the tracking frame position as the target image data. For each target image data of the search object obtained, the target image data can be processed using a multimodal model to generate a text description describing the search object in the target image data.

[0048] Step 230: Generate text description features of the search object based on the text description.

[0049] Among them, the text description feature can be a semantic feature of the image content description text used for the retrieval object. The image content description text can be obtained by performing text description on the target image data of the retrieval object through a multimodal model. The text description feature can be obtained by performing deep learning or word vector processing on the image content description text of the retrieval object. A retrieval object can have multiple text description features.

[0050] In an embodiment of the present invention, a text description of the search object can be extracted, and the text description can include one or more. It can be understood that a search object can be located in multiple image frames, and there can be multiple image content description texts. Semantic features can be extracted from the above multiple image content descriptions respectively, so as to obtain multiple text description features corresponding to a search object.

[0051] Exemplarily, for the search object, the text description and semantic features can be extracted, and the extraction can include but is not limited to counting the word frequency features, word frequency inverse document frequency, word embedding features and other features of the text description as text description features.

[0052] In some embodiments of the invention, the feature database may also include a category feature label for the search object. The category label feature may be a semantic feature of a classification label that reflects the type of object or person to which the search object belongs. The classification label corresponding to the category label feature may be included in the tracking result of the search object. The classification label may be generated in the detection phase of multi-target detection and tracking of the search object. The detection phase may locate different targets from an image or video and identify the type of the target. When constructing the feature database, the classification label may also be extracted from the tracking result of the search object. For a search object, all its text descriptions and classification labels may be subjected to semantic feature extraction, thereby obtaining the text description features corresponding to the text description of the search object, as well as the category label features.

[0053] Furthermore, based on the above-mentioned embodiments of the invention, the category label features and text description features may also include other text features, which may include but are not limited to word frequency features, part of speech features, syntactic features, text structure features, etc.

[0054] Step 240: construct a feature database based on the text description features of each search object.

[0055] In the embodiment of the present invention, the text description features of all search objects can be persistently stored, and the persistently stored information set can be used as a feature database. It can be understood that the above feature database can be generated by a professional database management system, or only the set of persistently stored text description features can be used as a feature database. Further, the feature database can include the search feature information of multiple different videos, that is, one feature database can correspond to multiple videos, and the search feature information of multiple search objects of multiple videos can be stored in the feature database.

[0056] Furthermore, when the search object includes multiple text description features, the multiple text description features can be associated and stored with the tracking ID of the search object so as to query and use the text description features. For example, if there is a text description feature that matches the text feature of the query statement, it is not necessary to match other text description features that belong to the same search object as the text description feature with the query statement, which can improve the matching efficiency of text description feature matching in the feature database.

[0057] Step 250: Obtain a query statement for the video to be retrieved, and generate text features of the query statement.

[0058] Step 260: Match the text description features of the feature database of the video to be retrieved according to the text features to obtain the target query object, wherein the text description features of the retrieval object in the feature database are determined based on the image content description text of the video to be retrieved.

[0059] Step 270: Generate a query result according to the target image of the target query object in the video to be retrieved, wherein the text description feature of the target query object corresponds to the image content description text of the target image.

[0060] According to an embodiment of the present invention, multi-target detection and tracking is performed on a video to be retrieved to obtain tracking results of different retrieval objects, target image data corresponding to the tracking results is determined in the video to be retrieved, and a text description of the target image data is obtained, and semantic features are extracted from the text description and the classification labels of the tracking results to obtain text description features and category label features of each retrieval object, and each text description feature and category label feature is saved as a feature database, a query statement corresponding to the video to be retrieved is obtained, and a text feature of the query statement is generated, and a target query object with matching text description features is searched for according to the text feature in the feature database of the video to be retrieved, and a target image of the target query object is determined in the video to be retrieved, wherein the image content description text of the target image corresponds to the target query object, and a query result corresponding to the target image is generated. The embodiment of the present invention determines the semantic features of the image content description text of the retrieval object between different image frames by performing multi-target detection and tracking on the video, and saves the semantic features to a feature database, which helps to improve the accuracy of feature queries of the video to be retrieved. The target query object can be quickly acquired by matching the text features of the query statement with the text description features in the feature database. There is no need to match all the text features with the text description features in the feature database, which can reduce the data processing amount of feature matching, improve video query efficiency, and reduce the performance overhead of the system.

[0061] In some embodiments of the invention, determining target image data of a search object in a video to be searched according to the tracking result and generating a text description of the target image data includes:

[0062] At least one image frame information is extracted from the tracking result of the retrieval object; the image frame corresponding to each image frame information is searched in the video to be retrieved as the target image data; and the multimodal model is called to process each image frame respectively to obtain a text description of each image frame.

[0063] Among them, the image frame information can indicate the information of the image frame to which the retrieved object belongs, and the image frame information can include the image frame name or the image frame serial number, etc. The multimodal model is a machine learning model that can process various types of data. The multimodal model can recognize and understand the image data and generate text descriptions in the image data. The multimodal model can include the Qwen-Vl model, the SmoVLM model, etc.

[0064] In an embodiment of the present invention, image frame information can be extracted from the corresponding tracking result for each retrieval object. One or more image frames can be extracted from the video to be retrieved according to the image frame information. That is, each detected object appears in multiple image frames. The above-mentioned one or more image frames can be recorded as the target image data of the detected object. The target image data can be input into a multimodal model for processing. The multimodal model can identify and understand the image content of the target image data, so as to generate a text description corresponding to the target image data. When the target image data of the detected object includes multiple image frames, the multimodal model can be used to process the multiple image frames of the retrieval object respectively to obtain the text description of each image frame. A retrieval object can obtain multiple text descriptions.

[0065] Exemplarily, the form of the text description can include the following forms:

[0066] 1. This is an ambulance with Chinese and English signs on its body, indicating that it is an emergency medical rescue vehicle. The words "Jiangsu 120 Emergency" are pasted on both the front and back of the vehicle, indicating that this is the 120 emergency service in Jiangsu. The word "AMBULANCE" is also printed on the vehicle, indicating that this is an ambulance. In addition, the logo of "Jiangling" is pasted on the vehicle, indicating that this is a vehicle produced by Jiangling Motors Corporation. The license plate number is "Su U·ABZ***", which is the registered license plate of the vehicle in China.

[0067] 2. The figure shows a fire truck with obvious fire-fighting equipment and facilities on its body.

[0068] 3. The vehicle in the figure is a green truck with white Chinese characters on its body, including "Post" and "Xiang A*****". There is a license plate at the rear of the truck, and the license plate number is "Xiang A*****".

[0069] In some embodiments of the invention, tracking box information can also be extracted from the tracking result of the retrieval object. The tracking box information can be information reflecting the position of the retrieval object in the image frame. The tracking box information can be represented in the form of image coordinates. For example, the tracking box information can include the upper left corner coordinates and the lower right corner coordinates. Image frame information can be extracted from the corresponding tracking result for each retrieval object. One or more image frames can be extracted from the video to be retrieved according to the image frame information, and partial images can be extracted from the image frames according to the tracking box information. The multimodal model can be used to describe the above-mentioned partial images, so as to obtain the text description of the retrieval object.

[0070] In some other embodiments of the invention, generating text description features of the retrieval object according to the text description includes:

[0071] Word embedding feature extraction is performed for each text description of the retrieval object to obtain at least one word embedding feature vector as a text description feature.

[0072] Specifically, word embedding features can be extracted from one or more text descriptions of the search object, and the word embedding features of each text description can be used as the third word embedding feature vector. For a search object, the extracted second word embedding feature vector can be used as a category label feature and each corresponding third word embedding feature vector can be used as a text description feature. It can be understood that the method of extracting word embedding features for category labels and text descriptions can be implemented through the Moka Massive Mixed Embedding (M3E) model.

[0073] In some embodiments of the invention, when the feature database includes category label features, it also includes: extracting word embedding features for the classification category of the retrieval object to obtain a word embedding feature vector as a text category label feature.

[0074] In an embodiment of the present invention, classification labels of the tracking results of all retrieval objects can be extracted. The classification labels can indicate the classification categories to which the retrieval objects belong. The classification labels can be generated during the processing of multi-target detection and tracking of the video to be detected. For example, in the detection phase of the above-mentioned multi-target detection and tracking, one or more detection objects in the video to be detected can be identified, and the classification category to which each detection object belongs can be determined. The detection objects in the above-mentioned detection phase are also the retrieval objects used for video retrieval in an embodiment of the present invention. The classification category of each retrieval object can be obtained in the detection phase of the multi-target detection and tracking of the video to be retrieved. The classification labels of all the above-mentioned retrieval objects can be subjected to word embedding feature extraction, and the extracted word embedding feature vector can be recorded as the second word embedding feature vector.

[0075] In some embodiments of the invention, a feature database is constructed based on the text description features of each search object, including:

[0076] Extract the image frame number and tracking mark of the search object in the tracking result; extract the video storage path of the video to be searched; for each search object, save the text description features, image frame number, tracking mark and video storage path of the search object as a description file, and save the description file to the feature database.

[0077] In an embodiment of the present invention, for each search object, the image frame number and tracking identifier in the tracking result can be extracted. In some embodiments of the invention, the tracking box information and other information can also be extracted in the tracking result to obtain the video storage path of the video to be searched. The above-collected image frame number, tracking identifier, video storage path and other information and the text description features of the search object can be saved as a description file. It can be understood that each search object can have its own corresponding description file. The description file corresponding to the search object can be saved to the feature database. It can be understood that since the search object can be located in multiple frames of images, multiple image frame numbers can be extracted in the tracking result of the search object, and one tracking identifier of the search object in the description file can correspond to multiple image frame numbers.

[0078] Embodiment 3

[0079] Figure 3 FIG. 1 is a flowchart of a video search method provided according to Embodiment 3 of the present invention. Based on the above embodiments, the present invention describes the process of generating query results. Figure 3 The method provided in the embodiment of the present invention specifically includes the following steps:

[0080] Step 310: Perform multi-target detection and tracking on the video to be retrieved to obtain a tracking result of at least one retrieval object.

[0081] Step 320: Determine the target image data of the search object in the video to be searched according to the tracking result, and generate a text description of the target image data.

[0082] Step 330: Generate text description features and category label features of the retrieval object according to the text description and the tracking result.

[0083] Step 340: construct a feature database based on the text description features and category label features of each search object.

[0084] Step 350: Obtain a query statement for the video to be retrieved, and generate text features of the query statement.

[0085] Step 360: Determine whether the text feature matches the category label feature of the search object in the feature database, wherein the category label feature is determined based on the detection result of multi-target detection and tracking of the video to be searched.

[0086] In an embodiment of the present invention, before matching the text feature with the text description feature of the feature database, the text feature can also be matched with the category label feature of each search object in the feature database to determine whether the text feature belongs to the classification category of the corresponding category label feature. If the text feature matches the category label feature of the search object in the feature database, the text feature can continue to be matched with the text description feature of the search object. If neither matches, the retrieval process of the video to be retrieved can be launched.

[0087] Step 370: Match the text description features of the feature database of the video to be retrieved according to the text features to obtain the target query object, wherein the text description features of the retrieval object in the feature database are determined based on the image content description text of the video to be retrieved.

[0088] Step 380: extract the target description file in the feature database, and extract the tracking mark of the target query object in the target description file, wherein the text description feature stored in the target description file is matched with the text feature.

[0089] In an embodiment of the present invention, when there is a match between a text description feature and a text feature, the description file storing the text description feature can be used as a target description file, the target description file can be stored in a feature database, and the tracking identifier of the target query object can be extracted from the target description file. It can be understood that the tracking identifier can be generated in the multi-target detection and tracking processing process of the video to be retrieved.

[0090] Step 390: Search the feature database for at least one other description file associated with the tracking identifier.

[0091] In the embodiment of the present invention, other description files corresponding to the tracking identifier may be searched in the feature database, and the correspondence may include that the tracking identifier of the target description file is the same as the tracking identifier of the other description files.

[0092] Step 3100: search for a target image as a query result according to the image frame number and video storage path saved in the target description file and other description files.

[0093] In an embodiment of the present invention, the image frame number and the video storage path can be read in the target description file and other description files, the target video can be read according to the video storage path, and the target video can be searched for one or more frames of images corresponding to the image frame number as the target image, and the above target image can be used as a query result. Furthermore, in some embodiments, the processing result of the target image can also be used as a query result. For example, the motion trajectory of the target query object in the target image can also be extracted as a query result, or text information or partial image in the target image can be extracted as a query result, etc.

[0094] The embodiment of the present invention performs multi-target detection and tracking on a video to be retrieved to obtain tracking results of different retrieval objects, determines target image data corresponding to the tracking results in the video to be retrieved, and obtains a text description of the target image data, extracts semantic features from the text description and the classification label of the tracking results, obtains text description features and category label features of each retrieval object, and saves each text description feature and category label feature as a feature database, generates text features of the query statement when a query statement of the video to be retrieved is obtained, determines that the text features match the category label features of the retrieval object in the feature database, determines the target query object by matching the text features with the feature information in the feature database, extracts a target description file in the feature database, extracts a tracking identifier of the target query object from the target description file, obtains other description files according to the tracking identifier, and generates a query result through the target image corresponding to the image frame number and the video storage path saved in the target description file and the other description files. The embodiment of the present invention performs multi-target detection and tracking on the video to determine the semantic features of the image content description text of the retrieval object between different image frames, and saves the semantic features in the feature database, which helps to improve the accuracy of feature query of the video to be retrieved.

[0095] In the present application scheme, when searching for a video, the text features of the query statement are first matched with the text description features in the description file stored in the feature database of the video to be retrieved. As long as the text description features in one description file are successfully matched, there is no need to match the text description features in other description files. At this time, a text description feature that matches the text feature in the feature database can be obtained, and then the corresponding tracking identifier is determined based on the matched text description feature. Other description files are obtained according to the tracking identifier. There is no need to match the text feature descriptions of other description files with the text features of the query statement respectively, which can reduce the performance overhead of feature matching and improve the video query efficiency of the system. In addition, since there are errors in the multimodal model's description of image content for the same or similar image frames, resulting in differences in text description features between different image frames, other description text methods are obtained through tracking identifiers, and by reducing the number of matches between text features and text feature descriptions, it is avoided that text features match text description features with large feature errors, which can reduce retrieval errors in the video search process and improve the accuracy of video search. In this embodiment, since the same target or search object may appear in different image frames (target images) of the video to be searched, if the posture or image background of the search object in different image frames (target images) is quite different, the multimodal model may get an incorrect text description when identifying the target image. Even if the search object actually exists in a certain image frame, the text description feature identified by the multimodal model does not have the feature information of the search object. In this case, when only the text feature is matched with the text feature description in the feature library, the matching result will be wrong and the image where the search object actually exists will be missed. Therefore, this solution only needs to introduce a tracking identifier after matching a corresponding text description feature in the feature database, and find all corresponding description files according to the tracking identifier, so as to determine all target images corresponding to the query statement; because the tracking identifier is obtained based on the tracking result, with the help of the target tracking technology for accurate tracking of the same target, the matching result will be wrong when only the text feature is matched with the text feature description in the feature library.

[0096] In some embodiments of the invention, it also includes: extracting the image frame number, video storage path and tracking frame information of the target description file and other description files; reading the target video in the video storage path, and extracting at least one image frame of the target video according to the image frame number; using the tracking frame information corresponding to the image frame as the target position of the target query object; arranging each target position in sequence as a motion trajectory according to the frame sequence number of the image frame; and outputting the motion trajectory as a query result.

[0097] Among them, the motion trajectory can be a trajectory reflecting the motion position of the target query object in the target image, and the motion trajectory can be composed of one or more positions of the target query object. In an embodiment of the present invention, the image frame number, video storage path, and tracking frame information stored in the target description file and other description files can be read. It can be understood that the above information corresponds to the target query object of the description file. The target video can be video data including the target query object, and the target video can be part or all of the video to be retrieved. For example, the video to be retrieved adopts distributed storage, and each part of the video is stored in a separate space area. The storage space of the part of the video to be retrieved can be read according to the video storage path, and the part of the video data stored in the storage space can be used as the target video. After reading the target video, one or more image frames can be extracted in the target video according to the image frame number.

[0098] Specifically, the corresponding tracking frame information can be determined for each image frame in the extracted tracking frame information. It can be understood that each tracking frame information can reflect the position of the target query object in an image, and the tracking frame information can be used as the target position. The correspondence between the image frame and the tracking frame information can be pre-configured in the description file. For example, if the image frame and the tracking frame information are located in the same record in the description file, then there is a correspondence between the image frame and the tracking frame information, or if the image frame and the tracking frame information have the same primary key in the description file, then there is a correspondence between the image frame and the tracking frame information. The image frames corresponding to each tracking frame information can be determined, and all the acquired tracking frame information can be arranged according to the frame sequence number of the image frame, that is, the target positions can be arranged, and the sequence of target positions formed by the above tracking frame information can be used as the motion trajectory of the target query object, and the above motion trajectory and the target image can be used as the query results corresponding to the query statement.

[0099] Based on the above-mentioned embodiment of the invention, a text description of the target image data is generated, including:

[0100] The target image data is sequentially input into the multimodal model for image content recognition to obtain a text description; wherein the multimodal model at least includes a Qwen-VL multimodal model.

[0101] In an embodiment of the present invention, text description can be realized by processing the target image data through the Qwen-VL multimodal model. The Qwen-VL model can be composed of a large language model, a visual encoder, a position-aware encoder, etc. The visual encoder in the Qwen-VL model can process the target image data into an image feature sequence, which can be processed in the large language model to understand the image content in the target image data and generate a corresponding text description. The position-aware encoder can perform cross-attention operations on the image feature sequence during the understanding of the target image data, enhance the importance of different positions in the image feature sequence, and thus enhance the accuracy of the text description.

[0102] Embodiment 4

[0103] The embodiment of the present invention provides a video search method, which performs semantic search on videos. The search process can be implemented based on a feature query library. Figure 4 , the generation of the feature query library may include the following process:

[0104] Step 1: Use the multi-target detection model to perform target detection on the input video file.

[0105] Step 2: Track the detected target and record the tracking track. It is understandable that the tracking track can be implemented by a multi-target tracking algorithm, and the tracking track may include the tracking ID number of the target, the detection frame position of the target, and the video frame number where the target is located. Multiple images of the same target can use the same tracking ID number to identify the same target, and images of different targets can be distinguished using different tracking ID numbers. In the tracking module of the multi-target tracking algorithm, the detected target can be named using its corresponding tracking ID number, frame number, and video source, so as to facilitate the storage of the above information in the tracking track.

[0106] Step 3: Extract the text description of each detection target. You can use the Qwen-VL series multimodal model to generate a paragraph description for each target. Each detection target can have multiple frames of images, so you can generate multiple text descriptions. Specifically, you can do this by: After tracking multiple targets in the video file, extract the detection image corresponding to each detection target, and input the multiple detection images of each detection target into the multimodal model in turn. Each detection image outputs the corresponding text description. If a detection target exists in multiple frames in the video file, then this detection target corresponds to multiple text descriptions. Each text description can be saved to a description file, and each description file can have a file ID. The file ID belonging to the same detection target can be associated with the tracking ID of the detection target. The association method may include the tracking ID of some or all detection targets in the file ID, or the file ID and the tracking ID are saved to a configuration information or configuration file.

[0107] Step 4: Input the text description of each detected target and the category label generated during the image detection process of each detected target in step 1 into the text-to-volume model M3E (Moka Massive Mixed Embedding) to generate a feature vector.

[0108] Step 5: For each detected target, create a description file based on the video path, image frame number, tracking ID number or target name used in the detection target tracking process, and the generated feature vector, and save the description file to the query library. It can be understood that each detected target can correspond to a tracking ID number during the tracking process.

[0109] In the embodiment of the present invention, after obtaining the query library (feature database) of the surveillance video, the semantic retrieval of the surveillance video can be realized through the query statement and the query library (feature database), see Figure 5 , the retrieval process may include the following steps:

[0110] Step 1: The query statement input by the user can be input into the text-to-volume model to generate a feature vector to be queried. The text-to-volume model can include an M3E model. Exemplarily, in a monitoring scenario, the query statement can include a person, vehicle or other object that the user needs to query in the monitoring video, and the query statement can include description information of the target.

[0111] Step 2, match the feature vector to be queried of the query warning with the feature vector in the description file in the query library. The matching process may include: matching the feature vector to be queried with the feature vector corresponding to each target category label in the query library to remove the situation where the category description is wrong when some multimodal models describe the target. If the match fails, the search process is exited. Otherwise, the feature vector to be queried is matched twice with the feature vector of the video image text description corresponding to the category label matched in the database. If the match fails, the search process is exited, and the subsequent steps are entered if the match succeeds. In the present application, as described in the above embodiment, it is also possible to directly match the feature vector to be queried in the query statement (i.e., the above text feature) with the feature vector in the description file in the feature database (i.e., the above text description feature) to obtain the corresponding matching result. The screening of the category label is equivalent to a preliminary screening of the category label obtained in advance according to the target detection and tracking results. When the category label screening is not performed and only the text features in the query statement are matched with the text description features in the feature database, each description file in the feature database does not include the category label of the target image.

[0112] Step 3: Obtain the file ID of the description file matched in the query library. It is understandable that each description file can set its own file ID during the generation process. When a file ID is obtained, other description files associated with the file ID, such as videos, images, etc., can be retrieved from the query library.

[0113] Embodiment 5

[0114] Figure 6 FIG. 1 is a schematic diagram of the structure of a video search device provided according to Embodiment 5 of the present invention. Figure 6 As shown, the device comprises:

[0115] The feature extraction module 410 is used to obtain a query statement of the video to be retrieved and generate text features of the query statement.

[0116] The object matching module 420 is used to match the text description features of the feature database of the video to be retrieved according to the text features to obtain a target query object, wherein the text description features of the retrieval object in the feature database are determined based on the image content description text of the video to be retrieved.

[0117] The result query module 430 is used to generate a query result according to the target image of the target query object in the video to be retrieved, wherein the text description feature of the target query object corresponds to the image content description text of the target image.

[0118] The embodiment of the present invention obtains a query statement corresponding to a video to be retrieved and generates text features of the query statement, searches for a target query object with matching text description features in a feature database of the video to be retrieved according to the text features, determines a target image of the target query object in the video to be retrieved, wherein the image content description text of the target image corresponds to the target query object, and generates a query result corresponding to the target image, thereby realizing automatic query of video data, improving data query efficiency, matching text description features in a feature database with text features of the query statement, quickly acquiring the target query object, without having to match all text features with text description features in the feature database, reducing the amount of data processing for feature matching, improving video query efficiency, and reducing system performance overhead.

[0119] Furthermore, based on the above-mentioned embodiment of the invention, a database module is also included, and the database module includes:

[0120] The tracking detection unit is used to perform multi-target detection and tracking on the video to be retrieved to obtain a tracking result of at least one retrieval object.

[0121] The text description unit is used to determine the target image data of the search object in the video to be searched according to the tracking result, and generate a text description of the target image data.

[0122] The retrieval feature unit is used to generate text description features and category label features of the retrieval object according to the text description and the tracking result.

[0123] A database unit is used to construct the feature database based on the text description features and the category label features of each of the search objects.

[0124] In some inventive embodiments, the result query module 430 includes:

[0125] A tracking identification unit is used to extract a target description file in the feature database and extract a tracking identification of the target query object from the target description file, wherein the text description feature stored in the target description file matches the text feature.

[0126] The file searching unit is configured to search the feature database for at least one other description file associated with the tracking identifier.

[0127] The result acquisition unit is used to search for the target image as the query result according to the image frame number and the video storage path saved in the target description file and the other description files.

[0128] In some embodiments of the invention, it also includes: a classification matching module, used to determine whether the text feature matches the category label feature of the search object in the feature database, wherein the category label feature is determined based on the tracking results of multi-target detection and tracking of the video to be retrieved.

[0129] In some embodiments of the invention, the text description unit is specifically used to:

[0130] Extracting at least one image frame information in the tracking result of the search object;

[0131] Searching for an image frame corresponding to each of the image frame information in the video to be retrieved as the target image data;

[0132] The multimodal model is called to process each of the image frames respectively to obtain the text description of each of the image frames.

[0133] In some embodiments of the invention, the retrieval feature unit is specifically used to:

[0134] Perform word embedding feature extraction on the classification label of the tracking result of the search object to obtain a second word embedding feature vector as the category label feature;

[0135] Word embedding feature extraction is performed on each of the text descriptions of the search object to obtain at least one third word embedding feature vector as the text description feature.

[0136] In some inventive embodiments, the database unit is used to:

[0137] Extract the image frame number and tracking identifier of the search object from the tracking result; extract the video storage path of the video to be searched;

[0138] For each of the search objects, the text description features, the category label features, the image frame number, the tracking identifier and the video storage path of the search object are saved as a description file, and the description file is saved to the feature database.

[0139] The video search device provided in the embodiment of the present invention can execute the video search method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0140] Embodiment 6

[0141] Figure 71 is a schematic diagram of the structure of an electronic device for implementing the video search method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0142] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0143] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0144] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a video search method.

[0145] In some embodiments, the video search method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the video search method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the video search method in any other suitable manner (e.g., by means of firmware).

[0146] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0148] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0150] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0151] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0152] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0153] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video search method, characterized in that: The method comprises: Obtaining a query statement of a video to be retrieved, and generating text features of the query statement; matching text description features of a feature database of the video to be retrieved according to the text features to obtain a target query object, wherein the text description features of the retrieval object of the feature database are determined based on the image content description text of the video to be retrieved; A query result is generated according to a target image of the target query object in the video to be retrieved, wherein a text description feature of the target query object corresponds to an image content description text of the target image.

2. The method according to claim 1, characterized in that: Also includes: Performing multi-target detection and tracking on the video to be retrieved to obtain a tracking result of at least one retrieval object; Determining target image data of the search object in the video to be searched according to the tracking result, and generating a text description of the target image data; Generating text description features of the search object according to the text description; The feature database is constructed based on the text description features of each of the search objects.

3. The method according to claim 1, characterized in that: The step of generating a query result according to the target image of the target query object in the video to be retrieved includes: Extracting a target description file in the feature database, and extracting a tracking mark of the target query object from the target description file, wherein the text description feature stored in the target description file matches the text feature; Searching the feature database for at least one other description file associated with the tracking identifier; The target image is searched as the query result according to the image frame number and the video storage path saved in the target description file and the other description files.

4. The method according to claim 1, characterized in that: The feature database also includes category label features of the retrieval object. Before obtaining the target query object by matching the text description features of the feature database of the video to be retrieved according to the text features, it also includes: determining that the text features match the category label features of the retrieval object in the feature database, wherein the category label features are determined based on the tracking results of multi-target detection and tracking of the video to be retrieved.

5. The method according to claim 2, characterized in that: The step of determining the target image data of the search object in the video to be searched according to the tracking result and generating a text description of the target image data includes: Extracting at least one image frame information in the tracking result of the search object; Searching for an image frame corresponding to each of the image frame information in the video to be retrieved as the target image data; The multimodal model is called to process each of the image frames respectively to obtain the text description of each of the image frames.

6. The method according to claim 2, characterized in that: The step of generating a text description feature of the search object according to the text description includes: Word embedding feature extraction is performed for each of the text descriptions of the retrieval object to obtain at least one word embedding feature vector as the text description feature.

7. The method according to claim 2, characterized in that: The constructing the feature database based on the text description features of each of the search objects includes: Extract the image frame number and tracking identifier of the search object from the tracking result; extract the video storage path of the video to be searched; For each of the search objects, the text description features, the image frame number, the tracking identifier and the video storage path of the search object are saved as a description file, and the description file is saved in the feature database.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the video search method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video search method according to any one of claims 1 to 7 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the video search method according to any one of claims 1 to 7.

Images