Monitoring video cross-modal semantic matching method and system and electronic equipment

Abstract

The invention provides a monitoring video cross-modal semantic matching method and system and electronic equipment, and relates to the technical field of computer vision, and the method comprises the steps: carrying out the target detection and tracking of a monitoring video, so as to extract the target information in a video frame, and filtering repeated targets; generating a structured text for describing the video content by using the visual large model; extracting keyword features and semantic features of the structured text, and constructing a retrieval database containing the keyword features, the semantic features and video time sequence information; extracting query keywords and query semantics of the text query; based on a mixed retrieval strategy, matching the query keywords with keyword features in a retrieval database, and matching the query semantics with semantic features in the retrieval database; and fusing the keyword matching result and the semantic matching result to obtain a final retrieval result, and returning a video clip corresponding to the final retrieval result. According to the method, high-precision retrieval of detail features such as colors, textures and carried articles is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a monitoring video cross-modal semantic matching method and device and electronic equipment. BACKGROUND

[0002] As the core data in the field of security and operation, the demand for cross-modal semantic retrieval of monitoring videos is increasingly urgent. Users need to quickly locate the target video segment through text queries (such as "find the person wearing blue overalls entering the machine room from 15:00 to 16:00"). The traditional cross-modal retrieval method represented by the CLIP model calculates the similarity by mapping text and images to the same multi-modal space, but there are three key defects in the monitoring video scene: 1. Inaccurate visual feature matching: the resolution of monitoring videos varies greatly, and the light changes frequently. The CLIP model has low matching accuracy for human / object attributes (such as clothing color and carried items), and often produces false positives, with an accuracy of only about 70%; 2. Low data processing efficiency: monitoring videos have high frame rates (25fps-30fps) and large data volumes. Processing each frame will generate a large amount of redundant data. The traditional method has no redundant data filtering mechanism, resulting in low text generation and retrieval efficiency; 3. Missing temporal correlation: user queries often contain time conditions, but the traditional method does not establish a correlation between text representation and video timing, and cannot accurately filter the target content in a specific time period.

[0003] In addition, the existing technology does not combine the characteristics of monitoring videos "centered on people / objects", and lacks a precise extraction and tracking mechanism for the target.

[0004] Therefore, a monitoring video cross-modal semantic matching method, device and electronic equipment are proposed. SUMMARY

[0005] The present application provides a monitoring video cross-modal semantic matching method, device and electronic equipment, which realizes high-precision retrieval of color, texture, carried items and other detailed features.

[0006] The present application provides a monitoring video cross-modal semantic matching method, comprising: target detection and tracking of the monitoring video to extract target information in the video frames and filter duplicate targets; based on the target information, generating structured text describing the video content using a visual large model; extracting keyword features and semantic features of the structured text, and constructing a retrieval database containing the keyword features, semantic features and video timing information; receiving a text query and extracting query keywords and query semantics of the text query; based on the hybrid retrieval strategy, matching the query keywords with keyword features in the retrieval database and matching the query semantics with semantic features in the retrieval database; fusing the keyword matching result and the semantic matching result to obtain a final retrieval result, and returning a video segment corresponding to the final retrieval result.

[0007] Optionally, the target detection and tracking of the monitoring video are performed to extract target information in the video frames and filter repeated targets, including: adopting a target detection algorithm to identify at least one target and its attribute information in the video frames; adopting a video object tracking algorithm to assign a unique tracking identifier to the same target appearing in consecutive video frames; based on the tracking identifier, filtering repeated detection results of the same target and retaining only target information in part of the frames for generating the structured text.

[0008] Optionally, the structured text describing the video content is generated by using a visual large model based on the target information, including: inputting the target information, the tracking identifier, and the corresponding video frames into the visual large model; wherein the target information includes target category, attribute, and location information; obtaining a text description in a structured data format output by the visual large model, the text description integrating target category, target attribute, target location, and scene relationship between targets.

[0009] Optionally, the keyword features and semantic features of the structured text are extracted, including: parsing the tracking identifier, target category words, and attribute words from the structured text as the keyword features, and assigning different weights to different types of keyword features; using a pre-trained language model to encode the whole or part of the structured text to generate the semantic feature vector.

[0010] Optionally, the retrieval database containing the keyword features, semantic features, and video time sequence information is constructed, including: storing the semantic feature vector, the keyword features and their weights, and the time stamp of the video frame corresponding to the semantic feature vector in association; wherein the retrieval database supports semantic retrieval based on vector similarity and accurate retrieval based on keyword weight.

[0011] Optionally, the matching of the query keyword and the keyword feature in the retrieval database and the matching of the query semantics and the semantic feature in the retrieval database based on the hybrid retrieval strategy comprises: calculating the coincidence degree and weight of the query keyword and the keyword feature to obtain a keyword matching score; calculating the similarity of the query semantics and the semantic feature to obtain a semantic similarity score; if the text query contains a time sequence condition, filtering data in the retrieval database according to the time sequence condition.

[0012] Optionally, the fusion of the keyword matching result and the semantic matching result to obtain a final retrieval result comprises: final retrieval score = β × keyword matching score + (1-β) × semantic similarity score + γ × time sequence matching score wherein β is a keyword weight coefficient and γ is a time sequence weight coefficient.

[0013] The present specification provides a monitoring video cross-modal semantic matching device, comprising: a first extraction module for target detection and tracking of a monitoring video to extract target information in a video frame and filter repeated targets; a generation module for generating a structured text describing video content based on the target information using a visual large model; a second extraction module for extracting keyword features and semantic features of the structured text and constructing a retrieval database containing the keyword features, semantic features and video time sequence information; a third extraction module for receiving a text query and extracting a query keyword and query semantics of the text query; a matching module for matching the query keyword with the keyword features in the retrieval database and matching the query semantics with the semantic features in the retrieval database based on a hybrid retrieval strategy; a fusion module for fusing keyword matching results and semantic matching results to obtain a final retrieval result and returning a video segment corresponding to the final retrieval result.

[0014] Optionally, the first extraction module comprises: adopting a target detection algorithm to identify at least one target and its attribute information in a video frame; adopting a video object tracking algorithm to assign a unique tracking identifier to the same target appearing in consecutive video frames; based on the tracking identifier, filtering repeated detection results of the same target and retaining target information in only part of the frames for generating the structured text.

[0015] Optionally, the generating module comprises: inputting the target information, the tracking identifier, and the corresponding video frame into the visual large model; wherein the target information comprises a category, attributes, and location information of the target; obtaining a text description output by the visual large model and organized in a structured data format, the text description integrating target category, target attribute, target location, and scene relationship between targets.

[0016] Optionally, the second extracting module comprises: parsing a tracking identifier, a target category word, and an attribute word from the structured text as the keyword feature, and assigning different weights to different types of the keyword feature; encoding the whole or part of the structured text using a pre-trained language model to generate the semantic feature vector.

[0017] Optionally, the second extracting module comprises: storing the semantic feature vector, the keyword feature and its weight, and the timestamp of the video frame corresponding to the semantic feature vector in association; wherein the retrieval database supports semantic retrieval based on vector similarity and accurate retrieval based on keyword weight.

[0018] Optionally, the matching module comprises: calculating the coincidence degree and weight of the query keyword and the keyword feature to obtain a keyword matching score; calculating the similarity of the query semantic and the semantic feature to obtain a semantic similarity score; if the text query contains a timing condition, filtering the data in the retrieval database according to the timing condition.

[0019] Optionally, the fusion module comprises: Final retrieval score = β × keyword matching score + (1-β) × semantic similarity score + γ × timing matching score wherein β is a keyword weight coefficient, and γ is a timing weight coefficient.

[0020] The specification also provides an electronic device, wherein the electronic device comprises: a processor; and a memory storing computer-executable instructions that, when executed, cause the processor to perform any of the above methods.

[0021] The specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, which when executed by a processor, implement any of the above methods.

[0022] In the present application, by introducing target detection and object tracking technology, the visual large model is provided with accurate and structured target attribute information as the basis for generation, fundamentally overcoming the problems of fuzzy attribute description and mismatch caused by relying on global visual features in traditional methods, and realizing high-precision retrieval of color, texture, carried articles and other detailed features. Using the object tracking mechanism to identify and filter redundant target information in continuous frames of video, the computational load of subsequent visual large model processing and text generation is greatly reduced. Combined with an efficient hybrid retrieval strategy, the system can achieve fast response in massive monitoring video data, effectively meeting the urgent needs of real-time or quasi-real-time retrieval in security and other scenarios. The temporal information of the video frame is deeply integrated into the whole process of text representation and retrieval matching, so that the system can accurately understand and respond to user's composite query containing time range, solving the key pain point of disconnection of temporal information in traditional cross-modal retrieval methods, and the retrieval result is more consistent with the actual application logic of monitoring video data. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0024] Figure 1 A principle diagram of a monitoring video cross-modal semantic matching method provided by an embodiment of the present application; Figure 2 A structure diagram of a monitoring video cross-modal semantic matching device provided by an embodiment of the present application; Figure 3 A structure diagram of an electronic device provided by an embodiment of the present application; Figure 4 A principle diagram of a computer-readable medium provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following description is used to disclose the present application so that those skilled in the art can implement the present application. The preferred embodiments in the following description are only as examples, and other obvious modifications can be thought by those skilled in the art. The basic principles of the present application defined in the following description can be applied to other embodiments, modifications, improvements, equivalents and other technical solutions without departing from the spirit and scope of the present application.

[0026] The accompanying drawings are incorporated in and constitute a part of this specification and will be understood by those skilled in the art. Figures 1-4 Exemplary embodiments of the present application are described more fully hereinafter with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the inventive concept to those skilled in the art. Like reference numerals refer to like elements throughout the specification.

[0027] In the case of a certain specific embodiment, the features, structures, characteristics or other details described do not exclude that they can be combined in a suitable manner in one or more other embodiments, in accordance with the technical concept of the present application.

[0028] In the description of the specific embodiments, the features, structures, characteristics or other details described are for the purpose of enabling a full understanding of the embodiments by those skilled in the art. However, it does not exclude that one or more of the specific features, structures, characteristics or other details can not be practiced by those skilled in the art without one or more of them.

[0029] The flowcharts shown in the drawings are only exemplary illustrations, and do not necessarily include all contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to the actual situation.

[0030] The block diagrams shown in the drawings are only functional entities, and do not necessarily have to correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] The term "and / or" or "and / or" includes all combinations of any one or more of the associated listed items.

[0032] Figure 1 A schematic diagram of a method for monitoring video cross-modal semantic matching provided by an embodiment of the present application, which can include: S110: target detection and tracking of the monitoring video to extract target information in the video frame and filter duplicate targets; Optionally, the S110 includes: Using a target detection algorithm to identify at least one target and its attribute information in the video frame; A video object tracking algorithm is used to assign a unique tracking identifier to the same target appearing in consecutive video frames; Based on the tracking identifier, the repeated detection results of the same target are filtered, and only the target information in part of the frames is retained for generating the structured text.

[0033] In the specific embodiments of the present specification, the target detection module uses a YOLOv8 model accelerated by TensorRT to perform inference on video frames on an NVIDIA A10 GPU, and the single-frame processing time can be controlled within 10 ms, ensuring real-time processing capability for 30fps video. The model is trained for monitoring scenarios and can accurately detect categories such as “person”, “vehicle”, “backpack”, “fire hydrant”, etc., and extract attributes such as color (e.g. “red”), texture, shape (e.g. “SUV”) of the target. The object tracking module integrates the DeepSORT algorithm, extracts the HOG and CNN appearance features of the target, and combines the motion trajectory predicted by Kalman filtering to calculate the correlation degree of the target between consecutive frames, and assigns a unique tracking ID such as “T005” to the same target. The filtering rule is as follows: the system maintains the attribute state of each tracking ID, and when the target with the same ID is detected in consecutive 5 frames (N=5) and the key attributes such as color and carried articles of the target do not change, only the subsequent text generation process of the 1st frame (K=1) is triggered, and the detection results of the subsequent 4 frames are directly filtered and not sent to the visual large model. This filtering mechanism can significantly reduce the computational load of text generation through testing.

[0034] S120: Based on the target information, a visual large model is used to generate structured text describing the video content; Optionally, the S120 comprises: The target information, the tracking identifier, and the corresponding video frame are input into the visual large model; wherein the target information includes the category, attributes, and location information of the target; The text description output by the visual large model in a structured data format is obtained, and the text description integrates the target category, target attributes, target location, and scene relationship between targets.

[0035] In the specific implementation of this specification, the image textification module receives attribute tags (such as "color: red, clothing: coat, backpack: black") from the target detection module, tracking IDs (such as "T001") from the object tracking module, and the corresponding original video frame images. This information is assembled into a prompt word and input into the LLaVA-1.5 model. The model's prompt word is specifically optimized, for example: "You are a surveillance video analysis model. Please generate a structured JSON description based on the provided target list and original images. Target list: [ID:T001, Category: Person, Attributes: Red Coat, Black Backpack, Position: Left Side of the Screen]." The output generated by the model is a strictly structured JSON object, for example: {"track_ids":[{"id":"T001","category":"Person","attributes":{"color":"Red","clothes":"Coat","bag":"Black Backpack"},"position":"Left Side of the Screen"}],"scene":"Shopping Mall Underground Parking Garage","description":"..."}. This structured output ensures accurate and unambiguous representation of target attributes, locations, and scene relationships, laying a solid foundation for subsequent retrieval.

[0036] S130: Extract the keyword features and semantic features of the structured text, and construct a retrieval database containing the keyword features, semantic features, and video temporal information; Optionally, S130 includes: The tracking identifier, target category words, and attribute words are parsed from the structured text as the keyword features, and different weights are assigned to different types of keyword features; The structured text, in whole or in part, is encoded using a pre-trained language model to generate the semantic feature vector.

[0037] In the detailed description of the present specification, the text feature processing module parses the JSON format text generated by the visual large model. For keyword extraction, a rule-based parser is used to directly extract "T001", "person", "red coat", "black backpack" and other word units from the track_ids and attributes fields of JSON, and according to the pre-defined weight table (track ID weight 1.5, target category word weight 1.0, color, texture and other attribute word weight 1.2, scene word weight 0.8), each keyword is assigned a weight. For semantic features, the description field (natural language description) in JSON is input into the pre-trained all-MiniLM-L6-v2Sentence-BERT model to generate a 384-dimensional semantic vector. At the same time, the timestamp (such as "14:30:25") is converted into the number of seconds in a day (52225 seconds), and the normalized value is concatenated to the semantic vector as an additional dimension, thereby realizing the integration of time sequence information.

[0038] Optionally, the S130 comprises: The semantic feature vector, the keyword feature and its weight, and the timestamp of the video frame corresponding to the semantic feature vector are stored in association; wherein the retrieval database supports semantic retrieval based on vector similarity and accurate retrieval based on keyword weight.

[0039] In the detailed description of the present specification, the system uses the open source vector database Milvus2.3 version. A Collection named video_embeddings is created in the database. The Collection contains the following fields: id (primary key), embedding (storing 384-dimensional Sentence-BERT semantic vector), keywords (storing keyword list such as ["T001", "person", "red coat", "black backpack"]), keyword_weights (storing weight list of corresponding keywords such as [1.5, 1.0, 1.2, 1.2]), timestamp (storing timestamp). Create IVF_FLAT index for the embedding field, with nlist parameter set to 1024 to accelerate large-scale vector similarity search. At the same time, create an inverted index for the keywords field, and use the expr (expression) function of Milvus to enable accurate filtering based on keywords and time when querying, for example "keywords in ['red coat', 'black backpack'] and timestamp between '2024-05-20 14:30:00' and '2024-05-20 14:45:00'".

[0040] S140: receiving a text query and extracting query keywords and query semantics of the text query; S150: matching the query keywords with keyword features in the retrieval database and matching the query semantics with semantic features in the retrieval database based on a hybrid retrieval strategy; Optionally, the S150 comprises: calculating coincidence degrees and weights of the query keywords and the keyword features to obtain keyword matching scores; calculating similarity degrees of the query semantics and the semantic features to obtain semantic similarity scores; if the text query contains a time sequence condition, filtering data in the retrieval database according to the time sequence condition.

[0041] In the specific embodiments of the present specification, the hybrid retrieval module executes two sub-processes in parallel. In the keyword matching process, the system first parses the user query, such as "14:30-14:45 person wearing a red coat", and extracts the keywords ["red coat", "person"] and the time sequence range. Then, a filter expression is submitted to the Milvus database to retrieve all records containing "red coat" and "person" in the keywords field and within the specified time range. For each matching record, the keyword matching score is calculated: assuming that the query keyword weights are 1.2 and 1.0, and the sum is 2.2, if a record matches both keywords, the score is (1.2+1.0) / 2.2=1.0. In the semantic matching process, the user query text (after removing the time condition) is input into the same Sentence-BERT model to generate a query vector. Then, on the subset of records preliminarily filtered by keywords and time, the cosine similarity between the query vector and the embedding vector in the library is calculated as the semantic similarity score. If the query text contains spatial relationship words such as "left side" and "next to", the system will identify these keywords and give more attention to the difference in the specific dimensions (determined by model analysis) of the semantic vector representing spatial relationships when calculating the cosine similarity, which is equivalent to increasing the weight of this part.

[0042] S160: fusing the keyword matching result and the semantic matching result to obtain a final retrieval result, and returning a video segment corresponding to the final retrieval result.

[0043] Optionally, the S160 comprises: Final retrieval score = β × keyword matching score + (1-β) × semantic similarity score + γ × time sequence matching score wherein β is a keyword weight coefficient and γ is a time sequence weight coefficient.

[0044] In the detailed description of the present specification, the result fusion module receives scores from keyword matching and semantic matching. The system presets fusion parameters β = 0.5 (keyword weight) and γ = 0.15 (timing weight). The calculation method of the timing matching score is: 1 - |frame timestamp - query time range midpoint| / (query time range duration / 2), which ensures that the frame at the center of the time range has the highest score. For example, the query time range is 14:30-14:45 (15 minutes long), and the frame timestamp is 14:31, then the timing matching score = 1 - |1| / (15 / 2)≈0.867. Finally, the total score of the frame for retrieval = 0.5*keyword matching score + 0.35*semantic similarity score + 0.15*0.867. These parameters β and γ are not fixed, and the performance optimization module will collect user feedback on the retrieval results (such as click rate, false positive report), and use gradient descent method to fine-tune these parameters on a validation set to optimize the overall retrieval effect.

[0045] Optionally, the specific way of filtering duplicate targets is: when the attributes of the targets corresponding to the same tracking identifier do not change in consecutive N frames, only the target information of the Kth frame is retained for generating the structured text, where N≥3 and K≤N.

[0046] In the detailed description of the present specification, the object tracking module maintains a target state table. For the target with tracking ID T001, the system continuously compares the attribute hash values of the target in different frames. Assuming that in the frame sequence F1, F2, F3, F4, F5, the attribute hash values of T001 are all the same (indicating that the attributes have not changed), and N is set to 5. Then, the system will only trigger text generation once in frame F1, and for frames F2 to F5, although the target is tracked, its information will not be sent to the visual large model. The parameter N can be adjusted adaptively according to the scene, for example, in the traffic checkpoint scene, the vehicle moves fast, the system sets N to 3 frames (about 100 ms); in the park patrol scene, the pedestrian moves slowly, the system sets N to 8 frames (about 267 ms). This adaptive adjustment is realized through a simple rule controller, which monitors the average motion speed of the target in the scene.

[0047] In the present application, by introducing object detection and object tracking technology, the visual large model is provided with accurate and structured object attribute information as the basis for generation, fundamentally overcoming the problem of fuzzy attribute description and mismatch caused by relying on global visual features in traditional methods, and realizing high-precision retrieval of detailed features such as color, texture, and carried items. Using the object tracking mechanism to identify and filter redundant target information in consecutive video frames, the computational load of subsequent visual large model processing and text generation is greatly reduced. Combined with an efficient hybrid retrieval strategy, the system can quickly respond in massive monitoring video data, effectively meeting the urgent needs of real-time or quasi-real-time retrieval in security and other scenarios. The temporal information of the video frame is deeply integrated into the whole process of text representation and retrieval matching, so that the system can accurately understand and respond to user queries containing time ranges, solving the key pain point of disconnection of temporal information in traditional cross-modal retrieval methods, and the retrieval results are more consistent with the actual application logic of monitoring video data.

[0048] Figure 2 A structural schematic diagram of a monitoring video cross-modal semantic matching device provided by an embodiment of the present application can include: A first extraction module 10 is configured to perform object detection and tracking on the monitoring video to extract target information in the video frame and filter duplicate targets. A generation module 20 is configured to generate structured text describing the video content based on the target information using a visual large model. A second extraction module 30 is configured to extract keyword features and semantic features of the structured text and construct a retrieval database containing the keyword features, semantic features, and video temporal information. A third extraction module 40 is configured to receive a text query and extract query keywords and query semantics of the text query. A matching module 50 is configured to match the query keywords with the keyword features in the retrieval database and match the query semantics with the semantic features in the retrieval database based on a hybrid retrieval strategy. A fusion module 60 is configured to fuse the keyword matching result and the semantic matching result to obtain a final retrieval result and return a video segment corresponding to the final retrieval result.

[0049] Optionally, the first extraction module 10 includes: A target detection algorithm is used to identify at least one target and its attribute information in the video frame. A video object tracking algorithm is used to assign a unique tracking identifier to the same target appearing in consecutive video frames. Based on the tracking identifier, the duplicate detection results of the same target are filtered, and only the target information in part of the frames is retained for generating the structured text.

[0050] Optionally, the generation module 20 comprises: inputting the target information, the tracking identifier and the corresponding video frame into the visual large model; wherein the target information comprises the category, attribute and position information of the target; obtaining the text description output by the visual large model in a structured data format, the text description integrating the target category, target attribute, target position and scene relationship between targets.

[0051] Optionally, the second extraction module 30 comprises: parsing the tracking identifier, target category word and attribute word from the structured text as the keyword feature, and assigning different weights to different types of keyword features; encoding the whole or part of the structured text using a pre-trained language model to generate the semantic feature vector.

[0052] Optionally, the second extraction module 30 comprises: storing the semantic feature vector, the keyword feature and its weight, and the timestamp of the video frame corresponding to the semantic feature vector in association; wherein the retrieval database supports semantic retrieval based on vector similarity and accurate retrieval based on keyword weight.

[0053] Optionally, the matching module 50 comprises: calculating the coincidence degree and weight of the query keyword and the keyword feature to obtain a keyword matching score; calculating the similarity of the query semantic and the semantic feature to obtain a semantic similarity score; if the text query contains a timing condition, filtering the data in the retrieval database according to the timing condition.

[0054] Optionally, the fusion module 60 comprises: Final retrieval score = β × keyword matching score + (1-β) × semantic similarity score + γ × timing matching score wherein β is the keyword weight coefficient, and γ is the timing weight coefficient.

[0055] The functions of the device of the embodiments of the present application have been described in the above-mentioned method embodiments, so if there is anything not described in detail in the description of the present embodiment, please refer to the relevant description in the foregoing embodiments, which will not be repeated here.

[0056] Based on the same inventive concept, the present specification also provides an electronic device.

[0057] The electronic device embodiment of the present application is described below, which can be regarded as a specific physical implementation of the above-mentioned method and device embodiments of the present application. For the details described in the electronic device embodiment of the present application, it should be regarded as a supplement to the above-mentioned method or device embodiments; for the details not disclosed in the electronic device embodiment of the present application, it can be implemented with reference to the above-mentioned method or device embodiments.

[0058] Figure 3 A structural schematic diagram of an electronic device is provided for the embodiment of the present specification. The electronic device 300 according to the embodiment of the present application is described below with reference to Figure 3 Figure 3 The displayed electronic device 300 is only an example and should not bring any limitation to the function and use range of the embodiment of the present application.

[0059] As shown in Figure 3 , the electronic device 300 is manifested in the form of a general computing device. The components of the electronic device 300 can include but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different system components (including the storage unit 320 and the processing unit 310), a display unit 340, etc.

[0060] The storage unit stores program codes, which can be executed by the processing unit 310, so that the processing unit 310 performs the steps according to various exemplary embodiments of the present application described in the processing method part of the present specification. For example, the processing unit 310 can perform the steps as shown in Figure 1

[0061] The storage unit 320 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 3201 and / or a cache memory unit 3202, and can further include a read-only memory (ROM) 3203.

[0062] The storage unit 320 can further include program / utilities 3204 with a set of (at least one) program modules 3205, such as but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof can include the implementation of a network environment.

[0063] The bus 330 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0064] ​​The electronic device 300 can also communicate with one or more external devices 400 such as a keyboard, a pointing device, a Bluetooth device, etc.; other devices that enable a user to interact with the electronic device 300; and / or any devices (e.g., a router, a modem, a peer device etc.) that enable the electronic device 300 to communicate with one or more other computing devices. Such communication can occur via an input / output (I / O) interface 350. Still yet, the electronic device 300 can communicate with one or more networks, such as a local area network (LAN), a general area network (WAN), and / or the Internet, via a network adapter 360. The network adapter 360 can communicate with the other components of the electronic device 300 via the bus 330. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with the electronic device 300. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc. Figure 3

[0065] From the above description of the embodiments of the present application, it is easy for those skilled in the art to understand that the exemplary embodiments described in the present application can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product. The software product can be stored in a computer readable storage medium (which can be a CD-ROM, an U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above-described methods according to the present application. When the computer program is executed by a data processing device, the computer readable medium enables the data processing device to implement the above-described methods of the present application, i.e., the method shown in FIG. 8. Figure 1

[0066] Figure 4 A schematic diagram of a computer readable medium according to an embodiment of the present application.

[0067] Implementation Figure 1 ​​A computer program of the illustrated method can be stored on one or more computer readable media. The computer readable media can be a readable signal medium or a readable storage medium. A readable storage medium, for example, can be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0068] The computer readable storage medium can include a data signal transported over a carrier wave and can be baseband or propagated along with carriers. The propagated carrier can take any suitable form, including but not limited to electro-magnetic, optical, or any suitable combination thereof. A computer readable medium can be any medium that can be read by a computer. Such mediums can include, but are not limited to, optical, electrical or magnetic tapes, optical, electrical, or magnetic disks, memory chips, or the like. The computer readable medium can be a medium that can be used to store or transport the program and / or data for use by or in connection with the instruction execution system, apparatus, or device.

[0069] The program code can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the audience computing device, partly on the audience computing device, as a stand-alone software package, partly on the audience computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the audience computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0070] In light of the above, the present application can be implemented in hardware, or implemented in software modules running on one or more processors, or implemented in a combination of the two. Those skilled in the art should understand that some or all of the functions of some or all of the components according to the embodiments of the present application can be implemented in practice using a general-purpose data processing device such as a microprocessor or a digital signal processor (DSP). The present application can also be implemented as a device or apparatus program (for example, a computer program and a computer program product) for performing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer readable medium or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0071] The above-described specific embodiments further illustrate the purposes, technical solutions and beneficial effects of the present application. It should be understood that the present application is not inherently related to any specific computer, virtual device or electronic device, and various general-purpose devices can implement the present application. The above-described specific embodiments are merely examples of the present application and are not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0072] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be mutually referred to. Each of the embodiments mainly describes the differences from other embodiments.

[0073] The above-described specific embodiments are merely examples of the present application and are not intended to limit the present application. Those skilled in the art can make various changes and modifications to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for monitoring video cross-modal semantic matching, characterized in that, The method comprises the following steps: target detection and tracking of surveillance video to extract target information in video frames and filter repeated targets; based on the target information, using a large visual model to generate structured text describing the video content; extracting keyword features and semantic features of the structured text and constructing a retrieval database containing the keyword features, semantic features and video timing information; receiving a text query and extracting query keywords and query semantics of the text query; based on a hybrid retrieval strategy, matching the query keywords with the keyword features in the retrieval database and matching the query semantics with the semantic features in the retrieval database; fusing keyword matching results and semantic matching results to obtain final retrieval results and returning video clips corresponding to the final retrieval results. 2.The method of claim 1, wherein, The target detection and tracking of surveillance video to extract target information in video frames and filter repeated targets comprises the following steps: using a target detection algorithm to identify at least one target and its attribute information in the video frame; using a video object tracking algorithm to assign a unique tracking identifier to the same target appearing in consecutive video frames; based on the tracking identifier, filtering repeated detection results of the same target and retaining only target information in part of the frames for generating the structured text.

3. The method of claim 2, wherein, The method of using a large visual model to generate structured text describing the video content based on the target information comprises the following steps: inputting the target information, tracking identifier and corresponding video frame into the large visual model; wherein the target information includes target category, attribute and location information; obtaining the text description output by the large visual model in a structured data format, which integrates target category, target attribute, target location and scene relationship between targets. 4.The method of claim 1, wherein, The method of extracting keyword features and semantic features of the structured text comprises the following steps: parsing the tracking identifier, target category words and attribute words from the structured text as the keyword features, and assigning different weights to different types of keyword features; using a pre-trained language model to encode the whole or part of the structured text to generate the semantic feature vector.

5. The method of claim 1, wherein, The method of constructing a retrieval database containing the keyword features, semantic features and video timing information comprises the following steps: storing the semantic feature vector, keyword features and their weights, and the timestamp of the video frame corresponding to the semantic feature vector in association; wherein the retrieval database supports semantic retrieval based on vector similarity and exact retrieval based on keyword weight.

6. The method of claim 1, wherein, The method of matching the query keywords with the keyword features in the retrieval database and matching the query semantics with the semantic features in the retrieval database based on a hybrid retrieval strategy comprises the following steps: calculating the coincidence degree and weight of the query keywords and the keyword features to obtain a keyword matching score; calculating the similarity of the query semantics and the semantic features to obtain a semantic similarity score; if the text query contains a timing condition, filtering the data in the retrieval database according to the timing condition.

7. The method of claim 6, wherein, The fusion keyword matching result and semantic matching result, get final search result, include: Final search score = beta x keyword matching score + (1-beta) x semantic similarity score + gamma x time sequence matching score Where, beta is keyword weight coefficient, gamma is time sequence weight coefficient.

8. A cross-modal semantic matching device for surveillance video, characterized in that, Include: The first extraction module is used for target detection and tracking of the monitoring video to extract target information in the video frame and filter repeated targets; The generation module is configured to generate structured text describing the video content based on the target information using a visual large model; The second extraction module is configured to extract keyword features and semantic features of the structured text and construct a search database containing the keyword features, semantic features and video time sequence information; The third extraction module is configured to receive a text query and extract query keywords and query semantics of the text query; The matching module is configured to match the query keywords with the keyword features in the search database and match the query semantics with the semantic features in the search database based on a hybrid search strategy; The fusion module is configured to fuse the keyword matching result and the semantic matching result to obtain a final search result and return a video segment corresponding to the final search result.

9. An electronic device, comprising: The electronic device includes: A processor; and a memory storing computer-executable instructions that, when executed, cause the processor to perform the method of any of claims 1-7.

10. A computer readable storage medium, wherein, The computer-readable storage medium stores one or more programs that, when executed by a processor, implement the method of any of claims 1-7.

Images