Intelligent search method and system based on edge event and multi-mode video tag

Abstract

The invention provides an intelligent search method and system based on an edge event and a multi-mode video tag, and the method comprises the steps: generating an alarm event by an edge device when a preset rule is triggered; the SAAS cloud platform performs label index management on the alarm event, and performs sampling frame extraction on video data in the alarm event to obtain a key frame picture; constructing a multi-modal video tag library according to the key frame pictures; analyzing the search statement by adopting an NLP algorithm to obtain an attribute value corresponding to the search statement; analyzing the search statement by adopting a structural vector database to obtain an attribute value corresponding to the key frame; based on the attribute value corresponding to the search statement, searching the attribute value corresponding to the closest key frame from a multi-modal video tag library; calculating the attribute value corresponding to the closest key frame to obtain a plurality of matching degrees; sorting the plurality of matching degrees according to a sequence from large to small, and selecting a value with the maximum matching degree; and more than multiple paths of cameras are monitored in real time, so that the automatic and intelligent management level is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent security technology, and in particular to intelligent search methods and systems based on edge events and multimodal video tags. Background Technology

[0002] In recent years, video surveillance has been widely used in various sectors of society, including industry, transportation, and smart communities. It is one of the important means of modern management and monitoring. How to improve management efficiency and automation level has become a major research topic in the development of intelligent video security.

[0003] Currently, in the field of intelligent security, such as intelligent IPC cameras, AI alarm events mainly target consumers (C-end), and mostly use edge-side target detection (such as people and vehicles), focusing on achieving proactive perception. However, this approach suffers from high equipment costs and the inability to reuse existing equipment. In monitoring scenarios for large communities, supermarkets, factories, and other industries, facing a large number of cameras and massive amounts of recorded data, a new system solution is proposed to address the pain points of low-cost equipment upgrades and the difficulty in detecting and managing large volumes of information, thus upgrading from traditional "passive monitoring" and "manual management" to "proactive perception" and "intelligent analysis."

[0004] Current smart IPC cameras have the following limitations: 1) Single algorithm: Due to limited computing power on the edge, a single algorithm is often tied to the device, making upgrades inconvenient; 2) Limited functionality: Primarily serving as a "sentinel," it cannot support more intelligent analysis tasks; 3) Limited data: It mainly generates simple alarm data, which is difficult to use for in-depth intelligent analysis. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide an intelligent search method and system based on edge events and multimodal video tags, so as to realize real-time monitoring, real-time analysis and rapid retrieval capabilities of multiple cameras, improve the level of automation and intelligent management, and solve the problems of traditional visual recognition, manual browsing, low efficiency and easy omission.

[0006] In a first aspect, embodiments of the present invention provide an intelligent search method based on edge events and multimodal video tags, the method comprising: When a preset rule is triggered, the edge device generates an alarm event; wherein, the alarm event includes the algorithm's structured data, keyframes, and video data; The edge device sends the alarm event to the SAAS cloud platform; The SaaS cloud platform performs tag index management on the alarm events and samples and extracts frames from the video data in the alarm events to obtain keyframe images. Construct a multimodal video tag library based on the keyframe images; The obtained search statement is parsed using an NLP algorithm to obtain the attribute value corresponding to the search statement; The search query is parsed using a structured vector database to obtain the attribute values ​​corresponding to the keyframes; Based on the attribute value corresponding to the search statement, search for the attribute value corresponding to the closest keyframe from the multimodal video tag library; The attribute values ​​corresponding to the closest keyframes are calculated to obtain multiple matching degrees; Sort the multiple matching scores in descending order and select the value with the highest matching score.

[0007] Furthermore, a multimodal video tag library is constructed based on the keyframe images, including: The keyframe images are used to generate custom labels and vectorized labels using a multimodal large model; The multimodal video tag library is constructed based on the custom tags and the vectorized tags.

[0008] Furthermore, the custom tags include tags from custom algorithms in the edge device and tags from custom classifications in the SaaS cloud platform; the vectorized tags are vectorized indexes of the keyframe images.

[0009] Furthermore, the attribute values ​​corresponding to the search statement include the value of the first object, the value of the first tag, and the first vector value, and the attribute values ​​corresponding to the keyframe include the value of the second object, the value of the second tag, and the second vector value.

[0010] Furthermore, based on the attribute value corresponding to the search query, the attribute value corresponding to the closest keyframe is searched from the multimodal video tag library, including: Based on the value of the first object, the value of the first tag, and the first vector value, the value of the second object, the value of the second tag, and the second vector value corresponding to the closest keyframe are searched from the multimodal video tag library.

[0011] Furthermore, the attribute values ​​corresponding to the closest keyframe are calculated to obtain multiple matching degrees, including: The matching degree is calculated according to the following formula:

[0012]

[0013] in, The matching degree is denoted as n, where n is the number of sampled video frames. The value of the first object. The value of the second object, The value of the first label. The value of the second label. For the first vector value, The second vector value, denoted as cosine similarity, and x, y, and z are tuning coefficients.

[0014] Secondly, embodiments of the present invention provide an intelligent search system based on edge events and multimodal video tags, the system comprising: The generation module is used to generate alarm events by the edge device when a preset rule is triggered; wherein, the alarm events include the algorithm's structured data, keyframes, and video data; The sending module is used by the edge device to send the alarm event to the SAAS cloud platform; The frame extraction module is used by the SAAS cloud platform to perform tag index management on the alarm events and to sample and extract frames from the video data in the alarm events to obtain keyframe images. The module is used to construct a multimodal video tag library based on the keyframe images; The first parsing module is used to parse the acquired search statement using an NLP algorithm to obtain the attribute value corresponding to the search statement; The second parsing module is used to parse the search statement using a structured vector database to obtain the attribute values ​​corresponding to the keyframes. The search module is used to search for the attribute value corresponding to the closest keyframe in the multimodal video tag library based on the attribute value corresponding to the search statement. The calculation module is used to calculate the attribute values ​​corresponding to the closest keyframe to obtain multiple matching degrees; The sorting module is used to sort the multiple matching degrees in descending order and select the value with the highest matching degree.

[0015] Furthermore, the building module is specifically used for: The keyframe images are used to generate custom labels and vectorized labels using a multimodal large model; The multimodal video tag library is constructed based on the custom tags and the vectorized tags.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described above.

[0017] Fourthly, embodiments of the present invention provide a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the method described above.

[0018] This invention provides an intelligent search method and system based on edge events and multimodal video tags, comprising: when a preset rule is triggered, an edge device generates an alarm event; wherein the alarm event includes structured data of the algorithm, keyframes, and video data; the edge device sends the alarm event to a SaaS cloud platform; the SaaS cloud platform performs tag index management on the alarm event and samples and extracts frames from the video data in the alarm event to obtain keyframe images; a multimodal video tag library is constructed based on the keyframe images; and the obtained search statement is parsed using an NLP algorithm to obtain the attributes corresponding to the search statement. The system parses the search query using a structured vector database to obtain the attribute values ​​corresponding to keyframes. Based on the attribute values ​​corresponding to the search query, it searches for the attribute values ​​corresponding to the closest keyframes in a multimodal video tag library. It calculates the attribute values ​​corresponding to the closest keyframes to obtain multiple matching degrees. The multiple matching degrees are sorted in descending order, and the value with the highest matching degree is selected. This enables real-time monitoring, real-time analysis, and rapid retrieval capabilities for multiple cameras, improving the level of automation and intelligent management, and solving the problems of low efficiency and easy omission in traditional visual recognition and manual review.

[0019] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 The flowchart of the intelligent search method based on edge events and multimodal video tags provided in Embodiment 1 of the present invention is shown below. Figure 2 This is a schematic diagram of a system architecture using edge-cloud provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the intelligent search process based on edge events and multimodal video tags provided in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the alarm event structure provided in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the overall data structure and search process provided in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of an intelligent search system based on edge events and multimodal video tags provided in Embodiment 2 of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] To facilitate understanding of this embodiment, the embodiments of the present invention will be described in detail below.

[0025] Example 1: Figure 1 The flowchart shows the intelligent search method based on edge events and multimodal video tags provided in Embodiment 1 of the present invention.

[0026] Reference Figure 1 The method includes the following steps: Step S101: When the preset rule is triggered, the edge device generates an alarm event; wherein, the alarm event includes the algorithm's structured data, key frames and video data; Step S102: The edge device sends the alarm event to the SAAS cloud platform; Specifically, refer to Figure 3 Edge devices complete tasks such as local business logic, algorithm detection, and data structuring, and then upload the structured data to the SaaS cloud platform.

[0027] When a preset rule is triggered, such as the appearance of a person, the edge device generates an alarm event. This event mainly contains information such as the algorithm's structured data, keyframes, and video data. The preset rule refers to a custom rule defined for different application scenarios. The overall structure of the alarm event is shown below. Figure 4 .

[0028] Step S103: The SAAS cloud platform performs tag index management on alarm events and samples and extracts frames from the video data in the alarm events to obtain keyframe images. Step S104: Construct a multimodal video tag library based on keyframe images; Step S105: The obtained search statement is parsed using an NLP algorithm to obtain the attribute values ​​corresponding to the search statement; Step S106: The search statement is parsed using a structured vector database to obtain the attribute values ​​corresponding to the keyframes; Step S107: Based on the attribute value corresponding to the search statement, find the attribute value corresponding to the closest keyframe from the multimodal video tag library; Step S108: Calculate the attribute values ​​corresponding to the closest keyframes to obtain multiple matching degrees; Step S109: Sort the multiple matching scores in descending order and select the value with the highest matching score.

[0029] Furthermore, step S104 includes the following steps: Step S201: Use the multimodal large model to generate custom labels and vectorized labels from the keyframe images; Here, when calling the multimodal large model to generate custom labels and vectorized labels, index enhancement generation is completed.

[0030] Step S202: Construct a multimodal video tag library based on custom tags and vectorized tags.

[0031] Furthermore, custom tags include tags from custom algorithms in edge devices and tags from custom categories in SaaS cloud platforms; vectorized tags are vectorized indexes of keyframe images.

[0032] Furthermore, the attribute values ​​corresponding to the search statement include the value of the first object, the value of the first tag, and the first vector value, while the attribute values ​​corresponding to the keyframe include the value of the second object, the value of the second tag, and the second vector value.

[0033] Furthermore, step S107 includes the following steps: Based on the value of the first object, the value of the first tag, and the value of the first vector, search the multimodal video tag library for the value of the second object, the value of the second tag, and the value of the second vector corresponding to the closest keyframe.

[0034] Furthermore, step S108 includes: The matching degree is calculated according to formulas (1) and (2): (1) (2) in, The matching degree is denoted by n, where n is the number of video frames sampled. The value of the first object. The value of the second object, The value of the first label. The value of the second label. The first vector value, For the second vector value, denoted as cosine similarity, and x, y, and z are tuning coefficients.

[0035] Reference Figure 2 On the edge: cameras, like sentinels everywhere, provide the entire system with the ability to sense if someone enters.

[0036] Edge side: Locally deployed edge devices (AI-BOX devices) provide computing power and network support. On the SaaS cloud platform, users can send algorithm packages to AI-BOX devices according to their own application scenarios to build their own AI scenario applications.

[0037] The AI-BOX device acts as a "sentinel," fulfilling its duties as a "sentinel" by assigning tasks and organizing information, ensuring privacy and security, and transmitting effective information to the "command center."

[0038] SAAS cloud platform: Enables intelligent management of all cameras, improves management efficiency, and is equivalent to the "command center" of the entire monitoring system.

[0039] Reference Figure 5 The multimodal video tag library includes custom tags and vectorized tags. Custom tags include tags based on edge device custom algorithms (people, vehicles) and tags based on multimodal custom classifications from the SaaS cloud platform (red clothes, red car), used to support precise searching. Vectorized tags are vectorized indexes of keyframe images, used to support general search capabilities and image-based search capabilities.

[0040] Video intelligent search platform: Using SaaS as the cloud platform and a structured vector database as the computing engine, it builds NLP video intelligent search capabilities.

[0041] In this embodiment, the edge-side algorithm detects alarm events and transmits the alarm event information to the SaaS cloud platform. Then, the SaaS platform completes the establishment of a multimodal video tag library for the cloud platform. Finally, through NLP technology and a structured vector database engine, the video matching degree is calculated, and the intelligent search video data is returned.

[0042] Compared with the prior art, this application has the following advantages: 1) Rich algorithms: Users can customize algorithms through the SAAS cloud platform to complete intelligent search functions and solve personalized customization needs in industry scenarios.

[0043] 2) Convenient upgrades: Algorithm updates can be directly distributed to edge devices via the SaaS cloud platform. For existing cameras, AI capabilities can be enabled simply by connecting to edge devices, leveraging existing technology and reducing upgrade costs.

[0044] 3) Rich data and unified management: It can simultaneously connect to 100+ cameras, enabling unified management and retrieval of data, providing intelligent management capabilities, and solving the pain points of traditional manual identification, low efficiency, and easy omissions.

[0045] Example 2: Figure 6 This is a schematic diagram of an intelligent search system based on edge events and multimodal video tags provided in Embodiment 2 of the present invention.

[0046] Reference Figure 6 The system includes: The generation module is used to generate alarm events by the edge device when a preset rule is triggered; the alarm events include the algorithm's structured data, keyframes, and video data. The sending module is used by edge devices to send alarm events to the SaaS cloud platform; The frame extraction module is used by the SAAS cloud platform to manage the tag index of alarm events and to sample and extract frames from the video data in the alarm events to obtain keyframe images. The building module is used to construct a multimodal video tag library based on keyframe images; The first parsing module is used to parse the acquired search statement using an NLP algorithm to obtain the attribute value corresponding to the search statement; The second parsing module is used to parse the search statement using a structured vector database to obtain the attribute values ​​corresponding to the keyframes; The search module is used to find the attribute value corresponding to the closest keyframe from the multimodal video tag library based on the attribute value corresponding to the search query. The calculation module is used to calculate the attribute values ​​corresponding to the closest keyframes to obtain multiple matching degrees; The sorting module is used to sort multiple matching scores in descending order and select the value with the highest matching score.

[0047] Furthermore, the building blocks are specifically used for: Use the keyframe images to generate custom labels and vectorized labels using a multimodal large model; A multimodal video tag library is constructed based on custom tags and vectorized tags.

[0048] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the intelligent search method based on edge events and multimodal video tags provided in the above embodiments.

[0049] This invention also provides a computer-readable medium having processor-executable non-volatile program code, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the intelligent search method based on edge events and multimodal video tags described above.

[0050] The computer program product provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0051] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0052] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0053] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0054] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0055] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A smart search method based on edge events and multi-modal video tags, characterized in that, The method includes: When a preset rule is triggered, the edge device generates an alarm event; wherein, the alarm event includes the algorithm's structured data, keyframes, and video data; The edge device sends the alarm event to the SAAS cloud platform; The SaaS cloud platform performs tag index management on the alarm events and samples and extracts frames from the video data in the alarm events to obtain keyframe images. Construct a multimodal video tag library based on the keyframe images; The obtained search statement is parsed using an NLP algorithm to obtain the attribute value corresponding to the search statement; The search query is parsed using a structured vector database to obtain the attribute values ​​corresponding to the keyframes; Based on the attribute value corresponding to the search statement, search for the attribute value corresponding to the closest keyframe from the multimodal video tag library; The attribute values ​​corresponding to the closest keyframes are calculated to obtain multiple matching degrees; Sort the multiple matching scores in descending order and select the value with the highest matching score. 2.The smart search method based on edge events and multi-modal video tags of claim 1, wherein, A multimodal video tag library is constructed based on the keyframe images, including: The keyframe images are used to generate custom labels and vectorized labels using a multimodal large model; The multimodal video tag library is constructed based on the custom tags and the vectorized tags. 3.The smart search method based on edge events and multi-modal video tags of claim 2, wherein, The custom tags include tags from custom algorithms in the edge device and tags from custom categories in the SaaS cloud platform; the vectorized tags are vectorized indexes of the keyframe images. 4.The smart search method based on edge events and multi-modal video tags of claim 1, wherein, The attribute values ​​corresponding to the search statement include the value of the first object, the value of the first tag, and the first vector value. The attribute values ​​corresponding to the keyframe include the value of the second object, the value of the second tag, and the second vector value. 5.The smart search method based on edge events and multi-modal video tags of claim 4, wherein, Based on the attribute value corresponding to the search query, the attribute value corresponding to the closest keyframe is searched from the multimodal video tag library, including: Based on the value of the first object, the value of the first tag, and the first vector value, the value of the second object, the value of the second tag, and the second vector value corresponding to the closest keyframe are searched from the multimodal video tag library. 6.The smart search method based on edge events and multi-modal video tags of claim 5, wherein, The attribute values ​​corresponding to the closest keyframe are calculated to obtain multiple matching degrees, including: The matching degree is calculated according to the following formula: wherein, is the matching degree, n is the number of sampled video frames, is the value of the first object, is the value of the second object, is the value of the first label, is the value of the second label, is the first vector value, is the second vector value, is the cosine similarity, x, y, z are tuning coefficients.

7. An intelligent search system based on edge events and multimodal video tags, characterized in that, The system includes: The generation module is used to generate alarm events by the edge device when a preset rule is triggered; wherein, the alarm events include the algorithm's structured data, keyframes, and video data; The sending module is used by the edge device to send the alarm event to the SAAS cloud platform; The frame extraction module is used by the SAAS cloud platform to perform tag index management on the alarm events and to sample and extract frames from the video data in the alarm events to obtain keyframe images. The module is used to construct a multimodal video tag library based on the keyframe images; The first parsing module is used to parse the acquired search statement using an NLP algorithm to obtain the attribute value corresponding to the search statement; The second parsing module is used to parse the search statement using a structured vector database to obtain the attribute values ​​corresponding to the keyframes. The search module is used to search for the attribute value corresponding to the closest keyframe in the multimodal video tag library based on the attribute value corresponding to the search statement. The calculation module is used to calculate the attribute values ​​corresponding to the closest keyframe to obtain multiple matching degrees; The sorting module is used to sort the multiple matching degrees in descending order and select the value with the highest matching degree.

8. The intelligent search system based on edge events and multimodal video tags according to claim 7, characterized in that, The building module is specifically used for: The keyframe images are used to generate custom labels and vectorized labels using a multimodal large model; The multimodal video tag library is constructed based on the custom tags and the vectorized tags.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 6.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the method according to any one of claims 1 to 6.

Images