Security human-vehicle multi-mode retrieval method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computer information retrieval, and discloses a security person and vehicle multi-mode retrieval method and device, electronic equipment and a storage medium, and the method comprises the following steps: arranging various sensor equipment in a security area; uploading the collected video and image to a cloud object for storage; performing image text feature extraction by using the large model; receiving an image or text request uploaded by a user, calling the corresponding large model to carry out image text feature extraction, and carrying out feature retrieval through the vector database; an Elasticsearch database is used for further query, screening item information set by a user is combined, and the device comprises a data acquisition module, a management platform service module, a data storage module, a large model feature extraction module and a feature retrieval and matching module. According to the method, a clock callback detection and compensation mechanism, machine ID unified configuration and microsecond-level timestamp serial number generation are introduced, so that the technical effect of avoiding ID conflicts and time sequence errors is effectively achieved.

Description

Technical Field

[0001] This application relates to the technical field of computer information retrieval, and specifically to a security multi-modal retrieval method, device, electronic device, and storage medium for people and vehicles. Background Art

[0002] In the modern security field, target recognition and behavior analysis are important technical tasks, especially in scenarios such as "finding people" and "finding vehicles". With the continuous progress of artificial intelligence technology, retrieval engines based on image, text, and other multi-modal information are gradually applied to security monitoring systems to improve the intelligence level and working efficiency of the systems.

[0003] Existing image processing technologies mainly rely on deep learning models based on convolutional neural networks, which can perform object detection and recognition. However, with the continuous change of requirements, the accuracy of single image recognition technology is gradually limited in complex environments. In recent years, the proposal of cross-modal large models such as Chinese-clip has brought significant progress through joint feature learning of image and text information. These large models can usually perform semantic alignment between images and texts and use text descriptions to guide the understanding of image content, thereby improving the accuracy and efficiency of multi-modal retrieval.

[0004] However, the existing Chinese-clip large models have limitations in network structure design and parameter optimization. When dealing with large-scale image and text data, existing models have problems such as high consumption of computing resources and long model training time. The existing technology has not fully decoupled and optimized the feature extraction of image and text information. There are certain errors in the cross-modal semantic alignment between images and texts in traditional models, resulting in inaccurate retrieval results. Moreover, since the snowflake algorithm depends on the combination of timestamp, machine ID, and serial number to generate IDs, it is easily affected by clock rollback problems in a distributed environment, resulting in ID conflicts. Therefore, the present invention provides a security multi-modal retrieval method, device, electronic device, and storage medium to solve the deficiencies existing in the prior art. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the purpose of this application is to provide a security multi-modal retrieval method, device, electronic device, and storage medium, which solves the problems of low multi-modal retrieval accuracy, performance bottleneck, and poor system scalability in the security field.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: A security multi-modal retrieval method for people and vehicles, including the following steps: Arrange a variety of sensor devices in the security area, collect video and image data through the sensor devices, and save the sensor device information into the Elasticsearch database; Upload the collected videos and images to the cloud object storage, and generate corresponding feature IDs through an improved Snowflake algorithm; Use a large model to extract image text features, and store the extracted image text features and the generated feature IDs in the Milvus vector database; Receive the images or text requests uploaded by users, call the corresponding large model to extract image text features, and perform feature retrieval through the vector database to obtain the top N matching feature IDs; According to the feature IDs obtained from the feature retrieval, further query using the Elasticsearch database, and combine the filter option information set by the user to return the most matching retrieval results.

[0007] Preferably, the multiple sensor devices include a high-definition camera and an infrared camera. The multiple sensor devices collect images, videos, and sensor data through a data acquisition system, and upload and save the sensor device information through cloud storage. The sensor device information includes device ID, device type, device location, and device status, and stores this information in the Elasticsearch database.

[0008] Preferably, the generation of corresponding feature IDs through the improved Snowflake algorithm includes the following steps: By comparing the current timestamp with the last generated timestamp, if the current timestamp is less than the last timestamp, adjust the timestamp to compensate for the rollback problem to avoid generating duplicate IDs; Adopt a distributed hashing method to assign a unique machine ID to each device to avoid ID conflicts between different devices; Through the microsecond-level timestamp and sequence number generation mechanism, ensure that each generated ID is unique and conflict-free under high-concurrency requests; If clock rollback occurs, the calculation formula for the new timestamp is: ; Wherein, is the current timestamp; is the timestamp used for the last generated ID; is the preset time offset; is the new timestamp after compensation.

[0009] Preferably, the use of a large model to extract image text features includes the following steps: Extract features from the collected images through a large model, and convert the images into feature vectors in the vector space; Perform natural language processing on the text uploaded by users through a large model, extract text features, and convert the features into feature vectors in the vector space; Optimize the semantic alignment between images and texts using a cross-modal attention mechanism to ensure that the image and text features have a small distance in a common feature space.

[0010] Preferably, storing the extracted image text features and the generated feature IDs in the vector database includes the following steps: Store the image and text features extracted by the large model in the Milvus vector database to ensure the efficient storage of feature data and subsequent fast retrieval; Store information such as the feature ID, feature data corresponding to each image and text, and the cloud storage link of the image or text in the Elasticsearch database for fast access during retrieval; Through the efficient vector indexing algorithm of Milvus, quickly match the top N feature IDs with higher similarity.

[0011] Preferably, calling the corresponding large model to extract image text features and performing feature retrieval through the vector database includes the following steps: Receive the image or text request uploaded by the user, and perform corresponding large model feature extraction according to the type of the request; Connect to the Milvus vector database, perform retrieval based on the extracted feature vectors, and obtain the top N feature IDs most relevant to the user request; Further screen in the Elasticsearch database in combination with the screening options set by the user and return the most matching retrieval results.

[0012] Preferably, the Snowflake algorithm includes: Calculation of the timestamp and sequence number for generating the ID. If there are multiple requests to generate the ID, calculate through the following formula: ; Among them, represents the timestamp field in milliseconds; represents the device or node number; represents the incrementing sequence number for generating the ID within the same millisecond; represents the generated ID feature.

[0013] The optimization formula for cross-modal alignment, by calculating the distance between the image and the text in the common space, uses the following error function for optimization: ; Among them, represents the L2 norm; is the image feature vector; is the text feature vector.

[0014] An anti-theft vehicle and pedestrian multi-modal retrieval device is also provided, including: A data acquisition module, configured to acquire video image data within a security area and upload it to a management platform service module in real time; A management platform service module, configured to receive user retrieval input information, process the registration and storage logic of acquisition devices, and undertake the preprocessing and storage logic of acquired data; A data storage module, including: A Postgresql database, configured to store the registration information of acquisition devices; A cloud object storage, configured to store acquired video and picture data; A vector database Milvus, configured to store picture feature vectors extracted by a large model feature extraction module; An Elasticsearch database, configured to store picture features, picture cloud storage links, and picture source information; A large model feature extraction module, configured to extract features from pictures and text information; A feature retrieval and matching module, configured to receive a retrieval request forwarded by the management platform service module; An optimization and fault tolerance module, configured to perform clock rollback detection and compensation during device registration and data acquisition, and adopt an improved timestamp adjustment method.

[0015] An electronic device is also provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it can implement an anti-theft vehicle and pedestrian multi-modal retrieval method.

[0016] A storage medium is also provided, storing a computer program. When the computer program is executed by a processor, it can implement an anti-theft vehicle and pedestrian multi-modal retrieval method.

[0017] In summary, the present application includes at least one of the following beneficial technical effects: 1. The present invention adopts a technical solution of generating a unique feature ID using an improved snowflake algorithm. By introducing a clock rollback detection and compensation mechanism, unified configuration of machine IDs, and generation of microsecond-level timestamp serial numbers, the technical effects of avoiding ID conflicts and timing errors are effectively achieved. Compared with the prior art that directly relies on the traditional snowflake algorithm and is prone to ID duplication and data consistency problems in a multi-node high-concurrency environment, the problems of limited system scalability and insufficient stability are solved.

[0018] 2. The present invention applies a large-scale vision-language model for image and text feature extraction, and introduces a cross-modal attention mechanism to achieve alignment of the semantic spaces of images and texts, thereby achieving the technical effects of consistent graphic and text semantics and significantly improved retrieval accuracy. Different from the existing methods based on shallow features or simple keyword matching in the prior art, the present invention effectively overcomes the problems of deep semantic mismatch between images and texts and irrelevant retrieval results.

[0019] 3. The present invention uses a two-stage retrieval architecture that first performs efficient preliminary retrieval using a vector database and then combines an Elasticsearch database for screening and optimization, successfully achieving the technical effects of fast retrieval speed and support for multi-dimensional filtering. Compared with the existing solutions that solely rely on vector approximate retrieval or single full-text index retrieval in the prior art, the present invention significantly improves the problems of low recall rate and poor precision in large-scale data retrieval.

[0020] 4. The user retrieval request of the present invention supports multiple screening options, including time period, location, device ID category, and is combined with weighted fusion of feature similarity scores and screening matching scores, achieving the technical effects of accurate retrieval results and reasonable sorting overall. In contrast, the existing solutions usually can only perform single-dimensional filtering, easily suffering from defects such as chaotic results or weak relevance. The present invention effectively makes up for such deficiencies and significantly improves the retrieval experience in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of the method steps of this application; Figure 2 is a diagram of the device architecture of this application; Figure 3 is a schematic diagram of an electronic device of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The following is a further detailed description of this application in conjunction with the attached Figure 1 - attached Figure 3 , to further elaborate on this application.

[0023] Please refer to the attached Figure 1 , the embodiments of the present invention provide a security vehicle and pedestrian multi-modal retrieval method, including the following steps: S1. Arrange a variety of sensor devices within the security area, collect video and image data through the sensor devices, and save the sensor device information into the Elasticsearch database; S2. Upload the collected video and images to cloud object storage, and generate corresponding feature IDs through an improved snowflake algorithm; S3. Use a large model for image and text feature extraction, and store the extracted image and text features and the generated feature IDs into the Milvus vector database; S4. Receive the images or text requests uploaded by the user, call the corresponding large model to extract image and text features, and perform feature retrieval through the vector database to obtain the top N matching feature IDs. S5. Further query using the Elasticsearch database based on the feature IDs obtained from the feature retrieval, and combine with the screening option information set by the user to return the most matching retrieval results.

[0024] For step S1, in this embodiment, to achieve efficient and multi-modal retrieval of targets such as people and vehicles in the security area, it is first necessary to establish a complete basic data collection and device information management system to ensure that the subsequent feature extraction and retrieval processing processes have accurate and rich input data sources. By deploying a variety of sensor devices in the security area, the sensor devices at least include high-definition cameras and infrared cameras. Generally, high-definition cameras are mainly used to collect high-resolution image and video data during the day and in well-lit environments, while infrared cameras are suitable for target detection and tracking at night or in low-light environments.

[0025] In a possible implementation, the sensor devices transmit data to the central management platform through a wired network or a wireless communication network (such as 5G, Wi-Fi, LoRa, etc.) to achieve real-time or near-real-time data collection and reporting. As an option, some sensors can be pre-equipped with edge computing units for preprocessing the original video stream, such as target detection, motion analysis, etc., to reduce the occupancy of the transmission bandwidth.

[0026] Specifically, when each sensor device is registered, it needs to collect and record its basic information, including but not limited to: Device ID: Used to uniquely identify each sensor device, generally a globally unique identifier (UUID) assigned by the system; Device Type: Indicates the category of the sensor, such as high-definition camera, infrared camera, thermal imager, etc.; Device Location: Used to describe the physical deployment location of the device, which can adopt longitude and latitude coordinates, address descriptions, or building / floor information; Device Status: Records the device operation status information, such as online, offline, faulty, under maintenance, etc.

[0027] In some embodiments, the device registration information can be further extended to include installation time, camera orientation angles (such as pitch angle, yaw angle), coverage area description, affiliated management unit number, etc., to facilitate subsequent device status management and data traceability.

[0028] Generally, all the collected device basic information and real-time sensor status are uniformly processed in a standardized manner through the management platform and stored in the Elasticsearch database. As a distributed high-performance retrieval engine, the Elasticsearch database can support the rapid retrieval, update, and query of a large amount of device information, effectively supporting various complex query requirements such as subsequent classification by device and screening by geographical area.

[0029] In one implementation, after receiving the device data, the management platform encapsulates the device information in the standard JSON format.

[0030] In this embodiment, to further ensure the reliability of the system, a data signature verification field can also be attached when uploading device information, such as a signature value generated based on the SHA256 or HMAC algorithm, to prevent the risk of data tampering.

[0031] As an option, to improve the scalability of the system, when deploying devices, it is supported to access a large number of sensor devices at one time through a batch registration interface, and at the same time, different information item templates are defined for different types of devices. For example, an infrared camera can be attached with a heating sensitivity level field, while a high-definition camera can be attached with parameters such as maximum resolution and frame rate.

[0032] In another possible implementation, to facilitate subsequent feature extraction and matching, the device registration information can also record device acquisition capability parameters, such as basic data like image width and height, supported compression encoding formats (such as H.264, H.265), and acquisition frequency. These data can be used as auxiliary filtering conditions during subsequent retrieval optimization.

[0033] For step S2, in this embodiment, after completing the data acquisition of the sensor device and storing the device information in the database, in order to achieve the standardized management of image and video data and the orderly development of subsequent feature extraction work, it is necessary to perform unified storage processing on the collected image and video files and synchronously generate corresponding feature identification IDs. Based on this, combined with the improved Snowflake algorithm, it is ensured that the generated feature IDs have uniqueness, timeliness, and fault tolerance characteristics in a high-concurrency environment, laying a foundation for subsequent multi-modal feature extraction and retrieval.

[0034] In this embodiment, the collected video stream and captured images are first uploaded to the cloud object storage system after standardization processing (such as encoding conversion, format unification). Generally, the upload uses an interface based on the HTTPS or other encrypted communication protocols to ensure the security and integrity of the data transmission process. As an option, some high-frequency acquisition devices can enable the breakpoint resumption mechanism to cope with the interruption risk caused by network fluctuations.

[0035] Specifically, after the upload is completed, the management platform system generates a unique corresponding feature ID for each piece of image or video data. To ensure the uniqueness of the feature ID in a large-scale distributed system and avoid ID conflict problems caused by abnormal system clocks, an improved Snowflake algorithm is adopted in this embodiment for feature ID generation.

[0036] In a possible implementation manner, the improved Snowflake algorithm specifically includes the following technical features: First, the system compares the current timestamp with the timestamp used for the last ID generation . If it is detected that the current timestamp is less than , that is, when there is a system clock rollback, the timestamp compensation mechanism is triggered. The new compensated timestamp is calculated according to the following formula: ; where, is a preset time offset, usually set to the minimum tolerable unit (such as 1 millisecond).

[0037] In general, when the system time does not roll back, the current timestamp is directly used for ID generation without additional adjustment. As an option, can be dynamically set to adapt to the abnormal time drift tolerance in different system environments.

[0038] Secondly, the specific generation of the ID is combined by bitwise operations according to the following formula: where, represents the timestamp field in milliseconds, occupying 41 bits, identifying the ID generation moment; represents the device or node number, occupying 10 bits, and the uniqueness is ensured by using distributed hashing or registry center allocation methods; represents the incrementing sequence number of the ID generated within the same millisecond, occupying 12 bits, and when the sequence number reaches the maximum value, it needs to wait for the next millisecond to generate again.

[0039] In some embodiments, to further enhance the robustness of the system, the allocation of the machine ID can be managed using distributed coordination components such as Zookeeper and Etcd to ensure that there are no machine ID conflict problems between different nodes.

[0040] In a specific embodiment, when the sequence number overflows and the current millisecond remains unchanged, the system can choose to spin wait or sleep until the next millisecond to ensure the timeliness and uniqueness of the ID.

[0041] It should be noted that by adopting the improved Snowflake algorithm described above, the problem of ID duplication caused by factors such as system clock rollback, device ID conflicts, and high-concurrency requests can be effectively avoided, thus ensuring that each data file uploaded to the cloud storage has a unique and traceable feature identifier.

[0042] Generally, the generated will be recorded in the object storage system in the form of metadata, or synchronously saved to the management platform database for subsequent association and matching with image text features.

[0043] As an option, in order to improve the stability of ID generation in case of system exceptions, a redundant check code (such as CRC32 check value) can be attached when generating the ID to quickly verify the correctness during the ID generation process.

[0044] In another possible implementation, if there are multiple sub-modules (such as video module, image module) at the data acquisition end, independent sub-segments can be set for each module to quickly identify the data source module during subsequent analysis.

[0045] For step S3, in this embodiment, after the standardized storage of image and video data and the generation of unique feature IDs, in order to achieve efficient retrieval based on feature vectors in the subsequent stage, it is necessary to perform in-depth semantic feature extraction processing on the uploaded data. By applying advanced large-scale pre-trained models to extract the deep features of images and texts, and using cross-modal attention mechanisms to optimize the feature alignment process, the retrieval accuracy and the overall performance of the system are further improved. This technical solution combines the application suggestions of the Chinese-CLIP model proposed in the customer's briefing and the cross-modal alignment optimization method, ensuring the semantic richness and expression consistency of the feature extraction results. In this embodiment, for the collected image data, a large-scale vision-language pre-trained model (such as Chinese-CLIP) is used for feature extraction processing. Generally, the image first undergoes a standard preprocessing process, including size scaling, normalization processing (such as scaling pixel values to the [0,1] interval), and color space unification (such as unifying to the RGB space) to ensure the input consistency of the subsequent feature extraction process.

[0046] As an option, in some embodiments, different preprocessing strategies can also be used for data from different device sources (such as high-definition camera images, infrared camera images) to further improve the robustness of feature extraction.

[0047] Specifically, after the image is preprocessed, it is input into the visual encoder part of the Chinese-CLIP model (such as the ViT-B / 16 structure). The deep semantic features of the image are extracted through multiple layers of Transformer encoders, and the output is an image feature vector with a fixed dimension, denoted as: ; where is a vector with dimension D, and D is generally taken as 512, 768 or higher, depending on the model scale.

[0048] In a possible implementation, for the text description information uploaded by the user, the system also uses the text encoder in the same Chinese-CLIP model for feature extraction. The text is first tokenized, stop words are removed, and uniformly encoded into an input vector of a fixed length, and then the corresponding text feature vector is output through the Transformer encoder, denoted as: .

[0049] In some embodiments, if the text content is a structured retrieval request (such as "a male wearing black clothes, smoking and making a phone call on Renmin Road from 21:00 to 22:00 on March 2024"), it can be preliminarily structured by a large language model (such as GPT), and key information such as location, time, and action is extracted before feature vector quantization encoding.

[0050] Generally, to achieve semantic consistency between the image and the text in the vector space, this embodiment introduces a cross-modal attention mechanism. Specifically, by constructing a bidirectional attention weight matrix, a fine-grained correspondence relationship is established between the image region features and the text Token features to enhance the deep semantic alignment ability between vision and language.

[0051] In one implementation, to optimize cross-modal feature alignment, the error loss function is defined as follows: ; where represents the L2 norm (i.e., the square of the Euclidean distance). This optimization objective aims to minimize the distance between the image feature vector and the text feature vector in the common semantic space, thereby improving the accuracy of image-text matching.

[0052] As an option, to further optimize the distribution of the feature space, a contrastive learning loss (such as InfoNCE loss) can also be introduced in some embodiments to increase the distance difference between positive and negative sample pairs.

[0053] After feature extraction is completed, this embodiment associates and binds the extracted image feature vector and text feature vector with the corresponding generated and stores them uniformly in the Milvus vector database.

[0054] In a possible implementation, to improve the vector retrieval performance, the Milvus database constructs feature indexes using indexing algorithms such as HNSW (Hierarchical Navigable Small World) or IVF+PQ (Inverted File with Product Quantization), so that it can still maintain millisecond-level response in the retrieval of millions of feature vectors.

[0055] It should be noted that when storing features, the system also records auxiliary information such as the cloud object storage link (Storage URL) of the image or text data, the feature generation timestamp, and the feature source device ID, which facilitates the quick association of the original data during subsequent retrieval.

[0056] For step S4, in this embodiment, after the image text feature extraction and vectorized storage are completed, to meet the multi-modal retrieval requirements based on user input, the system receives user requests, performs feature extraction, vector retrieval, and preliminary feature matching, ensuring that the system can accurately locate the image or text data closest to the user input while responding efficiently. This embodiment fully considers the balance between retrieval performance and retrieval accuracy during design, and combines the multi-stage retrieval strategy of vector priority retrieval and filter option-assisted optimization proposed in the customer brief.

[0057] In this embodiment, the user uploads the image to be retrieved or enters a text description through the front-end interface, and the system receives the request and parses the request type. Generally, image requests directly enter the image feature extraction process, while text requests enter the text feature extraction process, and pre-deployed large models are called for processing respectively.

[0058] As an option, to meet different business requirements, audio retrieval requests can also be supported in the system design. By transcribing audio data into text first and then processing it in the same way as text retrieval.

[0059] Specifically, for image requests, this embodiment uses a large model (such as the Chinese-CLIP visual encoder) to extract features from the uploaded image. The extracted image feature vector is denoted as: ; where is a D-dimensional feature vector, which is consistent with the dimension of the image feature vectors stored in the library.

[0060] For text requests, this embodiment extracts features from the input text through the Chinese-CLIP text encoder to obtain a text feature vector, denoted as: .

[0061] In a possible implementation, to improve the retrieval accuracy, before extracting text features, content regularization can be performed through a natural language preprocessing module, such as synonym normalization, time and location entity extraction, etc.

[0062] After completing the query feature extraction, in this embodiment, the pymilvus client is used to connect to the Milvus vector database, and feature retrieval is performed based on similarity calculation. Generally, the inner product (Inner Product) or cosine similarity (Cosine Similarity) is used as the metric for similarity calculation. Taking cosine similarity as an example, the similarity formula is as follows: ; where is the query feature vector, is the feature vector stored in the database, represents the L2 norm of the vector.

[0063] During retrieval, the system returns the top features records most similar to from Milvus, obtaining the corresponding feature ID set, denoted as: ; where is the upper limit of the return quantity set by the system or selectable by the user, such as 10, 50, or 100.

[0064] As an option, to further improve the recall rate of the preliminary retrieval, in some embodiments, a similarity threshold (such as 0.7) can be set, and only the feature ID results with similarity higher than this threshold are retained.

[0065] In a specific implementation, during the query process, a feature clustering mechanism (such as K-Means or HNSW subgraph partitioning) can be enabled in parallel to accelerate the feature retrieval speed and reduce the interference of irrelevant features.

[0066] Generally, after completing the Milvus vector retrieval, in this embodiment, the final retrieval result is not directly returned immediately, but the obtained feature ID set is handed over to the subsequent steps for further multi-dimensional comprehensive screening through Elasticsearch in combination with the filter option information to improve the retrieval accuracy and user experience.

[0067] It should be noted that the retrieval and matching between the query feature vector extracted in this step and the vector library features are completely based on the consistency of the pre-established feature space, ensuring that the similarity can be directly calculated between the image and the text in the common vector space, and avoiding the matching error problem caused by feature heterogeneity in traditional retrieval.

[0068] For step S5, in this embodiment, after the preliminary feature matching based on the Milvus vector database is completed, in order to further improve the accuracy and multi-dimensional relevance of the retrieval results, a detailed retrieval and screening mechanism for the Elasticsearch database in combination with the feature ID is introduced. By comprehensively applying vector retrieval and multi-condition structured retrieval, this step effectively makes up for the limitations of pure vector matching under complex retrieval requirements and realizes an efficient transition from rough retrieval to precise retrieval.

[0069] Generally, the system uses this ID set as a query condition to initiate a batch retrieval request (Batch Query) to the Elasticsearch database. In a specific embodiment, the retrieval request adopts a Bool Query structure and is comprehensively filtered in combination with the multi-functional screening conditions set by the user.

[0070] Specifically, the user can set the following screening options when submitting the retrieval request: Retrieval category (such as person, vehicle, other targets); Retrieval time period (start_date, end_date, start_time, end_time); Snapshot device ID or device type; Location range or keyword; Specific attribute features (such as wearing color, behavior action).

[0071] Generally, to improve the query performance, in this embodiment, inverted indexes are pre-established in Elasticsearch for fields such as feature ID, category, time, and device ID, and reasonable sharding and replication strategies are configured according to the data volume to ensure low-latency response under high-concurrency retrieval.

[0072] As an option, the system also supports a weighted query mechanism based on weights, that is, different matching priorities are set for different screening options. For example, the time matching score weight can be higher than the device matching score weight, so as to be closer to the user's focus when retrieving and sorting.

[0073] In a specific embodiment, if the retrieval request contains structured text screening (such as location keywords, action descriptions, etc.), then in combination with the characteristics of the Elasticsearch full-text search engine, through Query or perform fuzzy matching and relevance scoring based on the BM25 scoring model.

[0074] To further optimize the retrieval accuracy, this embodiment also introduces a comprehensive sorting formula (comprehensive score formula), which is specifically as follows: ; Among them, is the final sorting score; is the similarity score in the vector retrieval stage; is the screening and matching score (such as category matching, time matching score); 、 is the weight coefficient set by the system, satisfying 。

[0075] Generally, can be set to 0.7, is set to 0.3, but it can also be flexibly adjusted according to the actual application scenario.

[0076] As an option, in some embodiments, to reduce the query load, paging query (Scroll API) or cursor query (Search After) can also be used to gradually load the retrieval results in the scenario of large-scale data return, improving the system stability.

[0077] The security person-vehicle multi-modal retrieval engine device described below can be correspondingly referred to the security person-vehicle multi-modal retrieval engine method described above.

[0078] Please refer to the appendix Figure 2 , the present invention also provides a security person-vehicle multi-modal retrieval engine device, including: A data acquisition module, which is used to acquire video image data in the security area and upload it to the management platform service module in real time; A management platform service module; which is used to receive the user's retrieval input information, process the registration and storage logic of the acquisition device, and undertake the preprocessing and storage logic of the acquired data; A data storage module; including: A Postgresql database, which is used to store the registration information of the acquisition device; Cloud object storage, which is used to store the acquired video and picture data; A vector database Milvus, which is used to store the picture feature vectors extracted by the large model feature extraction module; An Elasticsearch database, which is used to store picture features, picture cloud storage links, and picture source information; A large model feature extraction module; which is used to extract features from pictures and text information; A feature retrieval and matching module; which is used to receive the retrieval request forwarded by the management platform service module; An optimization and fault tolerance module; which is used to perform clock rollback detection and compensation during device registration and data acquisition, and adopts an improved timestamp adjustment method.

[0079] The device of this embodiment can be used to execute the above method embodiment, and its principle and technical effect are similar, so it will not be elaborated here.

[0080] An electronic device described below can be correspondingly referred to the multi-modal retrieval engine method for security of people and vehicles described above.

[0081] Please refer to the attached Figure 3 , The present invention also provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it can execute the above method.

[0082] The present invention also provides a storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the above method.

[0083] Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM for short), electrically erasable programmable read-only memory (EEPROM for short), erasable programmable read-only memory (EPROM for short), programmable read-only memory (PROM for short), read-only memory (ROM for short), magnetic memory, flash memory, magnetic disk or optical disk.

[0084] The embodiments of the specific implementation manners are all preferred embodiments of the present application, and do not limit the protection scope of the present application accordingly. The same components are denoted by the same reference numerals. Therefore, all equivalent changes made according to the structure, shape, and principle of the present application should be covered within the protection scope of the present application.

Claims

1. A multi-modal retrieval method for security of people and vehicles, characterized in that, Including the following steps: Arrange a variety of sensor devices within the security area, collect video and image data through the sensor devices, and save the sensor device information to the Elasticsearch database; Upload the collected videos and images to the cloud object storage, and generate corresponding feature IDs through an improved Snowflake algorithm; Use a large model to extract image text features, and store the extracted image text features and the generated feature IDs in the Milvus vector database; Receive the image or text request uploaded by the user, call the corresponding large model to extract image text features, and perform feature retrieval through the vector database to obtain the top N matching feature IDs; According to the feature IDs obtained from the feature retrieval, use the Elasticsearch database for further query, and combine the screening option information set by the user to return the most matching retrieval result.

2. The security vehicle and pedestrian multi-modal retrieval method according to claim 1, wherein The variety of sensor devices include high-definition cameras and infrared cameras. The variety of sensor devices collect images, videos, and sensor data through a data acquisition system, and upload and save the sensor device information through cloud storage. The sensor device information includes device ID, device type, device location, and device status, and stores this information in the Elasticsearch database.

3. The security vehicle and pedestrian multi-modal retrieval method according to claim 1, characterized in that The generation of the corresponding feature ID through the improved Snowflake algorithm includes the following steps: By comparing the current timestamp with the last generated timestamp, if the current timestamp is less than the last timestamp, adjust the timestamp to compensate for the rollback problem to avoid generating duplicate IDs; Adopt a distributed hash method to assign a unique machine ID to each device to avoid ID conflicts between different devices; Through the microsecond-level timestamp and sequence number generation mechanism, ensure that each generated ID is unique and conflict-free under high-concurrency requests; If clock rollback occurs, the new timestamp calculation formula is: ； wherein, is the current timestamp; is the timestamp used for generating the ID last time; is the preset time offset; is the compensated new timestamp.

4. The security vehicle and pedestrian multi-modal retrieval method according to claim 1, wherein The use of the large model for image text feature extraction includes the following steps: Extract features from the collected images through the large model, and convert the images into feature vectors in the vector space; Perform natural language processing on the text uploaded by the user through the large model, extract text features, and convert these features into feature vectors in the vector space; Use a cross-modal attention mechanism to optimize the semantic alignment between images and texts, ensuring that the image and text features have a small distance in the common feature space.

5. The security vehicle and pedestrian multi-modal retrieval method according to claim 1, wherein The storage of the extracted image text features and the generated feature IDs in the vector database includes the following steps: Store the image and text features extracted by the large model in the Milvus vector database to ensure the efficient storage of feature data and subsequent fast retrieval; Store information such as the feature ID, feature data corresponding to each image and text, and the cloud storage link of the image or text in the Elasticsearch database for quick access during retrieval; Through the efficient vector indexing algorithm of Milvus, quickly match the top N feature IDs with higher similarity.

6. The security vehicle and pedestrian multi-modal retrieval method according to claim 1, characterized in that The calling of the corresponding large model for image text feature extraction and feature retrieval through the vector database includes the following steps: Receive the images or text requests uploaded by users, and perform corresponding large model feature extraction according to the types of the requests; Connect to the Milvus vector database, retrieve based on the extracted feature vectors, and obtain the top N feature IDs most relevant to the user requests; Further screen and return the most matching retrieval results in the Elasticsearch database in combination with the screening options set by the user.

7. The security vehicle and pedestrian multi-modal retrieval method according to claim 3, wherein, The snowflake algorithm includes: Calculation of the timestamp and sequence number for generating the ID. If there are multiple requests for generating the ID, calculate through the following formula: ； Among them, represents a timestamp field in milliseconds; represents a device or node number; represents an incrementing serial number for generating an ID within the same millisecond; represents the generated ID feature; Optimization formula for cross-modal alignment. By calculating the distance between images and texts in the common space, use the following error function for optimization: ； Among them, represents the L2 norm; is the image feature vector; is the text feature vector.

8. A security vehicle and pedestrian multimodal retrieval device, applied to the security vehicle and pedestrian multimodal retrieval method according to any one of claims 1-7, characterized in that, Including: Data acquisition module, used to acquire video image data in the security area and upload it to the management platform service module in real time; Management platform service module; Responsible for receiving the user's retrieval input information, processing the registration and storage logic of the acquisition devices, and undertaking the preprocessing and storage logic of the acquired data; Data storage module; including: Postgresql database, used to store the registration information of the acquisition devices; Cloud object storage, used to store the acquired video and picture data; Vector database Milvus, used to store the picture feature vectors extracted by the large model feature extraction module; Elasticsearch database, used to store picture features, picture cloud storage links, and picture source information; Large model feature extraction module; used to extract features from picture and text information; Feature retrieval and matching module; used to receive the retrieval requests forwarded by the management platform service module; Optimization and fault tolerance module; used to perform clock rollback detection and compensation during device registration and data acquisition, and adopt an improved timestamp adjustment method.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the security vehicle and pedestrian multi-modal retrieval method according to any one of claims 1-7.

10. A storage medium, characterized in that, Store a computer program, and when the computer program is executed by the processor, it implements the security vehicle and pedestrian multi-modal retrieval method according to any one of claims 1-7.

Images