Security person and vehicle multi-modal retrieval method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of computer information retrieval, and discloses a security person-vehicle multi-modal retrieval method and device, an electronic equipment and a storage medium, the method comprising the following steps: arranging multiple sensor devices in a security area; uploading collected videos and images to a cloud object storage; using a large model to extract image text features; receiving an image or text request uploaded by a user, calling a corresponding large model to extract image text features and performing feature retrieval through a vector database; further querying by using an Elasticsearch database, and combining screening item information set by the user, the device comprises a data acquisition module, a management platform service module, a data storage module, a large model feature extraction module and a feature retrieval and matching module. By introducing a clock rollback detection and compensation mechanism, a machine ID unified configuration and a microsecond-level timestamp serial number generation, technical effects of avoiding ID conflicts and timing errors are effectively achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer information retrieval, in particular to a security person-vehicle multi-modal retrieval method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the modern security field, target recognition and behavior analysis are important technical tasks, especially in the context of "finding people" and "finding cars". With the continuous progress of artificial intelligence technology, retrieval engines based on images, texts and other multi-modal information are gradually applied to security monitoring systems, aiming to improve the intelligent level and work efficiency of the system.

[0003] Existing image processing technology mainly relies on deep learning models based on convolutional neural networks, which can detect and recognize objects. However, with the continuous change of demand, the accuracy of single image recognition technology in complex environments is gradually limited. In recent years, the introduction of cross-modal large models such as Chinese-clip has brought significant progress through joint feature learning of image and text information. These large models can usually perform semantic alignment between images and texts, and use text descriptions to guide the understanding of image content, thereby improving the accuracy and efficiency of multi-modal retrieval.

[0004] However, the existing Chinese-clip large model has limitations in network structure design and parameter optimization. The existing model has problems of large consumption of computing resources and long model training time when processing large-scale image and text data. The feature extraction of image and text information in the existing technology has not been completely decoupled and optimized, and there is a certain error in the cross-modal semantic alignment between images and texts in the traditional model, resulting in inaccurate retrieval results. In addition, since the snowflake algorithm relies on the combination of time stamp, machine ID and serial number to generate ID, it is easy to be affected by the clock rollback problem in a distributed environment, resulting in ID conflict, therefore, the present application provides a security person-vehicle multi-modal retrieval method, device, electronic equipment and storage medium to solve the deficiencies in the prior art. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application aims to provide a security person-vehicle multi-modal retrieval method, device, electronic equipment and storage medium, which solves the problems of low multi-modal retrieval accuracy, performance bottleneck and poor system scalability in the security field.

[0006] To achieve the above purpose, the present application is implemented by the following technical scheme: a security person-vehicle multi-modal retrieval method, comprising the following steps:

[0007] A plurality of sensor devices are arranged in a security area, video and image data are collected by the sensor devices, and sensor device information is saved to an Elasticsearch database;

[0008] The collected video and image are uploaded to a cloud object storage, and a corresponding feature ID is generated by an improved snowflake algorithm;

[0009] A large model is used to extract image text features, and the extracted image text features and the generated feature ID are stored in a Milvus vector database;

[0010] A user uploaded image or text request is received, a corresponding large model is called to extract image text features, and a vector database is used for feature retrieval to obtain the top N matching feature IDs;

[0011] According to the feature ID obtained by feature retrieval, an Elasticsearch database is further queried, combined with the user-set filtering option information, and the most matching search result is returned.

[0012] Preferably, the plurality of sensor devices include high-definition cameras and infrared cameras, the plurality of sensor devices collect image, video and sensor data through a data collection system, and save sensor device information through cloud storage upload, the sensor device information includes device ID, device type, device location and device state, and the information is stored in an Elasticsearch database.

[0013] Preferably, the generation of the corresponding feature ID by the improved snowflake algorithm includes the following steps:

[0014] By comparing the current timestamp with the last generated timestamp, if the current timestamp is less than the last timestamp, the timestamp is adjusted to compensate for the rollback problem, and to avoid generating duplicate IDs;

[0015] A distributed hash method is used to assign a unique machine ID to each device to avoid ID conflicts between different devices;

[0016] A microsecond timestamp and sequence number generation mechanism are used to ensure that each generated ID is unique and does not conflict under high concurrency requests;

[0017] If clock rollback occurs, the new timestamp calculation formula is:

[0018] ;

[0019] Where, is the current timestamp; is the timestamp used by the last generated ID; is a preset time offset; The new timestamp after compensation.

[0020] Preferably, the image text feature extraction using a large model includes the following steps:

[0021] Feature extraction of the collected image through a large model, converting the image into a feature vector in a vector space;

[0022] Natural language processing of the user-uploaded text through a large model, extracting text features and converting the features into feature vectors in a vector space;

[0023] Optimizing semantic alignment between images and text using cross-modal attention mechanism, ensuring that image and text features have small distances in the common feature space.

[0024] Preferably, the storing of the extracted image text features and generated feature IDs in the vector database includes the following steps:

[0025] Storing the image and text features extracted through a large model in the Milvus vector database, ensuring efficient storage and subsequent fast retrieval of feature data;

[0026] Storing the feature ID, feature data, and cloud storage link of each image and text, and other information in the Elasticsearch database for quick access during retrieval;

[0027] Through the efficient vector indexing algorithm of Milvus, quickly matching the top N feature IDs with higher similarity.

[0028] Preferably, the calling of corresponding large models for image text feature extraction and feature retrieval through the vector database includes the following steps:

[0029] Receiving user-uploaded image or text requests, and performing corresponding large model feature extraction according to the request type;

[0030] Connecting the Milvus vector database, retrieving based on the extracted feature vectors, and obtaining the top N feature IDs most relevant to the user request;

[0031] Further filtering in the Elasticsearch database combined with user-set filtering options and returning the most matching retrieval results.

[0032] Preferably, the Snowflake algorithm includes:

[0033] Timestamp and sequence number calculation of generated ID, if multiple requests generate ID, then calculate by the following formula:

[0034] ;

[0035] wherein, represents a timestamp field in milliseconds; represents a device or node number; represents a self-incrementing sequence number of the ID generated within the same millisecond; represents a generated ID feature.

[0036] The optimization formula of cross-modal alignment optimizes by calculating the distance of images and texts in the common space using the following error function:

[0037] ;

[0038] wherein, represents an L2 norm; is an image feature vector; is a text feature vector.

[0039] The security person-vehicle multi-modal retrieval device is also provided, comprising:

[0040] A data acquisition module is configured to acquire video image data in a security area and upload the data to a management platform service module in real time;

[0041] A management platform service module is configured to receive user search input information, process the registration and storage logic of the acquisition device, and undertake the preprocessing and storage logic of the collected data;

[0042] A data storage module comprises:

[0043] A Postgresql database is configured to store the registration information of the acquisition device;

[0044] A cloud object storage is configured to store the collected video and picture data;

[0045] A vector database Milvus is configured to store the picture feature vectors extracted by the large model feature extraction module;

[0046] An Elasticsearch database is configured to store the picture features, picture cloud storage links, and picture source information;

[0047] A large model feature extraction module is configured to extract features from pictures and text information;

[0048] A feature search and matching module is configured to receive search requests forwarded by the management platform service module;

[0049] An optimization and fault tolerance module is configured to detect and compensate clock rollback during device registration and data acquisition, and adopt an improved timestamp adjustment method.

[0050] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the security person-vehicle multi-modal retrieval method can be realized.

[0051] The application further provides a storage medium storing a computer program, and when the processor executes the computer program, the security person-vehicle multi-modal retrieval method can be realized.

[0052] In summary, the present application includes at least one of the following beneficial technical effects:

[0053] 1. The improved snowflake algorithm is used to generate a unique feature ID, which effectively avoids ID conflicts and timing errors by introducing a clock rollback detection and compensation mechanism, a unified machine ID configuration, and a microsecond-level timestamp serial number generation. Compared with the prior art that directly relies on the traditional snowflake algorithm, the scheme is prone to ID duplication and data consistency problems in a multi-node high-concurrency environment, which solves the problems of limited system scalability and insufficient stability.

[0054] 2. The application applies a large-scale visual-language model to extract image and text features, and introduces a cross-modal attention mechanism to align the image and text semantic spaces, thereby achieving consistent image-text semantics and significantly improving retrieval accuracy. Unlike traditional methods based on shallow features or simple keyword matching, the present application effectively overcomes the problem of deep semantic mismatch between images and text and irrelevant retrieval results.

[0055] 3. The present application successfully realizes fast retrieval and supports multi-dimensional filtering through a two-level retrieval architecture that first uses a vector database for efficient preliminary retrieval and then combines Elasticsearch database for screening optimization. Compared with the prior art that is simply based on vector approximate retrieval or single full-text index retrieval, the present application significantly improves the problems of low recall rate and poor precision in large-scale data retrieval.

[0056] 4. The user retrieval request of the present application supports multiple filtering options, including time period, location, and device ID category, and is combined with feature similarity score and screening matching score weighting fusion, which overall achieves accurate retrieval results and reasonable sorting. The existing scheme usually only filters in a single dimension, which is prone to result confusion or weak correlation. The present application effectively makes up for such deficiencies and significantly improves the retrieval experience in practical applications. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a method step flowchart of the present application;

[0058] Figure 2is a device architecture diagram of the present application;

[0059] Figure 3 is an electronic device schematic diagram of the present application. DETAILED DESCRIPTION

[0060] The following description will be made in conjunction with the accompanying drawings Figure 1 - the accompanying drawings Figure 3 The present application will be further described in detail.

[0061] Please refer to the accompanying drawings Figure 1 The embodiment of the present application provides a security and protection person-vehicle multi-modal retrieval method, which comprises the following steps:

[0062] S1, arranging a plurality of sensor devices in a security and protection area, collecting video and image data through the sensor devices, and saving sensor device information to an Elasticsearch database;

[0063] S2, uploading the collected video and image to a cloud object storage, and generating a corresponding feature ID through an improved snowflake algorithm;

[0064] S3, using a large model to extract image text features, and storing the extracted image text features and the generated feature ID in a Milvus vector database;

[0065] S4, receiving a user uploaded image or text request, calling a corresponding large model to extract image text features, and performing feature retrieval through the vector database to obtain the top N matched feature IDs;

[0066] S5, using the Elasticsearch database to further query according to the feature ID obtained through the feature retrieval, combining the user set filtering item information, and returning the most matched retrieval result.

[0067] For step S1, in the embodiment, in order to realize efficient and multi-modal retrieval of people and vehicles in the security and protection area, a perfect basic data collection and device information management system needs to be established first, so as to ensure that the subsequent feature extraction and retrieval processing process has accurate and rich input data sources. A plurality of sensor devices are deployed in the security and protection area, and the sensor devices at least include a high-definition camera and an infrared camera. Generally, the high-definition camera is mainly used to collect high-resolution image and video data in daylight and sufficient light environment, while the infrared camera is suitable for target detection and tracking in night or low-illumination environment.

[0068] In one possible implementation, the sensor device transmits data to the central management platform through a wired network or a wireless communication network (such as 5G, Wi-Fi, LoRa, etc.), realizing real-time or quasi-real-time data collection and reporting. As an alternative, some sensors can be pre-installed with edge computing units to pre-process raw video streams, such as target detection and motion analysis, to reduce transmission bandwidth occupancy.

[0069] Specifically, each sensor device needs to collect and record its basic information when registering, including but not limited to:

[0070] Device ID (Device ID): used to uniquely identify each sensor device, generally a globally unique identifier (UUID) assigned by the system;

[0071] Device Type (Device Type): indicates the category of the sensor, such as high-definition camera, infrared camera, thermal imager, etc.;

[0072] Device Location (Device Location): used to describe the physical deployment location of the device, which can use latitude and longitude coordinates, address description, or building / floor information;

[0073] Device Status (Device Status): records device status information, such as online, offline, fault, maintenance, etc.

[0074] In some embodiments, device registration information can be further extended to include installation time, camera orientation angle (such as pitch angle, yaw angle), coverage area description, and management unit number, etc., to facilitate subsequent device status management and data traceability.

[0075] Generally, all collected device basic information and real-time sensor status are standardized by the management platform and stored in the Elasticsearch database. Elasticsearch database, as a distributed high-performance retrieval engine, can support fast retrieval, update and query of massive device information, effectively supporting subsequent complex query requirements such as device classification and geographic area filtering.

[0076] In one implementation, the management platform encapsulates device information in standard JSON format after receiving device data.

[0077] In this embodiment, to further ensure the reliability of the system, a data signature verification field can be attached when uploading device information, such as a signature value generated based on SHA256 or HMAC algorithm, to prevent data tampering risks.

[0078] Alternatively, to improve system scalability, a large number of sensor devices can be accessed at one time through a batch registration interface during device deployment, and differentiated information item templates are defined for different types of devices. For example, an infrared camera can have a heat sensitivity level field added, and a high-definition camera can have a maximum resolution, frame rate, and other parameters added.

[0079] In another possible implementation, to facilitate subsequent feature extraction and matching, device registration information can also record device acquisition capability parameters, such as image width and height, supported compression encoding formats (such as H.264, H.265), acquisition frequency, and other basic data. These data can be used as auxiliary filtering conditions during subsequent search optimization.

[0080] For step S2, in this embodiment, after the data acquisition of the sensor device and the storage of the device information are completed, to realize standardized management of image and video data and orderly development of subsequent feature extraction work, the collected image and video files need to be uniformly stored and processed, and corresponding feature identification IDs are generated synchronously. Based on this, an improved scheme of the snowflake algorithm is used to ensure that the generated feature IDs have uniqueness, time sequence, and fault resistance in a high-concurrency environment, laying a foundation for subsequent multi-modal feature extraction and search.

[0081] In this embodiment, the collected video stream and snapshot images are first standardized (such as encoding conversion and format unification) and then uploaded to a cloud object storage system. Generally, an interface based on HTTPS or other encryption communication protocols is used for uploading to ensure the security and integrity of the data transmission process. Alternatively, a breakpoint resume mechanism can be enabled for some high-frequency acquisition devices to address the risk of interruption caused by network fluctuations.

[0082] Specifically, after uploading is completed, the management platform system generates a unique corresponding feature ID for each image or video data. To ensure that the feature ID has uniqueness in a large-scale distributed system and avoid ID conflict problems caused by system clock abnormalities, an improved snowflake algorithm is used to generate the feature ID in this embodiment.

[0083] In one possible implementation, the improved snowflake algorithm specifically includes the following technical features:

[0084] First, the system compares the current timestamp with the timestamp used for generating the last ID . If it is detected that the current timestamp is less than , that is, there is a system clock rollback, a timestamp compensation mechanism is triggered. The new timestamp after compensation is calculated according to the following formula:

[0085] ;

[0086] wherein, is a preset time offset, usually set as the minimum tolerable unit (e.g. 1 millisecond).

[0087] In general cases, when the system time does not occur backtracking, the current timestamp is directly used for ID generation without additional adjustment. Alternatively, it can be dynamically set to adapt to the abnormal time drift tolerance in different system environments.

[0088] Secondly, the specific generation of the ID is combined by bit operation according to the following formula:

[0089]

[0090] wherein, represents the timestamp field in milliseconds, occupying 41 bits, indicating the ID generation time; represents the device or node number, occupying 10 bits, and the uniqueness is ensured by distributed hash or registration center allocation; represents the self-incrementing sequence number of the ID generated in the same millisecond, occupying 12 bits, and when the sequence number reaches the maximum value, it needs to wait for the next millisecond to generate.

[0091] In some embodiments, to further improve the robustness of the system, the allocation of machine ID can be managed by distributed coordination components such as Zookeeper, Etcd, etc., to ensure that there is no machine ID conflict problem between different nodes.

[0092] In a specific embodiment, when the sequence number overflows and the current millisecond does not change, the system can choose to spin or sleep until the next millisecond to ensure the timing and uniqueness of the ID.

[0093] It should be noted that the improved snowflake algorithm can effectively avoid the ID duplication problem caused by system clock backtracking, device ID conflict, high concurrency request and other factors, thereby ensuring that each data file uploaded to the cloud storage has a unique and traceable feature identifier.

[0094] In general cases, the generated will be recorded in the object storage system in the form of metadata, or saved synchronously to the management platform database, so as to be associated and matched with the image text features in the future.

[0095] Alternatively, to improve the stability of ID generation in abnormal system conditions, a redundant check code (such as CRC32 check value) can be attached when generating the ID, which is used to quickly verify the correctness of the ID generation process.

[0096] In another possible implementation, if there are multiple sub-modules (such as a video module and an image module) in the data collection end, an independent sub-section can be set for each module to quickly identify the data source module during subsequent analysis.

[0097] For step S3, after the standardized storage of image and video data and the generation of unique feature IDs, deep semantic feature extraction processing of the uploaded data is needed to achieve efficient retrieval based on feature vectors. By applying advanced large-scale pre-training models to extract deep features of images and text, and using cross-modal attention mechanisms to optimize feature alignment, the retrieval accuracy and overall system performance are further improved. This technical solution combines the Chinese-CLIP model application suggestion and cross-modal alignment optimization method proposed by the client, ensuring the semantic richness and consistency of the feature extraction results,

[0098] In this embodiment, for the collected image data, a large-scale visual language pre-training model (such as Chinese-CLIP) is used for feature extraction processing. Generally, the image first undergoes standard preprocessing procedures, including size scaling, normalization processing (such as scaling the pixel value to the [0, 1] interval), and color space unification (such as unification to the RGB space), to ensure the consistency of the input in the subsequent feature extraction process.

[0099] As an option, different preprocessing strategies can be used for data from different devices (such as high-definition camera images and infrared camera images) in some embodiments to further improve the robustness of feature extraction.

[0100] Specifically, after preprocessing, the image is input into the visual encoder part of the Chinese-CLIP model (such as the ViT-B / 16 structure), and the deep semantic features of the image are extracted through multiple layers of Transformer encoder, outputting a fixed-dimensional image feature vector, denoted as: ; where is a vector of dimension D, generally taking 512, 768, or higher, depending on the model size.

[0101] In one possible implementation, for the text description information uploaded by the user, the system also uses the text encoder in the same Chinese-CLIP model for feature extraction. The text is first processed by word segmentation and stop word removal, and then encoded into a fixed-length input vector. Then, the Transformer encoder outputs the corresponding text feature vector, denoted as: .

[0102] ​In some embodiments, if the text content is a structured search request (for example, "male in black clothes smoking and making a phone call on People Road from 21:00 to 22:00 on March 24, 2024"), a large language model such as GPT can be used for preliminary structured processing to extract key information such as location, time, and action, and then perform feature vectorization encoding.

[0103] Generally, to achieve semantic consistency of images and text in the vector space, the present embodiment introduces a cross-modal attention mechanism. Specifically, by constructing a bidirectional attention weight matrix, a fine-grained correspondence is established between image region features and text Token features to enhance the deep semantic alignment capability of vision and language.

[0104] In an implementation manner, to achieve cross-modal feature alignment optimization, the error loss function is defined as follows:

[0105] ;

[0106] wherein, represents the L2 norm (i.e., the square of the Euclidean distance). The optimization objective aims to minimize the distance between image feature vectors and text feature vectors in the common semantic space, thereby improving the image-text matching accuracy.

[0107] As an option, to further optimize the distribution of the feature space, a contrast learning loss (such as InfoNCE loss) can also be introduced in some embodiments to expand the distance difference between positive sample pairs and negative sample pairs.

[0108] After feature extraction, the extracted image feature vectors and text feature vectors are associated and bound with the corresponding generated , and are uniformly stored in the Milvus vector database.

[0109] In a possible implementation manner, to improve the vector retrieval performance, the Milvus database uses index algorithms based on HNSW (Hierarchical Navigable Small World) or IVF+PQ (Inverted File with Product Quantization) to construct feature indexes, so that the response time can still be maintained at milliseconds level in the retrieval of millions of feature vectors.

[0110] It should be noted that when storing features, the system also records the cloud object storage link (Storage URL) of the image or text data, the feature generation timestamp, the feature source device ID, and other auxiliary information, which facilitates quick association with the original data during subsequent retrieval.

[0111] As for step S4, in this embodiment, after completing image text feature extraction and vector storage, in order to realize the multi-modal retrieval demand based on user input, the user request is received, feature extraction, vector retrieval and preliminary feature matching are carried out, so as to ensure that the system can accurately locate the image or text data closest to the user input while responding efficiently. In the design of this embodiment, the balance between retrieval performance and retrieval accuracy is fully considered, and the multi-stage retrieval strategy based on vector priority retrieval and screening item auxiliary optimization proposed by the customer is combined.

[0112] In this embodiment, the user uploads the image to be retrieved or inputs the text description through the front-end interface, and the system receives the request and analyzes the request type. Generally, the image request directly enters the image feature extraction process, and the text request enters the text feature extraction process, respectively calling the pre-deployed large model for processing.

[0113] As an option, to adapt to different business needs, the system design can also support audio retrieval requests by first transcribing the audio data into text and then processing it according to the text retrieval method.

[0114] Specifically, for image requests, this embodiment uses a large model (such as Chinese-CLIP visual encoder) to extract features from the uploaded image. The extracted image feature vector is denoted as: ;

[0115] wherein, is a D-dimensional feature vector, which is consistent in dimension with the image feature vectors stored in the library.

[0116] For text requests, this embodiment extracts features from the input text using a Chinese-CLIP text encoder to obtain a text feature vector, denoted as: .

[0117] In one possible implementation, to improve retrieval accuracy, natural language preprocessing modules can be used to normalize the content before text feature extraction, such as synonym normalization, time and place entity extraction, etc.

[0118] After completing the query feature extraction, this embodiment connects to the Milvus vector database through the pymilvus client and performs feature retrieval based on similarity calculation. Generally, inner product (Inner Product) or cosine similarity (Cosine Similarity) is used as an indicator for similarity calculation. Taking cosine similarity as an example, the similarity formula is as follows:

[0119] ;

[0120] wherein, To query the feature vector, Store the feature vector in the database, The L2 norm of the vector.

[0121] At the time of retrieval, the system returns the top Similar feature records from Milvus to the user, and obtains the corresponding feature ID set, denoted as:

[0122] ;

[0123] Where, The system sets or the user can select the upper limit of the return quantity, such as 10, 50, or 100.

[0124] As an option, to further improve the recall rate of the preliminary retrieval, a similarity threshold (such as 0.7) can be set in some embodiments, and only the feature ID results with a similarity higher than the threshold are retained.

[0125] In a specific implementation, the feature clustering mechanism (such as K-Means or HNSW subgraph division) can be enabled in parallel during the query process to speed up the feature retrieval and reduce irrelevant feature disturbances.

[0126] Generally, after completing the Milvus vector retrieval, this embodiment does not directly return the final retrieval result, but instead passes the obtained feature ID set to the subsequent step for further multi-dimensional comprehensive screening through Elasticsearch combined with filter information, to improve the retrieval accuracy and user experience.

[0127] It should be noted that the retrieval and matching between the query feature vector extracted in this step and the feature vectors in the vector library is completely based on the consistency of the feature space in the early stage, ensuring that the image and the text can be directly calculated for similarity in the common vector space, avoiding the matching error problem caused by feature heterogeneity in traditional retrieval.

[0128] For step S5, in this embodiment, after completing the preliminary feature matching based on the Milvus vector database, a detailed retrieval and filtering mechanism of the Elasticsearch database combined with the feature ID is introduced to further improve the accuracy and multi-dimensional relevance of the retrieval result. By comprehensively using vector retrieval and multi-condition structured retrieval, this step effectively compensates for the limitations of pure vector matching in complex retrieval requirements, and realizes an efficient transition from coarse retrieval to fine retrieval.

[0129] ​Generally, the system uses the ID set as a query condition to initiate a Batch Query to the Elasticsearch database. In an embodiment, the retrieval request adopts a Bool Query structure and combines the multifunctional filtering conditions set by the user for comprehensive filtering.

[0130] Specifically, the user can set the following filtering options when submitting a retrieval request:

[0131] retrieval category (e.g., person, vehicle, other target);

[0132] retrieval time period (start_date, end_date, start_time, end_time);

[0133] snap device ID or device type;

[0134] location range or keyword;

[0135] specific attribute feature (e.g., clothing color, behavior action).

[0136] Generally, to improve query performance, the embodiment pre-establishes inverted indexes for feature ID, category, time, device ID, etc. fields in Elasticsearch, and configures reasonable sharding and replication strategies according to the data volume to ensure low latency response under high concurrency retrieval.

[0137] As an option, the system also supports a weight-based weighted query mechanism, i.e., setting different matching priority for different filtering options. For example, the time matching score weight can be higher than the device matching score weight, so as to be closer to the user's focus point in retrieval sorting.

[0138] In an embodiment, if the retrieval request contains structured text filtering (e.g., location keyword, action description, etc.), the Elasticsearch full-text search engine feature can be combined to perform fuzzy matching and relevance scoring through query or based on the BM25 scoring model.

[0139] To further optimize retrieval accuracy, the embodiment also introduces a comprehensive sorting formula (comprehensive score formula), which is as follows:

[0140] ;

[0141] wherein, is the final sorting score; is the similarity score in the vector retrieval stage; is the filtering matching score (e.g., category matching, time matching score); , The weight coefficient set for the system satisfies .

[0142] In general, It can be 0.7, It is 0.3, but it can also be adjusted flexibly according to the actual application scene.

[0143] As an option, in some embodiments, to reduce query load, a paging query (ScrollAPI) or a cursor query (Search After) can also be used to load the search results step by step in a large-scale data return scenario, improving system stability.

[0144] The security person-vehicle multi-modal retrieval engine device described below can be mutually corresponding to the security person-vehicle multi-modal retrieval engine method described above.

[0145] Please refer to the attached Figure 2 , the present application also provides a security person-vehicle multi-modal retrieval engine device, comprising:

[0146] The data acquisition module is used for acquiring video image data in the security area and uploading to the management platform service module in real time;

[0147] The management platform service module is used for receiving user search input information, processing the registration and storage logic of the acquisition device, and undertaking the preprocessing and storage logic of the collected data;

[0148] The data storage module comprises:

[0149] The Postgresql database is used for storing the registration information of the acquisition device;

[0150] The cloud object storage is used for storing the collected video and picture data;

[0151] The vector database Milvus is used for storing the picture feature vectors extracted by the large model feature extraction module;

[0152] The Elasticsearch database is used for storing the picture features, picture cloud storage links and picture source information;

[0153] The large model feature extraction module is used for feature extraction of picture and text information;

[0154] The feature retrieval and matching module is used for receiving the search request forwarded by the management platform service module;

[0155] The optimization and fault tolerance module is used for clock rollback detection and compensation in the device registration and data acquisition process, and an improved timestamp adjustment method is adopted.

[0156] The device can be used to execute the method embodiments described above, and has similar principles and technical effects, which will not be repeated here.

[0157] An electronic device described below can be referred to the security and protection person and vehicle multi-modal retrieval engine method described above.

[0158] Please refer to the accompanying Figure 3 The application also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the method described above can be executed.

[0159] The application also provides a storage medium, which stores a computer program, and when the computer program is executed by a processor, the method described above can be executed.

[0160] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0161] The embodiments of the specific implementation are the preferred embodiments of the application, and are not intended to limit the protection scope of the application, wherein the same parts are indicated by the same reference numerals. Therefore, any equivalent changes made according to the structure, shape, principle of the application should be covered within the protection scope of the application.

Claims

1. A multimodal retrieval method for security personnel and vehicles, characterized by: The following steps are involved: Deploy various sensor devices within the security area, collect video and image data through the sensor devices, and save the sensor device information to the Elasticsearch database; Upload the collected videos and images to the cloud object storage and generate corresponding feature IDs using the improved snowflake algorithm; Use the large model to extract image text features, and store the extracted image text features and generated feature IDs in the Milvus vector database; Receive image or text requests uploaded by users, call the corresponding large model to extract image and text features, and perform feature retrieval through the vector database to obtain the feature IDs of the top N matches; Based on the feature ID obtained from the feature retrieval, the Elasticsearch database is used for further query, and the best matching search results are returned based on the filter information set by the user. The multiple sensor devices include high-definition cameras and infrared cameras. The multiple sensor devices collect images, videos, and sensor data through a data acquisition system, and upload and save sensor device information through cloud storage. The sensor device information includes device ID, device type, device location, and device status, and the information is stored in an Elasticsearch database; The generation of the corresponding feature ID by the improved snowflake algorithm includes the following steps: By comparing the current timestamp with the last generated timestamp, if the current timestamp is less than the last timestamp, the timestamp is adjusted to compensate for the callback problem and avoid generating duplicate IDs; Use distributed hashing to assign a unique machine ID to each device to avoid ID conflicts between different devices; Through the microsecond timestamp and serial number generation mechanism, each generated ID is guaranteed to be unique and non-conflicting under high concurrency requests. If the clock is rolled back, the new timestamp calculation formula is: new timestamp =max(current timestamp ,last timestamp )+time offset ; Among them, current timestamp is the current timestamp; last timestamp The timestamp used for the last ID generation; time offset For the preset time offset; new timestamp The new timestamp after compensation.

2. The multimodal retrieval method for security personnel and vehicles according to claim 1 is characterized in that: The image text feature extraction using the large model includes the following steps: The collected image is subjected to feature extraction through a large model, and the image is converted into a feature vector in a vector space; The large model performs natural language processing on the text uploaded by the user, extracts text features, and converts the features into feature vectors in the vector space; A cross-modal attention mechanism is used to optimize the semantic alignment between images and text, ensuring that image and text features have a small distance in a common feature space.

3. The multimodal retrieval method for security personnel and vehicles according to claim 1, characterized in that: The step of storing the extracted image text features and the generated feature ID in the vector database comprises the following steps: The image and text features extracted by the large model are stored in the Milvus vector database to ensure efficient storage and subsequent fast retrieval of feature data; The feature ID, feature data, and cloud storage link information corresponding to each image and text are stored in the Elasticsearch database for fast access during retrieval. Through Milvus's efficient vector indexing algorithm, the top N feature IDs with high similarity are quickly matched.

4. The multimodal retrieval method for security personnel and vehicles according to claim 1, characterized in that: The calling of the corresponding large model to extract image text features and performing feature retrieval through the vector database includes the following steps: Receive image or text requests uploaded by users and perform large model feature extraction based on the type of request; By connecting to the Milvus vector database, we search based on the extracted feature vectors to obtain the top N feature IDs most relevant to the user request. Combined with the filter items set by the user, further filter in the Elasticsearch database and return the most matching search results.

5. The multimodal retrieval method for security personnel and vehicles according to claim 1 is characterized in that: The snowflake algorithm includes: the timestamp and serial number calculation of the generated ID. If there are multiple requests to generate IDs, the calculation is performed using the following formula: unique ID =timestamp<<22+machine ID <<12+sequence number ; Among them, timestamp represents the timestamp field in milliseconds; machine ID Indicates the device or node number; sequence number Indicates the auto-increment serial number of the ID generated within the same millisecond; unique ID Indicates the generated ID feature; The optimization formula for cross-modal alignment calculates the distance between image and text in the common space and optimizes using the following error function: error function =||image embedding -text embedding || 2 ; Among them, ||·|| 2 represents the L2 norm; image embedding is the image feature vector; text embedding is the text feature vector.

6. A multimodal search device for security personnel and vehicles, applied to the multimodal search method for security personnel and vehicles according to any one of claims 1 to 5, characterized in that: include: Data acquisition module, used to collect video image data within the security area and upload it to the management platform service module in real time; Management platform service module; Responsible for receiving user search input information, processing the registration and storage logic of the collection equipment, and undertaking the pre-processing and storage logic of the collected data; Data storage module; including: Postgresql database, used to store the registration information of the collection equipment; Cloud object storage, used to store collected video and image data; The vector database Milvus is used to store the image feature vectors extracted by the large model feature extraction module; Elasticsearch database, used to store image features, image cloud storage links, and image source information; Large model feature extraction module; used to extract features from images and text information; Feature retrieval and matching module; used to receive retrieval requests forwarded by the management platform service module; Optimization and fault tolerance module; used for clock dialback detection and compensation during device registration and data collection, using an improved timestamp adjustment method.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the multimodal retrieval method for security personnel and vehicles as described in any one of claims 1 to 5.

8. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the multimodal retrieval method for human and vehicle security as described in any one of claims 1 to 5 is implemented.

Images