Cross-modal retrieval method fusing human body attribute features and vector embedding

Through a cross-modal search method that integrates human attribute features and vector embedding, the search difficulty of monitoring data under different conditions is solved, efficient and accurate monitoring data retrieval is achieved, and the stability and user experience of the system are improved.

CN119829799APending Publication Date: 2025-04-15SHANGHAI WONDERTEK SOFTWARE CORP LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411916565.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-12-24
Publication Date
2025-04-15

AI Technical Summary

Technical Problem

The processing and retrieval of monitoring data becomes difficult in large-scale monitoring networks, especially under different formats, resolutions, perspectives and environmental conditions. Traditional methods are prone to missed detection, missed detection and manual fatigue, affecting the accuracy and efficiency of retrieval.

Method used

A cross-modal search method that integrates human attribute features and vector embedding is adopted to extract face and human feature vectors through deep learning models, and combine human structured attribute features to realize image retrieval and cross-module text retrieval, improving matching accuracy and recognition capabilities.

Benefits of technology

It improves the accuracy and efficiency of surveillance data retrieval, and can quickly and accurately retrieve personnel information in the video under multiple cameras, reduce false detection and missed detection, and improve the stability and user experience of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119829799A_ABST
    Figure CN119829799A_ABST
Patent Text Reader

Abstract

The invention discloses a cross-modal retrieval method fusing human body attribute features and vector embedding. The method comprises the following steps: regularly extracting pictures from a monitoring video stream, and coding and caching; extracting a face feature vector, a human body feature vector, a text vector and a human body structured attribute feature of the picture; and associating the face feature vector with the human body feature vector to realize face fusion, and storing the face fusion in a vector library. And associating the human body structured attribute features with the vector data, and storing the human body structured attribute features and the vector data in a search engine. According to the method, feature extraction and vectorization are carried out on an input image or character, vector data are matched with data in a vector library, then filtering is carried out through a search engine according to human body structured attribute features, and a search result is obtained. According to the method, the human face and human body fusion technology is introduced, the human body attribute features are structured, and the deep learning model is combined, so that the personnel information in the video can be quickly and accurately retrieved under multiple cameras, and the video in the corresponding time period can be efficiently and stably generated for playback.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of monitoring data processing, and in particular relates to a cross-modal retrieval method integrating human attribute features with vector embedding. Background Art

[0002] With the continuous development and popularization of technology, cameras have become an indispensable part of our lives. From public places to private areas, the number of cameras has increased exponentially, which has provided unprecedented convenience for monitoring and security. However, this large-scale monitoring network has also brought a severe challenge: the processing and retrieval of monitoring data has become extremely difficult.

[0003] Since surveillance data may have different formats, resolutions, viewing angles, and environmental conditions, this poses a greater challenge to data processing and matching. For example, if you want to find a person's video playback from a video, the traditional data processing method may use manual playback, which requires manual search frame by frame or segment by segment, which requires a lot of time and human resources. Especially in large-scale video surveillance systems, it may take a long time to find the target person. In addition, humans are prone to fatigue during long video playback, and it is difficult to concentrate. They may miss the appearance of the target person or important details, which may lead to missing important information or misidentifying the target person. At the same time, people's subjective judgment may be affected by factors such as emotions and fatigue, resulting in instability in the search results.

[0004] In order to solve the above problems, the existing technology adopts the video retrieval method based on similarity measurement, but there are still problems of false detection and missed detection. Especially in complex scenes, such as lighting changes, occlusion, motion blur, etc., the accuracy of retrieval may be affected. In some cases, video retrieval requires cross-modal matching, that is, matching text queries with video content. This may face problems such as modality mismatch and semantic gap, leading to technical problems of poor retrieval effect. Summary of the invention

[0005] The technical purpose of the present invention is to provide a cross-modal retrieval method that integrates human attribute features and vector embedding to solve the problem of poor monitoring data retrieval effect.

[0006] To solve the above problems, the technical solution of the present invention is: A cross-modal retrieval method integrating human attribute features and vector embedding includes the following steps: Continuously monitor the video stream, extract pictures regularly, and cache the extracted picture codes in the message queue; Use the deep learning model to extract information from cached images, including face feature vectors, body feature vectors, and convert images into corresponding text vectors. Use the logistic regression model to obtain structural attribute features of the human body. By detecting the position of the human body coordinate points in the image, the facial feature vector and the human body feature vector are correlated with each other, and combined with their position information in the image, face fusion is achieved and stored in the vector library; Using the coordinate points of the human body in the image, the structured attribute features of the human body are associated with the vector data, and the structured data of the human body attributes are stored in the search engine; A corresponding search method is selected according to the input target to be matched, including image retrieval and cross-module text retrieval; for image retrieval, a picture vector is obtained by extracting features and vectorizing the input image, and the picture vector is matched with the data in the vector library, and then the search engine is used to filter according to the structured attribute characteristics of the human body to obtain the search results; for cross-module text retrieval, a text vector is obtained by extracting features and vectorizing the input text, and the text vector is matched with the data in the vector library, and then the search engine is used to filter according to the structured attribute characteristics of the human body to obtain the search results.

[0007] In the process of extracting pictures, each extracted picture is Base64 encoded, and then the encoded picture is cached in the message queue.

[0008] Specifically, in the step of extracting information from the cached image, the extracted facial feature vector is also normalized, and the calculation formula is as follows: Face feature vector: , where the subscript of v indicates the order of the element in the vector; Calculate the L2 norm: ; Calculate the normalization factor: ; Compute the normalized vector: ; The facial feature vector Multiply each element in by the normalization factor to get a normalized vector.

[0009] Among them, the structural attribute characteristics of the human body include clothing style, clothing color and character gender.

[0010] Among them, image retrieval specifically includes: Based on the video screenshots or pictures selected by the user, a deep learning model is used to extract human body vectors and face vectors; The extracted face vector is subjected to cosine similarity calculation with the face feature vector in the vector library. After the calculation is completed, the face feature vectors are sorted according to the similarity; The matched facial feature vector is sent to the human body deduplication model for deduplication processing; Calculate the cosine similarity between the extracted human body vector and the human body feature vector in the vector library, and sort the human body feature vectors after deduplication; The matched human feature vector is associated with the corresponding face feature vector, and the cosine similarity is calculated with the face vector of the input video screenshot or picture, and the matching results are sorted; The human coordinates in the query results are associated with the human coordinates stored in the search engine library by intersection and comparison calculation, and compared and filtered in combination with the structural attribute features of the human body in the search engine library.

[0011] Among them, cross-module text search specifically includes: Based on the text information input by the user, a deep learning model is used to extract the vector features of the text; Compare the extracted vector features with the text vectors in the vector library and sort the results using cosine similarity calculation; The structured data selected by the user is then fused with the structured attribute features of the human body in the search engine library, and the fused features are matched and sorted using the intersection sorting method to obtain the results that best match the user's needs.

[0012] Further preferably, after the retrieval is completed, the search results will be displayed in the form of thumbnails and sorted by dots, displaying the information of the video pictures, including the video location, video time, body clothing style, body clothing color and gender; by clicking on the marked dotted area, the video can be reviewed and the clips before and after the corresponding time point can be played.

[0013] An electronic device includes a processor and a memory, the memory stores computer instructions that can be executed on the processor, and the processor is used to call the computer instructions in the memory to execute the cross-modal retrieval method that integrates human attribute features and vector embedding as described above.

[0014] A storage medium is used to store computer instructions, wherein when the computer instructions are executed, the device where the storage medium is located is controlled to execute the cross-modal retrieval method that integrates human attribute features and vector embedding as described above.

[0015] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art: By correlating the features of the human face and the human body, the system can more accurately identify and match the target, thereby improving the overall matching accuracy. The present invention also introduces the integrated learning of the structured attributes of the human body and vector embedding, which can better capture the diversity and complexity of human targets, improve the recognition ability of human targets in different scenes and environments, and perform secondary retrieval and filtering based on attributes such as human clothing color, style and gender, thereby improving the matching accuracy.

[0016] Therefore, the present invention introduces face and body fusion technology and structured human attribute features, combined with a deep learning model, can quickly and accurately retrieve personal information in the video under multiple cameras, thereby efficiently and stably generating video playback of the corresponding time period. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0018] Figure 1 A simplified flowchart of a cross-modal retrieval method integrating human attribute features and vector embedding according to the present invention; Figure 2 The present invention is a flow chart of a cross-modal retrieval method integrating human attribute features and vector embedding. DETAILED DESCRIPTION

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0020] In order to simplify the drawings, only the parts related to the present invention are schematically shown in each figure, and they do not represent the actual structure of the product. In addition, in order to simplify the drawings and facilitate understanding, in some figures, only one of the parts with the same structure or function is schematically drawn or marked. In this article, "one" not only means "only one", but also means "more than one".

[0021] The following is a further detailed description of a cross-modal retrieval method integrating human attribute features and vector embedding proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description and claims.

[0022] See also Figure 1 and Figure 2 This embodiment provides a cross-modal retrieval method integrating human attribute features and vector embedding, which may specifically include the following steps: First, the video stream is continuously monitored and pictures are periodically extracted, that is, the video is intercepted to obtain multiple images. During the picture extraction process, each extracted picture is Base64 encoded and then cached in the message queue.

[0023] Then, a deep learning model is used to extract information from the cached image, including facial feature vectors, human feature vectors, and converting images into corresponding text vectors. The above deep learning model mainly uses the CLIP-ReID model, in which the backbone model is replaced with dinov2, so that the model pays more attention to the area on the human body, and the loss function refers to the loss data used by the image retrieval model, paying more attention to the relevance of details, such as shoe textures, patterns on T-shirts, etc. For the facial feature vector, it is further normalized, and the calculation formula is as follows: Face feature vector: , where the subscript of v indicates the order of the element in the vector.

[0024] Calculate the L2 norm: .

[0025] Calculate the normalization factor: .

[0026] Compute the normalized vector: .

[0027] Among them, the face feature vector Multiplying each element in by the normalization factor gives us the normalized vector we need.

[0028] Preferably, in this step, a logistic regression model is also used to obtain structural attribute characteristics of the human body, and the structural attribute characteristics of the human body include clothing style, clothing color, and character gender.

[0029] Next, by detecting the position of the human body coordinate points in the image, the facial feature vector is correlated with the human body feature vector, and combined with their position information in the image to achieve face fusion. At the same time, the relevant information is stored in the vector library for subsequent retrieval and analysis.

[0030] Then, the human body coordinate points in the image are used to associate the structured attribute features of the human body with the vector data. Through this association, the structured data of the human body attributes can be stored in the elasticsearch search engine, thereby realizing the fusion of human body attribute features. This fusion method can improve the understanding and analysis capabilities of subsequent retrieval of human body attributes, making it possible to more accurately identify and extract the characteristic information of the human body. At this point, the preparation steps before retrieval are completed.

[0031] Then, enter the search step. The search can execute different search strategies according to the content input by the user. The user can input a picture format file or enter text to perform image search and cross-module text search respectively.

[0032] For image retrieval, we use deep learning models to extract features and vectorize input images to obtain image vectors, match the image vectors with the data in the vector library, i.e. calculate similarity, and sort by similarity. Then, we merge the face and body vector features and sort them to further improve the matching accuracy. We use the Intersection over Union (IoU) algorithm to associate the structural attribute features of the human body for secondary filtering to ensure the quality and accuracy of the final result. For cross-module text retrieval, a deep learning model is used to extract features and vectorize the input text to obtain a text vector. The text vector is matched with the text feature vector in the vector library, that is, similarity is calculated, and the text is sorted according to the similarity. At the same time, based on the human attribute structured data selected by the user, human attribute features are retrieved from the search engine library. The retrieved human attribute features will be filtered by intersection with the structured data selected by the user, and sorted according to the intersection results to obtain more accurate results.

[0033] Among them, image retrieval specifically includes the following steps: Based on the video screenshots selected by the user or the image files uploaded, a deep learning model is used to extract human body vectors and face vectors.

[0034] The extracted face vector is subjected to cosine similarity calculation with the face feature vector in the vector library. After the calculation is completed, the face feature vectors are sorted by similarity. The calculation formula is as follows: in, Representation vector With vector The dot product of Representation vector The L2 norm of Represents the vector The L2 norm of the face similarity threshold is 0.35 by default, and the similarity threshold interface can be set.

[0035] Next, the matched facial feature vector is sent to the human body deduplication model for deduplication processing to ensure that images of the same person with different human attributes are retained in the search results, thereby improving the diversity and accuracy of the search results. Among them, since the vector library will store both face vectors and human body vectors and realize association based on imgBoxId, the associated human body vector can be queried through the face vector, and then the human body deduplication model can be used to obtain different human body attributes of the same person. In addition, the above-mentioned human body deduplication model can identify and match the same person based on images taken by different cameras. The principle is to use a deep learning model to learn a suitable distance metric space through a large number of human body images captured from multiple camera perspectives, and map the input human body image to a vector in this space. In this space, samples of the same category are as close as possible, and samples of different categories are as far away as possible. By measuring the similarity of the two vectors (generally cosine similarity), it is determined whether the two samples are of the same category.

[0036] Subsequently, the cosine similarity between the extracted human body vector and the human body feature vector in the vector library is calculated, and the human body images with human body feature vectors after deduplication are sorted to ensure that the returned results best match the requirements.

[0037] The matched human feature vector is associated with the corresponding face feature vector of the query, and the cosine similarity is calculated with the face vector of the input video screenshot or picture, and the matching results are further sorted to improve the matching accuracy. The purpose of the similarity calculation here is to put the results with high similarity in front of the search results, and the default matching threshold is 0.35.

[0038] Finally, the human coordinates in the query results are associated with the human coordinates stored in the search engine library by the intersection-over-union (IOU) calculation, and compared and filtered with the structural attributes of the human body in the search engine library (clothing style, top and bottom color, and gender, etc.). The data with low confidence is excluded through secondary retrieval filtering to ensure the accuracy of the returned results. Among them, the human coordinates in the vector library are obtained based on the CLIP-ReID model, and the human coordinates stored in the search engine library are obtained based on the CLIP model.

[0039] The specific steps for cross-module text search are as follows: Based on the text information entered by the user, a deep learning model is used to extract the vector features of the text.

[0040] Then, the extracted vector features are compared with the text vectors in the vector library, and the results are sorted using cosine similarity calculation, which is calculated as follows: in, Representation vector With vector The dot product of Representation vector The L2 norm of Represents the vector The L2 norm of .

[0041] Then, the search is performed based on the structured data selected by the user and combined with the human body attribute features. The structured data selected by the user is fused with the human body structured attribute features in the search engine library to obtain more accurate search results. The fusion in this step mainly means that when searching, not only the feature vector after the image is converted into text is searched, but also the structured data in the search engine library (such as the color of the upper and lower clothes) is matched and searched, and sorted by similarity. Then, the intersection sorting method is used to match and sort the fused features to obtain the results that best match the user's needs. This optimization measure will improve the search efficiency and accuracy of the system and provide users with more personalized and accurate results.

[0042] Preferably, after the search is completed, the search results are displayed in thumbnail form and sorted by dots, and the information of the video images is displayed, including but not limited to the video location, video time, human clothing style, human clothing color and gender. The user can click on the marked dotted area to review the video and play the clips before and after the corresponding time point.

[0043] Preferably, this embodiment provides an electronic device, which includes a processor and a memory, wherein the memory stores computer instructions that can be executed on the processor, and the processor is used to call the computer instructions in the memory to execute the cross-modal retrieval method that integrates human attribute features and vector embedding as described above.

[0044] Further preferably, the present embodiment provides a storage medium, the storage medium being used to store computer instructions, wherein when the computer instructions are executed, the device where the storage medium is located is controlled to execute the cross-modal retrieval method that integrates human attribute features and vector embedding as described above.

[0045] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Even if various changes are made to the present invention, if these changes fall within the scope of the claims of the present invention and their equivalents, they still fall within the protection scope of the present invention.

Claims

1. A cross-modal retrieval method integrating human attribute features and vector embedding, characterized in that: The steps include: Continuously monitor the video stream, extract pictures regularly, and cache the extracted picture codes in the message queue; Use the deep learning model to extract information from cached images, including face feature vectors, body feature vectors, and convert images into corresponding text vectors. Use the logistic regression model to obtain structural attribute features of the human body. By detecting the position of the human body coordinate points in the image, the facial feature vector and the human body feature vector are correlated with each other, and combined with their position information in the image, face fusion is achieved and stored in the vector library; Using the coordinate points of the human body in the image, the structured attribute features of the human body are associated with the vector data, and the structured data of the human body attributes are stored in the search engine; A corresponding search method is selected according to the input target to be matched, including image retrieval and cross-module text retrieval; for image retrieval, a picture vector is obtained by extracting features and vectorizing the input image, and the picture vector is matched with the data in the vector library, and then the search engine is used to filter according to the structured attribute characteristics of the human body to obtain the search results; for cross-module text retrieval, a text vector is obtained by extracting features and vectorizing the input text, and the text vector is matched with the data in the vector library, and then the search engine is used to filter according to the structured attribute characteristics of the human body to obtain the search results.

2. The cross-modal retrieval method integrating human attribute features and vector embedding according to claim 1 is characterized in that: During the image extraction process, each extracted image is Base64 encoded and then cached in the message queue.

3. The cross-modal retrieval method integrating human attribute features and vector embedding according to claim 1 is characterized in that: In the step of extracting information from the cached image, the extracted facial feature vector is also normalized, and the calculation formula is as follows: Face feature vector: , where the subscript of v indicates the order of the element in the vector; Calculate the L2 norm: ; Calculate the normalization factor: ; Compute the normalized vector: ; The facial feature vector Multiply each element in by the normalization factor to get a normalized vector.

4. The cross-modal retrieval method integrating human attribute features and vector embedding according to claim 1, characterized in that: The structural attribute features of the human body include clothing style, clothing color and gender of the person.

5. The cross-modal retrieval method integrating human attribute features and vector embedding according to claim 1, characterized in that: Image retrieval specifically includes: Based on the video screenshots or pictures selected by the user, a deep learning model is used to extract human body vectors and face vectors; The extracted face vector is subjected to cosine similarity calculation with the face feature vector in the vector library. After the calculation is completed, the face feature vectors are sorted according to the similarity; The matched facial feature vector is sent to the human body deduplication model for deduplication processing; Calculate the cosine similarity between the extracted human body vector and the human body feature vector in the vector library, and sort the human body feature vectors after deduplication; The matched human feature vector is associated with the corresponding face feature vector, and the cosine similarity is calculated with the face vector of the input video screenshot or picture, and the matching results are sorted; The human coordinates in the query results are associated with the human coordinates stored in the search engine library by intersection and comparison calculation, and compared and filtered in combination with the structural attribute features of the human body in the search engine library.

6. The cross-modal retrieval method integrating human attribute features and vector embedding according to claim 1, characterized in that: Cross-module text search specifically includes: Based on the text information input by the user, a deep learning model is used to extract the vector features of the text; Compare the extracted vector features with the text vectors in the vector library and sort the results using cosine similarity calculation; The structured data selected by the user is then fused with the structured attribute features of the human body in the search engine library, and the fused features are matched and sorted using the intersection sorting method to obtain the results that best match the user's needs.

7. The cross-modal retrieval method integrating human attribute features and vector embedding according to claim 1, characterized in that: After the retrieval is completed, the search results will be displayed in the form of thumbnails and sorted by dots, showing the information of the video pictures, including the video location, video time, body clothing style, body clothing color and gender; by clicking on the marked dotted area, you can review the video and play the clips before and after the corresponding time point.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores computer instructions that can be executed on the processor, and the processor is used to call the computer instructions in the memory to execute the cross-modal retrieval method that integrates human attribute features and vector embedding as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is used to store computer instructions, wherein when the computer instructions are executed, the device where the storage medium is located is controlled to execute the cross-modal retrieval method integrating human attribute features and vector embedding as described in any one of claims 1 to 7.

Citation Information

Cited By

  • Intelligent picture retrieval method and system based on large model, vectorization and face recognition

    CN120670610A