Identity recognition method and system based on multi-modal memory storage and dynamic matching

Abstract

The invention relates to the technical field of identity recognition, and particularly discloses an identity recognition method and system based on multi-modal memory storage and dynamic matching, and the method comprises the steps: firstly constructing an identity feature multi-modal database which is used for storing an identity feature record containing an identity ID, a serial number and image features of a plurality of subdivided visual angles; thirdly, capturing a target individual image by using a multi-view camera, constructing a query condition by performing view classification and image feature extraction on each view image, and searching an identity feature record meeting the condition in a database to obtain a query result; and counting the occurrence frequency of each identity ID in the query result so as to select the identity ID with the highest occurrence frequency as a final recognition result. According to the method, by combining the multi-view image features, front face dependence can be effectively relieved, and the accuracy and robustness of identity recognition are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of identity recognition, and more particularly, to an identity recognition method and system based on multi-modal memory storage and dynamic matching. BACKGROUND

[0002] In today's rapid development of social informatization and intelligentization, accurate and efficient identity recognition technology plays a crucial role in many fields such as public security, financial services, access control, and smart city management. It not only serves as the first line of defense for social security, but also is a key link to improve operational efficiency and optimize user experience. Traditional identity recognition methods, such as those based on facial recognition, have achieved remarkable results under certain conditions, but they highly depend on obtaining clear and frontal individual facial images. In actual application scenarios, such as complex security monitoring environments or non-cooperative personnel access management, individuals often exhibit varying postures and perspectives, and facial information may be obscured, blurred, or at non-ideal angles, resulting in a significant decrease in single facial recognition accuracy and making it difficult to meet actual needs.

[0003] To overcome the limitations of single biometric recognition, multi-modal identity recognition technology has emerged, attempting to utilize multiple sources of information such as faces, gaits, postures, and clothing for comprehensive discrimination. However, existing multi-modal methods mostly use a single type of feature independently, lacking a unified storage and retrieval mechanism, resulting in low system robustness and difficulty in achieving long-term usable identity memory and dynamic updates.

[0004] Therefore, an optimized identity recognition method and system based on multi-modal memory storage and dynamic matching are expected. SUMMARY

[0005] To solve the above technical problems, the present application is proposed. The embodiments of the present application provide an identity recognition method and system based on multi-modal memory storage and dynamic matching, which first constructs an identity feature multi-modal database for storing identity feature records containing identity ID, serial number, and image features of multiple subdivided perspectives. Then, a multi-perspective camera is used to capture target individual images, and query conditions are constructed by classifying each perspective image and extracting image features. The database is searched for identity feature records that meet the conditions to obtain query results, and the occurrence frequencies of each identity ID in the query results are counted to select the identity ID with the highest occurrence frequency as the final recognition result. This method effectively alleviates frontal face dependence by combining multi-perspective image features, improving the accuracy and robustness of identity recognition.

[0006] Accordingly, according to one aspect of the present application, an identity recognition method based on multi-modal memory storage and dynamic matching is provided, which includes:

[0007] construct an identity feature multimodal database, each identity feature record in the identity feature multimodal database containing an identity ID, a serial number, and a set of subdivided perspective category image feature vectors;

[0008] obtain an individual image to be identified;

[0009] input the individual image to be identified into the trained perspective pre-classification model to obtain a perspective category recognition result, and construct a query condition based on the perspective category recognition result;

[0010] search in the identity feature multimodal database according to the query condition to obtain a query result;

[0011] count the occurrence frequency of each identity ID in the query result, and select the identity ID with the highest occurrence frequency as the final recognition result.

[0012] According to another aspect of the present application, an identity recognition system based on multimodal memory storage and dynamic matching is provided, which comprises:

[0013] a database construction module for constructing an identity feature multimodal database, each identity feature record in the identity feature multimodal database containing an identity ID, a serial number, and a set of subdivided perspective category image feature vectors;

[0014] an individual image to be identified obtaining module for obtaining an individual image to be identified;

[0015] a perspective category recognition module for inputting the individual image to be identified into the trained perspective pre-classification model to obtain a perspective category recognition result, and constructing a query condition based on the perspective category recognition result;

[0016] a search query module for searching in the identity feature multimodal database according to the query condition to obtain a query result;

[0017] a recognition result confirmation module for counting the occurrence frequency of each identity ID in the query result, and selecting the identity ID with the highest occurrence frequency as the final recognition result.

[0018] Compared with the prior art, the identity recognition method and system based on multi-modal memory storage and dynamic matching provided by the application firstly constructs an identity feature multi-modal database for storing identity feature records containing an identity ID, a serial number and image features of multiple subdivided perspectives. Then, a multi-perspective camera is used to capture target individual images, and a query condition is constructed by performing perspective classification and image feature extraction on each perspective image, an identity feature record meeting the condition is found in the database to obtain a query result, and the occurrence frequency of each identity ID in the query result is counted to select the identity ID with the highest occurrence frequency as the final recognition result. The method can effectively alleviate the dependence on the front face by combining multi-perspective image features, and improve the accuracy and robustness of identity recognition. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description of embodiments of the present application, when taken in conjunction with the accompanying drawings. The drawings provided in the present application are used to provide further understanding of the embodiments of the present application, and constitute a part of the specification, and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 A flowchart of the identity recognition method based on multi-modal memory storage and dynamic matching according to the embodiments of the present application.

[0021] Figure 2 A data flow diagram of the identity recognition method based on multi-modal memory storage and dynamic matching according to the embodiments of the present application.

[0022] Figure 3 A flowchart of step S1 in the identity recognition method based on multi-modal memory storage and dynamic matching according to the embodiments of the present application.

[0023] Figure 4 A flowchart of step S3 in the identity recognition method based on multi-modal memory storage and dynamic matching according to the embodiments of the present application.

[0024] Figure 5 A flowchart of step S43 in the identity recognition method based on multi-modal memory storage and dynamic matching according to the embodiments of the present application.

[0025] Figure 6 A flowchart of step S433 in the identity recognition method based on multi-modal memory storage and dynamic matching according to the embodiments of the present application.

[0026] Figure 7 A block diagram of the identity recognition system based on multi-modal memory storage and dynamic matching according to the embodiments of the present application. DETAILED DESCRIPTION

[0027] Hereinafter, the example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. It is worth noting that in this application, all the actions of obtaining data are carried out in accordance with the corresponding data protection regulations and policies of the country where the device is located, and with the authorization given by the corresponding device owner.

[0028] Embodiment 1

[0029] Figure 1 A flowchart of the identity recognition method based on multi-modal memory storage and dynamic matching according to the embodiments of the present application. Figure 2 A data flow diagram of the identity recognition method based on multi-modal memory storage and dynamic matching according to the embodiments of the present application. As shown in Figure 1 and Figure 2 The identity recognition method based on multi-modal memory storage and dynamic matching according to the embodiments of the present application includes the following steps: S1, constructing an identity feature multi-modal database, each identity feature record in the identity feature multi-modal database contains an identity ID, a serial number and a set of subdivided view angle category image feature vectors; S2, obtaining an individual image to be identified; S3, inputting the individual image to be identified into a trained view angle pre-classification model to obtain a view angle category recognition result, and constructing a query condition based on the view angle category recognition result; S4, searching the identity feature multi-modal database according to the query condition to obtain a query result; S5, counting the occurrence frequency of each identity ID in the query result, and selecting the identity ID with the highest occurrence frequency as the final recognition result.

[0030] In the above identity recognition method based on multi-modal memory storage and dynamic matching, the step S1, constructing an identity feature multi-modal database, each identity feature record in the identity feature multi-modal database contains an identity ID, a serial number and a set of subdivided view angle category image feature vectors. It can be understood that the traditional face recognition technology highly depends on clear and frontal face images, while in actual complex scenes, individual posture and view angle are variable, face is easily occluded and blurred, which leads to a significant decrease in recognition accuracy, and cannot meet the actual demand. In order to fully utilize the individual gait, body posture and clothing features for identity discrimination, and to construct a unified storage framework and efficient retrieval mechanism to realize long-term available identity memory and dynamic update, the present application integrates multi-view image information to comprehensively describe individual features, constructs an identity feature multi-modal database, and thus realizes unified storage and management of multi-modal identity features.

[0031] Figure 3This is a flowchart of step S1 in the identity recognition method based on multimodal memory storage and dynamic matching according to an embodiment of this application. Figure 3 As shown, step S1 includes: S11, capturing target individual image data of the first target individual in the first scene using a multi-view camera; S12, performing image separation and subdivided view category labeling on the target individual image data to obtain a set of target individual local view subdivided category images; S13, extracting image features of each target individual local view subdivided category image from the set of target individual local view subdivided category images to obtain a set of target individual local subdivided category image visual feature encoding vectors as a set of subdivided view category image feature vectors of the first target individual in the first scene.

[0032] Specifically, step S11 involves capturing image data of the first target individual in a first scene using a multi-view camera. It should be understood that traditional identity recognition schemes often rely on images from a single perspective or of specific quality, making it difficult to adapt to the variability of individual postures and angles in the real world, resulting in the inability to effectively acquire key identity information in many scenarios. Therefore, in order to obtain raw data that comprehensively reflects the multifaceted visual characteristics of an individual in the real environment from the source, this application captures image data of the target individual using a multi-view camera to achieve comprehensive feature recording of the individual from multiple perspectives. In practical applications, multiple cameras at different angles need to be reasonably arranged in different scenarios (such as security monitoring scenarios, access control scenarios, etc.), and multiple cameras need to work simultaneously to capture image data of the first target individual in the first scene from multiple perspectives, ensuring that the collected images cover the front, side, and back of the target individual, as well as different parts and different states (such as static and walking states). Based on this, the problem of a single data source in traditional methods is effectively solved, enabling the collected data to more comprehensively reflect individual characteristics.

[0033] Specifically, the step S12 is to perform image separation and subdivided view class labeling on the target individual image data to obtain a set of target individual local view subdivided class images. It should be understood that the captured target individual image data contains information of multiple views and parts, and different parts or views (such as front, side, back, face, half body, and full body) of the same individual have different contributions to identity recognition and different applicable feature extraction methods, which is not conducive to subsequent feature extraction and storage. Therefore, in order to arrange the complex image data into a form convenient for processing and utilization, based on the principles of target detection, image segmentation, and semantic classification, the target individual is accurately separated from the background through a series of preprocessing operations on the captured target individual image data, and the separated individual image is classified and labeled in detail according to the view and body part it presents. In one specific example of the present application, the set of target individual local view subdivided class images includes front face images, side face images, head back images, front half body images, side half body images, back half body images, full body front images, full body side images, full body back images, front gait images, side gait images, and back gait images. Specifically, for the captured target individual image data, first, a target detection algorithm (such as YOLO, Faster R-CNN) is used to locate the individual in the image, and an image segmentation technique (such as MaskR-CNN) is used to accurately extract the pixel area of the individual, removing irrelevant background. Subsequently, a semantic classification algorithm is used to classify and label the extracted individual image according to the view (such as front, side, and back) and body part (such as face, upper body, and full body) it presents, to divide the overall visual information of the individual into a series of standardized, semantically clear (view and content) local images, obtaining a set of target individual local view subdivided class images. In this way, each target individual local view subdivided class image can clearly correspond to specific human identity information, facilitating subsequent targeted feature extraction and improving data processing efficiency and accuracy.

[0034] Specifically, the step S13 extracts the image features of each target individual local perspective fine classification category image in the set of target individual local perspective fine classification category images to obtain a set of target individual local fine classification category image visual feature encoding vectors as a set of fine division perspective category image feature vectors of the first target individual in the first scene. It should be understood that the image itself is high-dimensional pixel data, which is inefficient for comparison and storage and difficult to capture deep semantic information. Therefore, in order to convert the separated and labeled various local perspective fine classification category images into a compact and distinctive numerical representation form, facilitate efficient storage and fast retrieval of matches in the database, the present application extracts the key information in the image by using the corresponding feature extraction model for different types of fine division perspective images based on deep learning feature extraction technology, and maps each image to a low-dimensional feature vector space to generate a set of target individual local fine classification category image visual feature encoding vectors.

[0035] In a specific example of the present application, the step S13 includes: for the front face image and the side face image in the set of target individual local perspective fine classification category images, extracting the image features of the front face image and the side face image by using a face recognition model; for other fine classification category images in the set of target individual local perspective fine classification category images except the front face image and the side face image, extracting the image features thereof by using a visual large model.

[0036] It should be understood that a face image contains more abundant biometric information than other images, such as facial feature shapes and textures, and has high specificity and stability for identity recognition. Therefore, for the frontal face image and the side face image in the set of local perspective fine classification images of the target individual, the present application uses an advanced face recognition model DeepFace to extract the facial features. The DeepFace model is based on a deep convolutional neural network structure, which is trained on a large amount of face image data. Through multi-level feature learning and abstraction of frontal face images and side face images, it can accurately capture key feature points in face images, such as the shapes and positions of eyes, nose, and mouth, as well as subtle features such as skin texture and facial contour, thereby generating a highly discriminative and robust face feature representation. As for other fine classification images in the set of local perspective fine classification images of the target individual, such as various angle half / full body images, back-of-head images, and gait images, it is necessary to focus on capturing the overall contour, posture, and clothing features of the human body. Therefore, for non-face images, the present application uses a general visual large model ViT model to extract image features. It should be understood that the ViT model is based on the Transformer architecture, which divides the image into a series of small blocks (patches) when processing the image, and captures the global dependency between these small blocks through a self-attention mechanism, thereby being able to more comprehensively understand the overall layout and contextual information in the image. Therefore, when using the ViT model to extract features from half / full body images, back-of-head images, and gait images, the overall contour, posture, and clothing features of the human body can be effectively captured, generating an image feature representation with rich semantic information. In this way, for different types of fine classification images, the corresponding feature extraction model is used to extract the key information in the image, thereby constructing a set of local fine classification image visual feature encoding vectors of the target individual, and providing strong data support for subsequent identity recognition.

[0037] Finally, a complete identity feature record of the first target individual is formed by associating and storing a set of subdivided perspective category image feature vectors of the first target individual in the first scene with its corresponding unique identity ID. In addition, the identity feature multi-modal database also supports incremental learning and updating, and can append newly identified individual identity data to the database in real time, avoiding excessive reliance on static base libraries. That is, each target individual corresponds to a unique identity ID, and in actual multi-modal identity recognition scenarios, image data generated by the same target individual at different times and in different scenes can be continuously captured and processed, and after identity verification, a set of subdivided perspective category image feature vectors generated by the individual in the corresponding scene are associated and stored with the unique identity ID of the individual, forming a new identity feature record. In this way, the identity feature record of the target individual can be continuously updated and improved, ensuring the timeliness and comprehensiveness of the identity feature data.

[0038] In the above-mentioned identity recognition method based on multi-modal memory storage and dynamic matching, the step S2 of acquiring the individual image to be identified. That is, in actual application, the visual image of the individual to be identified in the current scene is also captured by a multi-perspective camera to obtain image information of different perspectives and body parts of the individual to be identified as basic data for subsequent identity query and recognition.

[0039] In the above-mentioned identity recognition method based on multi-modal memory storage and dynamic matching, the step S3 of inputting the individual image to be identified into the trained perspective pre-classification model to obtain a perspective category recognition result, and constructing a query condition based on the perspective category recognition result. That is, in order to selectively extract the most relevant and most distinctive features from the individual image to be identified, and construct an efficient query request matching the storage format of each identity feature record in the database, the present application further performs perspective analysis on the individual image to be identified through image classification and target-oriented feature extraction, and guides the subsequent feature extraction and query construction process according to the analysis result, to realize efficient feature matching.

[0040] Figure 4 The flowchart of step S3 in the identity recognition method based on multi-modal memory storage and dynamic matching according to the embodiments of the present application is shown in FIG. 3. Figure 4 As shown in FIG. 3, the step S3 includes: S31, inputting the individual image to be identified into the trained perspective pre-classification model to obtain a combination of perspective type labels as the perspective category recognition result; S32, based on the combination of perspective type labels, calling the face recognition model and / or the visual large model to extract image features of the individual image to be identified to obtain a plurality of subdivided perspective category feature query conditions; and S33, combining the plurality of subdivided perspective category feature query conditions to obtain the query condition.

[0041] Specifically, first, since the individual image to be identified contains various perspective information, and the identity feature multimodal database stores feature vectors corresponding to specific subdivided perspective categories. Therefore, the present application first uses the trained perspective pre-classification model to pre-classify the individual image to be identified. Here, the trained perspective pre-classification model is consistent with the perspective pre-classification model used in the construction process of the identity feature multimodal database. After a large amount of labeled data training, it can accurately identify various perspective types contained in the individual image to be identified, such as front, side, back, etc., and body part information, such as face, upper body, full body, etc., and output the corresponding combination of perspective type labels as a guide for subsequent feature extraction and query construction. Then, according to the results of perspective pre-classification, for the front or side face image contained in the individual image to be identified, the face recognition model DeepFace is called to extract its facial features, generating front face feature query conditions and side face feature query conditions; and for other perspective category images contained in the individual image to be identified, the visual large model ViT is called to extract its overall contour, posture, clothing, etc. Feature, generate the corresponding subdivided perspective category feature query condition. Finally, the various subdivided perspective category feature query conditions are combined to construct the complete individual identity feature information to be identified as the final query condition. In this way, for different perspective and body part image information, the most suitable feature extraction model can be used to construct a query condition that matches the identity feature record stored in the database, thereby providing strong support for subsequent identity recognition and improving the relevance and effectiveness of the query.

[0042] In the above-mentioned identity recognition method based on multimodal memory storage and dynamic matching, the step S4 searches the identity feature multimodal database according to the query condition to obtain a query result. That is, in order to query the most similar registered identity record of the individual to be identified in the identity feature multimodal database, the present application is based on the principle of vector similarity matching and threshold filtering, by calculating the similarity between the individual identity feature information to be identified and each identity feature record stored in the identity feature multimodal database, and the identity feature record with a similarity exceeding a predetermined threshold is taken as a candidate identity record, to further narrow down the query range, and obtain a preliminary query result.

[0043] In an embodiment of the present application, the step S4 comprises: S41A, for each of the sub-view category feature query conditions in the query conditions, performing a Top-K search in the identity feature multimodal database to obtain a plurality of search results; S42A, merging the plurality of search results to obtain a set of candidate identity feature records. That is, for each sub-view category feature query condition, such as a front face feature, a side face feature, a half-body image feature, etc., the present application performs a Top-K search in the identity feature multimodal database. Here, K is a predetermined positive integer, indicating that for each query condition, the database returns K identity feature records most similar to it. For example, the K identity feature records with the highest similarity can be selected by calculating the cosine similarity between the sub-view category feature query condition and each sub-view category image feature vector stored in the identity feature multimodal database. In this way, the identity records most similar to the individual to be identified in each single view and body part can be preliminarily screened out. Then, since a Top-K search is independently performed for each sub-view in the query condition, a plurality of search result lists are obtained, which may contain duplicate records and point to different identity IDs. Therefore, in order to form a unified preliminary candidate pool containing all potential matching records for subsequent more accurate comprehensive evaluation, the present application merges and removes the plurality of result lists obtained from different views, and collects all identity feature records hit by at least one view search to obtain a set of candidate identity feature records. For example, if the Top-K results of the front face contain records A, B, C, and the Top-K results of the front gait contain records B, D, E, then the merged candidate set will contain A, B, C, D, E. In this way, preliminary matching information from different view dimensions can be effectively summarized, providing a clear object range for subsequent refined similarity calculation and screening.

[0044] In another embodiment of the present application, the step S4 comprises: S41B, constructing a joint search condition based on all the sub-perspective category feature query conditions in the query condition; S42B, performing a rough similarity matching in the identity feature multimodal database based on the joint search condition to obtain a set of candidate identity feature records. That is, all the sub-perspective category feature query conditions of the individual to be identified are combined to form a joint search condition as a whole, and a rough overall joint search is performed in the identity feature multimodal database. For example, all the sub-perspective category feature query conditions are concatenated at the feature level to obtain a joint feature vector containing all the perspective feature information. Then, in the identity feature multimodal database, the multi-perspective features of each registered identity record are also jointly processed to form a complete registered identity feature, and the overall feature similarity between the joint search condition and each complete registered identity feature in the identity feature multimodal database is calculated to screen out the k registered identity records with the highest overall similarity to the individual to be identified as candidate identity records, thereby forming a set of candidate identity feature records.

[0045] Since the candidate identity feature records screened out above are based on single-perspective or overall rough approximate matching, there may be a case of false matching. Therefore, on the basis of the set of candidate identity feature records screened out above, the step S4 further comprises: S43, traversing each candidate identity feature record in the set of candidate identity feature records, and calculating the feature accurate similarity between each candidate identity feature record and the query condition to obtain a set of feature accurate similarities; S44, based on the comparison between the set of feature accurate similarities and a preset threshold, screening the identity feature records that meet the preset condition from the set of candidate identity feature records as the query result. More specifically, in one specific example of the present application, the step S43 comprises: calculating the accurate sub-similarity between each sub-perspective category feature query condition in the query condition and the corresponding sub-perspective category image feature vector in the candidate identity feature record to obtain a plurality of accurate sub-similarities; and calculating the weighted sum of the plurality of accurate sub-similarities to obtain the feature accurate similarity.

[0046] That is, for each of the perspective features in the candidate identity feature record, find the corresponding same-perspective feature query condition in the query condition (if the perspective feature exists in the query condition), and then calculate the exact subdivision similarity between the pair of same-perspective features (for example, using the Mahalanobis distance metric function). Then, calculate the weighted sum of the exact subdivision similarities of each perspective as the feature exact similarity of the candidate identity feature record. Here, different weights can be set for different perspective features to reflect their importance in identity recognition. For example, the face feature can be given a higher weight due to its high discriminability and stability in identity recognition; the back image may contain less effective identification information than the front image and side image, and therefore be given a lower weight. In this way, the contributions of different perspective features to identity recognition can be considered comprehensively, the similarity between the candidate identity feature record and the individual to be identified can be more accurately evaluated, and the risk of false matching can be further reduced. Next, in order to ensure that the final output matching result has a high enough confidence, the present application is based on the principle of threshold decision, by setting a reasonable similarity threshold (for example, setting the preset threshold to 0.75), and filtering the exact similarity scores of each candidate identity feature record with this preset threshold, only keeping the identity feature records with feature exact similarity scores higher than the preset threshold as the final query result, thereby ensuring the accuracy and reliability of identity recognition.

[0047] In the above identity recognition method based on multi-modal memory storage and dynamic matching, the step S5 is to count the occurrence frequencies of each identity ID in the query result, and select the identity ID with the highest occurrence frequency as the final recognition result. It should be understood that, considering that the database can store multiple identity feature records of the same individual at different time periods and in different scenarios, the query result can contain multiple identity feature records of the same individual. Therefore, in order to make a final identity determination from the multiple candidate identity feature records that pass the similarity screening, the present application extracts the identity ID associated with each identity feature record in the query result, counts the occurrence frequencies of each identity ID in the query result, and selects the identity ID with the highest occurrence frequency as the identity recognition result of the individual to be identified. It should be understood that, if multiple identity record instances with high similarity to the query feature all point to the same identity ID, it indicates that the individual corresponding to the identity ID has high consistency with the individual to be identified in multiple perspectives and feature dimensions, thereby further enhancing the accuracy and reliability of identity recognition. In addition, if there are multiple identity IDs with the same highest occurrence frequency (for example, once, i.e., there is only one identity record of the target individual in the database, so that each candidate identity feature record screened based on the similarity threshold comparison corresponds to a different identity ID), a secondary sorting can be performed based on the feature accurate similarity of each candidate identity feature record, and the identity ID with the highest feature accurate similarity is selected as the final recognition result. In this way, not only can the situation of multiple records of the same individual in the database be effectively dealt with, but also in the presence of noise or interference information, the robustness of identity recognition can be improved through the majority voting mechanism.

[0048] Embodiment 2

[0049] In particular, considering that there is a certain correlation between image features of different perspectives in the identity feature record, which has a certain synergistic effect on the identity recognition of the target individual, and the traditional method calculates the feature accurate similarity between the candidate identity feature record and the query condition by calculating the accurate similarity of each subdivided perspective category feature query condition in the query condition and the corresponding vector in the candidate identity feature record, and then obtaining the final feature accurate similarity by weighted summation. This way simply combines image features of different perspectives by weighting, and fails to fully utilize the internal relationship and synergistic effect between each subdivided perspective category feature, resulting in inaccurate calculation of the feature accurate similarity. In this embodiment, a feature accurate similarity calculation method based on deep learning technology is proposed, which uses a deep neural network model to jointly interact and learn the multi-perspective features of the candidate identity feature record and the query condition, to deeply mine the internal relationship and synergistic effect between different perspective features, and to achieve more accurate feature similarity calculation.

[0050] Figure 5 This is a flowchart of step S43 in the identity recognition method based on multimodal memory storage and dynamic matching according to Embodiment 2 of this application. Figure 5 As shown, step S43 includes: S431, concatenating all the subdivided viewpoint category feature query conditions in the query conditions to obtain a query condition concatenated encoding vector; S432, concatenating multiple subdivided viewpoint category image feature vectors in the candidate identity feature records corresponding to the query conditions to obtain a candidate identity feature multimodal concatenated encoding vector; S433, inputting the query condition concatenated encoding vector and the candidate identity feature multimodal concatenated encoding vector into a precise similarity measurement network to obtain the feature precise similarity.

[0051] Specifically, in step S431, all the subdivided viewpoint category feature query conditions in the query conditions are concatenated to obtain a query condition concatenated encoding vector. It should be understood that the aforementioned similarity measurement method, which calculates the similarity of each modality feature separately and then linearly combines them, is insufficient to fully explore the deep interactions and nonlinear dependencies between different modalities, thus limiting the accuracy of similarity determination. To enable subsequent deep models to comprehensively process various viewpoint features and learn complex intermodal relationships, this application uses an early feature fusion method to concatenate all effective subdivided viewpoint category feature query conditions (i.e., feature vectors extracted from images of the individual to be identified from different viewpoints) in the query conditions to form a query condition concatenated encoding vector. This lays the foundation for subsequent learning of cross-viewpoint feature interactions and avoids the model only analyzing isolated viewpoints.

[0052] Specifically, in step S432, the image feature vectors of multiple subdivided viewpoint categories corresponding to the query conditions in the candidate identity feature record are concatenated to obtain a candidate identity feature multimodal concatenated encoding vector. Similarly, based on the feature representation alignment principle, this application extracts image feature vectors corresponding to each subdivided viewpoint in the query conditions from the candidate identity feature record, and concatenates and merges them to form a candidate identity feature multimodal concatenated encoding vector. This maintains the consistency of the feature dimension structure and ensures the correspondence of the feature structure between the candidate identity feature multimodal concatenated encoding vector and the query condition concatenated encoding vector, facilitating subsequent query response interaction.

[0053] Specifically, the step S433 inputs the query condition concatenated encoding vector and the candidate identity feature multi-modal concatenated encoding vector into the precise similarity measurement network to obtain the feature precise similarity. Specifically, since the traditional similarity indicators such as Euclidean distance, cosine similarity and the like belong to shallow nonlinear measurement, they cannot sufficiently learn the complex matching relationship between two multi-modal feature sets. Especially when different modal features are context-related. Therefore, in order to deeply mine the context-aware and fine-grained similarity relationship between the query condition concatenated encoding vector and the candidate identity feature multi-modal concatenated encoding vector, the application further utilizes a deep learning algorithm to construct a precise similarity measurement network to learn the high-order nonlinear fine-grained feature interaction relationship and deep context association between the query condition concatenated encoding vector and the candidate identity feature multi-modal concatenated encoding vector, thereby decoding a more precise feature similarity measurement.

[0054] Figure 6 The flowchart of step S433 in the identity recognition method based on multi-modal memory storage and dynamic matching according to Embodiment 2 of the application is shown in FIG. 13. As shown in FIG. 13, the step S433 includes: S4331, performing local granularity-based query response encoding on the query condition concatenated encoding vector and the candidate identity feature multi-modal concatenated encoding vector to obtain a sequence of query condition-candidate identity feature local query response encoding matrices; S4332, performing query response feature reasoning propagation encoding on the sequence of query condition-candidate identity feature local query response encoding matrices to obtain a candidate identity query response feature encoding vector; and S4333, performing feature decoding on the candidate identity query response feature encoding vector to obtain the feature precise similarity. Figure 6

[0055] In a specific example of the application, the step S4331 includes: first, performing rank reconstruction based on feature value size on the query condition concatenated encoding vector and the candidate identity feature multi-modal concatenated encoding vector to obtain a query condition feature rank reconstruction encoding vector and a candidate identity feature rank reconstruction encoding vector, which is expressed by a formula as follows:

[0056]

[0057]

[0058] wherein, denotes the query condition concatenated encoding vector, denotes the candidate identity feature multi-modal concatenated encoding vector, denotes a sorting operation on vector elements, denotes the query condition feature rank reconstruction encoding vector, denotes the candidate identity feature rank reconstruction encoding vector.​

[0059] It should be understood that the physical position of the feature elements in the query condition concatenated encoding vector and the candidate identity feature multimodal concatenated encoding vector is not necessarily related to their discriminative strength. Therefore, in order to eliminate the interference of randomness in feature arrangement and make the model focus on the intensity distribution pattern of feature values, this application, based on the principle of feature strength priority response, sorts the query condition concatenated encoding vector and the candidate identity feature concatenated encoding vector in descending order according to the size of the feature values, and generates query condition feature rank reconstruction encoding vector and candidate identity feature rank reconstruction encoding vector respectively. This allows for the analysis of feature similarity from a unified intensity distribution perspective, avoiding mismatches caused by differences in the original feature order.

[0060] Next, the query condition feature rank reconstruction encoding vector and the candidate identity feature rank reconstruction encoding vector are subjected to uniform modal segmentation to obtain the sequence of query condition local feature rank reconstruction encoding vectors and the sequence of candidate identity local feature rank reconstruction encoding vectors, which are expressed by the formula:

[0061]

[0062]

[0063] in, , , and These represent the 1st, 2nd, and 3rd rank-reconstructed encoding vectors of the local features of the query conditions, respectively. The and the first Reconstructing the encoding vector based on the rank of local features for each query condition The number of encoding vectors reconstructed from the rank of the local features of the query condition. , , and These represent the 1st, 2nd, and 3rd rank reconstruction encoding vectors of the local features of the candidate identity, respectively. The and the first Reconstructing encoding vectors from the rank of local features of each candidate identity This represents the feature segmentation function.

[0064] That is, since directly performing global similarity calculation ignores the fine-grained interaction mode of the local feature interval (for example, the cooperative discriminability of the high-response feature cluster and the low-response feature cluster). Therefore, in order to capture the local fine-grained feature interaction mode between the query condition and the candidate identity feature, the application reconstructs the query condition feature rank order encoding vector and the candidate identity feature rank order encoding vector into multiple equal-length subvector segments along the feature dimension, generates a query condition local feature rank order reconstruction encoding vector sequence and a candidate identity local feature rank order reconstruction encoding vector sequence, in order to independently analyze the feature distribution of different intensity intervals, for example, the high-response interval may correspond to the significant identity mark (such as the face key point), and the low-response interval may imply auxiliary discriminant information (such as clothing texture), so as to more finely depict the local similarity and difference between features.

[0065] Then, each group of corresponding query condition local feature rank order reconstruction encoding vectors and candidate identity local feature rank order reconstruction encoding vectors in the sequence of query condition local feature rank order reconstruction encoding vectors and the sequence of candidate identity local feature rank order reconstruction encoding vectors are input into an interaction reasoning unit to obtain a sequence of query condition-candidate identity feature local query response encoding matrices, which is expressed by the formula:

[0066]

[0067] Wherein, represents a linear transformation matrix, represents a ReLU activation function, represents matrix multiplication, represents and the query condition-candidate identity feature local query response encoding matrix between

[0068] That is, in order to quantify the fine-grained correlation mode between each group of corresponding local feature segments in the sequence of query condition local feature rank order reconstruction encoding vectors and the sequence of candidate identity local feature rank order reconstruction encoding vectors, the application designs an interaction reasoning unit to deeply interact model each pair of query condition local feature rank order reconstruction encoding vectors and candidate identity local feature rank order reconstruction encoding vectors, to explicitly encode the complex interaction mode within the local interval, thereby generating a sequence of query condition-candidate identity feature local query response encoding matrices.

[0069] In a preferred embodiment of this application, step S4332 includes: first, based on the inter-block coupling probability between the local feature rank reconstruction encoding vector and the candidate identity local feature rank reconstruction encoding vector corresponding to each group, performing feature sparsity regularization on each query condition-candidate identity feature local query response encoding matrix in the sequence to obtain an optimized sequence of query condition-candidate identity feature local query response encoding matrices.

[0070] Specifically, considering that different segmentation granularities (i.e., row vector length L in the matrix) directly affect the granularity of local block interaction modeling during the generation of the query condition-candidate identity feature local query response encoding matrix, that is, when L is too large, the query condition-candidate identity feature local query response encoding matrix may introduce noise due to containing too many redundant feature components, causing a decrease in the saliency of important features. Based on this, in order to suppress noise interference in the query condition-candidate identity feature local query response encoding matrix, this application introduces dynamic regularization constraints based on a point process stochastic model, so that each query condition-candidate identity feature local query response encoding matrix can automatically adjust its spatial measure according to its feature intensity distribution, thereby balancing the granularity and generalization ability of local interaction modeling, and thus solving the problem of rigid feature representation caused by fixed segmentation granularity.

[0071] Specifically, firstly, the size of the local region (i.e., the row vector length) of the query condition-candidate identity feature local query response encoding matrix is ​​determined. As a local size strength constraint, the subspace measure representation (i.e., F-norm) of the query condition-candidate identity feature local query response encoding matrix is... It should follow a Poisson-like relation:

[0072]

[0073] in, To represent factorial, Denotes the Frobenius norm of a matrix. express The length of the row vector, This represents an exponential function with the natural constant as its base. is a Poisson distribution parameter, representing the expected density of random events in the local interaction between query conditions and candidate identity features.

[0074] Therefore, the Poisson distribution parameters can be solved. .

[0075] Thus, by introducing a local size strength constraint... The mean and expected value are In the case of point process random model generation, with and Describing the boundary association representation of the spatial dimension allows us to further determine the inter-feature block coupling probability between the query condition features and candidate identity features of two local domains:

[0076]

[0077] in, express and The coupling probability between feature blocks. This indicates taking the absolute value. This indicates the calculation of the L2 norm.

[0078] Then, the coupling probability between the feature blocks is used. To determine the size of the local area Perform adaptive parameter calibration:

[0079]

[0080] in, Indicates the optimized .

[0081] That is, while strictly ensuring that the expected degree is In this case, the normalization of the global spatial metric is determined by normalizing the fluctuations in the mean expected connection density of each row. Finally, the optimized... Reconstrain the query conditions - candidate identity feature local query response encoding matrix Characteristic sparsity:

[0082]

[0083] in, express The corresponding optimized query condition-candidate identity feature local query response encoding matrix. Here, by forcing the low-rank representation of the query condition-candidate identity feature local query response encoding matrix to adaptively match the feature intensity distribution, the ordered interaction information in the local spatial dimension can avoid local overfitting, thereby improving the overall expression effect of the sequence of the query condition-candidate identity feature local query response encoding matrix.

[0084] Then, the sequence of the optimized query condition-candidate identity feature local query response encoding matrix is ​​input into an LSTM transitive encoding module with an attention mechanism to obtain the candidate identity query response feature encoding vector, which is expressed by the formula:

[0085]

[0086]

[0087]

[0088] wherein, denotes the query condition-candidate identity feature local query response encoding matrix between denotes the query condition-candidate identity feature local query response encoding matrix between denotes the query condition-candidate identity feature local query response encoding matrix between denotes the matrix flattening operation, denotes the query condition-candidate identity feature local query response encoding vector obtained by unfolding, denotes the exponential function operation with base e, is a feature modulation function based on attention mechanism, denotes the LSTM model, denotes the candidate identity query response feature encoding vector.

[0089] Since the sequence of the optimized query condition-candidate identity feature local query response encoding matrix contains fragmented query condition-candidate identity interaction information, direct weighted average will lose the context dependence across intervals (for example, the dominant features of the high response interval may affect the discriminant weight of the low response interval). Therefore, in order to integrate the global interaction pattern between the query condition and the candidate identity feature and capture the semantic evolution law in the feature intensity attenuation process, the present application uses the LSTM network to perform transfer coding on the sequence of the optimized query condition-candidate identity feature local query response encoding matrix, and introduces an attention mechanism to weight the interaction feature importance of the sequence of the optimized query condition-candidate identity feature local query response encoding matrix before transfer coding, in order to strengthen the role of key features in the context dependence, thereby realizing dynamic context aggregation of query condition-candidate identity local interaction information, and obtaining the candidate identity query response feature encoding vector.

[0090] In a specific example of this application, step S4333 involves feature decoding of the candidate identity query response feature encoding vector to obtain the precise feature similarity. Here, to quantify the interaction response information of the candidate identity features relative to the query conditions contained in the candidate identity query response feature encoding vector into a specific precise feature similarity index, this application introduces a fully connected layer and a Sigmoid activation function into the output layer of the aforementioned LSTM network to perform a nonlinear transformation on the candidate identity query response feature encoding vector and map it to the [0,1] interval to obtain a numerical value representing the precise feature similarity. In this way, by accurately modeling the complex interaction relationship between the query conditions and candidate identity features, high accuracy and robustness of the similarity measurement can be ensured.

[0091] In summary, the identity recognition method based on multimodal memory storage and dynamic matching according to the embodiments of this application is explained. First, a multimodal identity feature database is constructed to store identity feature records containing identity ID, serial number, and image features from multiple subdivided perspectives. Next, images of the target individual are captured using a multi-view camera, and query conditions are constructed by classifying the images from each perspective and extracting image features. Identity feature records that meet the conditions are searched in the database to obtain query results. The frequency of each identity ID in the query results is counted, and the identity ID with the highest frequency is selected as the final recognition result. This method, by combining multi-view image features, can effectively alleviate reliance on frontal faces and improve the accuracy and robustness of identity recognition.

[0092] Furthermore, this application also provides an identity recognition system based on multimodal memory storage and dynamic matching.

[0093] Figure 7 This is a block diagram of an identity recognition system based on multimodal memory storage and dynamic matching according to an embodiment of this application. Figure 7 As shown, the identity recognition system 100 based on multimodal memory storage and dynamic matching according to an embodiment of this application includes: a database construction module 100, used to construct an identity feature multimodal database, wherein each identity feature record in the identity feature multimodal database includes an identity ID, a serial number, and a set of subdivided viewpoint category image feature vectors; a candidate individual image acquisition module 120, used to acquire an image of the candidate individual; a viewpoint category recognition module 130, used to input the candidate individual image into a trained viewpoint pre-classification model to obtain a viewpoint category recognition result, and construct query conditions based on the viewpoint category recognition result; a search query module 140, used to search the identity feature multimodal database according to the query conditions to obtain query results; and a recognition result confirmation module 150, used to count the frequency of occurrence of each identity ID in the query results, and select the identity ID with the highest frequency of occurrence as the final recognition result.

[0094] Here, those skilled in the art can understand that the specific operations of each module in the above identity recognition system based on multi-modal memory storage and dynamic matching have been described in detail in the above description of the identity recognition method based on multi-modal memory storage and dynamic matching according to Figures 1 to 6 Therefore, the repeated description will be omitted.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An identity recognition method based on multimodal memory storage and dynamic matching, characterized in that, include: Construct an identity feature multimodal database, wherein each identity feature record in the identity feature multimodal database contains an identity ID, a serial number, and a set of image feature vectors for subdivided viewpoint categories; Acquire images of the individuals to be identified; The individual image to be identified is input into the trained viewpoint pre-classification model to obtain viewpoint category recognition results, and query conditions are constructed based on the viewpoint category recognition results; Based on the query conditions, a search is performed in the identity feature multimodal database to obtain the query results; The frequency of each identity ID in the query results is counted, and the identity ID with the highest frequency is selected as the final identification result.

2. The identity recognition method based on multimodal memory storage and dynamic matching according to claim 1, characterized in that, Constructing a multimodal database of identity features, including: Capture image data of the first target individual in the first scene using a multi-view camera; The target individual image data is subjected to image separation and subdivided viewpoint category labeling to obtain a set of target individual local viewpoint subdivided category images; The image features of each target individual's local view sub-categorized image are extracted from the set of target individual local view sub-categorized images to obtain a set of target individual local sub-categorized image visual feature encoding vectors, which are used as a set of sub-view category image feature vectors for the first target individual in the first scene.

3. The identity recognition method based on multimodal memory storage and dynamic matching according to claim 2, characterized in that, The set of target individual local perspective subcategories of images includes frontal face images, side face images, back of head images, frontal half-body human images, side half-body human images, back half-body human images, full-body frontal images, full-body side images, full-body back images, frontal gait images, side gait images, and back gait images.

4. The identity recognition method based on multimodal memory storage and dynamic matching according to claim 3, characterized in that, Extracting image features from each of the target individual's local viewpoint sub-categorization images in the set of target individual local viewpoint sub-categorization images, including: For the frontal and side face images in the set of local view subcategories of the target individual, the image features of the frontal and side face images are extracted using a face recognition model; For the target individual's local view subcategories of images, excluding frontal and side-view images, other subcategories are extracted using a large visual model.

5. The identity recognition method based on multimodal memory storage and dynamic matching according to claim 4, characterized in that, The image of the individual to be identified is input into a trained viewpoint pre-classification model to obtain viewpoint category recognition results, and query conditions are constructed based on the viewpoint category recognition results, including: The individual image to be identified is input into the trained viewpoint pre-classification model to obtain a combination of viewpoint type labels as the viewpoint category recognition result. Based on the combination of the viewpoint type labels, the face recognition model and / or the large visual model are retrieved to extract the image features of the individual image to be identified in order to obtain multiple subdivided viewpoint category feature query conditions; The query conditions are obtained by combining the multiple segmented perspective category feature query conditions.

6. The identity recognition method based on multimodal memory storage and dynamic matching according to claim 5, characterized in that, Based on the query conditions, a search is performed in the identity feature multimodal database to obtain query results, including: For each of the subdivided perspective category feature query conditions in the query conditions, a Top-K search is performed in the identity feature multimodal database to obtain multiple search results; The multiple search results are merged to obtain a set of candidate identity feature records; Traverse each candidate identity feature record in the set of candidate identity feature records, and calculate the feature precision similarity between each record and the query condition to obtain a set of feature precision similarities; Based on the comparison between the set of precise similarity of the features and a preset threshold, the set of candidate identity feature records is selected to meet the preset conditions as the query result.

7. The identity recognition method based on multimodal memory storage and dynamic matching according to claim 5, characterized in that, Based on the query conditions, a search is performed in the identity feature multimodal database to obtain query results, including: Based on all the subdivided perspective category feature query conditions in the query conditions, construct joint search conditions; Based on the joint search conditions, coarse similarity matching is performed in the identity feature multimodal database to obtain a set of candidate identity feature records; Traverse each candidate identity feature record in the set of candidate identity feature records, and calculate the feature precision similarity between each record and the query condition to obtain a set of feature precision similarities; Based on the comparison between the set of precise similarity of the features and a preset threshold, the set of candidate identity feature records is selected to meet the preset conditions as the query result.

8. The identity recognition method based on multimodal memory storage and dynamic matching according to claim 6 or 7, characterized in that, Iterate through each candidate identity feature record in the set of candidate identity feature records, and calculate the precise feature similarity between each record and the query condition to obtain a set of precise feature similarities, including: Calculate the precise subdivision similarity between each subdivision view category feature query condition in the query conditions and the corresponding subdivision view category image feature vector in the candidate identity feature record to obtain multiple precise subdivision similarities; The weighted sum of the multiple precise segmentation similarities is calculated to obtain the precise similarity of the features.

9. The identity recognition method based on multimodal memory storage and dynamic matching according to claim 6 or 7, characterized in that, Iterate through each candidate identity feature record in the set of candidate identity feature records, and calculate the precise feature similarity between each record and the query condition to obtain a set of precise feature similarities, including: Concatenate all the subdivided perspective category feature query conditions in the query conditions to obtain the query condition concatenated encoding vector; The image feature vectors of multiple subdivided viewpoint categories corresponding to the query conditions in the candidate identity feature record are concatenated to obtain the candidate identity feature multimodal concatenated encoding vector; The query condition concatenated encoding vector and the candidate identity feature multimodal concatenated encoding vector are input into the precise similarity measurement network to obtain the precise similarity of the features.

10. An identity recognition system based on multimodal memory storage and dynamic matching, characterized in that, include: The database construction module is used to construct an identity feature multimodal database. Each identity feature record in the identity feature multimodal database contains an identity ID, a serial number, and a set of image feature vectors for subdivided viewpoint categories. The individual image acquisition module is used to acquire images of the individuals to be identified. The viewpoint category recognition module is used to input the image of the individual to be identified into the trained viewpoint pre-classification model to obtain the viewpoint category recognition result, and to construct query conditions based on the viewpoint category recognition result; The search query module is used to search the identity feature multimodal database according to the query conditions to obtain query results; The identification result confirmation module is used to count the frequency of each identity ID in the query results and select the identity ID with the highest frequency as the final identification result.

Images