Object recognition method and device, electronic equipment and storage medium

Abstract

The invention discloses an object recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a multi-view image of a to-be-recognized object, extracting a multi-view feature from the multi-view image, determining the multi-view matching similarity between the multi-view feature and the multi-view template feature of each object in a retrieval library through employing a multi-view matching strategy, and carrying out the recognition of the to-be-recognized object. And determining an identification result of the to-be-identified object based on each multi-view matching similarity. According to the method, the multi-view matching strategy is adopted, the multi-view features of the to-be-recognized object are matched with the multi-view template features of all the objects in the retrieval library, and accurate recognition of the to-be-recognized object is effectively achieved under the conditions that the angle change of the to-be-recognized object in the image is large, the view angle is random, and especially the to-be-recognized object is locally shielded.

Description

Technical Field

[0001] The present invention relates to the technical field of object recognition, and more specifically, to an object recognition method, device, electronic device and storage medium. Background Art

[0002] In life and production, objects (such as animals and people) need to be identified for various purposes such as business needs and safety needs. Existing object recognition is usually based on collected images, but when the object angle in the collected object image varies greatly, the viewing angle is random, or even partial occlusion occurs, it is easy to cause object recognition failure. Summary of the invention

[0003] In view of this, the present invention discloses an object recognition method, device, electronic device and storage medium, so as to achieve accurate recognition of the object to be recognized even when the angle of the object to be recognized in the image varies greatly and the viewing angle is random, especially when the object to be recognized is partially occluded.

[0004] An object recognition method, comprising:

[0005] Acquire multi-view images of the object to be identified;

[0006] Extracting multi-view features from the multi-view images;

[0007] Adopting a multi-view matching strategy, determining the multi-view matching similarity between the multi-view feature and the multi-view template feature of each object in the retrieval library;

[0008] The recognition result of the object to be recognized is determined based on the multi-view matching similarities.

[0009] Optionally, extracting multi-view features from the multi-view images includes:

[0010] Extracting the multi-view features from the multi-view images using a feature extraction network;

[0011] Wherein, the feature extraction network has: an explicit supervised spatial attention module and an automatic aligned cross attention module;

[0012] The explicit supervised spatial attention module is used to perform a masking operation in the feature extraction process to suppress the background area features of the multi-view image;

[0013] The automatic alignment cross-attention module is used to perform feature segmentation and align local area features in the multi-view image to obtain the multi-view features.

[0014] Optionally, the explicit supervised spatial attention module takes the intermediate features extracted by the backbone network from the multi-view image as input, obtains the local-global perceived attention vector after channel compression and deformation reorganization, and the attention vector is reorganized by a multi-layer perceptron and deformation to obtain spatial attention;

[0015] The automatic alignment cross-attention module is used to take the image features processed by the spatial attention as input, divide the image features into grid blocks, obtain multiple local area features of the multi-view image, calculate the cross-attention between the multiple local area features, and aggregate local features with similar semantics to the corresponding query, align the local area features, and obtain the multi-view features.

[0016] Optionally, the adopting of a multi-view matching strategy to determine the multi-view matching similarity between the multi-view feature and the multi-view template feature of each object in the retrieval library includes:

[0017] Using a multi-view matching strategy to determine the similarity between the multi-view feature and the multi-view template feature of each object in the search library, and obtaining a set of similarity values ​​corresponding to each object;

[0018] An average similarity value is calculated for the set of similarity values, and the obtained average similarity value is used as the multi-view matching similarity between the object to be identified and the corresponding object.

[0019] Optionally, extracting multi-view features from the multi-view images, and using a multi-view matching strategy to determine a multi-view matching similarity between the multi-view features and multi-view template features of each object in a retrieval library includes:

[0020] The multi-view features are extracted from the multi-view images using a pre-trained object recognition model, and a multi-view matching strategy is adopted to determine the multi-view matching similarity between the multi-view features and the multi-view template features of each object in the retrieval library.

[0021] Optionally, the training process of the object recognition model includes:

[0022] The first stage of self-supervised pre-training of the object recognition model is performed based on the ImageNet1K dataset to obtain the backbone network parameters and the parameters of the explicit supervised spatial attention module;

[0023] Acquire an object recognition dataset, wherein the object recognition dataset includes multiple individuals of each object and multi-view images of each individual, and each of the multi-view images is annotated with an object bounding box and a mask label;

[0024] Using the backbone network parameters and the explicit supervision spatial attention module parameters as initialization parameters, using the object recognition data set as a training data set, performing the second stage task head training of the object recognition model, and obtaining the automatic alignment cross attention module parameters;

[0025] The backbone network parameters, the explicit supervision spatial attention module parameters and the automatic alignment cross attention module parameters are used as initialization parameters, and the object recognition dataset is used as a training dataset to perform the third stage end-to-end training of the object recognition model to obtain the object recognition model.

[0026] Optionally, the determining the recognition result of the object to be recognized based on each of the multi-view matching similarities includes:

[0027] Taking each of the multi-view matching similarities as a final probability score that the object to be identified is the object corresponding to the multi-view matching similarity, and determining a maximum final probability score from all the final probability scores;

[0028] The target object corresponding to the maximum final probability score is determined as the recognition result of the object to be recognized.

[0029] Optionally, determining the recognition result of the object to be recognized based on the multi-view matching similarities includes:

[0030] Inputting the multi-view features into a customized classification module to obtain a classification score for each object in the search library by which the object to be identified is classified;

[0031] Taking a weighted average of the multi-view matching similarities and the classification scores corresponding to each of the objects, to obtain a final probability score of the object to be identified being the object;

[0032] Determine a maximum final probability score from all of the final probability scores;

[0033] The target object corresponding to the maximum final probability score is determined as the recognition result of the object to be recognized.

[0034] Optionally, determining the target object corresponding to the maximum final probability score as the recognition result of the object to be recognized includes:

[0035] Determine whether the maximum value of the final probability score is greater than a similarity threshold;

[0036] If yes, the target object corresponding to the maximum final probability score is determined as the recognition result of the object to be recognized.

[0037] Optionally, the training process of the customized classification module includes:

[0038] Obtain a customized fine-tuning dataset of the image to be identified, wherein the customized fine-tuning dataset includes: unlabeled images and labeled images;

[0039] When the semi-supervised training method is used for customized training, a pseudo label is assigned to each of the unlabeled images, and a true label of each of the labeled images is determined;

[0040] Extracting target multi-view features from the customized fine-tuning dataset, and performing customized classification on the target multi-view features to obtain target classification scores for each image to be identified in the customized fine-tuning dataset;

[0041] For the pseudo label of each of the unlabeled images, the true label of each of the labeled images, and the classification scores of each target, the cross entropy loss function is used to calculate the loss function value, and the parameters of the customized classification module are trained through the feedback propagation algorithm to obtain the customized classification module.

[0042] An object recognition device, comprising:

[0043] An image acquisition unit, used to acquire multi-view images of the object to be identified;

[0044] A feature extraction unit, used to extract multi-view features from the multi-view image;

[0045] A similarity matching unit, used to adopt a multi-view matching strategy to determine the multi-view matching similarity between the multi-view feature and the multi-view template features of each object in the retrieval library;

[0046] The recognition unit is used to determine the recognition result of the object to be recognized based on the similarities of each multi-view matching.

[0047] An electronic device, comprising: a memory and a processor;

[0048] The memory is used to store at least one instruction;

[0049] The processor is used to execute the at least one instruction to implement the object recognition method described above.

[0050] A computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the object recognition method described above is implemented.

[0051] It can be seen from the above technical solutions that the present invention discloses an object recognition method, device, electronic device and storage medium, which obtains a multi-view image of an object to be recognized, extracts multi-view features from the multi-view image, and determines the multi-view matching similarity between the multi-view features and the multi-view template features of each object in the retrieval library by adopting a multi-view matching strategy, so as to determine the recognition result of the object to be recognized based on each multi-view matching similarity. The present invention adopts a multi-view matching strategy, and by matching the multi-view features of the object to be recognized with the multi-view template features of each object in the retrieval library, it effectively solves the problem of accurate recognition of the object to be recognized when the angle of the object to be recognized in the image changes greatly and the perspective is random, especially when the object to be recognized is partially occluded. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without paying creative work.

[0053] Figure 1 A flow chart of an object recognition method disclosed in an embodiment of the present invention;

[0054] Figure 2 A structural diagram of a feature extraction network disclosed in an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of an object recognition process disclosed in an embodiment of the present invention;

[0056] Figure 4 A customized training, reasoning, and data collection flow chart disclosed in an embodiment of the present invention;

[0057] FIG5 (a) is a schematic diagram of the first stage of self-supervised pre-training of an object recognition model disclosed in an embodiment of the present invention;

[0058] FIG5( b ) is a schematic diagram of the second-stage task head training of an object recognition model disclosed in an embodiment of the present invention;

[0059] FIG5 (c) is a schematic diagram of the third stage end-to-end training of an object recognition model disclosed in an embodiment of the present invention;

[0060] Figure 6 A schematic diagram of the structure of an object recognition device disclosed in an embodiment of the present invention;

[0061] Figure 7 The present invention is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The embodiment of the present invention discloses an object recognition method, device, electronic device and storage medium, which obtains a multi-view image of an object to be recognized, extracts multi-view features from the multi-view image, and determines the multi-view matching similarity between the multi-view features and the multi-view template features of each object in a retrieval library by adopting a multi-view matching strategy, so as to determine the recognition result of the object to be recognized based on each multi-view matching similarity. The present invention adopts a multi-view matching strategy, and by matching the multi-view features of the object to be recognized with the multi-view template features of each object in the retrieval library, it effectively solves the problem of accurate recognition of the object to be recognized when the angle of the object to be recognized in the image changes greatly and the perspective is random, especially when the object to be recognized is partially occluded.

[0064] See also Figure 1 , a flow chart of an object recognition method disclosed in an embodiment of the present invention, the method comprising:

[0065] Step S101: Acquire multi-view images of an object to be identified.

[0066] The objects to be identified in this application include but are not limited to animals, people, vehicles, etc.

[0067] Multi-view images refer to multiple images of the object to be identified in the same scene captured from different viewpoints, wherein the posture of the object to be identified in the same scene can be fixed or not.

[0068] Step S102: extracting multi-view features from the multi-view image.

[0069] The present application extracts corresponding viewing features from each viewing image of a multi-view image, thereby obtaining multi-view features of an object to be identified, so that when the angle of the object to be identified in the image varies greatly and the viewing angle is random, especially when the object to be identified is partially occluded, the multi-view features can still well reflect the characteristics of the object to be identified.

[0070] Taking the object to be identified as an animal as an example, the multi-view features extracted from the multi-view image may be color features, texture features and shape features.

[0071] Color characteristics: Different animals may have different color characteristics at different viewing angles. For example, some animals may have bright color contrast at certain viewing angles, while the colors may be dull or difficult to distinguish at other viewing angles.

[0072] Texture features: Animal skin, hair or scales will show different texture features at different viewing angles. These texture features can be used to distinguish different types of animals or identify the same animal at different viewing angles.

[0073] Shape characteristics: the body size, head shape, limb length, etc. of an animal will change under different viewing angles, but some basic shape characteristics (such as head outline, tail shape, etc.) may remain consistent under multiple viewing angles.

[0074] Step S103: adopt a multi-view matching strategy to determine the multi-view matching similarity between the multi-view feature and the multi-view template features of each object in the retrieval library.

[0075] The present application stores multi-view template features of different objects in the retrieval library in advance. For example, the retrieval library stores multi-view template features of different breeds of cats, multi-view template features of different breeds of dogs, and so on.

[0076] In practical applications, multi-view images of each object can be entered into the retrieval library, and then the corresponding multi-view template features can be extracted and stored from the multi-view images corresponding to each object using a feature extraction network.

[0077] In the present application, the multi-perspective features of the object to be identified include multiple different perspective features, and the multi-perspective template features of each object in the retrieval library include multiple different perspective template features. By matching the multiple different perspective features of the object to be identified with the multiple different perspective template features of each object in the retrieval library using a multi-perspective matching strategy, the multi-perspective matching similarity of the object to be identified and each object in the retrieval library can be obtained.

[0078] By matching the multi-view features of the object to be identified with the multi-view template features of each object in the retrieval library using a multi-view matching strategy, we can effectively solve the problem of large angle changes and random perspectives of the object to be identified in the image, especially when the object to be identified is partially occluded.

[0079] Step S104: determining a recognition result of the object to be recognized based on the multi-view matching similarities.

[0080] Specifically, the recognition result of the object to be recognized may be determined according to the numerical values ​​of each multi-view matching similarity, including: taking the target object corresponding to the maximum multi-view matching similarity as the recognition result of the object to be recognized.

[0081] In summary, the present invention discloses an object recognition method, which obtains a multi-view image of an object to be recognized, extracts multi-view features from the multi-view image, and determines the multi-view matching similarity between the multi-view features and the multi-view template features of each object in a retrieval library by adopting a multi-view matching strategy, so as to determine the recognition result of the object to be recognized based on each multi-view matching similarity. The present invention adopts a multi-view matching strategy, and by matching the multi-view features of the object to be recognized with the multi-view template features of each object in the retrieval library, it effectively solves the problem of accurate recognition of the object to be recognized when the angle of the object to be recognized in the image changes greatly and the perspective is random, especially when the object to be recognized is partially occluded.

[0082] In one embodiment, step S102 may specifically include:

[0083] A feature extraction network is used to extract multi-view features from multi-view images.

[0084] Wherein, the feature extraction network has: an explicit supervised spatial attention module and an automatic aligned cross attention module;

[0085] The explicit supervised spatial attention module is used to perform a masking operation during the feature extraction process to suppress the background area features of the multi-view image.

[0086] The automatic alignment cross-attention module is used to perform feature segmentation and align local area features in the multi-view image to obtain the multi-view features.

[0087] For details, see Figure 2 The structure diagram of the feature extraction network shown in Figure 2 As can be seen from the figure, the explicitly supervised spatial attention module takes the intermediate features extracted by the backbone network from the multi-view image as input, and obtains the local-global perceived attention vector after channel compression and deformation (reshaping). The attention vector is reorganized through a multi-layer perceptron and deformation to obtain spatial attention.

[0088] During the feature extraction network training process, the learning goal of the explicit supervised spatial attention module is to assign high weights to pixel positions in the target area and low weights to background areas. Therefore, spatial attention enables the feature extraction network to suppress interference from background areas.

[0089] The automatic alignment cross-attention module is used to take the image features processed by the spatial attention as input, divide the image features into grid blocks, obtain multiple local area features of the multi-view image, calculate the cross-attention between the multiple local area features, and aggregate local features with similar semantics to the corresponding query, align the local area features, and obtain the multi-view features.

[0090] Specifically, the automatic alignment cross attention module takes the image features processed by spatial attention as input, first divides the image features into n local area features of multi-view images according to the grid, and in order to align the local area features when calculating the matching similarity, the automatic alignment cross attention module will generate a built-in parameter, namely Figure 2 The cross attention of m queries q is used in the training, and then the n local area features are used as the key k and value v of the cross attention, and the cross attention between the n local area features is calculated, and the local features with similar semantics are aggregated to the corresponding query q. Among them, the built-in parameter query q is randomly initialized at the beginning of training and is optimized together with other network parameters during the training process.

[0091] Preferably, the number n of local area features is 16, and the number m of queries q is 4.

[0092] The explicitly supervised spatial attention module and automatically aligned cross-attention module designed in the present invention effectively suppress problems caused by large changes in the object's posture, especially severe background interference (for example, severe background interference in the bounding box caused by large movements of the animal's tail and limbs) and misalignment of various parts of the object.

[0093] The three-stage training strategy of the feature extraction network designed in the present invention effectively improves the alignment effect of the automatic alignment cross-attention module.

[0094] The explicitly supervised spatial attention and automatically aligned cross-attention modules used in the feature extraction network of the present invention can further improve the performance by stacking or increasing the channel dimension.

[0095] In one embodiment, step S103 may specifically include:

[0096] Using a multi-view matching strategy to determine the similarity between the multi-view feature and the multi-view template feature of each object in the search library, and obtaining a set of similarity values ​​corresponding to each object;

[0097] An average similarity value is calculated for the set of similarity values, and the obtained average similarity value is used as the multi-view matching similarity between the object to be identified and the corresponding object.

[0098] See Figure 3As shown in the schematic diagram of the object recognition process, the retrieval library stores the multi-view template features of each object, adopts a multi-view matching strategy, and uses the cosine similarity index to determine the similarity between the multi-view features of the object to be identified and the multi-view template features of each object in the retrieval library, and obtains a set of similarity values ​​corresponding to each object.

[0099] For an example, see Figure 3 , adopting the multi-view matching strategy, using the cosine similarity algorithm to calculate the similarity between the multi-view features of the object to be identified and the multi-view template features of object 1, and obtain a set of similarity values, which are: S 11 , S 12 , …, S 1n , where n is the number of multi-view template features in object 1. 11 , S 12 , …, S 1n Remove the similarity average and get the similarity mean , and the similarity mean As the multi-view matching similarity between the object to be identified and object 1.

[0100] Adopting the multi-view matching strategy, the cosine similarity algorithm is used to calculate the similarity between the multi-view features of the object to be identified and the multi-view template features of object 2, and a set of similarity values ​​are obtained, which are: S 21 , S 22 , …, S 2n , where n is the number of multi-view template features in object 2. 21 , S 22 , …, S 2n Remove the similarity average and get the similarity mean , and the similarity mean As the multi-view matching similarity between the object to be identified and object 2.

[0101] Among them, the multi-view matching strategy designed by the present invention adopts the cosine similarity index to measure the matching degree between the object to be identified and each object in the retrieval library. The cosine similarity index can be replaced by other similarity indicators or distance metrics, such as Euclidean distance, Mahalanobis distance, Bhattacharyya distance, etc.

[0102] In one embodiment, step S104 may specifically include:

[0103] Taking each of the multi-view matching similarities as a final probability score that the object to be identified is the object corresponding to the multi-view matching similarity, and determining a maximum final probability score from all the final probability scores;

[0104] The target object corresponding to the maximum final probability score is determined as the recognition result of the object to be recognized.

[0105] from Figure 3 It can be seen that the multi-view matching strategy can be used to obtain the multi-view matching similarity between the object to be identified and each object in the retrieval library. Each multi-view matching similarity can be used as the final probability score between the object to be identified and the corresponding object, and the target object corresponding to the maximum value of the final probability score is determined as the recognition result of the object to be identified, that is, the recognition result of the object to be identified is: the object to be identified is the target object.

[0106] In order to improve the recognition accuracy of the object to be recognized, a similarity threshold th can be set to determine whether the maximum value of the final probability score is greater than the similarity threshold. If so, the target object corresponding to the maximum value of the final probability score is determined as the recognition result of the object to be recognized, and the recognition of the object to be recognized is determined to be successful.

[0107] On the contrary, when the maximum value of the final probability score is not greater than the similarity threshold, it is determined that the recognition of the object to be recognized has failed, that is, it is determined that the object to be recognized does not exist in the search library.

[0108] In one embodiment, the present application also provides a customized fine-tuning strategy to significantly improve the model's ability to identify key focus objects. When the focus object is relatively fixed, the model can continuously learn the object's surface features and adapt to the changes in the focus object's surface.

[0109] Based on this, step S104 may specifically include:

[0110] Inputting the multi-view features into a customized classification module to obtain a classification score for each object in the search library for the object to be identified;

[0111] Taking a weighted average of the multi-view matching similarities and the classification scores corresponding to each of the objects, to obtain a final probability score of the object to be identified being the object;

[0112] Determine a maximum final probability score from all of the final probability scores;

[0113] The target object corresponding to the maximum final probability score is determined as the recognition result of the object to be recognized.

[0114] For details, see Figure 3 , input the object to be identified into the customized classification module, and obtain the classification score of each object in the search library for the object to be identified. For example, the classification score of the object to be identified as object 1 in the search library is , the object to be identified is classified as object 2 in the retrieval library and the classification score is .

[0115] The multi-view matching similarity and classification score corresponding to each object are weighted averaged to obtain the final probability score of the object to be identified.

[0116] Assume that the weighted weight of multi-view matching similarity is α, the weighted weight of classification score is (1-α), and the multi-view matching similarity corresponding to object 1 is , the classification score corresponding to object 1 is , the multi-view matching similarity corresponding to object 1 is The product of α and the classification score corresponding to object 1 is Sum the product with (1-α) to get the final probability score that the object to be identified is object 1 .

[0117] The multi-view matching similarity corresponding to object 2 is , the classification score corresponding to object 2 is , the multi-view matching similarity corresponding to object 2 is The product of α and the classification score corresponding to object 2 is Sum the product with (1-α) to get the final probability score that the object to be identified is object 2 .

[0118] Maximize all final probability scores The corresponding target object is determined as the recognition result of the object to be recognized.

[0119] It should be noted that the weighted weight α is 1 before the customized fine-tuning is completed, indicating that the object to be identified is confirmed only by relying on the multi-view matching results. After the customized fine-tuning is completed, the weighted weight α is 0.5.

[0120] In order to improve the recognition accuracy of the object to be recognized, a similarity threshold th can be set to determine whether the maximum value of the final probability score is greater than the similarity threshold. If so, the target object corresponding to the maximum value of the final probability score is determined as the recognition result of the object to be recognized, and the recognition of the object to be recognized is determined to be successful.

[0121] On the contrary, when the maximum value of the final probability score is not greater than the similarity threshold, it is determined that the recognition of the object to be recognized has failed, that is, it is determined that the object to be recognized does not exist in the search library.

[0122] It should be noted that the customized fine-tuning strategy further designed in the present invention enables the entire object recognition model to continuously learn the characteristics of individual objects that require long-term attention, and strengthen the model's ability to distinguish the identities of specific individual objects. Figure 4As shown in the figure, during the algorithm operation, the customized fine-tuning process will be repeatedly executed at a certain time interval. The order of each execution is: first, customized data collection, then customized training process, after the customized training is completed, the new customized model can be used to execute the customized reasoning process.

[0123] The specific process of a round of customized fine-tuning is as follows:

[0124] (1) Obtain a customized fine-tuning dataset of images to be recognized, wherein the customized fine-tuning dataset includes: unlabeled images and labeled images.

[0125] Specifically, during the algorithm operation, the images to be identified are randomly sampled and randomly pushed to users at a certain ratio for object information annotation (such as animal identity annotation), and the "image-annotation" is stored as a labeled image in the customized fine-tuning dataset. Considering the feasibility, the number of images to be identified pushed to users should be very small, and most of the customized fine-tuning data are unlabeled images.

[0126] (2) When using a semi-supervised training method for customized training, a pseudo label is assigned to each of the unlabeled images, and a true label of each of the labeled images is determined.

[0127] Specifically, when the amount of data in the obtained customized fine-tuning dataset reaches the standard, the customized training process begins. Each customized training is performed for 60 training rounds, and the learning rate is fixed at 0.01. Since there are a large number of unlabeled images and a small number of labeled images in the customized fine-tuning dataset, the customized training adopts a semi-supervised training method. In each iteration, training data is sampled from labeled images and unlabeled images at a preset ratio. For labeled images, the real labels are used directly. For unlabeled images, pseudo labels need to be assigned to them first, and then the pseudo labels are used for training.

[0128] In order to assign pseudo labels to unlabeled images, firstly, feature extraction and customized classification are performed on the unlabeled images to obtain the scores of each category. When the maximum score is greater than the score threshold ts, the pseudo label corresponding to the maximum score is assigned to the unlabeled image. If the scores are all less than the score threshold ts, the unlabeled image is considered invalid and does not participate in this iterative training. The customized classification module in the present invention adopts a multi-layer perceptron composed of 3 fully connected layers, and the score threshold ts is 0.7.

[0129] (3) Extracting target multi-view features from the customized fine-tuning dataset, and performing customized classification on the target multi-view features to obtain a target classification score for each image to be identified in the customized fine-tuning dataset.

[0130] (4) For each pseudo label of the unlabeled image, the true label of each labeled image, and each target classification score, a cross entropy loss function is used to calculate the loss function value, and the parameters of the customized classification module are trained through the feedback propagation algorithm to obtain the customized classification module.

[0131] Among them, the parameters of the feature extraction module used to extract the target multi-view features from the customized fine-tuning dataset are fixed.

[0132] In actual applications, after the customized classification module is trained, the customized classification module can be used for object recognition. The complete reasoning process combines multi-view matching similarity and classification scores , weighted average is performed with the weight of α to obtain the final probability score of the object to be identified as the object, the maximum final probability score is determined from all the final probability scores, and the target object corresponding to the maximum final probability score is determined as the recognition result of the object to be identified.

[0133] It should be noted that the customized fine-tuning of the present invention adopts a semi-supervised training method, which only requires a small amount of annotation to effectively and continuously learn the apparent features of the focus target, thereby further improving the performance of the entire object recognition model.

[0134] Among them, the semi-supervised training method used in the customized fine-tuning strategy of the present invention can be replaced by other semi-supervised training methods, such as FixMatch, MixMatch, FreeMatch, etc.

[0135] In one embodiment, step S103 may specifically include:

[0136] The multi-view matching strategy is adopted using a pre-trained object recognition model to determine the multi-view matching similarity between the multi-view features and the multi-view template features of each object in the retrieval library.

[0137] The training process of the object recognition model is divided into three stages: self-supervised pre-training stage, task head training stage, and end-to-end training stage.

[0138] The schematic diagrams of the three-stage training are shown in Figure 5 (a), Figure 5 (b) and Figure 5 (c) respectively. The overall process is as follows:

[0139] (1) The first stage of self-supervised pre-training of the object recognition model is performed based on the ImageNet1K dataset to obtain the backbone network parameters and the parameters of the explicit supervised spatial attention module.

[0140] The self-supervised pre-training method adopted by the present invention is the BarlowTwins algorithm, which only requires positive sample pairs and does not require the use of negative sample pairs for training. The feature dimension of the BarlowTwins algorithm used to calculate the contrast loss is 8192, the number of batch data for each iteration is 256, and the training rounds total 300 rounds. In one training iteration, the same input image is first data enhanced twice, and then the features of the enhanced image are extracted through the backbone network, explicit supervision spatial attention module, global average pooling and fully connected layer, and then the features of the same image and different data enhancements are used to construct positive sample pairs, and the contrast loss of all positive sample pairs in this iteration is calculated, and the model is trained based on the back-propagation algorithm. For a more detailed introduction to the training method, please refer to the prior art BarlowTwins self-supervised pre-training.

[0141] The BarlowTwins algorithm is a self-supervised learning algorithm that aims to learn a common representation that is invariant to input variations by reducing redundancy between output representations.

[0142] Among them, the BarlowTwins self-supervised pre-training method can be replaced by other pre-training methods, such as fully supervised pre-training, MOCO self-supervised pre-training, SimCLR self-supervised pre-training, BYOL self-supervised pre-training, etc.

[0143] The ImageNet1K dataset has a hierarchical directory structure, where each category corresponds to a subdirectory that contains all images belonging to that category. This organizational structure makes data access and processing relatively simple and efficient.

[0144] (2) Obtain an object recognition dataset, where the object recognition dataset includes multiple individuals of each object and multi-view images of each individual, and each of the multi-view images is annotated with an object bounding box and a mask label.

[0145] The current training data for object recognition is limited in both quantity and quality. Therefore, before starting subsequent training, data collection that meets the application scenario is required. The data set is required to cover multiple objects (for example, multiple animals), each object has multiple individuals (for example, each animal has multiple individuals), each individual has images from multiple angles and multiple postures, and the bounding box and mask label of the object in the image are annotated. Since the mask annotation project is large, the present invention adopts an interactive annotation method, that is, evenly annotating several points on the object, and then automatically generating masks with the help of a large segmentation model such as SAM (Segment Anything Model).

[0146] (3) Using the backbone network parameters and the explicit supervised spatial attention module parameters as initialization parameters and the object recognition dataset as a training dataset, the second stage task head training of the object recognition model is performed to obtain the automatic alignment cross attention module parameters.

[0147] Specifically, the backbone network and explicit supervision spatial attention module parameters obtained from the first stage of training are used as initialization parameters, and the obtained object recognition dataset is used as the training dataset to start the second stage of task head training. During the task head training process, the backbone network parameters and the explicit supervision spatial attention module parameters are kept fixed, and only the parameters of the automatic alignment cross attention module are adjusted. In each training iteration, N anchor point samples are first sampled, and then a positive sample and b negative sample are sampled for each anchor point sample to form the batch data of this iteration. Then, features are extracted from these samples respectively, and positive and negative sample pairs are constructed. The triplet loss is calculated, and the network training is supervised to bring the positive sample pairs closer and push the negative sample pairs farther in the feature space.

[0148] The triplet loss used in this invention is defined as:

[0149]

[0150] In the formula, is the triplet loss, is the cosine distance between the i-th anchor point sample and the a positive samples sampled in this iteration, is the cosine distance between the anchor point sample and the b negative samples sampled, N is the number of anchor point samples, represents a real matrix with 1 row and a columns, represents a real matrix with 1 row and b columns.

[0151] In the present invention, N is 256, a is 5, b is 15, the number of training rounds is 10, the random gradient descent method is used for training, the learning rate is 0.2, and the triple loss weight is 0.001.

[0152] (4) Using the backbone network parameters, the explicit supervision spatial attention module parameters and the automatic alignment cross attention module parameters as initialization parameters, and using the object recognition dataset as a training dataset, performing the third stage end-to-end training of the object recognition model to obtain the object recognition model.

[0153] The backbone network parameters and explicit supervision spatial attention module parameters obtained from the first stage training, as well as the cross-attention module parameters obtained from the second stage training are used as initialization parameters, and the acquired object recognition data set is used as the training data set to start the third stage end-to-end training. In the end-to-end training process, all network parameters are trained simultaneously, and mask label information is added on the basis of the second stage loss function. MSE (Mean Squared Error) loss is used to supervise the spatial attention module to learn the ability to focus on the target area and suppress background interference. The weight of MSE loss in the present invention is 1.

[0154] After the above three stages of training are completed, the object recognition model designed by the present invention has the ability to recognize objects. After inputting the multi-view image of the object to be recognized, the object recognition result can be obtained through the multi-view matching steps (i.e., feature extraction and cosine similarity calculation).

[0155] Corresponding to the above method embodiment, the present invention also discloses an object recognition device.

[0156] See also Figure 6 , a schematic diagram of the structure of an object recognition device disclosed in an embodiment of the present invention, the device may include:

[0157] An image acquisition unit 201 is used to acquire multi-view images of an object to be identified;

[0158] The objects to be identified in this application include but are not limited to animals, people, vehicles, etc.

[0159] Multi-view images refer to multiple images of the object to be identified in the same scene captured from different viewpoints, wherein the posture of the object to be identified in the same scene can be fixed or not.

[0160] A feature extraction unit 202, configured to extract multi-view features from the multi-view image;

[0161] The present application extracts corresponding viewing features from each viewing image of a multi-view image, thereby obtaining multi-view features of an object to be identified, so that when the angle of the object to be identified in the image varies greatly and the viewing angle is random, especially when the object to be identified is partially occluded, the multi-view features can still well reflect the characteristics of the object to be identified.

[0162] Taking the object to be identified as an animal as an example, the multi-view features extracted from the multi-view image may be color features, texture features and shape features.

[0163] A similarity matching unit 203 is used to adopt a multi-view matching strategy to determine the multi-view matching similarity between the multi-view feature and the multi-view template features of each object in the search library;

[0164] The present application stores multi-view template features of different objects in the retrieval library in advance. For example, the retrieval library stores multi-view template features of different breeds of cats, multi-view template features of different breeds of dogs, and so on.

[0165] In practical applications, multi-view images of each object can be entered into the retrieval library, and then the corresponding multi-view template features can be extracted and stored from the multi-view images corresponding to each object using a feature extraction network.

[0166] In the present application, the multi-perspective features of the object to be identified include multiple different perspective features, and the multi-perspective template features of each object in the retrieval library include multiple different perspective template features. By matching the multiple different perspective features of the object to be identified with the multiple different perspective template features of each object in the retrieval library using a multi-perspective matching strategy, the multi-perspective matching similarity of the object to be identified and each object in the retrieval library can be obtained.

[0167] By matching the multi-view features of the object to be identified with the multi-view template features of each object in the retrieval library using a multi-view matching strategy, we can effectively solve the problem of large angle changes and random perspectives of the object to be identified in the image, especially when the object to be identified is partially occluded.

[0168] The recognition unit 204 is used to determine the recognition result of the object to be recognized based on the multi-view matching similarities.

[0169] Specifically, the recognition result of the object to be recognized may be determined according to the numerical values ​​of each multi-view matching similarity, including: taking the target object corresponding to the maximum multi-view matching similarity as the recognition result of the object to be recognized.

[0170] In summary, the present invention discloses an object recognition device, which obtains a multi-view image of an object to be recognized, extracts multi-view features from the multi-view image, and determines the multi-view matching similarity between the multi-view features and the multi-view template features of each object in a retrieval library by adopting a multi-view matching strategy, so as to determine the recognition result of the object to be recognized based on each multi-view matching similarity. The present invention adopts a multi-view matching strategy, and by matching the multi-view features of the object to be recognized with the multi-view template features of each object in the retrieval library, it effectively solves the problem of accurate recognition of the object to be recognized when the angle of the object to be recognized in the image changes greatly and the perspective is random, especially when the object to be recognized is partially occluded.

[0171] In one embodiment, the feature extraction unit 202 is further configured to:

[0172] Extracting the multi-view features from the multi-view images using a feature extraction network;

[0173] Wherein, the feature extraction network has: an explicit supervised spatial attention module and an automatic aligned cross attention module;

[0174] The explicit supervised spatial attention module is used to perform a masking operation in the feature extraction process to suppress the background area features of the multi-view image;

[0175] The automatic alignment cross-attention module is used to perform feature segmentation and align local area features in the multi-view image to obtain the multi-view features.

[0176] The explicit supervised spatial attention module uses the intermediate features extracted from the multi-view images by the backbone network as input, obtains the local-global perceived attention vector after channel compression and deformation reorganization, and obtains the spatial attention after the attention vector is reorganized by a multi-layer perceptron and deformation;

[0177] The automatic alignment cross-attention module is used to take the image features processed by the spatial attention as input, divide the image features into grid blocks, obtain multiple local area features of the multi-view image, calculate the cross-attention between the multiple local area features, and aggregate local features with similar semantics to the corresponding query, align the local area features, and obtain the multi-view features.

[0178] In one embodiment, the similarity matching unit 203 is further used to:

[0179] Using a multi-view matching strategy to determine the similarity between the multi-view feature and the multi-view template feature of each object in the search library, and obtaining a set of similarity values ​​corresponding to each object;

[0180] An average similarity value is calculated for the set of similarity values, and the obtained average similarity value is used as the multi-view matching similarity between the object to be identified and the corresponding object.

[0181] In one embodiment, the feature extraction unit 202 is further configured to:

[0182] The multi-view features are extracted from the multi-view images using a pre-trained object recognition model.

[0183] In one embodiment, the similarity matching unit 203 is further used to:

[0184] Using a pre-trained object recognition model and a multi-view matching strategy, the multi-view matching similarity between the multi-view feature and the multi-view template feature of each object in the retrieval library is determined, including:

[0185] The multi-view features are extracted from the multi-view images using a pre-trained object recognition model, and a multi-view matching strategy is adopted to determine the multi-view matching similarity between the multi-view features and the multi-view template features of each object in the retrieval library.

[0186] In one embodiment, the object recognition device further includes: a recognition model training module.

[0187] The recognition model training module is used to:

[0188] The first stage of self-supervised pre-training of the object recognition model is performed based on the ImageNet1K dataset to obtain the backbone network parameters and the parameters of the explicit supervised spatial attention module;

[0189] Acquire an object recognition dataset, wherein the object recognition dataset includes multiple individuals of each object and multi-view images of each individual, and each of the multi-view images is marked with an object bounding box and a mask label;

[0190] Using the backbone network parameters and the explicit supervision spatial attention module parameters as initialization parameters, using the object recognition data set as a training data set, performing the second stage task head training of the object recognition model, and obtaining the automatic alignment cross attention module parameters;

[0191] The backbone network parameters, the explicit supervision spatial attention module parameters and the automatic alignment cross attention module parameters are used as initialization parameters, and the object recognition dataset is used as a training dataset to perform the third stage end-to-end training of the object recognition model to obtain the object recognition model.

[0192] In one embodiment, the identification unit 204 is further configured to:

[0193] Taking each of the multi-view matching similarities as a final probability score that the object to be identified is the object corresponding to the multi-view matching similarity, and determining a maximum final probability score from all the final probability scores;

[0194] The target object corresponding to the maximum final probability score is determined as the recognition result of the object to be recognized.

[0195] In one embodiment, the identification unit 204 is further configured to:

[0196] Inputting the multi-view features into a customized classification module to obtain a classification score of each object in the search library for the object to be identified;

[0197] Taking a weighted average of the multi-view matching similarities and the classification scores corresponding to each of the objects, to obtain a final probability score of the object to be identified being the object;

[0198] Determine a maximum final probability score from all of the final probability scores;

[0199] The target object corresponding to the maximum final probability score is determined as the recognition result of the object to be recognized.

[0200] In one embodiment, the identification unit 204 is further configured to:

[0201] Determine whether the maximum value of the final probability score is greater than a similarity threshold;

[0202] If yes, the target object corresponding to the maximum final probability score is determined as the recognition result of the object to be recognized.

[0203] In one embodiment, the object recognition device may further include: a customized classification module training unit.

[0204] The customized classification module training unit is also used for:

[0205] Obtain a customized fine-tuning dataset of the image to be identified, wherein the customized fine-tuning dataset includes: unlabeled images and labeled images;

[0206] When the semi-supervised training method is used for customized training, a pseudo label is assigned to each of the unlabeled images, and a true label of each of the labeled images is determined;

[0207] Extracting target multi-view features from the customized fine-tuning dataset, and performing customized classification on the target multi-view features to obtain target classification scores for each image to be identified in the customized fine-tuning dataset;

[0208] For the pseudo label of each of the unlabeled images, the true label of each of the labeled images, and the classification scores of each target, the cross entropy loss function is used to calculate the loss function value, and the parameters of the customized classification module are trained through the feedback propagation algorithm to obtain the customized classification module.

[0209] Corresponding to the above embodiment, Figure 7 As shown, the present invention also discloses an electronic device, which may include: a processor 1 and a memory 2;

[0210] The processor 1 and the memory 2 communicate with each other via a communication bus 3;

[0211] Processor 1, configured to execute at least one instruction;

[0212] Memory 2, used to store at least one instruction;

[0213] The processor 1 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0214] The memory 2 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0215] The processor executes at least one instruction to implement the steps shown in the object recognition method embodiment.

[0216] Corresponding to the above embodiment, the present invention further discloses a computer-readable storage medium, which stores at least one instruction. When the at least one instruction is executed by a processor, the steps shown in the object recognition method embodiment are implemented.

[0217] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0218] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0219] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An object recognition method, characterized in that: include: Acquire multi-view images of the object to be identified; Extracting multi-view features from the multi-view images; Adopting a multi-view matching strategy, determining the multi-view matching similarity between the multi-view feature and the multi-view template feature of each object in the retrieval library; The recognition result of the object to be recognized is determined based on the multi-view matching similarities.

2. The object recognition method according to claim 1, characterized in that: The extracting multi-view features from the multi-view images comprises: Extracting the multi-view features from the multi-view images using a feature extraction network; Wherein, the feature extraction network has: an explicit supervised spatial attention module and an automatic aligned cross attention module; The explicit supervised spatial attention module is used to perform a masking operation in the feature extraction process to suppress the background area features of the multi-view image; The automatic alignment cross-attention module is used to perform feature segmentation and align local area features in the multi-view image to obtain the multi-view features.

3. The object recognition method according to claim 2, characterized in that: The explicit supervision spatial attention module takes the intermediate features extracted from the multi-view images by the backbone network as input, obtains the local-global perceived attention vector after channel compression and deformation reorganization, and obtains the spatial attention after the attention vector is reorganized by a multi-layer perceptron and deformation; The automatic alignment cross-attention module is used to take the image features processed by the spatial attention as input, divide the image features into grid blocks, obtain multiple local area features of the multi-view image, calculate the cross-attention between the multiple local area features, and aggregate local features with similar semantics to the corresponding query, align the local area features, and obtain the multi-view features.

4. The object recognition method according to claim 1, characterized in that: The multi-view matching strategy is adopted to determine the multi-view matching similarity between the multi-view feature and the multi-view template feature of each object in the search library, including: Using a multi-view matching strategy to determine the similarity between the multi-view feature and the multi-view template feature of each object in the search library, and obtaining a set of similarity values ​​corresponding to each object; An average similarity value is calculated for the set of similarity values, and the obtained average similarity value is used as the multi-view matching similarity between the object to be identified and the corresponding object.

5. The object recognition method according to any one of claims 1 to 4, characterized in that: The step of extracting multi-view features from the multi-view image and using a multi-view matching strategy to determine the multi-view matching similarity between the multi-view features and the multi-view template features of each object in the retrieval library includes: The multi-view features are extracted from the multi-view images using a pre-trained object recognition model, and a multi-view matching strategy is adopted to determine the multi-view matching similarity between the multi-view features and the multi-view template features of each object in the retrieval library.

6. The object recognition method according to claim 5, characterized in that: The training process of the object recognition model includes: The first stage of self-supervised pre-training of the object recognition model is performed based on the ImageNet1K dataset to obtain the backbone network parameters and the parameters of the explicit supervised spatial attention module; Acquire an object recognition dataset, wherein the object recognition dataset includes multiple individuals of each object and multi-view images of each individual, and each of the multi-view images is marked with an object bounding box and a mask label; Using the backbone network parameters and the explicit supervision spatial attention module parameters as initialization parameters, using the object recognition data set as a training data set, performing the second stage task head training of the object recognition model, and obtaining the automatic alignment cross attention module parameters; The backbone network parameters, the explicit supervision spatial attention module parameters and the automatic alignment cross attention module parameters are used as initialization parameters, and the object recognition dataset is used as a training dataset to perform the third stage end-to-end training of the object recognition model to obtain the object recognition model.

7. The object recognition method according to claim 1, characterized in that: The determining the recognition result of the object to be recognized based on the multi-view matching similarities includes: Taking each of the multi-view matching similarities as a final probability score that the object to be identified is the object corresponding to the multi-view matching similarity, and determining a maximum final probability score from all the final probability scores; The target object corresponding to the maximum final probability score is determined as the recognition result of the object to be recognized.

8. The object recognition method according to claim 1, characterized in that: The determining the recognition result of the object to be recognized based on the multi-view matching similarities includes: Inputting the multi-view features into a customized classification module to obtain a classification score of each object in the search library for the object to be identified; Taking a weighted average of the multi-view matching similarities and the classification scores corresponding to each of the objects, to obtain a final probability score of the object to be identified being the object; Determine a maximum final probability score from all of the final probability scores; The target object corresponding to the maximum final probability score is determined as the recognition result of the object to be recognized.

9. The object recognition method according to claim 7 or 8, characterized in that: The step of determining the target object corresponding to the maximum final probability score as the recognition result of the object to be recognized includes: Determine whether the maximum value of the final probability score is greater than a similarity threshold; If yes, the target object corresponding to the maximum final probability score is determined as the recognition result of the object to be recognized.

10. The object recognition method according to claim 8, characterized in that: The training process of the customized classification module includes: Obtain a customized fine-tuning dataset of the image to be identified, wherein the customized fine-tuning dataset includes: unlabeled images and labeled images; When the semi-supervised training method is used for customized training, a pseudo label is assigned to each of the unlabeled images, and a true label of each of the labeled images is determined; Extracting target multi-view features from the customized fine-tuning dataset, and performing customized classification on the target multi-view features to obtain target classification scores for each image to be identified in the customized fine-tuning dataset; For the pseudo label of each of the unlabeled images, the true label of each of the labeled images, and the classification scores of each target, the cross entropy loss function is used to calculate the loss function value, and the parameters of the customized classification module are trained through the feedback propagation algorithm to obtain the customized classification module.

11. An object recognition device, characterized in that: include: An image acquisition unit, used to acquire multi-view images of the object to be identified; A feature extraction unit, used to extract multi-view features from the multi-view image; A similarity matching unit, used to adopt a multi-view matching strategy to determine the multi-view matching similarity between the multi-view feature and the multi-view template features of each object in the retrieval library; The recognition unit is used to determine the recognition result of the object to be recognized based on the similarities of each multi-view matching.

12. An electronic device, characterized in that: The electronic device comprises: a memory and a processor; The memory is used to store at least one instruction; The processor is used to execute the at least one instruction to implement the object recognition method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by the processor, the object recognition method according to any one of claims 1 to 10 is implemented.

Images