Object recognition model processing method and device, equipment, medium and program product

Abstract

The invention relates to an object recognition model processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a to-be-trained student model containing a double-branch feature extraction network, wherein the double-branch feature extraction network comprises a first feature extraction branch and a second feature extraction branch; respectively extracting style invariant features and identity discrimination features through a first feature extraction branch and a second feature extraction branch based on the sample image, then carrying out feature fusion, determining target features, and determining a first object prediction result based on the target features; calling an object shape feature extraction model, and determining an object shape feature based on the sample image so as to determine a second object prediction result; and on the basis of the target feature, the object shape feature, the first object prediction result and the second object prediction result, constructing the target loss so as to perform model training on the to-be-trained student model to obtain the object recognition model, thereby improving the accuracy of object recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to an object recognition model processing method and device, computer equipment, computer readable storage medium, and computer program product. BACKGROUND

[0002] With the continuous development of computer technology and Internet technology, the application scenarios of various visual recognition tasks are increasingly rich and complex, and object recognition has gradually become a core research direction in the field of computer vision that is extremely challenging and highly concerned, especially in the aspect of pedestrian re-identification, which exhibits important application value and research significance.

[0003] In related technologies, a model training is usually performed by using a modality generation method (such as MMN or X-modality) to obtain an object recognition model, for example, by aligning modality differences through generated intermediate modality images, using cross-modality paired data as a supervision signal to perform model training. However, when paired data is lacking or the amount of data is insufficient, it is difficult to ensure the recognition effect of the object recognition model, resulting in low accuracy of object recognition. SUMMARY

[0004] Therefore, it is necessary to provide an object recognition model processing method, device, computer equipment, computer readable storage medium, and computer program product that can improve the recognition effect of an object recognition model and improve the accuracy of object recognition.

[0005] In a first aspect, the present application provides an object recognition model processing method, comprising:

[0006] obtaining a student model to be trained containing a double-branch feature extraction network, wherein the double-branch feature extraction network comprises a first feature extraction branch and a second feature extraction branch;

[0007] based on a sample image, extracting style-invariant features and identity discriminative features through the first feature extraction branch and the second feature extraction branch respectively, fusing the style-invariant features and the identity discriminative features to determine target features corresponding to the sample image, and determining a first object prediction result about the sample image based on the target features;

[0008] calling an object shape feature extraction model as a teacher model, determining object shape features based on the sample image, and determining a second object prediction result about the sample image based on the object shape features;

[0009] constructing a target loss based on the target features, the object shape features, the first object prediction result, and the second object prediction result;

[0010] The student model to be trained is trained based on the target loss, and the trained student model is taken as an object recognition model used for object recognition.

[0011] In a second aspect, the present application further provides a processing device of an object recognition model, comprising:

[0012] The learning model acquisition module is configured to acquire a student model to be trained containing a double-branch feature extraction network, wherein the double-branch feature extraction network comprises a first feature extraction branch and a second feature extraction branch.

[0013] The first object prediction module is configured to extract style-invariant features and identity discriminative features from the sample image respectively through the first feature extraction branch and the second feature extraction branch, fuse the style-invariant features and the identity discriminative features, determine target features corresponding to the sample image, and determine a first object prediction result about the sample image based on the target features.

[0014] The second object prediction module is configured to call an object shape feature extraction model as a teacher model, determine object shape features based on the sample image, and determine a second object prediction result about the sample image based on the object shape features.

[0015] The target loss construction module is configured to construct a target loss based on the target features, the object shape features, the first object prediction result and the second object prediction result.

[0016] The model training module is configured to train the student model to be trained based on the target loss, and take the trained student model as an object recognition model used for object recognition.

[0017] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0018] The student model to be trained containing a double-branch feature extraction network is acquired, wherein the double-branch feature extraction network comprises a first feature extraction branch and a second feature extraction branch.

[0019] Style-invariant features and identity discriminative features are extracted from the sample image respectively through the first feature extraction branch and the second feature extraction branch, the style-invariant features and the identity discriminative features are fused, target features corresponding to the sample image are determined, and a first object prediction result about the sample image is determined based on the target features.

[0020] calling an object shape feature extraction model as a teacher model, determining an object shape feature based on the sample image, determining a second object prediction result about the sample image based on the object shape feature;

[0021] constructing a target loss based on the target feature, the object shape feature, the first object prediction result, and the second object prediction result;

[0022] performing model training on the to-be-trained student model based on the target loss, and taking the trained student model as an object recognition model, wherein the object recognition model is used for object recognition.

[0023] In a fourth aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the following steps:

[0024] obtaining a to-be-trained student model containing a double-branch feature extraction network, wherein the double-branch feature extraction network comprises a first feature extraction branch and a second feature extraction branch;

[0025] based on a sample image, extracting a style-invariant feature and an identity discriminative feature through the first feature extraction branch and the second feature extraction branch respectively, fusing the style-invariant feature and the identity discriminative feature to determine a target feature corresponding to the sample image, and determining a first object prediction result about the sample image based on the target feature;

[0026] calling an object shape feature extraction model as a teacher model, determining an object shape feature based on the sample image, determining a second object prediction result about the sample image based on the object shape feature;

[0027] constructing a target loss based on the target feature, the object shape feature, the first object prediction result, and the second object prediction result;

[0028] performing model training on the to-be-trained student model based on the target loss, and taking the trained student model as an object recognition model, wherein the object recognition model is used for object recognition.

[0029] In a fifth aspect, the present application further provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the following steps:

[0030] obtaining a to-be-trained student model containing a double-branch feature extraction network, wherein the double-branch feature extraction network comprises a first feature extraction branch and a second feature extraction branch;

[0031] The style-invariant feature and the identity discriminative feature are respectively extracted based on the sample image through the first feature extraction branch and the second feature extraction branch, the style-invariant feature and the identity discriminative feature are fused, the target feature corresponding to the sample image is determined, and the first object prediction result about the sample image is determined based on the target feature;

[0032] An object shape feature extraction model serving as a teacher model is called, an object shape feature is determined based on the sample image, and the second object prediction result about the sample image is determined based on the object shape feature;

[0033] The target loss is constructed based on the target feature, the object shape feature, the first object prediction result and the second object prediction result;

[0034] The student model to be trained is subjected to model training based on the target loss, and the trained student model serves as an object recognition model, which is used for object recognition.

[0035] The processing method, device, computer equipment, computer readable storage medium and computer program product of the object recognition model are capable of obtaining a student model to be trained containing a double-branch feature extraction network, the double-branch feature extraction network including a first feature extraction branch and a second feature extraction branch. The style-invariant feature and the identity discriminative feature are respectively extracted based on the sample image through the first feature extraction branch and the second feature extraction branch. Then, the style-invariant feature with unified style and the identity discriminative feature with identity discriminative details are obtained by fusing and filtering noise, which can realize the reservation of identity-related features and the directional filtering of noise, ensure that the target feature corresponding to the sample image can better learn the object prediction ability to determine the first object prediction result about the sample image. An object shape feature extraction model serving as a teacher model is called, an object shape feature is determined based on the sample image, and the second object prediction result about the sample image is determined based on the object shape feature. The target loss is constructed based on the target feature, the object shape feature, the first object prediction result and the second object prediction result. The student model to be trained is subjected to model training based on the target loss, which can not only learn from the features obtained from the two feature extraction branches for model self-learning, but also transfer the discriminative knowledge of the object shape space of the teacher model to the student model to be trained. In this way, without considering the data amount of the paired data, the model training effect can be ensured, and therefore, after the trained student model serves as an object recognition model, accurate object recognition can be performed based on the object recognition model with good training effect, and the accuracy of object recognition is improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the related art. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained from these drawings without creative labor.

[0037] Figure 1 An application environment diagram of the processing method of the object recognition model in one embodiment;

[0038] Figure 2 A flowchart of the processing method of the object recognition model in one embodiment;

[0039] Figure 3 A model architecture diagram of the student model to be trained in one embodiment;

[0040] Figure 4 A schematic diagram of the target feature determination process in one embodiment;

[0041] Figure 5 A specific model structure diagram of the object recognition model in one embodiment;

[0042] Figure 6 A structure block diagram of the processing device of the object recognition model in one embodiment;

[0043] Figure 7 An internal structure diagram of the computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purposes, technical solutions and advantages of the present application clearer, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0045] It should be noted that the terms "first", "second" and the like used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "multiple" used in the present application refers to two and more than two. The term "and / or" used in the present application refers to one of the solutions, or any combination of multiple solutions.

[0046] The processing method of the object recognition model provided by the embodiments of the present application can be applied to, for example Figure 1The application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The object recognition model processing method provided by the embodiments of the application can be executed by the terminal 102 or the server 104 alone, or by the terminal 102 and the server 104 cooperatively, and the specific implementation is not limited.

[0047] In one embodiment, the server 104 acquires a student model to be trained containing a double-branch feature extraction network, the double-branch feature extraction network including a first feature extraction branch and a second feature extraction branch; the server 104 extracts style-invariant features and identity discriminant features based on a sample image through the first feature extraction branch and the second feature extraction branch respectively, fuses the style-invariant features and the identity discriminant features, determines target features corresponding to the sample image, and determines a first object prediction result about the sample image based on the target features; calls an object shape feature extraction model as a teacher model, determines object shape features based on the sample image, and determines a second object prediction result about the sample image based on the object shape features; constructs a target loss based on the target features, the object shape features, the first object prediction result and the second object prediction result; the server 104 performs model training on the student model to be trained based on the target loss, and uses the trained student model as an object recognition model. The server 104 sends the object recognition model to the terminal 102 or stores it locally on the server 104, so that after receiving an object image to be recognized, the object recognition model can be called to perform object recognition on the object image to be recognized.

[0048] The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, unmanned aerial vehicles, low-altitude flying vehicles, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0049] In one exemplary embodiment, as Figure 2 shown, an object recognition model processing method is provided. The method is applied to a computer device (which can be a terminal 102 or a server 104). Figure 1The method is described by taking a terminal 102 (which can also be a server 104) as an example, and includes the following steps 202 to 210. Among them:

[0050] At step 202, a to-be-trained student model including a double-branch feature extraction network is obtained, and the double-branch feature extraction network includes a first feature extraction branch and a second feature extraction branch.

[0051] The to-be-trained student model is a to-be-trained object recognition model, and the double-branch feature extraction network includes two feature extraction branches. Illustratively, the double-branch feature extraction network can be a residual modal mitigation network. Illustratively, the double-branch feature extraction network includes a first feature extraction branch and a second feature extraction branch. The first feature extraction branch is used to extract style-invariant features, i.e., to learn and extract features that are invariant to image style and appearance, thereby suppressing modal-related noise. The modal can be understood as a mode, such as a visible light mode or an infrared mode, etc. Illustratively, the first feature extraction branch can be an instance normalization (IN) branch. The second feature extraction branch is used to extract discriminative features, i.e., features used to identify identity information, such as object identification. Illustratively, the second feature extraction branch can be a batch normalization (BN) branch.

[0052] In one embodiment, the to-be-trained student model can be a model constructed based on a Resnet (Residual neural network) architecture. Illustratively, the double-branch feature extraction network is added to the Resnet architecture to form the to-be-trained student model. Illustratively, as shown in FIG. 1, it is a model architecture diagram of the to-be-trained student model in one embodiment. The to-be-trained student model includes a pre-sequence feature extraction network, a double-branch feature extraction network, and a post-sequence feature extraction network connected in sequence. The pre-sequence feature extraction network can be used to extract basic features, i.e., directly extract basic features from the input image, and the post-sequence feature extraction network can be used to extract high-level features. The pre-sequence feature extraction network and the post-sequence feature extraction network belong to the network in the Resnet architecture. Figure 3

[0053] In one embodiment, the computer device directly obtains an initial model constructed based on Resnet, which can be a trained model. Then, the double-branch feature extraction network is inserted into the initial model to obtain the to-be-trained student model.

[0054] ​At step 204, based on the sample image, style-invariant features and identity discriminative features are respectively extracted through the first feature extraction branch and the second feature extraction branch, the style-invariant features and the identity discriminative features are fused, target features corresponding to the sample image are determined, and based on the target features, a first object prediction result about the sample image is determined.

[0055] The sample image is a single-modal image, that is, the sample image is an image obtained by photographing an object in a scene through a single-modal camera. The single-modal camera refers to a camera that only photographs a single mode, for example, a visible light mode camera, and the corresponding photographed sample image is a visible light mode image, for example, an infrared mode camera, and the corresponding photographed sample image is an infrared mode image. Therefore, it can be understood that the object recognition model trained by the embodiments of the present application is an object recognition process in a completely modal missing scene. The completely modal missing refers to a situation in which, in a long-time detection system, due to the automatic switching of visible light and infrared modes by the camera according to the lighting conditions, only a single mode (single mode, for example, visible light mode or infrared mode) is used to capture the same identity object.

[0056] The first object prediction result is a result of recognizing the object in the sample image by the student model to be trained.

[0057] In one embodiment, the computer device inputs the sample image into the pre-feature extraction network of the student model to be trained to obtain a feature map, inputs the feature map into the first feature extraction branch to obtain the style-invariant features, and inputs the feature map into the second feature extraction branch to obtain the identity discriminative features.

[0058] In one embodiment, after the target features are obtained by fusing the style-invariant features and the identity discriminative features, the target features are input into the post-feature extraction network to obtain recognition features, and based on the recognition features, the first object prediction result about the sample image is determined. In one embodiment, the output of the student model to be trained is also connected to the double-head classifier, and therefore, after the recognition features are input into the double-head classifier, the first object prediction result is obtained through a linear layer.

[0059] In one embodiment, the computer device directly superimposes the style-invariant features and the identity discriminative features to obtain the target features corresponding to the sample image.

[0060] In one embodiment, fusing the style-invariant features and the identity discriminative features to determine the target features corresponding to the sample image includes: based on the sample image, weights corresponding to the first feature extraction branch and the second feature extraction branch are respectively determined; and the style-invariant features and the identity discriminative features are weighted and processed according to the weights of the first feature extraction branch and the second feature extraction branch to determine the target features corresponding to the sample image.

[0061] Exemplarily, the computer device further obtains a scene identifier of the sample image, and determines the respective weights of the first feature extraction branch and the second feature extraction branch based on the scene identifier. For example, the computer device obtains a weight mapping table of the scene identifier and the feature extraction branch, and the weight mapping table includes a plurality of mapping relationships. Each mapping relationship corresponds to a scene identifier, and the mapping relationship includes the respective weights of the first feature extraction branch and the second feature extraction branch corresponding to the scene identifier. Therefore, after obtaining the scene identifier of the sample image, the respective weights of the first feature extraction branch and the second feature extraction branch corresponding to the obtained scene identifier are obtained by querying the weight mapping table. Exemplarily, the scene identifier can be obtained by performing scene recognition on the sample image, and is used to represent the scene of the sample image. The object recognition requirements of different scenes are different. For example, a certain scene pays more attention to style transfer, and the weight of the first feature extraction branch is higher than that of the second feature extraction branch. Another scene pays more attention to identity, and the weight of the first feature extraction branch is equal to that of the second feature extraction branch.

[0062] Exemplarily, the computer device determines the respective weights of the first feature extraction branch and the second feature extraction branch based on the feature map determined in the foregoing. The sum of the respective weights of the first feature extraction branch and the second feature extraction branch is 1, and the respective weights of the first feature extraction branch and the second feature extraction branch are negatively correlated, for example, the greater the weight of the first feature extraction branch, the smaller the weight of the second feature extraction branch. Therefore, in one embodiment, the double-branch feature extraction network further includes a weight adjustment network, which determines the respective weights of the first feature extraction branch and the second feature extraction branch based on the sample image, including: determining the feature map of the sample image; inputting the feature map into the weight adjustment network to output the respective weights of the first feature extraction branch and the second feature extraction branch.

[0063] The weight adjustment network is a network constructed based on an attention mechanism, and is used to adjust the weights of the two feature extraction branches in real time.

[0064] Exemplarily, after determining the feature map based on the foregoing process, the computer device further inputs the feature map into the weight adjustment network to output the respective weights of the first feature extraction branch and the second feature extraction branch.

[0065] In this way, the respective weights of the first feature extraction branch and the second feature extraction branch are dynamically allocated by the weight adjustment network to realize the process of dynamically matching the weights, and the accuracy of feature fusion is ensured to improve the accuracy of object prediction.

[0066] Therefore, in combination with Figure 4 The specific determination steps of the target feature are described as follows.Figure 4 Fig. 1 shows a schematic diagram of a target feature determination process in an embodiment. The computer device inputs a sample image into a pre-feature extraction network to obtain a feature map, inputs the feature map into a first feature extraction branch, a weight adjustment network, and a second feature extraction branch respectively to obtain a style-invariant feature, weights of the first and second feature extraction branches respectively, and an identity discriminative feature, performs weighted processing on the style-invariant feature and the identity discriminative feature according to the weights of the first and second feature extraction branches respectively, and determines a target feature corresponding to the sample image. For example, the feature map is obtained based on the sample image, the style-invariant feature is obtained through instance normalization operation of the first feature extraction branch based on the feature map, the discriminative feature is obtained through batch normalization operation of the second feature extraction branch based on the feature map, and the weight vector is generated by the weight adjustment network based on the feature map.

[0067] In this embodiment, the weights of the two feature extraction branches are adaptively adjusted based on the sample image to dynamically fuse the features of different feature extraction branches, so as to ensure the accuracy of feature fusion and improve the accuracy of object prediction.

[0068] Step 206: calling an object shape feature extraction model as a teacher model, determining an object shape feature based on the sample image, and determining a second object prediction result about the sample image based on the object shape feature.

[0069] The object shape feature extraction model is constructed based on a shape-guided consistency learning framework (SGCL), and the shape-guided consistency learning framework is used for shape-guided learning, i.e., knowledge distillation. The shape-guided learning refers to a technology that an object parsing map is used as an intermediate representation without modal difference, and discriminative knowledge in a shape space is migrated to an original image feature space (a feature space where a modal of an original image is located) through knowledge distillation.

[0070] Optionally, the computer device can call an object shape feature extraction model matched with a scene indicated by the scene identifier as the teacher model based on the scene identifier of the sample image.

[0071] ​​​​​​​​​​​Optionally, the computer device inputs the sample image into an object parsing model to obtain a shape representation, inputs the shape representation into an object shape feature extraction model to obtain an object shape feature. In the case where the object is a human, the object parsing model can be based on a SCHP model (Self-Correction Human Parsing), and the shape representation is used to represent each part in the object.

[0072] Optionally, the computer device inputs the object shape feature into a double-head classifier to obtain a second object prediction result.

[0073] In step 208, a target loss is constructed based on the target feature, the object shape feature, the first object prediction result, and the second object prediction result.

[0074] For example, the computer device constructs the target loss based on a difference between the target feature and the object shape feature, a difference between the first object prediction result and the second object prediction result, and a difference between the first object prediction result and a sample label of the sample image. The sample label of the sample image is used to indicate whether an object is recognized in the sample image, and if so, the object identity is determined.

[0075] In one embodiment, the target loss is constructed based on the target feature, the object shape feature, the first object prediction result, and the second object prediction result, including: constructing a first loss based on a difference between the target feature and the object shape feature; constructing a second loss based on the first object prediction result and the second object prediction result through cross-entropy processing; and fusing the first loss and the second loss to construct the target loss.

[0076] For example, if the first object prediction result and the second object prediction result are , where i is a sample number, the second loss is: ; N is the total number of samples, i.e. .

[0077] For example, the computer device determines the respective weights of the first loss and the second loss, weights the first loss and the second loss according to the respective weights of the first loss and the second loss, and obtains the target loss.

[0078] In the embodiment, the first loss is constructed by the difference between the target feature and the object shape feature to constrain the feature relationship consistency, the second loss is constructed by the cross entropy processing based on the first object prediction result and the second object prediction result to realize knowledge distillation, that is, the discriminative knowledge of the object shape space of the teacher model is migrated to the student model to be trained. Therefore, after the first loss and the second loss are fused, the model can be reasonably and accurately and effectively trained based on the target loss to ensure the training effect. In addition, when the model is trained based on the first loss and the second loss subsequently, the teacher-student double-branch distillation framework is actually constructed to realize cross-modal knowledge migration through the relationship similarity constraint and the soft label alignment with the object analysis graph as the intermediate modality.

[0079] In one embodiment, the method further comprises: obtaining an identity label to which the sample image belongs, and based on the identity label, obtaining a plurality of first other sample images and a plurality of second other sample images corresponding to the sample image, each first other sample image having the same identity label as the sample image, and each second other sample image having a different identity label from the sample image.

[0080] It should be noted that the sample image and the corresponding first other sample image can be understood as being obtained by taking the same object multiple times with the same single-modality camera, for example, for object 1, the visible light camera is used to take object 1 C1 times to obtain C1 images, which belong to the positive sample set, and the positive sample set includes the sample image and the first other sample image corresponding to the sample image. The sample image and the corresponding second other sample image can be understood as being obtained by taking different objects with different single-modality cameras, for example, for object 1, the visible light camera is used to take object 1 once to obtain the sample image, and for object 2, the infrared camera is used to take at least one other object at least once to obtain the second other sample image. The sample image and the second other sample image belong to the negative sample set. The positive sample set and the negative sample set are both the current batch of sample sets. wherein represents the camera label, is the total number of visible light cameras, is the total number of infrared cameras. Each camera subset contains image-label pairs, is a pedestrian image, is the corresponding identity label, is the total number of samples. Due to the time span factor of camera mode switching, there are no cross-modal positive sample pairs in the training set, that is, when and , all satisfy The training target of the present application is that the double-branch feature extraction network can extract modal-invariant feature representation from the image, for cross-modal object matching, for example, when the object is a person, cross-modal pedestrian matching is performed.

[0081] Therefore, the positive samples corresponding to the sample image are constructed using the same identity label, and the negative samples of the sample image are constructed using different identity labels, so as to enrich the sample data during model training and ensure the training effect.

[0082] In one example, after determining the target feature corresponding to each first other sample image, the object shape feature about each first other sample image, the target feature corresponding to each second other sample image, and the object shape feature about each second other sample image, a first loss is constructed based on the difference between the target feature and the object shape feature, including: constructing a first feature matrix based on the target feature corresponding to each first other sample image, the target feature corresponding to each second other sample image, and the target feature of the sample image; constructing a second feature matrix based on the object shape feature corresponding to each first other sample image, the object shape feature corresponding to each second other sample image, and the object shape feature of the sample image; calculating the product of the transpose matrix of the first feature matrix and the first feature matrix to obtain a first product; calculating the product of the transpose matrix of the second feature matrix and the second feature matrix to obtain a second product; determining the first loss between the target feature and the object shape feature based on the difference between the first product and the second product.

[0083] Wherein, based on the sample image, the first other sample image and the second other sample image corresponding to the sample image, a training set of the current batch is constructed, and the number of the training set is For example, the first feature matrix is Wherein, are the respective recognition features corresponding to the sample image, the first other sample and the second other sample, respectively, and B is The second feature matrix is Wherein, are the respective object shape features corresponding to the sample image, the first other sample and the second other sample, respectively. It should be noted that based on the target feature corresponding to each first other sample image, the target feature corresponding to each second other sample image, and the target feature of the sample image, the respective recognition features are obtained according to the subsequent feature extraction network.

[0084] The product of A and is calculated to obtain a first product, which can be regarded as a first similarity matrix , that is, ; and and the first product, the first similarity matrix can be regarded as ; then the first loss The first loss can be calculated by the following formula: .

[0085] In this embodiment, the first feature matrix is constructed by each target feature in the sample set to calculate the similarity between them, to obtain the first product, the second feature matrix is constructed based on the object shape features of the sample set, to calculate the similarity between them, to obtain the second product, so that the student model to be trained can learn the discriminative knowledge of the object shape space of the teacher model.

[0086] In step 210, the model to be trained is trained based on the target loss, and the trained student model is used as an object recognition model, which is used for object recognition.

[0087] Exemplarily, the model to be trained is trained with the minimum target loss as the training target, and the trained student model is used as an object recognition model, which is used for object recognition. For example, in the case of an object being a person, the object recognition model can be used for re-identification of pedestrians.

[0088] In an actual training process, the performance verification in the complete modal missing scene is tested by using the reconstruction dataset. SYSU-MM01-CMM: from the original 395 identities, 197 identities of visible light images (11384) and 198 identities of infrared images (5874) are randomly selected to construct a training set, totaling 17258 images; the test set retains 96 identities of 3803 infrared query images and 3010 visible light gallery images. RegDB-CMM: randomly retain single modal images (visible light or infrared) from the original 206 identities, to construct 2060 training images; the test set uses 2060 full modal images (including bidirectional visible light→infrared / infrared→visible light test). In the deployment implementation, ResNet-50 is used as the backbone network, the global average pooling layer is replaced by the generalized mean pooling layer, and the residual modal relief network only acts on the original image branch, thereby constructing the student model to be trained. Among them, the batch size is 128, the optimizer is Adam, the initial learning rate is set to 0.0006, and the learning rate is reduced by 0.1 times at the 30th and 60th iterations, and a total of 100 iterations are trained. The object parsing network is pre-trained based on the PASCAL-Person-Part dataset, and is fine-tuned in the target domain to improve the parsing quality. In the ablation experiment, the performance improvement is shown in Table 1:

[0089] Table 1: Comparison table ​

[0090]

[0091] Referring to Table 1, the effectiveness of the object recognition model provided by the present application can be explained.

[0092] In the processing method of the object recognition model, the student model to be trained containing the double-branch feature extraction network is obtained, the double-branch feature extraction network includes a first feature extraction branch and a second feature extraction branch. Based on the sample image, the style-invariant feature and the identity-discriminative feature are extracted through the first feature extraction branch and the second feature extraction branch, respectively. Then, the style-invariant feature with unified style and the identity-discriminative feature with identity-discriminative details are fused to filter noise, so as to realize the reservation of identity-related features and the directional filtering of noise, and ensure that the target feature corresponding to the sample image can better learn the object prediction ability to determine the first object prediction result about the sample image. The object shape feature extraction model as the teacher model is called to determine the object shape feature based on the sample image, and determine the second object prediction result about the sample image based on the object shape feature. The target loss is constructed based on the target feature, the object shape feature, the first object prediction result and the second object prediction result. The model training is performed on the student model to be trained based on the target loss, which not only enables the model to learn from the features obtained from the two feature extraction branches respectively, but also enables the discriminative knowledge of the object shape space of the teacher model to be migrated to the student model to be trained. In this way, without considering the data amount of the paired data, the model training effect can be ensured, and therefore, after the trained student model is used as the object recognition model, accurate object recognition can be performed based on the object recognition model with good training effect, thereby improving the accuracy of object recognition.

[0093] In one embodiment, the method further comprises: obtaining an image to be recognized; determining the style-invariant feature and the identity-discriminative feature corresponding to the image to be recognized based on the image to be recognized through the first feature extraction branch and the second feature extraction branch of the trained object recognition model, respectively; fusing the style-invariant feature and the identity-discriminative feature corresponding to the image to be recognized to determine the target feature corresponding to the image to be recognized, and determining the object prediction result about the image to be recognized based on the target feature.

[0094] For example, the model structure of the trained object recognition model is as shown in Figure 5 Fig. 1, which is a specific model structure diagram of the object recognition model in one embodiment. The object recognition model includes a pre-sequence feature extraction network, a double-branch feature extraction network and a post-sequence feature extraction network connected in sequence. The double-branch feature extraction network includes a first feature extraction branch, a weight adjustment network and a second feature extraction branch.

[0095] Exemplarily, the computer device inputs the to-be-recognized image into a pre-feature extraction network of the trained object recognition model to obtain a feature map of the to-be-recognized image, inputs the feature map into the double-branch feature extraction network to be respectively input into the first feature extraction branch, the weight adjustment network and the second feature extraction branch in the double-branch feature extraction network, so as to obtain the style-invariant feature, the respective weights of the two feature extraction branches and the identity discrimination feature; the style-invariant feature and the identity discrimination feature corresponding to the to-be-recognized image are weighted according to the respective weights of the first feature extraction branch and the second feature extraction branch, to obtain a target feature corresponding to the to-be-recognized image. The target feature is input into a post-feature extraction network to obtain a recognition feature, and an object prediction result about the to-be-recognized image is determined according to the recognition feature.

[0096] In the embodiment, the object recognition model with good training effect is obtained by fusing the respective features obtained by the two feature extraction branches for model self-learning and the discriminative knowledge of the object shape space of the knowledge distillation teacher model, thereby improving the accuracy of object recognition. It should be noted that the double-branch feature extraction network dynamically allocates weights through the weight adjustment network, realizes self-adaptive feature decoupling, and solves the information loss problem caused by traditional fixed proportion fusion.

[0097] In one specific embodiment, the training phase and the inference phase are involved, and the specific process can refer to the following steps:

[0098] In the training phase:

[0099] Step 1: The computer device obtains a to-be-trained student model containing a double-branch feature extraction network, the double-branch feature extraction network including a first feature extraction branch, a second feature extraction branch and a weight adjustment network; based on a sample image, the computer device extracts a style-invariant feature and an identity discrimination feature through the first feature extraction branch and the second feature extraction branch respectively.

[0100] Step 2: The computer device determines a feature map of the sample image; the feature map is input into the weight adjustment network to output respective weights of the first feature extraction branch and the second feature extraction branch. The style-invariant feature and the identity discrimination feature are weighted according to the respective weights of the first feature extraction branch and the second feature extraction branch to determine a target feature corresponding to the sample image.

[0101] Step 3: The computer device determines a first object prediction result about the sample image based on the target feature. An object shape feature extraction model serving as a teacher model is called to determine an object shape feature based on the sample image, and determine a second object prediction result about the sample image based on the object shape feature.

[0102] Step 4: Obtain the identity label to which the sample image belongs, and based on the identity label, obtain a plurality of first other sample images and a plurality of second other sample images corresponding to the sample image, each first other sample image having the same identity label as the sample image, and each second other sample image having a different identity label from the sample image;

[0103] After determining the target feature corresponding to each first other sample image, the first object prediction result about each first other sample image, the target feature corresponding to each second other sample image, and the first object prediction result about each second other sample image, a first feature matrix is constructed based on the target feature corresponding to each first other sample image, the target feature corresponding to each second other sample image, and the target feature of the sample image; a second feature matrix is constructed based on the object shape feature corresponding to each first other sample image, the object shape feature corresponding to each second other sample image, and the object shape feature of the sample image; the product of the transpose matrix of the first feature matrix and the first feature matrix is calculated to obtain a first product; the product of the transpose matrix of the second feature matrix and the second feature matrix is calculated to obtain a second product; and the first loss between the target feature and the object shape feature is determined based on the difference between the first product and the second product.

[0104] Step 5: The computer device constructs a second loss by cross-entropy processing based on the first object prediction result and the second object prediction result; and fuses the first loss and the second loss to construct a target loss.

[0105] Step 6: The model training is performed on the to-be-trained student model by minimizing the target loss, the trained student model is taken as an object recognition model, and the object recognition model is used for object recognition.

[0106] Then, the inference phase is performed, and the specific process is as follows:

[0107] Step 7: The computer device obtains a to-be-recognized image; based on the feature map of the to-be-recognized image, the style-invariant feature and the identity discrimination feature corresponding to the to-be-recognized image are determined by the first feature extraction branch and the second feature extraction branch of the trained object recognition model, respectively; based on the feature map, the weights of the two feature extraction branches are determined by the weight adjustment network of the trained object recognition model, and the style-invariant feature and the identity discrimination feature corresponding to the to-be-recognized image are fused according to the respective weights to determine the target feature corresponding to the to-be-recognized image, and based on the target feature, the object prediction result about the to-be-recognized image is determined.

[0108] In the embodiment, by acquiring a to-be-trained student model containing a double-branch feature extraction network, the double-branch feature extraction network includes a first feature extraction branch and a second feature extraction branch; based on a sample image, style-invariant features and identity discriminative features are respectively extracted through the first feature extraction branch and the second feature extraction branch, then the style-invariant features and the identity discriminative features with identity discriminative details are fused to filter noise and unify styles, so that the identity-related features can be reserved and the noise can be filtered in a targeted manner, and the target features corresponding to the sample image can be obtained, which can better learn the object prediction ability to determine the first object prediction result about the sample image; an object shape feature extraction model serving as a teacher model is called, object shape features are determined based on the sample image, and a second object prediction result about the sample image is determined based on the object shape features; and the target loss is constructed based on the target features, the object shape features, the first object prediction result and the second object prediction result. The model training is performed on the to-be-trained student model based on the target loss, which can not only perform model self-learning from the features obtained from the two feature extraction branches respectively, but also migrate the discriminative knowledge of the object shape space of the teacher model to the to-be-trained student model, so that the model training effect can be ensured without considering the data amount of the paired data. Therefore, after the trained student model is used as an object recognition model, accurate object recognition can be performed based on the object recognition model with good training effect, and the accuracy of object recognition is improved.

[0109] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow indication, these steps are not necessarily executed in sequence according to the arrow indication. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0110] Based on the same inventive concept, the embodiments of the present application also provide an object recognition model processing device for implementing the object recognition model processing method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more object recognition model processing device embodiments provided below can refer to the limitations of the object recognition model processing method in the above text, which will not be repeated here.

[0111] In one exemplary embodiment, as shown in Figure 6 An object recognition model processing device 600 is provided, comprising a learning model obtaining module 602, a first object prediction module 604, a second object prediction module 606, a target loss constructing module 608, and a model training module 610, wherein:

[0112] The learning model obtaining module 602 is configured to obtain a student model to be trained containing a double-branch feature extraction network, the double-branch feature extraction network comprising a first feature extraction branch and a second feature extraction branch.

[0113] The first object prediction module 604 is configured to extract style-invariant features and identity discriminative features from the sample image through the first feature extraction branch and the second feature extraction branch respectively, fuse the style-invariant features and the identity discriminative features, determine target features corresponding to the sample image, and determine a first object prediction result about the sample image based on the target features.

[0114] The second object prediction module 606 is configured to call an object shape feature extraction model as a teacher model, determine object shape features based on the sample image, and determine a second object prediction result about the sample image based on the object shape features.

[0115] The target loss constructing module 608 is configured to construct a target loss based on the target features, the object shape features, the first object prediction result, and the second object prediction result.

[0116] The model training module 610 is configured to perform model training on the student model to be trained based on the target loss, and use the trained student model as an object recognition model, which is used for object recognition.

[0117] In one embodiment, the first object prediction module 604 is configured to determine respective weights of the first feature extraction branch and the second feature extraction branch based on the sample image, and perform weighted processing on the style-invariant features and the identity discriminative features according to the respective weights of the first feature extraction branch and the second feature extraction branch to determine the target features corresponding to the sample image.

[0118] In an embodiment, the dual-branch feature extraction network further comprises a weight adjustment network, and the first object prediction module 604 is configured to determine a feature map of the sample image; input the feature map into the weight adjustment network, and output respective weights corresponding to the first feature extraction branch and the second feature extraction branch.

[0119] In an embodiment, the target loss construction module 608 is configured to construct a first loss based on a difference between the target feature and the object shape feature; construct a second loss by cross-entropy processing based on the first object prediction result and the second object prediction result; and fuse the first loss and the second loss to construct the target loss.

[0120] In an embodiment, the apparatus further comprises an other sample acquisition module configured to acquire an identity label to which the sample image belongs, and acquire a plurality of first other sample images and a plurality of second other sample images corresponding to the sample image based on the identity label, each of the first other sample images having the same identity label as the sample image, and each of the second other sample images having a different identity label from the sample image; and after the target feature corresponding to each of the first other sample images, the object shape feature about each of the first other sample images, the target feature corresponding to each of the second other sample images, and the object shape feature about each of the second other sample images are determined, the target loss construction module 608 is configured to construct a first feature matrix based on the target feature corresponding to each of the first other sample images, the target feature corresponding to each of the second other sample images, and the target feature of the sample image; construct a second feature matrix based on the object shape feature corresponding to each of the first other sample images, the object shape feature corresponding to each of the second other sample images, and the object shape feature of the sample image; calculate a product of a transpose matrix of the first feature matrix and the first feature matrix to obtain a first product; calculate a product of a transpose matrix of the second feature matrix and the second feature matrix to obtain a second product; and determine the first loss between the target feature and the object shape feature based on a difference between the first product and the second product.

[0121] In an embodiment, the apparatus further comprises a model use module configured to acquire a to-be-identified image; determine a style-invariant feature and an identity discrimination feature corresponding to the to-be-identified image respectively by the first feature extraction branch and the second feature extraction branch of the trained object recognition model based on the to-be-identified image; fuse the style-invariant feature and the identity discrimination feature corresponding to the to-be-identified image to determine a target feature corresponding to the to-be-identified image; and determine an object prediction result about the to-be-identified image based on the target feature.

[0122] The modules in the processing apparatus of the object recognition model can be implemented by software, hardware, or a combination thereof. The modules can be embedded in the processor of the computer device in hardware form or independent of the processor, or stored in the memory of the computer device in software form to be invoked and executed by the processor.

[0123] In an example embodiment, a computer device is provided, which can be a server or a terminal. An internal structure diagram of the computer device can be as shown in Figure 7 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a processing method of an object recognition model.

[0124] Those skilled in the art can understand that Figure 7 The structure shown in the above

[0125] In an example embodiment, a computer device is provided, which can be a server or a terminal. An internal structure diagram of the computer device can be as shown in

[0126] In an example embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0127] In an example embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0128] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0129] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0130] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of these technical features is deemed to be within the scope of the present application.

[0131] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for processing an object recognition model, characterized in that, The method includes: Obtain a student model to be trained that includes a dual-branch feature extraction network, wherein the dual-branch feature extraction network includes a first feature extraction branch and a second feature extraction branch; Based on the sample image, style-invariant features and identity-discriminating features are extracted through the first feature extraction branch and the second feature extraction branch, respectively. The style-invariant features and identity-discriminating features are fused to determine the target features corresponding to the sample image. Based on the target features, a first object prediction result for the sample image is determined. The object shape feature extraction model, which serves as the teacher model, is invoked. Based on the sample image, object shape features are determined, and based on the object shape features, a second object prediction result is determined for the sample image. Based on the target features, the object shape features, the first object prediction result, and the second object prediction result, a target loss is constructed. The student model to be trained is trained based on the target loss, and the trained student model is used as an object recognition model for object recognition.

2. The method according to claim 1, characterized in that, The fusion of style-invariant features and identity-discriminating features determines the target features corresponding to the sample image, including: Based on the sample image, the weights corresponding to the first feature extraction branch and the second feature extraction branch are determined respectively. Based on the respective weights of the first feature extraction branch and the second feature extraction branch, the style-invariant features and identity-discriminating features are weighted to determine the target features corresponding to the sample image.

3. The method according to claim 2, characterized in that, The dual-branch feature extraction network further includes a weight adjustment network. The step of determining the weights corresponding to the first feature extraction branch and the second feature extraction branch based on the sample image includes: Determine the feature map of the sample image; The feature map is input into the weight adjustment network, and the weights corresponding to the first feature extraction branch and the second feature extraction branch are output.

4. The method according to claim 1, characterized in that, The step of constructing a target loss based on the target features, the object shape features, the first object prediction result, and the second object prediction result includes: Based on the difference between the target features and the object shape features, a first loss is constructed; Based on the prediction results of the first object and the prediction results of the second object, a second loss is constructed through cross-entropy processing; The first loss and the second loss are combined to construct the target loss.

5. The method according to claim 4, characterized in that, The method further includes: Obtain the identity tag to which the sample image belongs, and based on the identity tag, obtain multiple first other sample images and multiple second other sample images corresponding to the sample image. Each first other sample image has the same identity tag as the sample image, and each second other sample image has a different identity tag from the sample image. After determining the target features corresponding to each first other sample image, the object shape features for each first other sample image, the target features corresponding to each second other sample image, and the object shape features for each second other sample image, the step of constructing a first loss based on the difference between the target features and the object shape features includes: A first feature matrix is ​​constructed based on the target features corresponding to each first other sample image, the target features corresponding to each second other sample image, and the target features of the sample images. A second feature matrix is ​​constructed based on the object shape features corresponding to each first other sample image, the object shape features corresponding to each second other sample image, and the object shape features of the sample images. Calculate the product of the transpose of the first feature matrix and the first feature matrix to obtain the first product; Calculate the product of the transpose of the second characteristic matrix and the second characteristic matrix to obtain the second product; Based on the difference between the first product and the second product, a first loss is determined between the target feature and the object shape feature.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Acquire the image to be recognized; Based on the image to be identified, the style-invariant features and identity-discriminating features corresponding to the image to be identified are determined by the first feature extraction branch and the second feature extraction branch of the trained object recognition model, respectively. By fusing style-invariant features and identity-discriminating features corresponding to the image to be identified, target features corresponding to the image to be identified are determined, and based on the target features, object prediction results for the image to be identified are determined.

7. A processing apparatus for an object recognition model, characterized in that, The device includes: The learning model acquisition module is used to acquire a student model to be trained that includes a dual-branch feature extraction network, wherein the dual-branch feature extraction network includes a first feature extraction branch and a second feature extraction branch. The first object prediction module is used to extract style-invariant features and identity-discriminating features from a sample image through a first feature extraction branch and a second feature extraction branch, respectively, fuse the style-invariant features and identity-discriminating features, determine the target features corresponding to the sample image, and determine the first object prediction result for the sample image based on the target features. The second object prediction module is used to call the object shape feature extraction model, which serves as the teacher model, to determine the object shape features based on the sample image, and to determine the second object prediction result for the sample image based on the object shape features. The target loss construction module is used to construct a target loss based on the target features, the object shape features, the first object prediction result, and the second object prediction result; The model training module is used to train the student model to be trained based on the target loss, and to use the trained student model as an object recognition model, which is used for object recognition.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images