Object recognition model training methods, devices, electronic equipment and storage media

Abstract

This disclosure relates to a method, apparatus, electronic device, and storage medium for training an object recognition model. The method includes: extracting semantic features of sample images at a target semantic depth based on the object recognition model to be trained to obtain an initial semantic feature map at the target semantic depth; performing scale compression on the initial semantic feature map to obtain a first semantic feature map; extracting semantic features of sample images at the target semantic depth based on a teacher recognition model to obtain a second semantic feature map; performing frequency domain enhancement processing on the frequency domain features corresponding to the first semantic feature map to obtain a first frequency domain enhanced feature map; performing frequency domain enhancement processing on the frequency domain features corresponding to the second semantic feature map to obtain a second frequency domain enhanced feature map; and training the object recognition model based on the difference between the first and second frequency domain enhanced feature maps until a preset training termination condition is met. This disclosure reduces the computational resource consumption of the model while ensuring the recognition accuracy of the object recognition model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an object recognition model training method, apparatus, electronic device, and storage medium. Background Art

[0002] With the development of computer technology, object recognition models based on convolutional neural networks have been widely used in the field of object recognition (such as product recognition). However, since convolutional neural networks require a lot of computing resources during inference, the application of object recognition models on mobile terminals or embedded devices has brought great challenges.

[0003] Related technologies have adopted some targeted training methods to address the above problems. While these training methods can reduce the computational resource consumption of the object recognition model during inference to some extent, they cannot guarantee the model's recognition accuracy. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for training an object recognition model, to at least solve the problem in related technologies that it is impossible to ensure high recognition accuracy of the model while effectively reducing the computational resource consumption of the object recognition model's inference process. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a method for training an object recognition model is provided, comprising:

[0006] The object recognition model to be trained extracts semantic features of the sample image at the target semantic depth to obtain an initial semantic feature map of the target semantic depth; the object recognition model is used to identify the target object in the sample image.

[0007] The initial semantic feature map of the target semantic depth is subjected to scale compression processing to obtain the first semantic feature map of the target semantic depth;

[0008] Based on a pre-trained teacher recognition model, semantic features of the sample image at the target semantic depth are extracted to obtain a second semantic feature map at the target semantic depth; the teacher recognition model is used to identify target objects in the sample image;

[0009] The first semantic feature map is enhanced by performing frequency domain enhancement processing on the frequency domain features corresponding to the first semantic feature map to obtain a first frequency domain enhanced feature map of the target semantic depth; and the second semantic feature map is enhanced by performing frequency domain enhancement processing on the frequency domain features corresponding to the second semantic feature map to obtain a second frequency domain enhanced feature map of the target semantic depth.

[0010] Based on the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map of the target semantic depth, the object recognition model is trained until the preset training termination condition is met, and the trained object recognition model is obtained.

[0011] In one exemplary embodiment, the sample image corresponds to a reference category label; training the object recognition model based on the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map of the target semantic depth until a preset training termination condition is reached includes:

[0012] A first loss value is determined based on the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map;

[0013] The object recognition model performs object recognition processing on the sample image based on the first semantic feature map to obtain a first recognition result; a second loss value is determined based on the first recognition result and the reference category label corresponding to the sample image.

[0014] The teacher recognition model performs object recognition processing on the sample image based on the second semantic feature map to obtain a second recognition result;

[0015] Based on the first identification result and the second identification result, a third loss value is determined;

[0016] Based on the first loss value, the second loss value, and the third loss value, the model parameters of the object recognition model are adjusted until the preset training termination condition is met.

[0017] In one exemplary embodiment, the step of extracting semantic features of the sample image at the target semantic depth based on the object recognition model to be trained, to obtain an initial semantic feature map of the target semantic depth, includes:

[0018] The sample image is input into the object recognition model to be trained, and the sample image is processed by the shallow convolutional network of the object recognition model to obtain the feature map output by the shallow convolutional network; the shallow convolutional network includes a preset number of convolutional layers close to the input layer in the object recognition model;

[0019] Based on the feature map output by the shallow convolutional network, an initial semantic feature map of the target semantic depth is obtained.

[0020] In one exemplary implementation, obtaining the initial semantic feature map of the target semantic depth based on the feature map output by the shallow convolutional network includes:

[0021] The feature map output by the shallow convolutional network is scaled and compressed. The scaled and compressed feature map is then input into the middle convolutional network connected to the shallow convolutional network for convolution processing to obtain the feature map output by the middle convolutional network.

[0022] The feature maps output by the middle-layer convolutional network and / or the feature maps output by the shallow-layer convolutional network are used as the initial semantic feature maps for the target semantic depth.

[0023] In one exemplary implementation, determining the first loss value based on the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map includes:

[0024] The first sub-loss value is determined based on the difference between the first frequency domain enhanced feature map corresponding to the shallow convolutional network and the second frequency domain enhanced feature map corresponding to the shallow convolutional network.

[0025] The second sub-loss value is determined based on the difference between the first frequency domain enhanced feature map corresponding to the middle convolutional network and the second frequency domain enhanced feature map corresponding to the middle convolutional network.

[0026] The first loss value is determined based on the first sub-loss value and the second sub-loss value.

[0027] In one exemplary implementation, determining the first loss value based on the first sub-loss value and the second sub-loss value includes:

[0028] The third sub-loss value is determined based on the difference between the frequency domain enhanced feature map corresponding to the high-level convolutional network in the object recognition model and the frequency domain enhanced feature map corresponding to the high-level convolutional network in the teacher recognition model.

[0029] The first loss value is determined based on the first sub-loss value, the second sub-loss value, and the third sub-loss value;

[0030] The frequency domain enhanced feature map corresponding to the high-level convolutional network is obtained based on the feature map output by the high-level convolutional network, and the feature map output by the high-level convolutional network is obtained by performing scale compression processing on the feature map output by the connected middle-level convolutional network.

[0031] In one exemplary embodiment, the step of performing frequency domain enhancement processing on the first semantic feature map based on the frequency domain features corresponding to the first semantic feature map to obtain a first frequency domain enhanced feature map of the target semantic depth includes:

[0032] Perform a Fourier transform on the first semantic feature map to obtain the first frequency domain feature map corresponding to the first semantic feature map;

[0033] Based on the first frequency domain feature map and the first mask map corresponding to the first frequency domain feature map, a first enhanced frequency domain map is determined; an inverse Fourier transform is performed on the first enhanced frequency domain map to obtain the first frequency domain enhanced feature map.

[0034] The step of performing frequency domain enhancement processing on the second semantic feature map based on the frequency domain features corresponding to the second semantic feature map to obtain the second frequency domain enhanced feature map of the target semantic depth includes:

[0035] Perform a Fourier transform on the second semantic feature map to obtain the second frequency domain feature map corresponding to the second semantic feature map;

[0036] Based on the second frequency domain feature map and the second mask map corresponding to the second frequency domain feature map, a second enhanced frequency domain map is determined; an inverse Fourier transform is performed on the second enhanced frequency domain map to obtain the second enhanced frequency domain feature map.

[0037] According to a second aspect of the present disclosure, an object recognition model training apparatus is provided, comprising:

[0038] The first feature extraction unit is configured to extract semantic features of the sample image at the target semantic depth based on the object recognition model to be trained, and obtain an initial semantic feature map of the target semantic depth; the object recognition model is used to identify the target object in the sample image;

[0039] The scaling compression unit is configured to perform scaling compression processing on the initial semantic feature map of the target semantic depth to obtain a first semantic feature map of the target semantic depth.

[0040] The second feature extraction unit is configured to perform a pre-trained teacher recognition model to extract semantic features of the sample image at the target semantic depth, thereby obtaining a second semantic feature map at the target semantic depth; the teacher recognition model is used to identify target objects in the sample image.

[0041] The frequency domain enhancement processing unit is configured to perform frequency domain enhancement processing on the first semantic feature map based on the frequency domain features corresponding to the first semantic feature map to obtain a first frequency domain enhanced feature map of the target semantic depth; and to perform frequency domain enhancement processing on the second semantic feature map based on the frequency domain features corresponding to the second semantic feature map to obtain a second frequency domain enhanced feature map of the target semantic depth.

[0042] The training unit is configured to perform training on the object recognition model by performing a difference between a first frequency domain enhanced feature map and a second frequency domain enhanced feature map based on the target semantic depth, until a preset training termination condition is met, thereby obtaining a trained object recognition model.

[0043] In one exemplary embodiment, the sample image corresponds to a reference category label; the training unit includes:

[0044] The first loss determination unit is configured to determine a first loss value based on the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map.

[0045] The second loss determination unit is configured to perform object recognition processing on the sample image based on the first semantic feature map using the object recognition model to obtain a first recognition result; and to determine a second loss value based on the first recognition result and the reference category label corresponding to the sample image.

[0046] The first recognition unit is configured to perform object recognition processing on the sample image based on the second semantic feature map using the teacher recognition model to obtain a second recognition result;

[0047] The third loss determination unit is configured to determine a third loss value based on the first identification result and the second identification result;

[0048] The parameter adjustment unit is configured to adjust the model parameters of the object recognition model based on the first loss value, the second loss value, and the third loss value until a preset training termination condition is reached.

[0049] In one exemplary embodiment, the first feature extraction unit includes:

[0050] The first convolutional unit is configured to input the sample image into the object recognition model to be trained, and perform convolution processing on the sample image through the shallow convolutional network of the object recognition model to obtain the feature map output by the shallow convolutional network; the shallow convolutional network includes a predetermined number of convolutional layers close to the input layer in the object recognition model;

[0051] The feature extraction subunit is configured to perform an initial semantic feature map based on the feature map output by the shallow convolutional network to obtain the target semantic depth.

[0052] In one exemplary embodiment, the feature extraction subunit is specifically configured to perform scale compression processing on the feature map output by the shallow convolutional network, input the scale-compressed feature map into a mid-level convolutional network connected to the shallow convolutional network for convolution processing, and obtain the feature map output by the mid-level convolutional network; and use the feature map output by the mid-level convolutional network and / or the feature map output by the shallow convolutional network as the initial semantic feature map of the target semantic depth.

[0053] In one exemplary embodiment, the first loss determination unit includes:

[0054] The first sub-loss determination unit is configured to determine the first sub-loss value based on the difference between the first frequency domain enhanced feature map corresponding to the shallow convolutional network and the second frequency domain enhanced feature map corresponding to the shallow convolutional network.

[0055] The second sub-loss determination unit is configured to determine the second sub-loss value based on the difference between the first frequency domain enhanced feature map corresponding to the middle convolutional network and the second frequency domain enhanced feature map corresponding to the middle convolutional network.

[0056] The fourth loss determination unit is configured to determine the first loss value based on the first sub-loss value and the second sub-loss value.

[0057] In one exemplary embodiment, the fourth loss determination unit includes:

[0058] The third sub-loss determination unit is configured to determine the third sub-loss value based on the difference between the frequency domain enhanced feature map corresponding to the high-level convolutional network in the object recognition model and the frequency domain enhanced feature map corresponding to the high-level convolutional network in the teacher recognition model.

[0059] The fifth loss determination unit is configured to determine the first loss value based on the first sub-loss value, the second sub-loss value, and the third sub-loss value;

[0060] The frequency domain enhanced feature map corresponding to the high-level convolutional network is obtained based on the feature map output by the high-level convolutional network, and the feature map output by the high-level convolutional network is obtained by performing scale compression processing on the feature map output by the connected middle-level convolutional network.

[0061] In one exemplary embodiment, the frequency domain enhancement processing unit includes:

[0062] The first frequency domain enhancement processing unit is configured to perform a Fourier transform on the first semantic feature map to obtain a first frequency domain feature map corresponding to the first semantic feature map; determine a first enhanced frequency domain map based on the first frequency domain feature map and a first mask map corresponding to the first frequency domain feature map; and perform an inverse Fourier transform on the first enhanced frequency domain map to obtain a first frequency domain enhanced feature map.

[0063] The second frequency domain enhancement processing unit performs a Fourier transform on the second semantic feature map to obtain a second frequency domain feature map corresponding to the second semantic feature map; determines a second enhanced frequency domain map based on the second frequency domain feature map and the second mask map corresponding to the second frequency domain feature map; and performs an inverse Fourier transform on the second enhanced frequency domain map to obtain a second frequency domain enhanced feature map.

[0064] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0065] processor;

[0066] Memory used to store the processor's executable instructions;

[0067] The processor is configured to execute the instructions to implement the object recognition model training method of the first aspect described above.

[0068] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the object recognition model training method of the first aspect described above.

[0069] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions, which, when executed by a processor, implement the object recognition model training method of the first aspect described above.

[0070] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0071] The semantic features of the target semantic depth in the sample image are extracted by the object recognition model and scaled to reduce the computational resource consumption of the object recognition model in the recognition process. The frequency domain enhanced feature map is used to realize the feature transfer between the teacher recognition model and the object recognition model to be trained, so that the recognition result of the object recognition model is basically consistent with the accuracy of the teacher recognition model. Thus, while ensuring the recognition accuracy of the object recognition model after training, the computational resource consumption of the model is effectively reduced.

[0072] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. Attached Figure Description

[0073] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0074] Figure 1This is a schematic diagram illustrating the application environment of an object recognition model training method according to an exemplary embodiment;

[0075] Figure 2 This is a flowchart illustrating an object recognition model training method according to an exemplary embodiment;

[0076] Figure 3 This is a schematic diagram illustrating the structure of an object recognition model to be trained according to an exemplary embodiment;

[0077] Figure 4 This is a flowchart illustrating another object recognition model training method according to an exemplary embodiment;

[0078] Figure 5 This is a flowchart illustrating another object recognition model training method according to an exemplary embodiment;

[0079] Figure 6 This is a schematic diagram of a first information transmission module according to an exemplary embodiment;

[0080] Figure 7 This is a structural block diagram of an object recognition model training device according to an exemplary embodiment;

[0081] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0082] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0083] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0084] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0085] Please see Figure 1The diagram illustrates an application environment for an object recognition model training method according to an exemplary embodiment. The application environment may include a terminal 110 and a server 120, which can be connected via a wired network or a wireless network.

[0086] Terminal 110 can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. Terminal 110 may have client software, such as an application (App), installed that provides image processing functions. This application can be a standalone application or a subroutine within an application. For example, the application may include short video applications, live streaming applications, etc. The image processing function may include identifying target objects in an image, such as identifying goods or other objects in an image.

[0087] Specifically, the terminal 110 can perform object recognition processing on the image to be recognized based on a pre-trained object recognition model to obtain the recognition result. The pre-trained object recognition model can be a lightweight convolutional neural network model that can run in the terminal 110.

[0088] Server 120 can provide background services for applications in terminal 110. These background services may include object recognition model training services. After training the object recognition model, server 120 can distribute the object recognition model to terminal 110. Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0089] Figure 2 This is a flowchart illustrating an object recognition model training method according to an exemplary embodiment, such as... Figure 2 As shown, the training method for this object recognition model includes the following steps.

[0090] In step S201, the semantic features of the sample image at the target semantic depth are extracted based on the object recognition model to be trained, and the initial semantic feature map of the target semantic depth is obtained.

[0091] The object recognition model to be trained is used to identify target objects in sample images. The target object can be set according to the actual application scenario. For example, in the product recognition scenario, the target object can be a product, and in the face recognition scenario, the target object can be a face, etc.

[0092] Sample images can be the original images of target objects in the training dataset. Each sample image has corresponding annotation information, which indicates the reference category of the target object in the sample image. For example, when the target object is a product, the reference category can include primary category (such as tops, pants, etc.) and secondary category (such as shirts, T-shirts, etc.).

[0093] The object recognition model to be trained can be a convolutional neural network (CNN) structure. A CNN is a deep neural network with convolutional structures. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0094] For example, the object recognition model to be trained can be a deep convolutional network such as a residual network (ResNet). ResNet, proposed in 2015, is a type of deep convolutional network that is easier to optimize than traditional convolutional neural networks and can improve accuracy by significantly increasing its depth. The core of ResNet is to address the side effects (degradation problem) of increasing depth, thus improving network performance simply by increasing network depth. ResNet typically contains many structurally identical sub-modules, and a number is usually used to represent the number of times a sub-module is repeated, such as in ResNet38.

[0095] Figure 3 The diagram shown is a structural schematic of an object recognition model to be trained according to an embodiment of this disclosure. Figure 3 As shown, the input and output consist of five sequentially connected convolutional network layers. The last convolutional network layer is connected to an average pooling layer, which in turn is connected to a classifier. The classifier includes a fully connected layer and an output layer. The fully connected layer has a number of neurons corresponding to the number of reference classes; for example, if there are 1000 reference classes, the fully connected layer would have 1000 neurons. The output layer can use the Softmax classification function. The following example... Figure 3 The convolutional network layers in the model are explained using an input image of size 224x224 as an example.

[0096] The first convolutional network layer has an input size of 112x112, with 64 convolutional kernels of 7x7 size. The second convolutional network layer has an input size of 56x56 and includes one max-pooling layer and three first convolutional modules. The max-pooling layer has 64 convolutional kernels of 3x3 size. Each first convolutional module includes three convolutional layers (with kernel sizes of 1x1, 3x3, and 1x1, and kernel numbers of 64, 64, and 25, respectively). 6) The input size of the third convolutional network layer is 28x28, consisting of one second convolutional module and three third convolutional modules. Each second convolutional module contains three convolutional layers (kernel sizes of 1x1, 3x3, and 1x1, with kernel numbers of 128, 128, and 512, and strides of 1, 2, and 1, respectively). Each third convolutional module contains three convolutional layers (kernel sizes of 1x1, 3x3, and 1x1, with kernel numbers of 128, 128, and 512, and a stride of 1 for each layer). The fourth convolutional network layer has an input size of 14x14 and includes one fourth convolutional module and five fifth convolutional modules. Each fourth convolutional module contains three convolutional layers (kernel sizes of 1x1, 3x3, and 1x1, with kernel numbers of 256, 256, and 1024, and strides of 1, 2, and 1, respectively). Each fifth convolutional module also contains three convolutional layers (kernel sizes of 1x1, 3x3, and 1x1, with kernel numbers of 256, 256, and 1024, and a stride of 1 for each layer). The fifth convolutional network layer has an input size of 7x7 and includes one sixth convolutional module and two seventh convolutional modules. The sixth convolutional module includes three convolutional layers (with kernel sizes of 1x1, 3x3, and 1x1, and the number of kernels of 512, 512, and 2048, and strides of 1, 2, and 1, respectively). Each seventh convolutional module includes three convolutional layers (with kernel sizes of 1x1, 3x3, and 1x1, and the number of kernels of 512, 512, and 2048, respectively, and a stride of 1 for each layer).

[0097] The target semantic depth can be set according to actual needs. For example, the target semantic depth can include low semantic depth, medium semantic depth, etc., with the semantic depth increasing sequentially.

[0098] In practical applications, a convolutional neural network (CNN) structure typically consists of shallow, mid-level, and high-level convolutional layers from input to output. The shallow convolutional layers are a predetermined number of layers located close to the input layer. This predetermined number can be set based on the total number of convolutional layers in the model; for example, if the model has three convolutional layers, the first layer can be designated as a shallow convolutional layer. The mid-level convolutional layers connect to the shallow layers, and the high-level convolutional layers connect to the mid-level layers. Therefore, the semantic features extracted by the shallow convolutional layers can be defined as low-depth semantic features (also known as low-level semantic feature maps), the semantic features extracted by the mid-level convolutional layers can be defined as mid-depth semantic features (also known as mid-level semantic feature maps), and the semantic features extracted by the high-level convolutional layers can be defined as high-depth semantic features (also known as high-level semantic feature maps).

[0099] by Figure 3 Taking the model structure shown as an example, the first and second convolutional network layers can be regarded as shallow convolutional networks, the third and fourth convolutional network layers as medium-level convolutional networks, and the fifth convolutional network layer as a high-level convolutional network. Then, the semantic features with low semantic depth can be the features output by the first and / or second convolutional network layers, and the semantic features with high semantic depth can be the features output by the fifth convolutional network layer. Therefore, the features output by the third and / or fourth convolutional network layers can both be regarded as semantic features with medium semantic depth.

[0100] It is understood that the above classification of shallow convolutional networks, mid-level convolutional networks, and high-level convolutional networks is merely an example and does not constitute a specific limitation on the embodiments of this disclosure.

[0101] In step S203, the initial semantic feature map of the target semantic depth is subjected to scale compression processing to obtain the first semantic feature map of the target semantic depth.

[0102] Size compression refers to reducing the resolution of the feature map to shrink its spatial size. In practice, the initial semantic feature map of the target semantic depth can be downsampled, and the degree of scale compression can be set according to actual needs to compress the size of the feature map while minimizing the loss of useful information.

[0103] By performing scale compression on the initial semantic feature map of the target semantic depth, the spatial scale of the compressed first semantic feature map can be reduced, avoiding the waste of computational resources caused by redundant spatial scales in subsequent model processing, thereby reducing the consumption of computational resources by the model in subsequent processing.

[0104] Considering that in practical applications, feature maps with lower semantic depth contain more redundant information, that is, scaling feature maps with lower semantic depth is more beneficial for reducing the consumption of computational resources during model processing. Therefore, the target semantic depth in this embodiment includes low semantic depth. Based on this, in an exemplary embodiment, the above step S201 may include:

[0105] The sample image is input into the object recognition model to be trained. The sample image is processed by the shallow convolutional network of the object recognition model to obtain the feature map output by the shallow convolutional network.

[0106] Based on the feature map output by the shallow convolutional network, an initial semantic feature map of the target semantic depth is obtained.

[0107] Specifically, when obtaining the initial semantic feature map of the target semantic depth from the feature map output by the shallow convolutional network, the feature map output by the shallow convolutional network can be directly used as the initial semantic feature map of the target semantic depth.

[0108] In this embodiment, the sample image is convolved by a shallow convolutional network of the object recognition model, and the initial semantic feature map of the target semantic depth is obtained based on the feature map output by the shallow convolutional network. This makes the target semantic depth include the low semantic depth, and the low semantic depth semantic feature map can be scaled to reduce the redundant information of the low semantic depth in subsequent computational resource consumption.

[0109] In an exemplary embodiment, obtaining the initial semantic feature map of the target semantic depth based on the feature map output by the shallow convolutional network may include: performing scale compression processing on the feature map output by the shallow convolutional network, and inputting the scale-compressed feature map into a middle convolutional network connected to the shallow convolutional network for convolution processing to obtain the feature map output by the middle convolutional network.

[0110] The feature maps output by the middle convolutional network and / or the feature maps output by the shallow convolutional network are used as the initial semantic feature maps for the target semantic depth.

[0111] In the above implementation, by using the feature map output by the intermediate convolutional network as the initial semantic feature map of the target semantic depth, the spatial scale of the semantic feature map of the intermediate semantic depth can be reduced in the subsequent process, thereby reducing the waste of computing resources caused by redundant information in the intermediate semantic depth.

[0112] In addition, in order to significantly reduce the computational resource consumption of the model processing, the target semantic depth can include both low semantic depth and medium semantic depth. That is, the feature maps output by the shallow convolutional network and the feature maps output by the medium convolutional network are obtained and used as the initial semantic feature maps of the target semantic depth. This can reduce the computational resource waste caused by the redundant information of the low semantic depth and the redundant information of the medium semantic depth, thereby significantly reducing the computational resource consumption of the model recognition process.

[0113] In step S205, the semantic features of the sample image at the target semantic depth are extracted based on the pre-trained teacher recognition model to obtain the second semantic feature map at the target semantic depth.

[0114] Among them, the pre-trained teacher recognition model is used to identify target objects in sample images.

[0115] The pre-trained teacher recognition model is a recognition model obtained after training on an object recognition task using a large amount of training data. Pre-trained teacher recognition models typically have high recognition accuracy. The number of model parameters in the pre-trained teacher recognition model can be greater than or equal to that of the object recognition model to be trained; that is, the object recognition model to be trained can be considered a student model of the pre-trained teacher recognition model.

[0116] In specific implementation, when the target semantic depth includes low semantic depth, the sample image can be input into the pre-trained teacher recognition model. The sample image is then convolved by the shallow convolutional network of the teacher recognition model to obtain the feature map output by the shallow convolutional network of the teacher recognition model. The feature map output by the shallow convolutional network of the teacher recognition model can be used as the second semantic feature map.

[0117] When the target semantic depth includes intermediate semantic depth, the feature map output by the shallow convolutional network can be convolved by the intermediate convolutional network connected to the shallow convolutional network in the teacher recognition model to obtain the feature map output by the intermediate convolutional network of the teacher recognition model. The feature map output by the intermediate convolutional network of the teacher recognition model can be used as the second semantic feature map.

[0118] When the target semantic depth includes both low and medium semantic depth, the feature maps output by the shallow convolutional network and the medium convolutional network of the teacher recognition model can be obtained and used as the second semantic feature map of the target semantic depth.

[0119] In step S207, the first semantic feature map is enhanced in the frequency domain based on the frequency domain features corresponding to the first semantic feature map of the target semantic depth to obtain the first frequency domain enhanced feature map of the target semantic depth; and the second semantic feature map is enhanced in the frequency domain based on the frequency domain features corresponding to the second semantic feature map of the target semantic depth to obtain the second frequency domain enhanced feature map of the target semantic depth.

[0120] Among them, frequency domain features can be obtained by transforming the semantic feature map to the frequency domain space.

[0121] In one exemplary implementation, performing frequency domain enhancement processing on the first semantic feature map based on the frequency domain features corresponding to the first semantic feature map to obtain a first frequency domain enhanced feature map of the target semantic depth may include:

[0122] Perform a Fourier transform on the first semantic feature map to obtain the first frequency domain feature map corresponding to the first semantic feature map;

[0123] Based on the first frequency domain feature map and the first mask map corresponding to the first frequency domain feature map, the first enhanced frequency domain map is determined;

[0124] Perform an inverse Fourier transform on the first enhanced frequency domain map to obtain the first enhanced frequency domain feature map.

[0125] The Fourier transform can be a two-dimensional discrete Fourier transform, thereby transforming the image from the image space to the frequency domain space. For example, the Fourier transform of a semantic feature map can be achieved using the following formula (1):

[0126] (1)

[0127] in, Represents points on the semantic feature map; M represents the semantic feature map; M represents the semantic feature map in the image coordinate system. The maximum value corresponding to the dimension; N represents the semantic feature map in the image coordinate system. The maximum value corresponding to the dimension; Represents a point in the frequency domain coordinate system; This represents the frequency domain feature map.

[0128] The first mask image corresponding to the first frequency domain feature map can be obtained by masking the first frequency domain feature map.

[0129] For example, masking can be implemented based on the following formula (2):

[0130] (2)

[0131] in, express The corresponding mask image; Represents the frequency domain feature map in the frequency domain coordinate system The maximum value corresponding to the dimension; Represents the frequency domain feature map in the frequency domain coordinate system The maximum value corresponding to the dimension; This represents the adjustment coefficient, which can be set according to actual needs, for example, it can be 0.5 or 1; The mask value is not equal to 1 and can be set according to actual needs.

[0132] In specific implementation, the enhanced frequency domain map can be obtained by calculating the product of the frequency domain feature map and the mask map corresponding to the frequency domain feature map. The exemplary calculation process can be expressed as the following formula (3):

[0133] (3)

[0134] in, This represents the enhanced frequency domain diagram.

[0135] The inverse Fourier transform is the inverse process of the Fourier transform, which can convert from the frequency domain space to the image space. For example, the inverse Fourier transform can be a two-dimensional discrete inverse Fourier transform. Specifically, following the aforementioned formulas (1) to (3), the inverse Fourier transform can be implemented using the following formula (4):

[0136] (4)

[0137] in, This represents the frequency domain enhanced feature map.

[0138] Then, when in the above formula (1) When the first semantic feature map is obtained, the first frequency domain enhanced feature map can be obtained by combining the above formulas (1) to (4).

[0139] Similarly, when in the above formula (1) When the second semantic feature map is obtained, the second frequency domain enhanced feature map can be obtained by combining the above formulas (1) to (4). That is to say, the frequency domain enhancement processing of the second semantic feature map based on the frequency domain features corresponding to the second semantic feature map in step S207 above, to obtain the second frequency domain enhanced feature map, may include:

[0140] Perform a Fourier transform on the second semantic feature map to obtain the second frequency domain feature map corresponding to the second semantic feature map;

[0141] The second enhanced frequency domain map is determined based on the second frequency domain feature map and the second mask map corresponding to the second frequency domain feature map;

[0142] Perform an inverse Fourier transform on the second enhanced frequency domain map to obtain the second enhanced frequency domain feature map.

[0143] In the above implementation, the semantic feature map (including the first semantic feature map and the second semantic feature map) is transformed onto the frequency domain feature map by Fourier transform. Then, a mask map is generated for different semantic feature maps. The product of the frequency domain feature map and the mask map is used as the feature map of the enhanced frequency domain. The enhanced frequency domain feature map is then subjected to inverse Fourier transform to obtain the enhanced frequency domain feature map.

[0144] In step S209, the object recognition model is trained based on the difference between the first frequency domain enhanced feature map of the target semantic depth and the second frequency domain enhanced feature map of the target semantic depth until the preset training termination condition is met.

[0145] Specifically, a first loss value can be determined based on the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map of the target semantic depth using a preset loss function. Then, the model parameters of the object recognition model to be trained can be adjusted based on the first loss value. The training can continue iteratively based on the adjusted model parameters until the preset training termination condition is met. The object recognition model corresponding to the model parameters at the end of training is then used as the final online object recognition model.

[0146] The difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map of the target semantic depth can be represented by the distance between them; the greater the distance, the greater the difference, and vice versa. This distance can be Euclidean distance, Manhattan distance, etc.

[0147] The preset loss function used to calculate the first loss value can be either the minimum absolute value deviation loss (i.e., the L1 loss function) or the minimum squared error loss (i.e., the L2 norm loss function). The L1 loss function minimizes the sum of the absolute differences between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map of the target semantic depth. The L2 norm loss function minimizes the sum of the squares of the differences between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map of the target semantic depth.

[0148] For example, the first loss value It can be expressed by the following formula (5):

[0149] (5)

[0150] in, Indicates the number of sample images; The first frequency domain enhanced feature map represents the semantic depth of the target; The second frequency domain enhanced feature map represents the semantic depth of the target.

[0151] In this embodiment of the disclosure, when adjusting the model parameters of the object recognition model to be trained based on the first loss value, stochastic gradient descent can be used to update the model parameters of the object recognition model to be trained. The preset training termination condition can be that the first loss value reaches a preset minimum loss value, which can be set based on practical experience; of course, the preset training termination condition can also be that the number of iterations reaches a preset iteration threshold, which can also be set based on practical experience, for example, it could be 100, etc.

[0152] In practical applications, the dimension of the second semantic feature map of the target semantic depth obtained based on the teacher recognition model and the dimension of the second semantic feature value of the target semantic depth obtained based on the object recognition model may be the same or different. When the dimensions differ, the dimension of the second semantic feature map of the target semantic depth obtained based on the teacher recognition model is usually larger than the dimension of the second semantic feature value of the target semantic depth obtained based on the object recognition model. Therefore, when determining the first loss value, dimensionality adjustment can be performed first to adjust the first frequency domain enhanced feature map and the second frequency domain enhanced feature map of the target semantic depth to the same dimension. Then, a preset loss function is used to determine the first loss value based on the difference between the two. Dimension adjustment can be achieved by convolutional processing of the feature maps using convolutional layers of appropriate size.

[0153] The above-described technical solution of this disclosure extracts semantic features of sample images at the target semantic depth using an object recognition model, and performs scale compression processing on these semantic features to obtain a first semantic feature map. This reduces the computational resource consumption of the object recognition model in subsequent recognition processes. To ensure a high recognition accuracy for the trained object recognition model, a second semantic feature map is obtained by further extracting semantic features of sample images at the target semantic depth using a pre-trained teacher recognition model. Frequency domain enhancement processing is then performed on the first and second semantic feature maps at the target semantic depth, respectively. The object recognition model is then trained based on the difference between the first and second frequency domain enhanced feature maps after frequency domain enhancement processing. This allows the object recognition model to learn the enhanced frequency domain information in the feature maps extracted by the teacher recognition model, narrowing the gap between the object recognition model and the teacher recognition model. Ultimately, the object recognition model effectively reduces computational resource consumption while maintaining recognition accuracy, thus facilitating the deployment of the trained object recognition model in scenarios such as embedded devices.

[0154] In one exemplary implementation, to further improve the recognition accuracy of the trained object recognition model, such as... Figure 4 As shown, step S209 above may include the following in a specific implementation:

[0155] In step S401, a first loss value is determined based on the difference between the first frequency domain enhanced feature map of the target semantic depth and the second frequency domain enhanced feature map of the target semantic depth.

[0156] In step S403, the object recognition model performs object recognition processing on the sample image based on the first semantic feature map to obtain a first recognition result; based on the first recognition result and the reference category label corresponding to the sample image, a second loss value is determined.

[0157] In step S405, the sample image is processed by the teacher recognition model based on the second semantic feature map to obtain the second recognition result.

[0158] In step S407, a third loss value is determined based on the first identification result and the second identification result.

[0159] In step S409, the model parameters of the object recognition model are adjusted based on the first loss value, the second loss value, and the third loss value until the preset training termination condition is met.

[0160] The preset loss function used to calculate the first loss value can be either the minimum absolute value deviation loss (i.e., the L1 loss function) or the minimum squared error loss (i.e., the L2 norm loss function). The L1 loss function minimizes the sum of the absolute differences between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map of the target semantic depth. The L2 norm loss function minimizes the sum of the squares of the differences between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map of the target semantic depth. In specific implementations, the first loss value can be calculated based on the aforementioned formula (5). .

[0161] Here, the first recognition result indicates the probability that the sample image predicted by the object recognition model belongs to the corresponding reference category, and the loss function used to calculate the second loss value can be the cross-entropy loss function. For example, the second loss value... It can be obtained through the following formula (6):

[0162] (6)

[0163] in, This represents the number of reference categories; n represents the number of sample images; The reference category label for sample image c; This represents the first recognition result of the object recognition model for the sample image c.

[0164] The second recognition result indicates the probability that the sample image predicted by the teacher recognition model belongs to the corresponding reference type. The loss function used to calculate the third loss value can be the KL divergence function. For example, the third loss value... It can be obtained through the following formula (7):

[0165] (7)

[0166] in, This represents the second recognition result of the teacher recognition model for sample image c; This represents the first recognition result of the object recognition model for the sample image c.

[0167] Specifically, when adjusting the model parameters of the object recognition model based on the first loss value, the second loss value, and the third loss value until the preset training termination condition is reached, weight values ​​can be assigned to the first loss value, the second loss value, and the third loss value respectively. Then, the first loss value, the second loss value, the third loss value, and the corresponding weight values ​​are weighted and summed to obtain the total loss value. The model parameters of the object recognition model are then adjusted based on the total loss value until the preset training termination condition is reached.

[0168] The above implementation method uses a first loss value to narrow the distance between the frequency domain enhancement features of the teacher recognition model and the object recognition model, thereby achieving information transmission. At the same time, it uses a second loss value and a third loss value to narrow the distance between the final classification features of the teacher recognition model and the object recognition model, thereby achieving more effective information transmission. Ultimately, the object recognition model trained based on the first loss value, the second loss value, and the third loss value has a recognition accuracy similar to that of the teacher recognition model, which further improves the recognition accuracy of the object recognition model while reducing the computational resource consumption of the object recognition model.

[0169] In one exemplary implementation, determining the first loss value based on the difference between a first frequency domain enhanced feature map of the target semantic depth and a second frequency domain enhanced feature map of the target semantic depth may include:

[0170] The first sub-loss value is determined based on the difference between the first frequency domain enhanced feature map corresponding to the shallow convolutional network and the second frequency domain enhanced feature map corresponding to the shallow convolutional network.

[0171] The second sub-loss value is determined based on the difference between the first frequency domain enhanced feature map corresponding to the middle convolutional network and the second frequency domain enhanced feature map corresponding to the middle convolutional network.

[0172] The first loss value is determined based on the first sub-loss value and the second sub-loss value.

[0173] Specifically, a first sub-loss value can be determined using a preset loss function, based on the difference between the first frequency domain enhanced feature map corresponding to the shallow convolutional network in the object recognition model and the second frequency domain enhanced feature map corresponding to the shallow convolutional network in the teacher recognition model. A second sub-loss value can then be determined using the same preset loss function, based on the difference between the first frequency domain enhanced feature map of the object in the middle convolutional network of the object recognition model and the second frequency domain enhanced feature map corresponding to the middle convolutional network in the teacher recognition model. Finally, the first loss value can be determined by combining the first and second sub-loss values; for example, the sum of the first and second sub-losses can be used as the first loss value.

[0174] The preset loss function used to calculate the first sub-loss value and the second sub-loss value can be the minimum absolute value deviation loss (i.e., the L1 loss function) or the minimum squared error loss (i.e., the L2 norm loss function).

[0175] In the above embodiments, by compressing the spatial scale of low-level and mid-level features and transmitting feature information for the frequency domain enhancement features corresponding to low-level and mid-level features respectively, the computational resource consumption of the object recognition model is significantly reduced while the recognition accuracy of the trained object recognition model is improved.

[0176] To further improve the accuracy of the object recognition model while reducing the computational resource consumption of the trained object recognition model during the recognition process, in an exemplary implementation, determining the first loss value based on the first sub-loss value and the second sub-loss value may further include:

[0177] The third sub-loss value is determined based on the difference between the frequency domain enhanced feature map corresponding to the high-level convolutional network in the object recognition model and the frequency domain enhanced feature map corresponding to the high-level convolutional network in the teacher recognition model.

[0178] The first loss value is determined based on the first sub-loss value, the second sub-loss value, and the third sub-loss value;

[0179] Among them, the frequency domain enhanced feature map corresponding to the high-level convolutional network is obtained based on the feature map output by the high-level convolutional network. The feature map output by the high-level convolutional network is obtained by scaling the feature map output by the connected middle-level convolutional network.

[0180] Specifically, after the sample image is input into the object recognition model, feature extraction is first performed through the shallow convolutional network of the object recognition model to obtain the feature map output by the shallow convolutional network. After scale compression, the scale-compressed feature map is used as the input to the middle convolutional network in the object recognition model. Feature extraction is performed through the middle convolutional network to obtain the feature map output by the middle convolutional network. After scale compression, the scale-compressed feature map is used as the input to the high-level convolutional network in the object recognition model. Feature extraction is performed through the high-level convolutional network to obtain the feature map output by the high-level convolutional network. Then, frequency domain enhancement is performed on the feature map output by the high-level convolutional network in the object recognition model to obtain the frequency domain enhanced feature map corresponding to the high-level convolutional network in the object recognition model.

[0181] In addition, after the sample image is input into the teacher recognition model, feature extraction is first performed through the shallow convolutional network of the teacher recognition model to obtain the feature map output by the shallow convolutional network. The feature map output by the shallow convolutional network is then used as the input to the middle convolutional network in the teacher recognition model, and feature extraction is performed through the middle convolutional network to obtain the feature map output by the middle convolutional network in the teacher recognition model. The feature map output by the middle convolutional network is then used as the input to the high-level convolutional network in the teacher recognition model, and feature extraction is performed through the high-level convolutional network to obtain the feature map output by the high-level convolutional network in the teacher recognition model. Then, frequency domain enhancement processing is performed on the feature map output by the high-level convolutional network in the teacher recognition model to obtain the frequency domain enhanced feature map corresponding to the high-level convolutional network in the teacher recognition model.

[0182] For details on frequency domain enhancement of the feature maps output by high-level convolutional networks, please refer to the aforementioned documentation. Figure 2 The relevant descriptions of step S207 in the method embodiment shown will not be repeated here.

[0183] The preset loss function used to calculate the third sub-loss value can be either the minimum absolute value deviation loss (i.e., the L1 loss function) or the minimum squared error loss (i.e., the L2 norm loss function).

[0184] Among them, determining the first loss value based on the first sub-loss value, the second sub-loss value, and the third sub-loss value can be done by using the sum of the first sub-loss value, the second sub-loss value, and the third sub-loss value as the first loss value.

[0185] The above implementation method, by adding information transmission of frequency domain enhancement features corresponding to high-level convolutional networks on the basis of low-level and mid-level features, can further improve the recognition accuracy of the object recognition model while reducing the consumption of computing resources by the trained object recognition model during the recognition process.

[0186] To gain a clearer understanding of the technical solutions in the embodiments of this disclosure, the following is in conjunction with... Figure 5 Let's illustrate this with a specific example.

[0187] like Figure 5 As shown, the sample images are processed by both the object recognition model to be trained and the teacher recognition model. For the object recognition model to be trained, low-level semantic feature maps of the sample images are extracted through the shallow convolutional network layers of the object recognition model. The low-level semantic feature map is then scaled down to obtain the compressed low-level semantic feature map. The compressed low semantic feature map The input is further fed into the middle convolutional network layer of the object recognition model for feature extraction to obtain the middle semantic feature map. And the semantic feature map of the middle layer Scale compression is performed to obtain the compressed mid-level semantic feature map. The compressed mid-level semantic feature map The input is further fed into the high-level convolutional network layers of the object recognition model for feature extraction to obtain high-level semantic feature maps. This high-level semantic feature map The sample image is further input into the classifier of the object recognition model for category prediction, resulting in the first recognition result. .

[0188] For the teacher recognition model, low-level semantic feature maps of sample images are extracted through the shallow convolutional network layers of the teacher recognition model. The low-level semantic feature map The input is further fed into the middle convolutional network layer of the teacher recognition model for feature extraction to obtain the middle semantic feature map. The middle-layer semantic feature map The data is further fed into the high-level convolutional network layers of the teacher recognition model for feature extraction to obtain high-level semantic feature maps. This high-level semantic feature map The sample image is then fed into the classifier of the teacher recognition model for category prediction, yielding a second recognition result. .

[0189] In addition, during the training process, information is transferred between the teacher recognition model and the object recognition model through the first information transfer module and the second information transfer module.

[0190] The first information transmission module is used to bridge the gap in frequency domain enhanced features between the teacher recognition model and the object recognition model. For example... Figure 5As shown, first information transmission modules can be set up corresponding to shallow convolutional network layers, mid-level convolutional network layers, and high-level convolutional network layers, respectively. The input to the first information transmission module corresponding to the shallow convolutional network layer includes the compressed low-level semantic feature map and the low-level semantic feature map corresponding to the teacher recognition model; the input to the first information transmission module corresponding to the mid-level convolutional network layer includes the compressed mid-level semantic feature map and the mid-level semantic feature map corresponding to the teacher recognition model; and the input to the first information transmission module corresponding to the high-level convolutional network layer includes the high-level semantic feature map of the object recognition model and the high-level semantic feature map of the teacher recognition model.

[0191] Specifically, such as Figure 6 The diagram shown is a schematic of the first information transmission module and a semantic feature map of the object recognition model. The corresponding frequency domain feature map is obtained after DFT (Fourier Transform). This frequency domain feature map and its corresponding mask image Multiplication yields an enhanced frequency domain map This enhanced frequency domain diagram The corresponding frequency domain enhanced feature map is obtained after IDFT (Inverse Fourier Transform). Then, the frequency domain enhanced feature map Upsampling (achievable through convolution) is performed to scale the feature map to a dimension appropriate for the frequency domain enhancement feature map corresponding to the teacher recognition model, resulting in... Similarly, the semantic feature map of the teacher identification model. The corresponding frequency domain feature map is obtained after DFT (Fourier Transform). This frequency domain feature map and its corresponding mask image Multiplication yields an enhanced frequency domain map This enhanced frequency domain diagram The corresponding frequency domain enhanced feature map is obtained after IDFT (Inverse Fourier Transform). Finally, the L1 loss function is used to enhance the feature map in the frequency domain corresponding to the object recognition model. Frequency domain enhanced feature map corresponding to the teacher recognition model The difference between them determines the loss value corresponding to the first information transmission module.

[0192] The second information delivery module is used to bridge the gap between the final classification features of the teacher recognition model and the object recognition model. In specific implementation, the second information delivery module can be implemented based on classification knowledge distillation, that is, using the difference between the first recognition result of the object recognition model and the second recognition result of the teacher recognition model, as well as the difference between the first recognition result of the object recognition model and the reference category label of the corresponding sample image, to determine the loss value corresponding to the second information delivery model.

[0193] Furthermore, the total loss used to train the object recognition model is the sum of the loss values ​​corresponding to the first information transmission module and the loss values ​​corresponding to the second information transmission module. Based on this total loss, the model parameters of the object recognition model are updated using gradient descent to iteratively train the object recognition model and obtain the final trained object recognition model.

[0194] Understandably, a well-trained object recognition model can be deployed to embedded devices such as terminals to identify target objects in the image to be processed.

[0195] In practice, the image to be processed can be input into an object recognition model. The shallow convolutional network layer of the object recognition model extracts features from the image to obtain a low-level semantic feature map. This low-level semantic feature map is then scaled to obtain a compressed low-level semantic feature map. This compressed low-level semantic feature map is then input into a mid-level convolutional network layer for feature extraction to obtain a mid-level semantic feature map. This mid-level semantic feature map is then scaled to obtain a compressed mid-level semantic feature map. This compressed mid-level semantic feature map is then input into a high-level convolutional network layer for feature extraction to obtain a high-level semantic feature map. Finally, this high-level semantic feature map is input into a classifier for classification processing to obtain the recognition result corresponding to the image to be processed. This recognition result indicates the category of the target object in the image to be processed.

[0196] Figure 7 This is a structural block diagram illustrating an object recognition model training device according to an exemplary embodiment. (Refer to...) Figure 7 The object recognition model training device 700 includes:

[0197] The first feature extraction unit 710 is configured to extract semantic features of the sample image at the target semantic depth based on the object recognition model to be trained, and obtain an initial semantic feature map of the target semantic depth; the object recognition model is used to identify the target object in the sample image;

[0198] The scaling compression unit 720 is configured to perform scaling compression processing on the initial semantic feature map of the target semantic depth to obtain the first semantic feature map of the target semantic depth.

[0199] The second feature extraction unit 730 is configured to perform a pre-trained teacher recognition model to extract semantic features of the sample image at the target semantic depth, thereby obtaining a second semantic feature map at the target semantic depth; the teacher recognition model is used to identify the target object in the sample image.

[0200] The frequency domain enhancement processing unit 740 is configured to perform frequency domain enhancement processing on the first semantic feature map based on the frequency domain features corresponding to the first semantic feature map to obtain a first frequency domain enhanced feature map of the target semantic depth; and to perform frequency domain enhancement processing on the second semantic feature map based on the frequency domain features corresponding to the second semantic feature map to obtain a second frequency domain enhanced feature map of the target semantic depth.

[0201] Training unit 750 is configured to perform training on the object recognition model by performing the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map based on the target semantic depth until the preset training termination condition is met, thereby obtaining the trained object recognition model.

[0202] In one exemplary embodiment, the sample images correspond to reference category labels; the training unit 750 includes:

[0203] The first loss determination unit is configured to determine a first loss value based on the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map.

[0204] The second loss determination unit is configured to perform object recognition processing on the sample image based on the first semantic feature map using the object recognition model to obtain a first recognition result; and determine a second loss value based on the first recognition result and the reference category label corresponding to the sample image.

[0205] The first recognition unit is configured to perform object recognition processing on the sample image based on the second semantic feature map using the teacher recognition model, and obtain a second recognition result;

[0206] The third loss determination unit is configured to determine a third loss value based on the first identification result and the second identification result;

[0207] The parameter adjustment unit is configured to adjust the model parameters of the object recognition model based on the first loss value, the second loss value, and the third loss value until the preset training termination condition is met.

[0208] In one exemplary embodiment, the first feature extraction unit includes:

[0209] The first convolutional unit is configured to input the sample image into the object recognition model to be trained, and perform convolution processing on the sample image through the shallow convolutional network of the object recognition model to obtain the feature map output by the shallow convolutional network.

[0210] The feature extraction subunit is configured to perform an initial semantic feature map of the target semantic depth based on the feature map output by the shallow convolutional network; the shallow convolutional network includes a predetermined number of convolutional layers close to the input layer in the object recognition model.

[0211] In one exemplary embodiment, the feature extraction subunit is specifically configured to perform scale compression processing on the feature map output by the shallow convolutional network, input the scale-compressed feature map into the mid-level convolutional network connected to the shallow convolutional network for convolution processing, and obtain the feature map output by the mid-level convolutional network; and use the feature map output by the mid-level convolutional network and / or the feature map output by the shallow convolutional network as the initial semantic feature map of the target semantic depth.

[0212] In one exemplary embodiment, the first loss determination unit includes:

[0213] The first sub-loss determination unit is configured to determine the first sub-loss value based on the difference between the first frequency domain enhanced feature map corresponding to the shallow convolutional network and the second frequency domain enhanced feature map corresponding to the shallow convolutional network.

[0214] The second sub-loss determination unit is configured to determine the second sub-loss value based on the difference between the first frequency domain enhanced feature map corresponding to the middle convolutional network and the second frequency domain enhanced feature map corresponding to the middle convolutional network.

[0215] The fourth loss determination unit is configured to determine the first loss value based on the first sub-loss value and the second sub-loss value.

[0216] In one exemplary embodiment, the fourth loss determination unit includes:

[0217] The third sub-loss determination unit is configured to perform a third sub-loss value based on the difference between the frequency domain enhanced feature map corresponding to the high-level convolutional network in the object recognition model and the frequency domain enhanced feature map corresponding to the high-level convolutional network in the teacher recognition model.

[0218] The fifth loss determination unit is configured to determine the first loss value based on the first sub-loss value, the second sub-loss value, and the third sub-loss value;

[0219] The frequency domain enhanced feature map corresponding to the high-level convolutional network is obtained based on the feature map output by the high-level convolutional network, which is obtained by scaling the feature map output by the connected middle-level convolutional network.

[0220] In one exemplary embodiment, the frequency domain enhancement processing unit 740 includes:

[0221] The first frequency domain enhancement processing unit is configured to perform a Fourier transform on the first semantic feature map to obtain a first frequency domain feature map corresponding to the first semantic feature map; determine a first enhanced frequency domain map based on the first frequency domain feature map and the first mask map corresponding to the first frequency domain feature map; and perform an inverse Fourier transform on the first enhanced frequency domain map to obtain a first frequency domain enhanced feature map.

[0222] The second frequency domain enhancement processing unit performs a Fourier transform on the second semantic feature map to obtain a second frequency domain feature map corresponding to the second semantic feature map; determines a second enhanced frequency domain map based on the second frequency domain feature map and the second mask map corresponding to the second frequency domain feature map; and performs an inverse Fourier transform on the second enhanced frequency domain map to obtain a second frequency domain enhanced feature map.

[0223] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0224] In one exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing processor-executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, it implements the object recognition model training method provided in any of the above embodiments.

[0225] The electronic device can be a terminal, a server, or a similar computing device. Taking a server as an example... Figure 8 This is a block diagram illustrating an electronic device for training an object recognition model according to an exemplary embodiment, such as... Figure 8As shown, the server 800 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 810 (CPUs 810 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 830 for storing data, and one or more storage media 820 (e.g., one or more mass storage devices) for storing application programs 823 or data 822. The memory 830 and storage media 820 may be temporary or persistent storage. The program stored in the storage media 820 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 810 may be configured to communicate with the storage media 820 and execute the series of instruction operations stored in the storage media 820 on the server 800. Server 800 may also include one or more power supplies 860, one or more wired or wireless network interfaces 850, one or more input / output interfaces 840, and / or one or more operating systems 821, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0226] The input / output interface 840 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 800. In one example, the input / output interface 840 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 840 may be a radio frequency (RF) module used for wireless communication with the Internet.

[0227] Those skilled in the art will understand that Figure 8 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 800 may also include... Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown.

[0228] In one exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 830 including instructions, which can be executed by a processor 810 of a server 800 to perform the above-described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0229] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the object recognition model training method provided in any of the above embodiments.

[0230] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0231] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for training an object recognition model, characterized in that, include: Based on the object recognition model to be trained, the semantic features of the sample image at the target semantic depth are extracted to obtain the initial semantic feature map of the target semantic depth; The object recognition model is used to identify target objects in the sample images; The initial semantic feature map of the target semantic depth is subjected to scale compression processing to obtain the first semantic feature map of the target semantic depth; Based on a pre-trained teacher recognition model, semantic features of the sample image at the target semantic depth are extracted to obtain a second semantic feature map at the target semantic depth; the teacher recognition model is used to identify target objects in the sample image; The first semantic feature map is enhanced by performing frequency domain enhancement processing on the frequency domain features corresponding to the first semantic feature map to obtain a first frequency domain enhanced feature map of the target semantic depth; and the second semantic feature map is enhanced by performing frequency domain enhancement processing on the frequency domain features corresponding to the second semantic feature map to obtain a second frequency domain enhanced feature map of the target semantic depth. Based on the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map of the target semantic depth, the object recognition model is trained until the preset training termination condition is met, and the trained object recognition model is obtained.

2. The method according to claim 1, characterized in that, The sample images correspond to reference category labels; the object recognition model is trained based on the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map of the target semantic depth until a preset training termination condition is reached, including: A first loss value is determined based on the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map; The object recognition model performs object recognition processing on the sample image based on the first semantic feature map to obtain a first recognition result; a second loss value is determined based on the first recognition result and the reference category label corresponding to the sample image. The teacher recognition model performs object recognition processing on the sample image based on the second semantic feature map to obtain a second recognition result; Based on the first identification result and the second identification result, a third loss value is determined; Based on the first loss value, the second loss value, and the third loss value, the model parameters of the object recognition model are adjusted until the preset training termination condition is met.

3. The method according to claim 2, characterized in that, The method of extracting semantic features of sample images at the target semantic depth based on the object recognition model to be trained, and obtaining the initial semantic feature map of the target semantic depth, includes: The sample image is input into the object recognition model to be trained, and the sample image is processed by the shallow convolutional network of the object recognition model to obtain the feature map output by the shallow convolutional network; the shallow convolutional network includes a preset number of convolutional layers close to the input layer in the object recognition model; Based on the feature map output by the shallow convolutional network, an initial semantic feature map of the target semantic depth is obtained.

4. The method according to claim 3, characterized in that, The initial semantic feature map of the target semantic depth is obtained based on the feature map output by the shallow convolutional network, including: The feature map output by the shallow convolutional network is scaled and compressed. The scaled and compressed feature map is then input into the middle convolutional network connected to the shallow convolutional network for convolution processing to obtain the feature map output by the middle convolutional network. The feature maps output by the middle-layer convolutional network and / or the feature maps output by the shallow-layer convolutional network are used as the initial semantic feature maps for the target semantic depth.

5. The method according to claim 4, characterized in that, The step of determining the first loss value based on the difference between the first frequency domain enhanced feature map and the second frequency domain enhanced feature map includes: The first sub-loss value is determined based on the difference between the first frequency domain enhanced feature map corresponding to the shallow convolutional network and the second frequency domain enhanced feature map corresponding to the shallow convolutional network. The second sub-loss value is determined based on the difference between the first frequency domain enhanced feature map corresponding to the middle convolutional network and the second frequency domain enhanced feature map corresponding to the middle convolutional network. The first loss value is determined based on the first sub-loss value and the second sub-loss value.

6. The method according to claim 5, characterized in that, Determining the first loss value based on the first sub-loss value and the second sub-loss value includes: The third sub-loss value is determined based on the difference between the frequency domain enhanced feature map corresponding to the high-level convolutional network in the object recognition model and the frequency domain enhanced feature map corresponding to the high-level convolutional network in the teacher recognition model. The first loss value is determined based on the first sub-loss value, the second sub-loss value, and the third sub-loss value; The frequency domain enhanced feature map corresponding to the high-level convolutional network is obtained based on the feature map output by the high-level convolutional network, and the feature map output by the high-level convolutional network is obtained by performing scale compression processing on the feature map output by the connected middle-level convolutional network.

7. The method according to any one of claims 1 to 6, characterized in that, The step of performing frequency domain enhancement processing on the first semantic feature map based on the frequency domain features corresponding to the first semantic feature map to obtain the first frequency domain enhanced feature map of the target semantic depth includes: Perform a Fourier transform on the first semantic feature map to obtain the first frequency domain feature map corresponding to the first semantic feature map; Based on the first frequency domain feature map and the first mask map corresponding to the first frequency domain feature map, a first enhanced frequency domain map is determined; an inverse Fourier transform is performed on the first enhanced frequency domain map to obtain the first frequency domain enhanced feature map. The step of performing frequency domain enhancement processing on the second semantic feature map based on the frequency domain features corresponding to the second semantic feature map to obtain the second frequency domain enhanced feature map of the target semantic depth includes: Perform a Fourier transform on the second semantic feature map to obtain the second frequency domain feature map corresponding to the second semantic feature map; Based on the second frequency domain feature map and the second mask map corresponding to the second frequency domain feature map, a second enhanced frequency domain map is determined; an inverse Fourier transform is performed on the second enhanced frequency domain map to obtain the second enhanced frequency domain feature map.

8. A training device for an object recognition model, characterized in that, include: The first feature extraction unit is configured to extract semantic features of the sample image at the target semantic depth based on the object recognition model to be trained, and obtain an initial semantic feature map of the target semantic depth; The object recognition model is used to identify target objects in the sample images; The scaling compression unit is configured to perform scaling compression processing on the initial semantic feature map of the target semantic depth to obtain a first semantic feature map of the target semantic depth. The second feature extraction unit is configured to perform a pre-trained teacher recognition model to extract semantic features of the sample image at the target semantic depth, thereby obtaining a second semantic feature map at the target semantic depth; the teacher recognition model is used to identify target objects in the sample image. The frequency domain enhancement processing unit is configured to perform frequency domain enhancement processing on the first semantic feature map based on the frequency domain features corresponding to the first semantic feature map to obtain a first frequency domain enhanced feature map of the target semantic depth; and to perform frequency domain enhancement processing on the second semantic feature map based on the frequency domain features corresponding to the second semantic feature map to obtain a second frequency domain enhanced feature map of the target semantic depth. The training unit is configured to perform training on the object recognition model by performing a difference between a first frequency domain enhanced feature map and a second frequency domain enhanced feature map based on the target semantic depth, until a preset training termination condition is met, thereby obtaining a trained object recognition model.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the object recognition model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the object recognition model training method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the object recognition model training method according to any one of claims 1 to 7.

Images