Method, apparatus for training image processing model and image processing

Abstract

The present disclosure provides a method and device for training an image processing model and image processing, and relates to the technical field of artificial intelligence, in particular to the technical field of deep learning and computer vision. The specific implementation scheme is as follows: inputting a sample image of a pre-acquired training sample into a pre-acquired initial image processing model to obtain a class activation mapping corresponding to the sample image and an attention heat map corresponding to a target feature extraction layer, wherein the training sample comprises the sample image and corresponding label information; training the initial image processing model based on minimizing the difference between the label information and the output of a full connection layer in the initial image processing model and an attention difference to obtain an image processing model, wherein the attention difference is determined based on the difference between the class activation mapping corresponding to the sample image and the attention heat map corresponding to the target feature extraction layer. Thus, guided learning of the attention weight of the attention mechanism is realized.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of deep learning and computer vision, and more particularly to a method and device for training an image processing model and image processing. BACKGROUND

[0002] With the development of deep learning and computer vision technology, attention mechanism has also been applied more and more widely.

[0003] In order to simulate the selective attention of human vision (by quickly scanning the global image, obtaining the target area that needs to be focused on, and then investing more attention resources in this area to obtain more detailed information of the target that needs to be focused on, thereby suppressing other useless information), the current attention mechanism in computer vision mainly adopts channel attention, pixel attention, multi-order attention and other ways to improve the image processing effect. SUMMARY

[0004] A method and device for training an image processing model and image processing are provided.

[0005] According to a first aspect, a method for training an image processing model is provided, which comprises: inputting a sample image of a pre-acquired training sample into a pre-acquired initial image processing model to obtain a class activation map corresponding to the sample image and an attention heat map corresponding to a target feature extraction layer, wherein the training sample comprises the sample image and corresponding label information, and the initial image processing model comprises a fully connected layer and a plurality of feature extraction layers; training the initial image processing model based on minimizing the difference between the label information and the output of the fully connected layer and an attention difference to obtain an image processing model, wherein the attention difference is determined based on the difference between the class activation map corresponding to the sample image and the attention heat map corresponding to the target feature extraction layer.

[0006] According to a second aspect, a method for processing an image is provided, which comprises: acquiring an image to be processed; inputting the image to be processed into a pre-trained image processing model to generate an image processing result, wherein the image processing model is obtained by the method for training an image processing model as described in any implementation manner of the first aspect.

[0007] According to a third aspect, an apparatus for training an image processing model is provided, the apparatus comprising: a generating unit configured to input a sample image of a pre-acquired training sample to a pre-acquired initial image processing model to obtain a class activation map corresponding to the sample image and an attention heat map corresponding to a target feature extraction layer, wherein the training sample comprises the sample image and corresponding label information, and the initial image processing model comprises a fully connected layer and a plurality of feature extraction layers; and a training unit configured to train the initial image processing model based on minimizing a difference between the label information and an output of the fully connected layer and an attention difference to obtain the image processing model, wherein the attention difference is determined based on a difference between the class activation map corresponding to the sample image and the attention heat map corresponding to the target feature extraction layer.

[0008] According to a fourth aspect, an apparatus for processing an image is provided, the apparatus comprising: an obtaining unit configured to obtain an image to be processed; and a processing unit configured to input the image to be processed to a pre-trained image processing model to generate an image processing result, wherein the image processing model is obtained by the method for training an image processing model as described in any implementation of the first aspect.

[0009] According to a fifth aspect, an electronic device is provided, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method as described in any implementation of the first aspect or the second aspect.

[0010] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being used to enable a computer to perform the method as described in any implementation of the first aspect or the second aspect.

[0011] According to a seventh aspect, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method as described in any implementation of the first aspect or the second aspect.

[0012] According to an eighth aspect, an autonomous vehicle is provided, the autonomous vehicle comprising the electronic device as described in the fifth aspect.

[0013] According to the technical solution of the present disclosure, the initial image processing model is trained based on the difference between the class activation map corresponding to the sample image and the attention heat map corresponding to the target feature extraction layer, the attention weight loss is introduced into the loss function of the model training, thereby realizing guided learning of the attention weight of the attention mechanism, making the learning process easier and more intuitive. Moreover, the attention mechanism proposed in the present disclosure only needs to introduce a weight parameter in the loss function, without adding additional structures, thereby having the advantages of simple implementation, small calculation amount, intuitive principle, etc. Moreover, since the difference term between the label information corresponding to the sample image and the final output result of the initial image processing model is still retained in the loss function of the model training, it will not have obvious adverse effects on the model training effect, thereby ensuring the model training effect.

[0014] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0016] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure;

[0017] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure;

[0018] Figure 3 is a schematic diagram of one application scenario of the method for training an image processing model that can implement the embodiments of the present disclosure;

[0019] Figure 4 is a schematic diagram of a device for training an image processing model according to an embodiment of the present disclosure;

[0020] Figure 5 is a schematic diagram of a device for processing images according to an embodiment of the present disclosure;

[0021] Figure 6 is a block diagram of an electronic device for implementing the method for training an image processing model or for processing images according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and include various details intended to facilitate understanding of the present disclosure. Thus, it should be apparent to those skilled in the art that various modifications and changes can be made in the embodiments described herein without departing from the scope and spirit of the present disclosure. Likewise, the description is not to be construed to be exhaustive or limiting as to the forms disclosed. Many modifications and variations of the embodiments described herein are possible in light of the above teachings. It is therefore understood that within the scope of the appended claims, the present disclosure can be practiced otherwise than as specifically described herein.

[0023] Figure 1 is a schematic diagram 100 illustrating a method for training an image processing model according to a first embodiment of the present disclosure. The method for training an image processing model comprises the following steps:

[0024] S101, inputting a sample image of a pre-acquired training sample to a pre-acquired initial image processing model to obtain a class activation mapping corresponding to the sample image and an attention heat map corresponding to a target feature extraction layer.

[0025] In the present embodiment, the execution subject of the method for training an image processing model can first acquire a training sample and an initial image processing model in various ways. The training sample can include a sample image and corresponding annotation information. The initial image processing model can include various deep learning networks for image processing. The initial image processing model can include a fully connected layer and a plurality of feature extraction layers. The plurality of feature extraction layers are usually similar in structure and are linearly connected, i.e. the output feature map of a shallow feature extraction layer is usually used as the input of the next layer (i.e. a deep feature extraction layer).

[0026] It should be noted that the execution subject can also pre-acquire a training sample set composed of a plurality of training samples, which is not limited herein.

[0027] In the present embodiment, the execution subject can input a sample image of a pre-acquired training sample to a pre-acquired initial image processing model to obtain a class activation mapping corresponding to the sample image and an attention heat map corresponding to a target feature extraction layer. The class activation mapping (CAM) can be used to represent a heat map obtained by calculating the weight of the fully connected layer in the initial image processing model with the feature map output by the last feature extraction layer in the initial image processing model. It can reflect the attention of the initial image processing model to different positions of the input sample image. Optionally, the class activation mapping can also include a method for optimization based thereon, such as Grad-CAM (Gradient-weighted Class Activation Mapping).

[0028] As an example, the class activation map can be represented by the following equation (1):

[0029]

[0030] wherein the C can be used to represent the generated class activation map corresponding to the current input sample image. The F j may be used to represent the j-th channel of the feature map output by the last layer of the initial image processing model for the training sample corresponding to the current input sample image. The n can be used to represent the total number of channels of the feature map output by the last layer of the initial image processing model. The ω ij may be used to represent the weight value of the j-th channel of the weight vector of the initial image processing model corresponding to the current input sample image belonging to the i-th class.

[0031] Thus, the class activation map is usually the same size as the feature map output by the last layer of the initial image processing model, and thus can reflect the attention of the initial image processing model to different positions of the input sample image.

[0032] In the present embodiment, the execution subject can generate the attention heat map based on the feature map output by the target feature extraction layer and the attention weight to be trained in various ways. As an example, the execution subject can multiply the feature map output by the target feature extraction layer with a mask of the same size element by element and then add the feature map to generate the attention heat map. Wherein each element in the mask can be used to represent the attention weight corresponding to the pixel point.

[0033] It should be noted that the target feature extraction layer can be any feature extraction layer pre-specified from the plurality of feature extraction layers included in the initial image processing model. As another example, the target feature extraction layer can be a feature extraction layer determined according to rules, for example, two adjacent feature extraction layers.

[0034] S102, based on minimizing the difference between the label information and the output of the fully connected layer and the attention difference, training the initial image processing model to obtain an image processing model.

[0035] In the embodiment, the execution subject can train the initial image processing model by adjusting the weight of the initial image processing model and the attention weight in various manners (e.g., gradient descent method, stochastic gradient descent method, etc.) based on the difference between the annotation information and the output of the full connection layer and the attention difference. After multiple adjustments as described above, the trained image processing model is finally obtained. The attention difference is determined based on the difference between the class activation map corresponding to the input sample image and the attention heat map corresponding to the target feature extraction layer.

[0036] In the embodiment, the execution subject can specifically perform the following training steps:

[0037] In the first step, it is determined whether the preset training completion condition is met based on the preset loss function.

[0038] In the embodiment, the execution subject can first calculate the loss value according to the preset loss function. The loss function can include an output result loss obtained based on the minimization of the difference between the annotation information and the output of the full connection layer, and an attention loss obtained based on the attention difference. The loss function can take various forms, such as L1 loss, L2 loss, etc. Then, the execution subject can determine whether the preset training completion condition is met. The preset training completion condition can include, but is not limited to, at least one of the following: the loss value is less than a preset threshold, the loss value converges, the training duration reaches a preset duration, and the training iteration number reaches a preset number.

[0039] It should be noted that the forward process of the initial image processing model can be to multiply the attention weight corresponding to each feature extraction layer and the full connection layer with the corresponding feature map and then add it to the original feature map.

[0040] As an example, when the channel attention mechanism is used, the output of the feature extraction layer in the initial image processing model can be represented by the following formula (2):

[0041] T i ′=(1+q i )·T i i∈[1,p](2)

[0042] Wherein, T i ′ can be used to represent the output of the feature extraction layer in the initial image processing model corresponding to the current input sample image. T i can be used to represent the i-th channel of the feature map corresponding to the current input sample image in the feature extraction layer of the initial image processing model. q iThe attention weight value corresponding to the i-th channel of the feature extraction layer in the initial image processing model can be used to represent the attention weight value. The p can be used to represent the total number of channels of the feature map output by the feature extraction layer in the initial image processing model.

[0043] In the second step, the weights and attention weights to be trained in the initial image processing model are adjusted in response to the determination that the condition is not satisfied.

[0044] In the present embodiment, in response to the determination that the training completion condition in the first step is not satisfied, the execution subject can adjust the weights and attention weights to be trained in the initial image processing model using various methods such as the gradient descent method and the back propagation method.

[0045] In the third step, the adjusted initial image processing model is determined as a new initial image processing model, and the step S101 and the training step are continued.

[0046] In the fourth step, in response to the determination that the condition is satisfied, the initial image processing model that has completed training is determined as the image processing model.

[0047] In the present embodiment, in response to the determination that the training completion condition in the first step is satisfied, the execution subject can determine the initial image processing model that has completed training as the image processing model.

[0048] The method provided by the above embodiments of the present disclosure trains the initial image processing model based on the difference between the class activation map corresponding to the sample image and the attention heat map corresponding to the target feature extraction layer, introduces the attention weight loss into the loss function of the model training, and thus realizes guided learning of the attention weight of the attention mechanism. Compared with the traditional training method without an independent attention loss function for the attention mechanism, the learning of the attention weight value in the present application is easier and more intuitive. Moreover, since the attention mechanism proposed by the present disclosure only needs to introduce a weight parameter into the loss function without adding additional structures, it has the advantages of simple implementation, small calculation amount, and intuitive principle. Moreover, since the difference term between the label information corresponding to the sample image and the final output result of the initial image processing model is still retained in the loss function of the model training, it will not have a significant adverse effect on the model training effect, thereby ensuring the model training effect.

[0049] In some optional implementations of the present embodiment, the target feature extraction layer can include a last feature extraction layer connected to the fully connected layer. Based on minimizing the difference between the label information and the output of the fully connected layer and the attention difference, the execution subject can train the initial image processing model by the following steps:

[0050] In the first step, the difference between the label information and the output of the fully connected layer is determined as a first difference.

[0051] In these implementations, the execution subject can determine the difference between the labeling information and the output of the fully connected layer as the first difference in various ways according to different actual application scenarios. As an example, the execution subject can determine whether the class indicated by the labeling information is consistent with the class indicated by the output of the fully connected layer. As another example, the execution subject can determine the size of the non-overlapping part between the region where the target indicated by the labeling information is located and the region where the target indicated by the output of the fully connected layer is located.

[0052] Secondly, the difference between the class activation map corresponding to the input sample image and the attention heat map corresponding to the last feature extraction layer connected with the fully connected layer is determined as the second difference.

[0053] In these implementations, as an example, the second difference can be represented by the following formula (3):

[0054] L = ||C-H k ||2 (3)

[0055] Wherein, the C can be used to represent the generated class activation map corresponding to the current input sample image. The H k can be used to represent the generated attention heat map of the last feature extraction layer corresponding to the current input sample image. The || ||2 can be used to represent the mean squared error (MSE).

[0056] Thirdly, the initial image processing model is trained based on minimizing the first difference and the second difference.

[0057] In these implementations, the execution subject can adjust the network parameters and attention weights in the initial image processing model by using machine learning methods based on minimizing the first difference and the second difference determined in the first step and the second step, so as to train the initial image processing model.

[0058] Based on the optional implementation, the scheme can minimize the attention difference as part of the loss function to make the attention heat map of the last feature extraction layer consistent with the class activation map of the entire network (i.e., the initial image processing model), so as to optimize the feature extraction effect of the feature extraction layer.

[0059] In some optional implementations of the present embodiment, the target feature extraction layer can include adjacent feature extraction layers. Based on minimizing the first difference and the second difference, the execution subject can also train the initial image processing model by the following steps:

[0060] First, a difference between attention heat maps corresponding to adjacent feature extraction layers is determined as a third difference.

[0061] In these implementations, as an example, the third difference described above can be represented by the following formula (4):

[0062] L = ||H i -H i-1 ||2 (4)

[0063] where H i , H i-1 may be used to represent the attention heat map of the generated feature extraction layer corresponding to the current layer (e.g., deep layer) and the adjacent previous layer (e.g., previous layer) of the feature extraction layer of the sample image inputted at present. The || ||2may be used to represent the mean squared error (MSE).

[0064] Second, based on minimizing the first difference, the second difference and the third difference, the initial image processing model is trained.

[0065] In these implementations, the execution subject may, based on minimizing the first difference, the second difference and the third difference (e.g., the sum of the three) determined above, adjust the network parameters and attention weights in the initial image processing model in a machine learning manner, thereby training the initial image processing model.

[0066] Based on the optional implementation described above, the present scheme can minimize the attention difference as part of the loss function to realize the attention heat map of the shallow feature extraction layer and the attention heat map of the deep feature extraction layer Figure 1 , thereby passing the information learned by the deep network to the shallow network, guiding the shallow network to focus on important regions in the input sample image, and further making the focus points of the entire network (i.e., the initial image processing model) consistent to improve the effect of the image processing model.

[0067] In some optional implementations of the present embodiment, the attention heat map corresponding to the target feature extraction layer can be generated according to the attention weights to be trained assigned to the feature map output by the target feature extraction layer by channel.

[0068] In these implementations, as an example, when the channel attention mechanism is adopted, the attention heat map corresponding to the target feature extraction layer can be represented by the following formula (5):

[0069]

[0070] wherein, the H can be used to represent the attention heat map of the generated target feature extraction layer corresponding to the current input sample image. The M i can be used to represent the i-th channel of the feature map output by the target feature extraction layer corresponding to the current input sample image. The q i can be used to represent the attention weight value of the i-th channel of the target feature extraction layer. The m can be used to represent the total number of channels of the feature map output by the target feature extraction layer.

[0071] Therefore, the attention heat map of the target feature extraction layer is usually the same as the length and width of the feature map output by the target feature extraction layer. By analogy with the class activation map, the attention heat map can reflect the attention of the target feature extraction layer to different positions of the input sample image.

[0072] Based on the optional implementation, the scheme can apply the channel domain attention mechanism to the attention heat map, thereby enriching the generation method of the attention heat map, and further enriching the method of training the image processing model based on the attention difference.

[0073] In some optional implementations of the embodiment, the attention difference can further include the difference between the class activation map corresponding to the input sample image and the attention heat map corresponding to the last layer of feature extraction layer connected with the fully connected layer, and the difference between the attention heat maps corresponding to the adjacent feature extraction layers.

[0074] In these implementations, as an example, the attention difference can also be represented by the following formula (6):

[0075]

[0076] wherein, the meanings represented by the symbols can be consistent with the foregoing corresponding descriptions, which will not be described here.

[0077] Based on the optional implementation, the scheme can minimize the attention difference as part of the loss function to realize that the attention heat map of the last layer of feature extraction layer is consistent with the class activation map of the entire network (i.e., the initial image processing model), and the attention heat map of the shallow feature extraction layer is consistent with the attention heat map of the deep feature extraction layer. Figure 1 Therefore, the information learned by the deep network can be transmitted to the shallow network, guiding the shallow network to focus on important regions in the input sample image, thereby making the attention points of the entire network (i.e., the initial image processing model) consistent, optimizing the feature extraction effect of the feature extraction layer, and improving the effect of the image processing model.

[0078] Continuing to refer to Figure 2 ,Figure 2 is a schematic diagram 200 according to a second embodiment of the present disclosure. The method for processing an image comprises the following steps:

[0079] S201, obtaining an image to be processed.

[0080] In this embodiment, the execution subject of the method for processing an image can obtain the image to be processed in various ways. Among them, the above-mentioned image to be processed can include various images that can be processed by a deep learning network, which is not limited here.

[0081] In this embodiment, as an example, the above-mentioned image to be processed can be an image containing a person. As another example, the above-mentioned image to be processed can be a road condition image taken by an autonomous vehicle. The above-mentioned execution subject can obtain the above-mentioned image to be processed from a local or communicatively connected electronic device.

[0082] S202, inputting the image to be processed into a pre-trained image processing model to generate an image processing result.

[0083] In this embodiment, the above-mentioned execution subject can input the image to be processed obtained in the above-mentioned step S201 into a pre-trained image processing model to generate an image processing result corresponding to the image to be processed. Among them, the above-mentioned image processing model can be trained by the method for training an image processing model as described in any of the preceding embodiments. The above-mentioned image processing result can correspond to the above-mentioned image processing model. As an example, when the above-mentioned image processing model is a face recognition model, the above-mentioned image processing result can be used to represent the information of the person displayed in the face image. As another example, when the above-mentioned image processing model is a lane detection model, the above-mentioned image processing result can be used to indicate the position of the lane line displayed in the image.

[0084] From Figure 2 It can be seen that the flow 200 of the method for processing an image in this embodiment embodies the step of using the image processing model trained by the method for training an image processing model to process the image. Therefore, the scheme described in this embodiment can improve the effect of image processing by using the image processing model trained by introducing the improved attention mechanism.

[0085] Continuing to refer to Figure 3 , Figure 3 is a schematic diagram of an application scenario of the method for training an image processing model according to an embodiment of the present disclosure. In Figure 3In the application scenario, the execution subject (such as a server) for training the image processing model can input the sample image 3011 in the pre-acquired training sample 301 to the pre-acquired initial image processing model 302 to obtain the class activation map 304 corresponding to the sample image 3011 and the attention heat map 304 corresponding to the target feature extraction layer in the initial image processing model 302. Based on the result difference 305 between the labeled information 3012 in the training sample 301 and the output result of the initial image processing model 302 and the attention difference 306, the network weight in the initial image processing model 302 and the attention weight for generating the class activation map and the attention heat map are adjusted, and finally the image processing model is obtained after multiple rounds of adjustment.

[0086] At present, one of the existing technologies is usually to introduce the attention mechanism in the channel domain attention or attention mask and the like. However, since there is no term containing the attention loss in the loss function, the learning of the weight of the attention mechanism cannot be guided, and the learning process of the attention weight is difficult. The method provided by the above embodiment of the present disclosure trains the initial image processing model based on the difference between the class activation map corresponding to the sample image and the attention heat map corresponding to the target feature extraction layer, introduces the attention weight loss into the loss function of the model training, thereby realizing the guided learning of the attention weight of the attention mechanism, making the learning process easier and more intuitive. Moreover, since the attention mechanism proposed by the present disclosure only needs to introduce the weight parameter in the loss function and does not add additional structures, the method has the advantages of simple implementation, small calculation amount and intuitive principle. Moreover, since the difference term between the labeled information corresponding to the sample image and the final output result of the initial image processing model is still retained in the loss function of the model training, the model training effect will not be obviously adversely affected, thereby ensuring the model training effect.

[0087] Further reference is made to Figure 4 As an implementation of the method shown in the above figures, the present disclosure provides one embodiment of an apparatus for training an image processing model, which corresponds to the method embodiment shown in Figure 1 The apparatus can be applied in various electronic devices.

[0088] As Figure 4As shown, the device 400 for training an image processing model provided in this embodiment includes a generation unit 401 and a training unit 402. The generation unit 401 is configured to input a sample image of a pre-acquired training sample to a pre-acquired initial image processing model to obtain a class activation map corresponding to the sample image and an attention heat map corresponding to a target feature extraction layer, where the training sample includes the sample image and corresponding label information, and the initial image processing model includes a fully connected layer and a plurality of feature extraction layers; and the training unit 402 is configured to train the initial image processing model based on minimizing a difference between the label information and an output of the fully connected layer and an attention difference to obtain the image processing model, where the attention difference is determined based on a difference between the class activation map corresponding to the sample image and the attention heat map corresponding to the target feature extraction layer.

[0089] In the device 400 for training an image processing model in this embodiment, the specific processing of the generation unit 401 and the training unit 402 and the technical effects brought by the same can be respectively referred to the corresponding descriptions of steps S101 and S102 in the corresponding embodiment. Figure 1 The related descriptions of steps S101 and S102 in the corresponding embodiment are not repeated here.

[0090] In some optional implementation manners of this embodiment, the target feature extraction layer can include a last layer feature extraction layer connected to the fully connected layer. The training unit can include a first determination module (not shown in the figure) configured to determine a difference between the label information and the output of the fully connected layer as a first difference; a second determination module (not shown in the figure) configured to determine a difference between the class activation map corresponding to the input sample image and the attention heat map corresponding to the last layer feature extraction layer connected to the fully connected layer as a second difference; and a training module (not shown in the figure) configured to train the initial image processing model based on minimizing the first difference and the second difference.

[0091] In some optional implementation manners of this embodiment, the target feature extraction layer can include adjacent feature extraction layers. The training module can be further configured to determine a difference between the attention heat maps corresponding to the adjacent feature extraction layers as a third difference, and train the initial image processing model based on minimizing the first difference, the second difference and the third difference.

[0092] In some optional implementation manners of this embodiment, the attention heat map corresponding to the target feature extraction layer can be generated according to the attention weights to be trained assigned to the feature maps output by the target feature extraction layer in a channel.

[0093] The apparatus provided in the above embodiments of this disclosure trains an initial image processing model by the training unit 402 based on the difference between the class activation map corresponding to the sample image generated by the generation unit 401 and the attention heatmap corresponding to the target feature extraction layer. Attention weight loss is introduced into the loss function of the model training, thereby achieving guided learning of the attention weights of the attention mechanism, making the learning process easier and more intuitive. Furthermore, since the attention mechanism proposed in this disclosure only requires introducing weight parameters into the loss function without adding any extra structure, it has advantages such as simple implementation, low computational cost, and intuitive principle. Moreover, since the loss function of the model training still retains the difference term between the annotation information corresponding to the sample image and the final output result of the initial image processing model, it will not have a significant adverse effect on the model training effect, thus ensuring the model training effect.

[0094] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an apparatus for processing images, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0095] like Figure 5 As shown, the image processing apparatus 500 provided in this embodiment includes an acquisition unit 501 and a processing unit 502. The acquisition unit 501 is configured to acquire an image to be processed; the processing unit 502 is configured to input the image to be processed into a pre-trained image processing model to generate an image processing result. The image processing model is obtained using the method for training an image processing model as described in the foregoing embodiments.

[0096] In this embodiment, the specific processing of the acquisition unit 501 and the processing unit 502 in the image processing apparatus 500 and the resulting technical effects can be referred to respectively. Figure 2 The relevant descriptions of steps S201 and S202 in the corresponding embodiments will not be repeated here.

[0097] The apparatus provided in the above embodiments of this disclosure processes the image acquired by the acquisition unit 501 by the image processing model trained by the processing unit 502 using the method of training an image processing model. Since the image processing model trained by introducing an improved attention mechanism is used, the image processing effect can be improved.

[0098] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and it does not violate public order and good morals.

[0099] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, a computer program product and an autonomous vehicle.

[0100] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0101] The autonomous vehicle provided by the present disclosure can include the above electronic device as shown in Figure 6

[0102] As shown in Figure 6 The device 600 includes a computing unit 601 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0103] Various components in the device 600 are connected to the I / O interface 605, including an input unit 606, such as a keyboard, a mouse, etc., an output unit 607, such as various types of displays, a speaker, etc., a storage unit 608, such as a magnetic disk, an optical disk, etc., and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0104] ​The computing unit 601 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as a method for training an image processing model or a method for processing an image. For example, in some embodiments, the method for training an image processing model or the method for processing an image can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the method for training an image processing model or the method for processing an image described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the method for training an image processing model or the method for processing an image by any other appropriate means, such as by means of firmware.

[0105] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0106] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0107] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0108] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0109] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0110] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions between them occurring over a communication network. The relationship between client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers incorporating blockchain.

[0111] It should be understood that the steps shown in the various forms above can be reordered, added to, or removed. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, without limitation herein, so long as the desired results of the technology disclosed in the present disclosure are achieved.

[0112] The specific embodiments described above are not intended to be limiting. One of skill in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments described above without departing from the spirit and principles of this disclosure. Any further modifications, changes, or improvements that come within the spirit and principles of the disclosure are intended to fall within the scope of the disclosure.

Claims

1. A method for training an image processing model, comprising: The sample images of the pre-acquired training samples are input into the pre-acquired initial image processing model to obtain the class activation map corresponding to the sample image and the attention heatmap corresponding to the target feature extraction layer. The training samples include sample images and corresponding annotation information. The initial image processing model includes a fully connected layer and multiple feature extraction layers. The target feature extraction layer includes a last feature extraction layer connected to the fully connected layer. The class activation map and the feature map output by the last feature extraction layer have the same length and width, which can reflect the attention of the initial image processing model to different positions of the input sample image. The difference between the annotation information and the output of the fully connected layer is determined as the first difference; the difference between the class activation map corresponding to the input sample image and the attention heatmap corresponding to the last feature extraction layer connected to the fully connected layer is determined as the second difference; the initial image processing model is trained based on minimizing the first difference and the second difference to obtain the image processing model.

2. The method according to claim 1, wherein, The target feature extraction layer includes adjacent feature extraction layers; as well as The step of training the initial image processing model based on minimizing the first difference and the second difference includes: The difference between the attention heatmaps corresponding to adjacent feature extraction layers is determined as the third difference; The initial image processing model is trained based on minimizing the first difference, the second difference, and the third difference.

3. The method according to any one of claims 1-2, wherein, The attention heatmap corresponding to the target feature extraction layer is generated by assigning weights to be trained according to the channels of the feature map output by the target feature extraction layer.

4. A method for processing an image, comprising: Obtain the image to be processed; The image to be processed is input into a pre-trained image processing model to generate an image processing result, wherein the image processing model is obtained by the method for training an image processing model as described in any one of claims 1-3.

5. An apparatus for training an image processing model, comprising: The generation unit is configured to input sample images of pre-acquired training samples into a pre-acquired initial image processing model to obtain a class activation map corresponding to the sample image and an attention heatmap corresponding to the target feature extraction layer. The training samples include sample images and corresponding annotation information. The initial image processing model includes a fully connected layer and multiple feature extraction layers. The target feature extraction layer includes a last feature extraction layer connected to the fully connected layer. The class activation map and the feature map output by the last feature extraction layer have the same length and width, which can reflect the attention of the initial image processing model to different positions of the input sample image. The training unit includes: The first determining module is configured to determine the difference between the annotation information and the output of the fully connected layer as a first difference; The second determining module is configured to determine the difference between the class activation map corresponding to the input sample image and the attention heatmap corresponding to the last feature extraction layer connected to the fully connected layer as the second difference; The training module is configured to train the initial image processing model based on minimizing the first difference and the second difference to obtain the image processing model.

6. The apparatus according to claim 5, wherein, The target feature extraction layer includes adjacent feature extraction layers; and The training module is further configured to: The difference between the attention heatmaps corresponding to adjacent feature extraction layers is determined as the third difference; The initial image processing model is trained based on minimizing the first difference, the second difference, and the third difference.

7. The apparatus according to any one of claims 5-6, wherein, The attention heatmap corresponding to the target feature extraction layer is generated by assigning attention weights to be trained according to the channel of the feature map output by the target feature extraction layer.

8. An apparatus for processing images, comprising: The acquisition unit is configured to acquire the image to be processed. The processing unit is configured to input the image to be processed into a pre-trained image processing model to generate an image processing result, wherein the image processing model is obtained by the method for training an image processing model as described in any one of claims 1-3.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-4.

12. An autonomous vehicle, including the electronic equipment as claimed in claim 9.

Images