Image processing method, computer readable medium and electronic device

Abstract

The application discloses an image processing method, a computer readable medium and an electronic device. The method comprises: performing first feature extraction on a to-be-processed image to obtain a first feature map; performing preset-time second feature extraction on the first feature map to output a plurality of second feature maps of different feature scales; each second feature extraction comprises batch normalization processing and group normalization processing; performing feature fusion on the plurality of second feature maps of different feature scales to obtain a fused feature map; obtaining the categories of pixel points in the fused feature map; performing image mask processing on the pixel points in the fused feature map according to the categories of the pixel points in the fused feature map to obtain a mask image. The application improves the generalization performance of image processing through batch normalization, saves computing resources through group normalization, and balances the generalization performance of image processing and computing resources.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an image processing method, a computer readable medium and an electronic device. BACKGROUND

[0002] Image processing has always been a research hotspot as an important content in computer vision technology. In image processing, some contents in an image are often extracted or recognized, for example, animals, vehicles, persons, objects with specific attributes, etc. in an image are detected. At present, a neural network model is usually used to recognize objects with specific attributes in an image, that is, an image is input into a trained neural network model, and the neural network model segments different categories of objects in the image to realize recognition of objects with specific attributes. The neural network model needs to be trained using a large number of training sample images. In the training process, due to the complex and diverse categories of objects, and the inability to collect training data of all categories, the generalization ability of image processing is poor, and it cannot be applied to different categories of images, which needs to be improved.

[0003] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0004] The purpose of the present application is to provide an image processing method, a computer readable medium and an electronic device to optimize the poor generalization ability of image processing in related technologies.

[0005] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0006] According to an aspect of an embodiment of the present application, an image processing method is provided, comprising:

[0007] performing first feature extraction on a to-be-processed image to obtain a first feature map;

[0008] performing second feature extraction on the first feature map for a preset number of times to output a plurality of second feature maps of different feature scales; each second feature extraction includes batch normalization processing and group normalization processing;

[0009] performing feature fusion on the plurality of second feature maps of different feature scales to obtain a fused feature map;

[0010] obtaining the category of a pixel point in the fused feature map;

[0011] performing image mask processing on the pixel point in the fused feature map according to the category of the pixel point in the fused feature map to obtain a mask image.

[0012] According to an aspect of an embodiment of the present application, an image processing apparatus is provided, comprising:

[0013] a first feature extraction module configured to perform first feature extraction on a to-be-processed image to obtain a first feature map;

[0014] a second feature extraction module configured to perform preset-number-of-times second feature extraction on the first feature map to output a plurality of second feature maps of different feature scales; each time of second feature extraction comprises batch normalization processing and group normalization processing;

[0015] a feature fusion module configured to perform feature fusion on the plurality of second feature maps of different feature scales to obtain a fused feature map;

[0016] a category obtaining module configured to obtain categories of pixel points in the fused feature map;

[0017] a mask image generation module configured to perform image mask processing on the pixel points in the fused feature map according to the categories of the pixel points in the fused feature map to obtain a mask image.

[0018] According to an aspect of an embodiment of the present application, a computer readable medium having a computer program stored thereon is provided, the computer program being executed by a processor to implement the image processing method in the above technical solution.

[0019] According to an aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute the image processing method in the above technical solution via execution of the executable instructions.

[0020] According to an aspect of an embodiment of the present application, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image processing method in the above technical solution.

[0021] In the technical scheme provided in the embodiments of the present application, the mask image is obtained through the first feature extraction, the second feature extraction of the preset number of times, the feature map fusion, the pixel point category recognition and the image mask processing on the to-be-processed image, the different category objects in the to-be-processed image are distinguished, and the normalization processing mode of alternately using the batch normalization processing and the group normalization processing is used in the image processing process, so that the generalization performance of the image processing can be improved through the batch normalization processing, and the calculation resources can be saved through the group normalization processing, so that the balance between the generalization performance of the image processing and the calculation resources is achieved, and the accuracy of the image processing can be ensured.

[0022] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory and are not restrictive of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application. It is readily apparent to one skilled in the art that the following figures are merely some embodiments of the present application, and other figures can be obtained from these figures without creative labor.

[0024] Figure 1 An exemplary system architecture block diagram to which the technical scheme of the present application is applied is schematically shown.

[0025] Figure 2 A schematic diagram of a self-moving device provided in an embodiment of the present application is schematically shown.

[0026] Figure 3 A flowchart of an image processing method provided in an embodiment of the present application is schematically shown.

[0027] Figure 4 A flowchart of the i-th feature extraction provided in an embodiment of the present application is schematically shown.

[0028] Figure 5A A flowchart of the second feature extraction provided in an embodiment of the present application is schematically shown.

[0029] Figure 5B A flowchart of the second feature extraction provided in another embodiment of the present application is schematically shown.

[0030] Figure 6 A schematic diagram of a feature map fusion process provided in an embodiment of the present application is schematically shown.

[0031] Figure 7A A schematic diagram of a to-be-processed image provided in an embodiment of the present application is schematically shown.

[0032] Figure 7B A schematic diagram of a mask image provided by an embodiment of the present application is shown.

[0033] Figure 8 A structural block diagram of an image processing apparatus provided by an embodiment of the present application is shown.

[0034] Figure 9 A structural block diagram of a computer system of an electronic device suitable for implementing an embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these example implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art.

[0036] The flowcharts shown in the drawings are merely illustrative and do not necessarily include all contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further broken down, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0037] Figure 1 An example system architecture block diagram to which the technical solutions of the present application are applied is shown.

[0038] As shown in Figure 1 , the system architecture 100 can include a terminal device 110, a network 120 and a server 130. The terminal device 110 can include a smartphone, a tablet computer, a notebook computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, a self-moving device, etc. The server 130 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The network 120 can be various connection types of communication media capable of providing a communication link between the terminal device 110 and the server 130, such as a wired communication link or a wireless communication link.

[0039] According to implementation needs, the system architecture in the embodiments of the present application can have any number of terminal devices, networks and servers. For example, the server 130 can be a server group composed of multiple server devices. In addition, the technical solutions provided by the embodiments of the present application can be applied to the terminal device 110, or can be applied to the server 130, or can be jointly implemented by the terminal device 110 and the server 130, and the present application does not specially limit this.

[0040] For example, the technical solution of the present application is implemented by the server 130. The to-be-processed image can be sent to the server 130 by the terminal device 110, and the server 130 performs first feature extraction on the to-be-processed image to obtain a first feature map. The first feature extraction can be to extract the image features of the to-be-processed image through convolution processing. Then, the server 130 performs second feature extraction for a preset number of times on the first feature map. In the process of the second feature extraction, batch normalization processing and group normalization processing are included, which refer to the normalization processing mode of data in the second feature extraction. For example, batch normalization processing can be used in the first few times of second feature extraction, and group normalization processing can be used in the last few times of second feature extraction. For another example, batch normalization processing and group normalization processing are alternately used in the normalization processing mode of the second feature extraction for a preset number of times. Through the second feature extraction for a preset number of times, a plurality of second feature maps of different scales are obtained. Next, the server 130 fuses the plurality of second feature maps of different scales to obtain a fused feature map, and then performs image mask processing according to the categories of the pixel points in the fused feature map to generate a mask image. The image mask processing is used to distinguish the target object recognized from the to-be-processed image from the background image in the to-be-processed image. For example, the to-be-processed image is an image with a grass region and a non-grass region, and the grass region is the target object. Therefore, the mask image can set the grass region and the non-grass region as different pixel values to distinguish them.

[0041] In an embodiment of the present application, the technical solution of the present application is implemented by the terminal device 110, for example, the terminal device 110 is a self-moving device. For example, Figure 2 A schematic diagram of a self-moving device provided by an embodiment of the present application is shown schematically. The self-moving device 200 can be various types of mobile robots, such as a sweeping robot, a mopping robot, a food delivery robot, a transport robot, a mowing robot, etc. The specific type and function of the self-moving device 200 are not limited in the embodiment of the present application. It can be understood that the self-moving device 200 in the embodiment can also include other devices with self-moving functions.

[0042] In an embodiment of the present application, the self-moving device 200 is provided with a camera 210 for capturing an environmental image around the self-moving device 200. The camera 210 can be fixed or non-fixed and rotatable, which is not limited in the embodiment of the present application. The environmental image captured by the camera 210 can be a color image, a black-and-white image, an infrared image, etc., which is not limited in the embodiment of the present application.

[0043] The camera 210 is connected to a control module 220 inside the self-moving device 200, which is configured to receive control instructions of the self-moving device 200 or generate various control instructions for the self-moving device 200. The control module 220 is also connected to driving components of the self-moving device 200, such as a steering shaft, steering wheels, motors, etc. of the self-moving device 200, for controlling the movement and steering of the self-moving device 200.

[0044] In the embodiments of the present application, the environment image captured by the camera 210 can be a to-be-processed image, which is transmitted to the control module 220 by the camera 210. After receiving the to-be-processed image, the control module 220 processes the to-be-processed image according to the image processing method provided in the embodiments of the present application to identify a target object in the to-be-processed image, and then adjusts the moving direction or path of the self-moving device 200 according to the position or region of the target object.

[0045] For example, the self-moving device 200 is a lawn mower robot, and the to-be-processed image captured by the camera 210 includes a grass region and a non-grass region. After processing the to-be-processed image according to the image processing method provided in the embodiments of the present application, the control module 220 identifies the grass region, and then controls the self-moving device 200 to work in the grass region.

[0046] The image processing method provided in the present application will be described in detail in combination with specific embodiments.

[0047] Figure 3 The flowchart of the image processing method provided in an embodiment of the present application is schematically shown in FIG. 3. As shown in FIG. 3, steps 310 to 350 are as follows. Figure 3

[0048] Step 310, first feature extraction is performed on the to-be-processed image to obtain a first feature map.

[0049] Specifically, the to-be-processed image can be a color image, a black-and-white image, a grayscale image, etc. The first feature extraction is performed to extract the image features of the to-be-processed image, so as to convert the to-be-processed image into the first feature map.

[0050] In an embodiment of the present application, the process of the first feature extraction includes: performing first convolution processing on the to-be-processed image to obtain a first convolution feature map; performing batch normalization processing on the first convolution feature map to obtain a batch normalization feature map; and performing activation processing and sampling processing on the batch normalization feature map to obtain the first feature map.

[0051] ​Specifically, first, the first convolutional processing is performed on the image to be processed, which refers to performing convolutional calculation on the image to be processed by a convolutional network of a certain size to obtain a first convolutional feature map. For example, a convolutional network with a channel number of 64, a convolution kernel size of 7*7, and a stride of 2 is used to perform the first convolutional processing on the image to be processed.

[0052] In the first convolutional feature map obtained by the first convolutional processing, the data ranges of the respective features can have a large difference. If the first convolutional feature map is directly used for subsequent processing, the data with a large difference will cause a large error, which in turn can easily lead to a reduction in the accuracy of the final image processing result. Therefore, in order to make the data ranges of the respective features in the first convolutional feature map consistent, the first convolutional feature map needs to be normalized.

[0053] In the embodiments of the present application, the first convolutional feature map is subjected to batch normalization (Batch Normalization, BN) processing to obtain a corresponding batch normalization feature map. Batch normalization processing is to divide the data into multiple batches and then perform normalization processing on each batch respectively.

[0054] In an embodiment of the present application, the first convolutional feature map can also be subjected to group normalization (Group Normalization) processing to obtain a corresponding group normalization feature map. Group normalization processing is to divide the features output by each channel of the convolutional network in the first convolutional processing into multiple groups, and then perform normalization processing on each group respectively.

[0055] After the normalization processing, the batch normalization feature map is subjected to activation processing and sampling processing to obtain a first feature map. The activation processing is to activate the batch normalization feature map by using an activation function, which can be a ReLU (Rectified Linear Unit) function, a sigmoid function (S-shaped function), a tanh function (hyperbolic tangent function), etc.

[0056] After the activation processing, the sampling processing is performed on the result of the activation processing. The sampling processing in the embodiments of the present application is mainly down-sampling processing, which is used to remove unnecessary redundant information in the first convolutional feature map in the subsequent processing process. For example, a 3*3 size, 2 stride pooling layer is used to perform max pooling to realize down-sampling, thereby obtaining the first feature map.

[0057] In step 320, the first feature map is subjected to second feature extraction a preset number of times to output a plurality of second feature maps of different feature scales. Each time of second feature extraction includes batch normalization processing and group normalization processing.

[0058] Specifically, after obtaining the first feature map, a second feature extraction is performed on the first feature map a preset number of times to extract multi-faceted or multi-angle, deeper-level feature information from the first feature map, resulting in multiple second feature maps with different scales. In this embodiment, batch normalization and group normalization are alternately performed to normalize the data during the second feature extraction process.

[0059] In one embodiment of this application, the second feature extractions of a preset number of times are performed sequentially; that is, the output of the previous second feature extraction is the input of the next second feature extraction. If each second feature extraction is considered as the function of a convolutional block, then the preset number of second feature extractions is implemented by multiple connected convolutional blocks, with the number of convolutional blocks being the same as the number of second feature extractions.

[0060] Since the process of extracting the second feature is similar each time, the following uses one second feature extraction to illustrate the implementation process of the second feature extraction with a preset number of extractions.

[0061] like Figure 4 As shown, the process of feature extraction for the i-th time includes steps 410 to 460, as detailed below:

[0062] Step 410: Use the output feature map of the (i-1)th second feature extraction as the input feature map of the i-th second feature extraction.

[0063] Since the output of the previous second feature extraction serves as the input for the next second feature extraction, the input feature map for the i-th second feature extraction is the output feature map for the (i-1)-th second feature extraction. Here, 2 ≤ i ≤ K, where K is the preset number of extractions. The input feature map for the first second feature extraction is the first feature map.

[0064] Step 420: Perform a second convolution on the input feature map to obtain the second convolution feature map.

[0065] Specifically, the second convolution process refers to performing convolution calculations on the input feature map of the i-th second feature extraction using a convolutional network of a certain size to obtain a second convolutional feature map. Generally, the kernel size of the convolutional network used for the second feature extraction is different from that used for the first feature extraction. For example, the kernel size used for the first feature extraction is 7*7, and the kernel size used for the second feature extraction is 3*3.

[0066] In the embodiments of the present application, the scale of the second convolution feature map can be adjusted by changing the stride in the second convolution process, without changing the size of the convolution kernel. Generally, when the stride in the second convolution process is doubled, the scale of the second convolution feature map is halved relative to the scale of the input feature map. For example, assuming that the scale of the input feature map is 56*56, and the stride of the second convolution is 2, the scale of the second convolution feature map is 28*28.

[0067] In the embodiments of the present application, the scale of the second convolution feature map also includes the number of channels, which can be pre-set. The number of channels of the second convolution feature map is the same as the number of channels set in the second convolution process. For example, if the number of channels set in the second convolution process is 128, the number of channels of the second convolution feature map is also 128. The number of channels of the second convolution feature map is also referred to as the dimension of the second convolution feature map.

[0068] In one embodiment of the present application, the convolution parameters of the second convolution process can be pre-set according to actual needs, and the convolution parameters of different second convolution processes can be different. The convolution parameters include the size of the convolution kernel of the convolution network, the number of channels, the stride, and the like. For example, the size of the convolution kernel of the first second convolution process can be set to be different from the size of the convolution kernel of the second second convolution process, the number of channels of the second second convolution process can be set to be different from the number of channels of the third second convolution process, the stride of the first second convolution process can be set to be different from the stride of the third second convolution process, and the like.

[0069] In step 430, group normalization processing and activation processing are performed on the second convolution feature map to obtain a group normalization result.

[0070] Specifically, the group normalization processing refers to dividing the feature data of each channel in the second convolution feature map into G groups, and then performing normalization processing on the G groups, as follows:

[0071]

[0072]

[0073]

[0074] wherein X represents the input feature data, C represents the number of channels; G represents the number of groups, g represents the number of subgroups after subdividing each group; W and H respectively represent the width and height of the second convolution feature map; n represents the nth batch, and ε is a pre-set small constant to prevent δ c (X)=0;X cnhw from being equal to 0; X c (X) represents the mean value of the feature data of channel c; δ c(X) represents the variance of the feature data of the channel c; X c represents the feature data of the channel c; represents X c The result after group normalization processing.

[0075] Briefly, group normalization processing is to divide the C channels of the feature data into G groups, so that each group has C / G channels, and then calculate the mean and variance of each group for normalization processing. The size of each group of feature data is represented as W*H*C / G. For example, assuming that the number of channels C of the second convolution feature map is 6, and the number of groups G of channel division is 2, then each group has 3 channels of feature data.

[0076] As can be seen from the calculation formula of group normalization, when the size of the feature map W*H is unchanged, the group normalization processing result is related to the number of channels C and the number of groups G of channel division, and is independent of the batch of feature data. Therefore, group normalization processing does not occupy too much video memory capacity.

[0077] After group normalization processing of the second convolution feature map, activation processing is continued, and the result of activation processing is recorded as the group normalization result. The activation function used in activation processing is the same as the activation function used in the first feature extraction process.

[0078] Step 440, performing third convolution processing on the group normalization result to obtain a third convolution feature map.

[0079] Specifically, the third convolution processing can use the same convolution network structure as the second convolution processing. For example, the convolution network structure of the second convolution processing is 3*3*64, which means that the convolution kernel size is 3*3 and the number of channels is 64. The third convolution processing can also use a convolution network structure of 3*3*64. The group normalization result is converted into a third convolution feature map through convolution calculation of the third convolution processing.

[0080] Step 450, performing batch normalization processing on the third convolution feature map to obtain a batch normalization result.

[0081] Specifically, batch normalization processing is to divide the data into multiple batches, and then normalize the data of each batch, as follows:

[0082]

[0083]

[0084]

[0085] Wherein, X represents the input feature data, C represents the number of channels; N represents the batch number, n represents the nth batch; g represents the number of subgroups after subdividing each group; W, H represent the width and height of the third convolution feature map respectively; ε is a small constant set in advance to prevent δ c (X) = 0; X cnhw (X) represents the feature data of the nth batch of the cth channel, the wth column and the hth row; u c (X) represents the mean of the feature data of channel c; δ c (X) represents the variance of the feature data of channel c; X c represents the feature data of channel c; represents X c After batch normalization processing. In short, batch normalization processing is to divide the feature data into N batches, and then calculate the mean and variance of each batch for normalization processing. The size of each batch of feature data is represented as W*H*N.

[0086] In batch normalization processing, the batch number N is to divide the data into N batches, and N is a preset value. The set value of N is 16, 32, 64, 128, etc. Generally, the smaller the value of N, the less the video memory capacity occupied during data calculation, but it will lead to the accuracy of the calculation result decreases, reduces the generalization ability of image processing; the larger the value of N, the larger the video memory capacity occupied during data calculation, also makes the accuracy of the calculation result increases, the generalization ability of image processing is enhanced.

[0087] It can be seen that if only batch normalization processing is used, more computing resources are needed to improve the generalization ability of image processing. However, the embodiment of the present application alternately uses the combination of batch normalization and group normalization, which can improve the generalization performance of image processing through batch normalization processing, and can save computing resources through group normalization processing, so as to balance the generalization performance of image processing and computing resources, while ensuring the accuracy of image processing results.

[0088] Step 460, fuse the input feature map with the batch normalization result to obtain the output feature map of the i-th second feature extraction.

[0089] Specifically, the fusion of the input feature map and the batch normalization result is the superposition of the input feature map and the batch normalization result. Assuming that the input feature map is denoted as X, the batch normalization result is the result obtained by performing convolution, normalization processing and other operations on the input feature map, denoted as F(X), and the fusion result of the two can be represented as F(X)+X. The fusion result is also the output feature map of the i-th second feature extraction.

[0090] In an embodiment of the present application, the process of fusing the input feature map and the batch normalization result specifically comprises: performing fourth convolution processing on the input feature map to obtain a fourth convolution feature map; and fusing the fourth convolution feature map and the batch normalization result to obtain the output feature map of the i-th second feature extraction.

[0091] When fusing the input feature map and the batch normalization result, in addition to directly superimposing the input feature map and the batch normalization result to obtain the fusion result, the fourth convolution processing can be performed on the input feature map first, and then the fourth convolution feature map obtained by the fourth convolution processing is fused with the batch normalization result. The fourth convolution processing is performed on the input feature map because the convolution processing in the i-th second feature extraction process changes the scale of the input feature map, so that the scale of the batch normalization result and the input feature map is inconsistent. Therefore, the fourth convolution processing is performed on the input feature map to make the scale of the input feature map and the batch normalization result consistent for superposition.

[0092] Exemplarily, Figure 5A The flowchart of the second feature extraction provided by an embodiment of the present application is schematically shown. As shown in the figure, Figure 5A The scale of the input feature map is 56*56*64, that is, the width*height is 56*56, and the dimension (or the number of channels) is 64. The second convolution processing adopts a convolution network structure of 3*3*128, that is, the size of the convolution kernel is 3*3, the number of channels is 128, and the stride s=2. After the input feature map is processed by the second convolution processing, the scale of the second convolution feature map obtained is 28*28*128. The group normalization processing and the activation processing do not change the scale of the feature map, and the third convolution processing adopts a convolution network structure of 3*3*128, which is the same as the convolution network structure of the second convolution processing, and the stride s=1, that is, the third convolution processing does not change the scale of the feature map, and the scale of the third convolution feature map output by the third convolution processing is still 28*28*128. The batch normalization processing does not change the scale of the feature map, and therefore the scale of the batch normalization result F(X) is 28*28*128. In order to make the scale of the two superimposed data consistent, the fourth convolution processing is performed on the input feature map X, the convolution network structure of the fourth convolution processing is 1*1*128, and the stride s=2, so that the fourth convolution feature map X' with the scale of 28*28*128 is obtained. Finally, the batch normalization result F(X) and the fourth convolution feature map X' are superimposed to obtain the output feature map F(X)+X', and the scale of the output feature map is 28*28*128.

[0093] In an embodiment of the present application, after the fourth convolution processing is performed on the input feature map, the structure of the fourth convolution processing is normalized to obtain the fourth convolution feature map, so that the data in the fourth convolution feature map is more standardized. The normalization method adopted here can adopt batch normalization processing or group normalization processing.

[0094] Exemplary, Figure 5B A flow chart of the second feature extraction provided by another embodiment of the application is schematically shown. As Figure 5B As shown, the scale of the input feature map is 56*56*64, the second convolution processing and the third convolution processing both adopt the convolution network structure of 3*3*64, and the stride s=1, so that the second convolution processing and the third convolution processing do not change the scale of the feature map, and the scale of the batch normalization result F(X) is 56*56*64. The scale of the input feature map X is the same as that of the batch normalization result F(X), so that the fourth convolution processing is not needed for the input feature map, and the batch normalization result F(X) can be directly superimposed with the input feature map X to obtain the output feature map F(X)+X, and the scale of the output feature map is 56*56*64.

[0095] In the technical solution of the application, multiple convolution processes are used, such as the first convolution processing, the second convolution processing, the third convolution processing, and the fourth convolution processing. These convolution processes refer to convolution calculation of input data by using convolution network of different sizes or convolution calculation of input data by using different convolution network structures. For example, the convolution network structure is represented by convolution kernel width*convolution kernel height*channel number*stride, the convolution network structure adopted by the first convolution processing is 7*7*64*2, the convolution network structure adopted by the second convolution processing is 3*3*128*2, the convolution network structure adopted by the third convolution processing is 3*3*128*1, and the convolution network structure adopted by the fourth convolution processing is 1*1*128*2.

[0096] In the technical solution of the application, the batch normalization processing and the group normalization processing are alternately performed throughout the whole process of the first feature extraction and the second feature extraction of the preset number of times, so that the normalization mode of the first second feature extraction is also the batch normalization processing and the group normalization processing alternately. In the embodiment of the application, the batch normalization processing is adopted in the first feature extraction, so that the normalization mode in the second feature extraction is the group normalization processing and the batch normalization processing alternately. If the group normalization processing is adopted in the first feature extraction, then the normalization mode in the second feature extraction is the batch normalization processing and the group normalization processing alternately.

[0097] According to the specific process of the second feature extraction above, each second feature extraction can obtain an output feature map, and after a preset number of second feature extractions, a plurality of output feature maps can be obtained, and finally a set number of feature maps can be selected from the plurality of output feature maps as the plurality of second feature maps of different feature scales required finally. For example, in the embodiment of the present application, the output feature maps of every two second feature extractions are taken as a second feature map. For example, there are 8 second feature extractions, and the output feature maps of the 2nd, 4th, 6th and 8th second feature extractions are taken as second feature maps. Generally, the output feature maps of the 4th, 6th and 8th second feature extractions can be selected as second feature maps.

[0098] Taking the output feature maps of every two second feature extractions as a second feature map is equivalent to taking every two second feature extractions as a convolution module, and then the output of each convolution module is a second feature map. In some embodiments, more times of second feature extraction can also be taken as a convolution module. For example, there are 4 convolution modules, the 1st convolution module includes 3 second feature extractions, the 2nd convolution module includes 4 second feature extractions, the 3rd convolution module includes 6 second feature extractions, and the 4th convolution module includes 3 second feature extractions.

[0099] In an embodiment of the present application, the number of second feature extractions can be pre-set or calculated according to relevant parameters. For example, the number of second feature extractions is obtained according to the ratio of the total number of channels of the first feature map to the preset group number, and the preset group number refers to the number of groups divided by each channel during group normalization processing. For example, the total number of channels of the first feature map is 64, and the preset group number is 4, so the number of second feature extractions is 16.

[0100] In an embodiment of the present application, after the last second feature extraction, the result of the last second feature extraction is input into a fully connected layer, and the feature map output by the fully connected layer can also be used as a second feature map.

[0101] Continuing to refer to Figure 3 , step 330, the second feature maps of different feature scales are fused to obtain a fused feature map.

[0102] Specifically, when fusing a plurality of second feature maps, the second feature maps are sequentially superimposed and fused according to the change of the scale of the second feature map to obtain a fused feature map.

[0103] In an embodiment of the present application, the plurality of second feature maps of different feature scales include a first sub-feature map, a second sub-feature map and a third sub-feature map, and the fusion process can refer to Figure 6 .

[0104] As Figure 6As shown, the scales of the first, second and third sub-feature maps are sequentially increased, and the three sub-feature maps constitute a feature pyramid structure. The first sub-feature map is first up-sampled to obtain a first up-sampled feature map. The first sub-feature map is up-sampled to expand the scale of the first sub-feature map, so as to obtain the first up-sampled feature map with the same scale as the second sub-feature map.

[0105] Then, as shown in Figure 6 , the first up-sampled feature map and the second sub-feature map are superimposed to obtain a superimposed feature map. The scale of the superimposed feature map is the same as that of the second sub-feature map. After obtaining the superimposed feature map, a convolution kernel with a set size can be used to perform convolution processing on the superimposed feature map, for example, a 3*3 convolution kernel is used to perform convolution processing on the superimposed feature map, so as to eliminate the aliasing effect caused by up-sampling. The aliasing effect refers to the overlapping of the data obtained by sampling due to unreasonable sampling frequency setting.

[0106] Next, as shown in Figure 6 , the superimposed feature map is up-sampled to obtain a second up-sampled feature map. This up-sampling is to expand the scale of the superimposed feature map, so that the scale of the second up-sampled feature map after up-sampling is the same as that of the third sub-feature map, so as to be superimposed subsequently. It can be understood that if the superimposed feature map is subjected to convolution processing in the previous step, this step is to up-sample the superimposed feature map subjected to convolution processing.

[0107] Finally, as shown in Figure 6 , the second up-sampled feature map and the third sub-feature map are superimposed to obtain a fusion feature map. After obtaining the fusion feature map, convolution processing can also be performed to eliminate the aliasing effect.

[0108] With reference back to Figure 3 , step 340, the class of the pixel point in the fusion feature map is obtained.

[0109] In an embodiment of the present application, the process of obtaining the class of the pixel point in the fusion feature map includes: obtaining the confidence of the pixel point in the fusion feature map, the confidence of the pixel point representing the probability that the pixel point belongs to the target class; when the confidence of the pixel point is greater than or equal to the confidence threshold corresponding to the target class, it is determined that the class of the pixel point belongs to the target class; when the confidence of the pixel point is less than the confidence threshold corresponding to the target class, it is determined that the class of the pixel point does not belong to the target class.

[0110] Specifically, after obtaining the fusion feature map, the category of each pixel point is determined based on the fusion feature map. The category of each pixel point can be determined by the confidence of each pixel point, where the confidence of a pixel point refers to the probability that the pixel point belongs to a target category, and the target category refers to the category to which a target object belongs. When the confidence of a pixel point is greater than or equal to a confidence threshold corresponding to the target category, it indicates that the pixel point has a high probability of being a pixel point of the target object, and thus it is determined that the pixel point belongs to the target category. When the confidence of a pixel point is less than the confidence threshold corresponding to the target category, it indicates that the pixel point has a low probability of being a pixel point of the target object, and thus it is determined that the pixel point does not belong to the target category.

[0111] For example, assuming that the target object is a human face, the category of a pixel point is determined by the confidence that the pixel point belongs to a pixel point of the human face. Generally, when the confidence that a pixel point belongs to a target object of a corresponding category is greater than a confidence threshold, it is determined that the category of the pixel point is the category of the target object. For example, the confidence threshold is 0.6, when the confidence that a pixel point belongs to a tree is greater than or equal to 0.6, it is determined that the category of the pixel point is a tree; when the confidence that a pixel point belongs to a tree is less than 0.6, it is determined that the category of the pixel point is not a tree.

[0112] With reference to Figure 3 , step 350, performing image mask processing on the pixel points in the fusion feature map according to the categories of the pixel points in the fusion feature map to obtain a mask image.

[0113] Specifically, the image mask processing refers to setting a corresponding pixel value for a pixel point according to the category of the pixel point, so as to distinguish pixel points of different categories. Specifically, when the category of a pixel point belongs to a target category, a first pixel value is generated for the pixel point; when the category of a pixel point does not belong to a target category, a second pixel value is generated for the pixel point. After setting the pixel value for each pixel point in the fusion feature map, a mask image is obtained. The mask image can then reflect target objects of different categories.

[0114] For example, Figure 7A An illustrative diagram of a to-be-processed image provided by an embodiment of the present application is schematically shown, and the target object to be recognized in the embodiment of the present application is a grassland in the to-be-processed image shown in Figure 7A , and the objects other than the grassland in the to-be-processed image shown in Figure 7A are non-target objects, such as the objects shown in the dashed box region in Figure 7A . The to-be-processed image shown in Figure 7A is processed by the image processing method provided by the present application to obtain a mask image as shown in Figure 7B . Figure 7BThe illustrated mask image distinguishes the target object from the non-target object, and the darker color area represents the non-target object, and the lighter color area represents the target object, i.e., the grass.

[0115] In an embodiment of the present application, the categories of the pixel points can be various, and different categories of the pixel points can be set with different pixel values. For example, when the category of the pixel point belongs to a first target category, a pixel value corresponding to the first target category is generated for the pixel point; when the category of the pixel point belongs to a second target category, a pixel value corresponding to the second target category is generated for the pixel point; when the category of the pixel point belongs to a third target category, a pixel value corresponding to the third target category is generated for the pixel point, and so on.

[0116] For example, the categories of the pixel points include flowers, trees, and stones. When the category of the pixel point is a flower, a pixel value corresponding to the flower is generated, which makes the pixel point visually displayed as red; when the category of the pixel point is a tree, a pixel value corresponding to the tree is generated, which makes the pixel point visually displayed as green; and when the category of the pixel point is a stone, a pixel value corresponding to the stone is generated, which makes the pixel point visually displayed as gray.

[0117] In the technical scheme provided in the embodiments of the present application, the mask image is obtained through the first feature extraction, the second feature extraction of the preset number of times, the feature map fusion, the pixel point category recognition, and the image mask processing, etc. on the to-be-processed image, and the different categories of objects in the to-be-processed image are distinguished, and the normalization processing mode of alternately using the batch normalization processing and the group normalization processing is used in the image processing process, so that the generalization performance of the image processing can be improved through the batch normalization processing, and the calculation resources can be saved through the group normalization processing, so as to balance the generalization performance and the calculation resources of the image processing, and the accuracy of the image processing can be ensured.

[0118] In an embodiment of the present application, the same training data and verification data are used to respectively verify the image processing method provided in the present application which alternately uses the batch normalization processing and the group normalization processing (hereinafter referred to as the BN+GN method), and the traditional image processing method which only uses the batch normalization processing (hereinafter referred to as the BN method), and the verification results are shown in Table 1.

[0119] Table 1 Verification results

[0120]

[0121] In Table 1, the column of "Comparison Category" represents the items for comparison of the two methods, box represents the target frame used for detecting the target object when performing instance segmentation by the image processing method, and mask represents the mask image. The threshold represents the confidence threshold used for identifying the category of the target object. The score reflects the accuracy of the result of identifying the target object. According to Table 1, it can be seen that, under the condition of using the same data and the same threshold, the score of the BN+GN method is obviously higher than that of the BN method, which indicates that the image processing method provided in the present application can effectively improve the accuracy of identifying the target object or performing instance segmentation.

[0122] It should be noted that although the steps of the method in the present application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps, etc.

[0123] The following describes an apparatus embodiment of the present application, which can be used to perform the image processing method in the above-mentioned embodiments of the present application. Figure 8 Fig. 1 schematically shows a structural block diagram of an image processing apparatus provided by an embodiment of the present application.

[0124] As shown in Fig. 1, the image processing apparatus provided by the embodiment of the present application comprises: Figure 8

[0125] The first feature extraction module 810 is configured to perform first feature extraction on the to-be-processed image to obtain a first feature map.

[0126] The second feature extraction module 820 is configured to perform preset-number-of-times second feature extraction on the first feature map to output a plurality of second feature maps of different feature scales. Each time of second feature extraction includes batch normalization processing and group normalization processing.

[0127] The feature fusion module 830 is configured to perform feature fusion on the plurality of second feature maps of different feature scales to obtain a fused feature map.

[0128] The category acquisition module 840 is configured to acquire the category of the pixel point in the fused feature map.

[0129] The mask image generation module 850 is configured to perform image mask processing on the pixel point in the fused feature map according to the category of the pixel point in the fused feature map to obtain a mask image.

[0130] In an embodiment of the present application, the first feature extraction module 810 is alternatively configured to:

[0131] ​performing first convolution processing on the to-be-processed image to obtain a first convolution feature map;

[0132] performing batch normalization processing on the first convolution feature map to obtain a batch normalization feature map;

[0133] performing activation processing and sampling processing on the batch normalization feature map to obtain the first feature map.

[0134] In an embodiment of the present application, the second feature extraction module 820 is specifically configured to:

[0135] inputting an output feature map of i-1th second feature extraction as an input feature map of ith second feature extraction; wherein 2≤i≤K, K is a preset number of times; and an input feature map of 1st second feature extraction is the first feature map;

[0136] performing second convolution processing on the input feature map to obtain a second convolution feature map;

[0137] performing group normalization processing and activation processing on the second convolution feature map to obtain a group normalization result;

[0138] performing third convolution processing on the group normalization result to obtain a third convolution feature map;

[0139] performing batch normalization processing on the third convolution feature map to obtain a batch normalization result;

[0140] fusing the input feature map and the batch normalization result to obtain an output feature map of ith second feature extraction.

[0141] In an embodiment of the present application, the second feature extraction module 820 is specifically further configured to:

[0142] performing fourth convolution processing on the input feature map to obtain a fourth convolution feature map;

[0143] fusing the fourth convolution feature map and the batch normalization result to obtain an output feature map of ith second feature extraction.

[0144] In an embodiment of the present application, the plurality of second feature maps of different feature scales include a first sub-feature map, a second sub-feature map and a third sub-feature map, and the scales of the first sub-feature map, the second sub-feature map and the third sub-feature map increase in turn; and the feature fusion module 830 is specifically configured to:

[0145] performing up-sampling on the first sub-feature map to obtain a first up-sampled feature map;

[0146] stacking the first up-sampled feature map and the second sub-feature map to obtain a stacked feature map;

[0147] upsample the superimposed feature map to obtain a second upsampled feature map;

[0148] superimpose the second upsampled feature map and the third sub-feature map to obtain the fusion feature map.

[0149] In an embodiment of the present application, the category obtaining module 840 is specifically configured to:

[0150] obtain a confidence of a pixel point in the fusion feature map, the confidence of the pixel point representing a probability that a category of the pixel point belongs to a target category;

[0151] when the confidence of the pixel point is greater than or equal to a confidence threshold corresponding to the target category, determine that the category of the pixel point belongs to the target category;

[0152] when the confidence of the pixel point is less than the confidence threshold corresponding to the target category, determine that the category of the pixel point does not belong to the target category.

[0153] In an embodiment of the present application, the mask image generating module 850 is specifically configured to:

[0154] when the category of the pixel point belongs to a target category, generate a first pixel value for the pixel point;

[0155] when the category of the pixel point does not belong to a target category, generate a second pixel value for the pixel point;

[0156] obtain a mask image based on the pixel values of the respective pixel points.

[0157] In an embodiment of the present application, the device further comprises:

[0158] a preset number of times determining module configured to obtain a total number of channels of the first feature map and a preset group number; calculate a ratio of the total number of channels and the preset group number, and take the ratio as a preset number of times of the second feature extraction.

[0159] The specific details of the image processing device provided in the embodiments of the present application have been described in detail in the corresponding method embodiments, and will not be repeated here.

[0160] Figure 9 The computer system structure block diagram of the electronic device for implementing the embodiments of the present application is schematically shown.

[0161] It should be noted that, Figure 9 The computer system 900 of the electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0162] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM). The RAM 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output interface 905 (I / O interface) is also connected to the bus 904.

[0163] The following components are connected to the input / output interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a local area network card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0164] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit 901, it performs various functions defined in the system of this application.

[0165] It should be noted that the computer readable medium shown in the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired or the like, or any suitable combination of the above.

Claims

1. An image processing method, characterized by, The method comprises the following steps: performing first feature extraction on a to-be-processed image to obtain a first feature map; performing second feature extraction on the first feature map for a preset number of times to output a plurality of second feature maps with different feature scales; each second feature extraction comprises batch normalization processing and group normalization processing; taking the output feature map of the (i-1)th second feature extraction as the input feature map of the ith second feature extraction; wherein 2≤i≤K, K is the preset number of times, and the input feature map of the first second feature extraction is the first feature map; performing second convolution processing on the input feature map to obtain a second convolution feature map; performing group normalization processing and activation processing on the second convolution feature map to obtain a group normalization result; performing third convolution processing on the group normalization result to obtain a third convolution feature map; performing batch normalization processing on the third convolution feature map to obtain a batch normalization result; fusing the input feature map and the batch normalization result to obtain the output feature map of the ith second feature extraction; performing feature fusion on the plurality of second feature maps with different feature scales to obtain a fused feature map; the feature fusion comprises sequentially superimposed fusion according to the change of the scale of the second feature map to obtain the fused feature map; obtaining the category of a pixel point in the fused feature map; performing image mask processing on the pixel points in the fused feature map according to the category of the pixel points in the fused feature map to obtain a mask image.

2. The image processing method of claim 1, wherein, The method comprises the following steps: performing first convolution processing on the to-be-processed image to obtain a first convolution feature map; performing batch normalization processing on the first convolution feature map to obtain a batch normalization feature map; performing activation processing and sampling processing on the batch normalization feature map to obtain the first feature map.

3. The image processing method of claim 1, wherein, The method comprises the following steps: performing fourth convolution processing on the input feature map to obtain a fourth convolution feature map; fusing the fourth convolution feature map and the batch normalization result to obtain the output feature map of the ith second feature extraction.

4. The image processing method of claim 1, wherein, The plurality of second feature maps with different feature scales comprise a first sub-feature map, a second sub-feature map and a third sub-feature map, the scales of the first sub-feature map, the second sub-feature map and the third sub-feature map increase in turn; the method comprises the following steps: performing up-sampling on the first sub-feature map to obtain a first up-sampled feature map; superimposing the first up-sampled feature map and the second sub-feature map to obtain a superimposed feature map; performing up-sampling on the superimposed feature map to obtain a second up-sampled feature map; superimposing the second up-sampled feature map and the third sub-feature map to obtain the fused feature map.

5. The image processing method of claim 1, wherein, The method comprises the following steps: obtaining the confidence of a pixel point in the fused feature map, the confidence of the pixel point representing the probability that the category of the pixel point belongs to a target category; When the confidence of the pixel point is greater than or equal to the confidence threshold corresponding to the target category, it is determined that the category of the pixel point belongs to the target category. When the confidence of the pixel point is less than the confidence threshold corresponding to the target category, it is determined that the category of the pixel point does not belong to the target category.

6. The image processing method of claim 5, wherein, The image mask processing of the pixel points in the fusion feature map according to the categories of the pixel points in the fusion feature map comprises: When the category of the pixel point belongs to a target category, a first pixel value is generated for the pixel point; When the category of the pixel point does not belong to a target category, a second pixel value is generated for the pixel point; The mask image is obtained based on the pixel values of the pixel points.

7. The image processing method of claim 1, wherein, The method further comprises: obtaining a total number of channels of the first feature map and a preset group number; calculating a ratio of the total number of channels to the preset group number, and taking the ratio as a preset number of times of the second feature extraction.

8. A computer readable medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the image processing method of any one of claims 1 to 7.

9. An electronic device, comprising: comprise: a processor; and a memory for storing executable instructions of the processor; wherein the processor executes the executable instructions to enable the electronic device to perform the image processing method of any one of claims 1 to 7.

Images