Image processing method, device, computer readable medium and electronic device

Abstract

The application discloses an image processing method, device, computer readable medium and electronic equipment. The method comprises: acquiring an image to be processed; performing feature extraction on the image to be processed to obtain a feature map to be processed; extracting dimension feature maps corresponding to each attention weight in the feature map to be processed according to the multi-dimensional attention weight, and fusing each dimension feature map to obtain a global feature map; performing image segmentation processing on the image to be processed according to the global feature map to obtain a target segmentation region of the image to be processed; and cross-fusing the feature map to be processed and the global feature map to obtain a detection result of each target object in the target segmentation region. The global information extracted by the technical scheme is more comprehensive and detailed, which is beneficial to image segmentation of large-scale objects, and can effectively shorten the time consumed by target detection and image segmentation, thereby improving the multi-target detection efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an image processing method and device, a computer readable medium and an electronic device. BACKGROUND

[0002] A visual perception system is an important component of a self-moving device. In some cases, after the self-moving device acquires an environment image, the self-moving device determines a working area in which the self-moving device needs to work and a plurality of target objects in the working area that affect the work of the self-moving device by performing target detection processing on the environment image through the visual perception system. When a plurality of target objects need to be detected, a plurality of target detection tasks are generally implemented in series to achieve multi-target detection. However, the self-moving device is generally an embedded device, and its data processing performance is limited. When target detection involves a plurality of serial target detection tasks, a large number of feature parameters need to be calculated, especially redundant feature parameters, which requires a large amount of computing resources, thereby reducing the target detection efficiency.

[0003] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0004] The present application aims to provide an image processing method, device, computer readable medium and electronic device to solve the problem of low multi-target detection efficiency in related technologies.

[0005] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0006] According to an aspect of an embodiment of the present application, an image processing method is provided, comprising:

[0007] obtaining a to-be-processed image;

[0008] performing feature extraction on the to-be-processed image to obtain a to-be-processed feature map;

[0009] extracting a dimension feature map corresponding to each attention weight in the to-be-processed feature map according to the multi-dimensional attention weight, and fusing each dimension feature map to obtain a global feature map;

[0010] performing image segmentation processing on the to-be-processed image according to the global feature map to obtain a target segmentation region of the to-be-processed image;

[0011] cross-fusing the to-be-processed feature map and the global feature map to obtain a detection result of each target object in the target segmentation region.

[0012] According to an aspect of some embodiments of the present application, an image processing apparatus is provided, comprising:

[0013] an image obtaining module configured to obtain a to-be-processed image;

[0014] a feature extraction module configured to perform feature extraction on the to-be-processed image to obtain a to-be-processed feature map;

[0015] an attention module configured to extract, according to a plurality of multi-dimensional attention weights, a dimension feature map corresponding to each of the attention weights in the to-be-processed feature map, and fuse each of the dimension feature maps to obtain a global feature map;

[0016] an image segmentation module configured to perform image segmentation processing on the to-be-processed image according to the global feature map to obtain a target segmentation region of the to-be-processed image;

[0017] a target detection module configured to cross-fuse the to-be-processed feature map and the global feature map to obtain a detection result of each target object in the target segmentation region.

[0018] According to an aspect of some embodiments of the present application, a computer readable medium having a computer program stored thereon is provided. The computer program, when executed by a processor, implements the image processing method in the above technical solutions.

[0019] According to an aspect of some embodiments of the present application, an electronic device is provided, comprising: a processor; and a memory configured to store executable instructions of the processor; wherein the processor executes the executable instructions to enable the electronic device to perform the image processing method in the above technical solutions.

[0020] According to an aspect of some embodiments of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the image processing method in the above technical solutions.

[0021] In the technical scheme provided in the embodiments of the present application, the feature extraction is performed on the to-be-processed image to obtain a to-be-processed feature map; then, the dimension feature maps corresponding to each attention weight are extracted according to the multi-dimensional attention weights, and each dimension feature map is fused to obtain a global feature map; since the dimension feature maps corresponding to each attention weight are extracted from the global information of the to-be-processed image in each dimension, the global feature map contains the global information of multiple dimensions of the to-be-processed image, the extracted global information is more comprehensive and detailed, and this is beneficial to the image segmentation of large-scale objects. After obtaining the global feature map, image segmentation is performed according to the global feature map to obtain a target segmentation region, and the to-be-processed feature map and the global feature map are cross-fused to obtain a detection result of multiple target objects. In this way, the image segmentation and the multi-target detection are simultaneously performed, compared with the processing mode of serially performing multi-target detection in the related art, the technical scheme of the present application can effectively shorten the time consumed by target detection and image segmentation, thereby improving the multi-target detection efficiency.

[0022] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate one embodiment consistent with the present application and, together with the description, serve to explain the principles of the application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0024] Figure 1 An exemplary system architecture block diagram to which the technical scheme of the present application is applied is schematically shown.

[0025] Figure 2 A schematic diagram of the application scenario of the technical scheme of the present application is schematically shown.

[0026] Figure 3 A flowchart of the image processing method provided in an embodiment of the present application is schematically shown.

[0027] Figure 4 A flowchart of the feature extraction on the to-be-processed image provided in an embodiment of the present application is schematically shown.

[0028] Figure 5 A flowchart of the acquisition process of the attention weight provided in an embodiment of the present application is schematically shown.

[0029] Figure 6 An acquisition process diagram of the global feature map provided in an embodiment of the present application is schematically shown.

[0030] Figure 7 An output process schematic diagram of the fourth feature map provided by an embodiment of the present application is schematically shown.

[0031] Figure 8 A structural schematic diagram of a network model for implementing the technical solution of the present application provided by an embodiment of the present application is schematically shown.

[0032] Figure 9 A structural block diagram of an image processing apparatus provided by an embodiment of the present application is schematically shown.

[0033] Figure 10 A computer system structural block diagram of an electronic device suitable for implementing an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0034] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive aspects to those skilled in the art. Like reference numerals may refer to like elements throughout.

[0035] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the

[0036] The block diagrams in the drawings show only the functional entities and not necessarily the physical separation of the functional entities. That is, the functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0037] The flow diagrams shown in the drawings are merely examples and not necessarily to be construed as having all content and processes / steps, nor necessarily to be executed in the order described. For example, some processes / steps can be further divided, and some processes / steps can be combined or partially combined, so that the actual execution order can be changed according to actual conditions.

[0038] Figure 1 An example system architecture block diagram to which the technical solution of the present application is applied is schematically shown.

[0039] AsFigure 1 As shown, the system architecture 100 can include a terminal device 110, a network 120 and a server 130. The terminal device 110 can include a smart phone, a tablet computer, a notebook computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, a self-moving device, and the like. The server 130 can be a stand-alone physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The network 120 can be a communication medium of various connection types capable of providing a communication link between the terminal device 110 and the server 130, for example, can be a wired communication link or a wireless communication link.

[0040] According to the implementation needs, the system architecture in the embodiments of the present application can have any number of terminal devices, networks and servers. For example, the server 130 can be a server group composed of multiple server devices. In addition, the technical solutions provided by the embodiments of the present application can be applied to the terminal device 110, can be applied to the server 130, or can be jointly implemented by the terminal device 110 and the server 130, and the present application does not make special limitations on this.

[0041] For example, the technical solutions of the present application are implemented by the terminal device 110, and the terminal device 110 is a self-moving device, such as a mobile robot. The terminal device 110 acquires a to-be-processed image and performs feature extraction on the to-be-processed image to obtain a to-be-processed feature map, which converts the to-be-processed image into a feature map. Then, the terminal device 110 extracts a dimension feature map corresponding to each attention weight in the to-be-processed feature map according to the multi-dimensional attention weight, and fuses each dimension feature map to obtain a global feature map, for example, superimposes each dimension feature map to obtain a global feature map. Next, the terminal device 110 performs image segmentation processing on the to-be-processed image according to the global feature map to obtain a target segmentation region of the to-be-processed image, and at the same time, the terminal device 110 cross-fuses the to-be-processed feature map and the global feature map to obtain a detection result of each target object in the target segmentation region. At this point, the terminal device 110 realizes both image segmentation and multi-target object detection through the technical solutions of the present application. The terminal device 110 can divide the work area according to the image segmentation result, and plan a work path according to the detected target objects.

[0042] For example, Figure 2A schematic diagram of an application scenario of the technical solution of the present application is shown. In the application scenario, the self-moving device 200 is the main body for implementing the technical solution of the present application. The self-moving device 200 can be various types of mobile robots, such as a sweeping robot, a mopping robot, a food delivery robot, a transport robot, a mowing robot, etc. The specific type and function of the self-moving device 200 are not limited in the embodiments of the present application. It can be understood that the self-moving device 200 in the embodiments can also include other devices with self-moving functions.

[0043] In the embodiments of the present application, the self-moving device 200 is provided with a camera 210 for capturing an environment image around the self-moving device 200. The camera 210 can be fixed or non-fixed and rotatable, which is not limited in the embodiments of the present application. The environment image captured by the camera 210 can be a color image, a black-and-white image, an infrared image, etc., which is not limited in the embodiments of the present application.

[0044] The camera 210 is connected to a control module 220 inside the self-moving device 200. The control module 220 is used to receive control instructions of the self-moving device 200 or generate various types of control instructions for the self-moving device 200. The control module 220 is also connected to the driving components of the self-moving device 200, such as the steering shaft, steering wheel, motor, etc. of the self-moving device 200, for controlling the movement and steering of the self-moving device 200.

[0045] In the embodiments of the present application, the environment image captured by the camera 210 can be used as a to-be-processed image. For example, as shown in Figure 2 The environment image is a to-be-processed image 230, and the camera 210 transmits the to-be-processed image 230 to the control module 220. After receiving the to-be-processed image 230, the control module 220 processes the to-be-processed image 230 according to the image processing method provided in the embodiments of the present application to obtain an image segmentation result and a target detection result. The image segmentation result represents various regions in the to-be-processed image 230, such as a working area 231 and a non-working area 232 as shown in Figure 2 The working area 231 is a target segmentation area of the to-be-processed image 230. The target detection result represents the detection result of each target object in the target segmentation area, such as a target object 233 in the working area 231 as shown in Figure 2 After identifying the working area 231 and the target object 233 in the to-be-processed image 230, the control module 220 can adjust the moving direction or path of the self-moving device 200 according to the range of the working area 231 and the position or area of the target object 233.

[0046] For example, the self-moving device 200 is a mowing robot, the work area 231 in the to-be-processed image 230 is a grass area, the non-work area 232 is a non-grass area, and the target object 233 is an obstacle in the grass area. After the control module 220 processes the to-be-processed image according to the image processing method provided in the embodiment of the present application, the grass area and the obstacle are recognized, and then the self-moving device 200 is controlled to work in the grass area and to avoid obstacles during the working process.

[0047] The image processing method provided in the present application will be described in detail in combination with a specific implementation process.

[0048] Figure 3 An illustrative flowchart of the image processing method provided in an embodiment of the present application is shown, which can be implemented by the image processing apparatus provided in the present application, and the image processing apparatus can be configured on a terminal device or a server, such as the terminal device 110 or the server 130 shown in Figure 1 The specific implementation process of the image processing method will be described below with the image processing apparatus as the execution subject. As shown in Figure 3 The image processing method provided in the embodiment of the present application includes steps 310 to 350, and the details are as follows.

[0049] Step 310, obtaining a to-be-processed image.

[0050] Specifically, the to-be-processed image represents an image that needs to be subjected to image segmentation and target detection, for example, an image in the forward direction of the self-moving device. The to-be-processed image can be a color image, a black-and-white image, a grayscale image, etc. The to-be-processed image can be an image obtained by the image processing apparatus in real time, or an image obtained by the image processing apparatus from other devices.

[0051] Step 320, performing feature extraction on the to-be-processed image to obtain a to-be-processed feature map.

[0052] Specifically, feature extraction refers to extracting image features in the to-be-processed image, and obtaining a to-be-processed feature map according to the extracted image features. Feature extraction can be implemented through a neural network, for example, the to-be-processed image is input into a trained neural network, and the neural network performs feature extraction on the to-be-processed image and outputs a corresponding to-be-processed feature map.

[0053] In an embodiment of the present application, the feature extraction on the image to be processed includes multiple nonlinear transformations on the image to be processed, which can map a low-dimensional space to a high-dimensional space and convert linearly inseparable data in the low-dimensional space to linearly separable data in the high-dimensional space. In an embodiment of the present application, one nonlinear transformation includes convolution processing, normalization processing and correction processing. The convolution processing is a convolution processing on input data by setting a convolution kernel with a certain size. The normalization processing is a batch normalization or group normalization processing on the output data of the convolution processing, and the correction processing is a calculation on the output data after the normalization processing by an activation function to obtain the output data of the nonlinear transformation, for example, the correction processing can be performed by a ReLU (Rectified Linear Unit) function.

[0054] In an embodiment of the present application, the multiple nonlinear transformations on the image to be processed are implemented by multiple nonlinear transformation groups, and the multiple nonlinear transformation groups are connected in series, that is, the output data of a previous nonlinear transformation group is the input data of a next nonlinear transformation group. For each nonlinear transformation group, including a first nonlinear transformation and a second nonlinear transformation, the specific implementation process is as follows: performing the first nonlinear transformation on the input data to obtain first to-be-processed data; performing channel splitting on the first to-be-processed data to obtain first channel data and second channel data; performing the second nonlinear transformation on the first channel data to obtain second to-be-processed data; and fusing the second channel data and the second to-be-processed data to obtain the output data of the nonlinear transformation group.

[0055] An exemplary, Figure 4 An embodiment of the present application is schematically shown in a flow chart of the feature extraction on the image to be processed. As Figure 4As shown, the feature extraction is performed on the to-be-processed image by two nonlinear transformation groups GhoostBlock to obtain a to-be-processed feature map. For each nonlinear transformation group GhoostBlock, the data processing process includes a first nonlinear transformation 410, a channel splitting 420, a second nonlinear transformation 430 and data fusion 440 in sequence. In the first nonlinear transformation 410, the input data is first processed by a convolution kernel with a size of 3*3 (Conv), then the output data of the convolution processing is processed by batch normalization (BN), and finally the output data of the batch normalization processing is modified by a ReLU function. The size of the output first to-be-processed data is (C, H, W). In the channel splitting (Split) 420, the first to-be-processed data is split into two channels, and the channel number of the first channel data and the second channel data is C / 2. The channel splitting does not affect the length and width of the data, so the size of the first channel data and the second channel data is (C / 2, H, W). The specific execution process of the second nonlinear transformation 430 is similar to that of the first nonlinear transformation 410, and the difference between them is that the convolution processing is different. The convolution processing of the second nonlinear transformation 430 is depth convolution processing (DWConv). After the depth convolution processing, the batch normalization processing and the modification processing are still performed. The size of the output second to-be-processed data of the second nonlinear transformation 430 is (C / 2, H, W). In the data fusion (Concat) 440, the second channel data and the second to-be-processed data are fused in the channel. Since the channel number of the two data is C / 2, the channel number of the fused data is C, and the size of the output data obtained finally is (C, H, W).

[0056] With reference to Figure 3 , step 330, according to the multi-dimensional attention weight, the dimension feature map corresponding to each attention weight in the to-be-processed feature map is extracted respectively, and each dimension feature map is fused to obtain a global feature map.

[0057] Specifically, this step is to further extract features from the to-be-processed feature map based on the multi-dimensional attention mechanism to obtain a dimension feature map corresponding to each attention dimension. One attention weight corresponds to one attention dimension. The dimension feature map corresponding to each attention dimension represents the global information of the to-be-processed feature map in the attention dimension. By fusing each dimension feature map, a global feature map representing multi-dimensional global information can be obtained.

[0058] In an embodiment of the present application, the attention dimension includes a spatial dimension and a channel dimension, and the attention weight includes a feature weight of the spatial dimension and a feature weight of the channel dimension. The process of obtaining the attention weight can refer to Figure 5 , specifically including:

[0059] Step 510, performing nonlinear transformation processing on the to-be-processed feature map to obtain a transformed feature map.

[0060] Specifically, the nonlinear transformation processing is the nonlinear transformation as mentioned above. First, the to-be-processed feature map is subjected to convolution processing according to a preset convolution kernel to obtain a first feature map. Then, the first feature map is subjected to batch normalization processing to obtain a second feature map. Finally, the second feature map is subjected to correction processing to obtain the transformed feature map.

[0061] In an embodiment of the present application, since the attention weight includes two kinds, two nonlinear transformation processes can be set to perform feature extraction on the to-be-processed feature map to obtain two transformed feature maps. Exemplarily, one nonlinear transformation process is denoted as (Conv->BN->ReLU), that is, it includes convolution processing, batch normalization processing and correction processing. Two nonlinear transformation processes (Conv->BN->ReLU) with a convolution kernel size of 3*3 can be used to perform feature extraction on the to-be-processed feature map with a size of (C, H, W) to obtain two transformed feature maps with a size of (C, H, W).

[0062] Step 520, performing spatial dimension pooling processing on the transformed feature map to obtain a spatial feature map.

[0063] Specifically, the spatial dimension pooling processing refers to performing feature extraction on features at different spatial positions in the transformed feature map to form a spatial feature map. In an embodiment of the present application, the spatial dimension pooling processing includes maximum pooling processing and average pooling processing, and therefore the spatial feature map includes a maximum-pooled spatial feature map and an average-pooled spatial feature map.

[0064] Step 530, performing channel dimension pooling processing on the transformed feature map to obtain a channel feature map.

[0065] Specifically, the channel dimension pooling processing refers to performing feature extraction on features at different channel positions in the transformed feature map to form a channel feature map. In an embodiment of the present application, the channel dimension pooling processing includes maximum pooling processing and average pooling processing, and therefore the spatial feature map includes a maximum-pooled channel feature map and an average-pooled channel feature map.

[0066] Step 540, performing channel compression and normalization processing on the spatial feature map to obtain a feature weight in the spatial dimension.

[0067] Specifically, the channel compression is implemented through convolution processing, and the normalization processing is implemented through an activation function. The channel compression and normalization processing on the spatial feature map is to splice the maximum-pooled spatial feature map and the average-pooled spatial feature map according to the channel, then perform convolution processing on the spliced result, and then normalize the convolution processing result to [0, 1] through an activation function to obtain the feature weight in the spatial dimension.

[0068] Step 550, channel compression and normalization processing is performed on the channel feature map to obtain the feature weight in the channel dimension.

[0069] Specifically, the process of obtaining the feature weight in the channel dimension is similar to the process of obtaining the feature weight in the spatial dimension. First, the maximum pooling channel feature map and the average pooling channel feature map are spliced according to the channel, then the spliced result is convolved, and then the convolution result is normalized to [0, 1] through an activation function to obtain the feature weight in the channel dimension.

[0070] In an embodiment of the present application, after obtaining the attention weight, the global feature map can be output according to the attention weight. The specific process is: convolving the transformed feature map according to the feature weight in the spatial dimension to obtain a spatial dimension feature map; convolving the transformed feature map according to the feature weight in the channel dimension to obtain a channel dimension feature map; and fusing the spatial dimension feature map and the channel dimension feature map to obtain the global feature map.

[0071] Specifically, convolving the transformed feature map according to the feature weight in the spatial dimension is to apply the feature weight in the spatial dimension to the transformed feature map in a feature reweighting manner to obtain the spatial dimension feature map. Similarly, the feature weight in the channel dimension is applied to the transformed feature map in a feature reweighting manner to obtain the channel dimension feature map. Finally, the spatial dimension feature map and the channel dimension feature map are fused to complete the extraction of global information, and the global feature map is obtained. The fusion process can be to superimpose the spatial dimension feature map and the channel dimension feature map in a bit-by-bit addition manner, and then sequentially perform convolution processing, batch normalization processing and rectification processing on the superimposed result to obtain the global feature map.

[0072] An exemplary, Figure 6 An exemplary diagram of the process of obtaining the global feature map provided by an embodiment of the present application is shown. As Figure 6 shown, the size of the feature map to be processed is (C, H, W). Two nonlinear transformation processes (Conv->BN->ReLU) with a convolution kernel size of 3*3 are used to process the feature map to be processed to obtain two transformed feature maps with a size of (C, H, W). Then, the feature weight in the spatial dimension is obtained according to one transformed feature map, and the feature weight in the channel dimension is obtained according to the other transformed feature map.

[0073] For the feature weight of the spatial dimension, firstly, the transformed feature map is respectively subjected to maximum pooling processing (MAX) and average pooling processing (MEAN) to obtain two spatial feature maps; then the two spatial feature maps are data fused (Concat) to be spliced into a spatial splicing feature map with a size of (2, H, W), and the spatial splicing feature map is subjected to convolution processing (Conv1*1) through a 1*1 convolution kernel to realize channel compression, and is subjected to normalization processing through a sigmoid (S-type) function to obtain the feature weight of the spatial dimension with a size of (1, H, W). Then the feature weight of the spatial dimension is applied to the transformed feature map to obtain a spatial dimension feature map with a size of (C, H, W).

[0074] For the feature weight of the channel dimension, firstly, the transformed feature map is respectively subjected to maximum pooling processing (MAX) and average pooling processing (MEAN) to obtain two channel feature maps; then the two channel feature maps are data fused (Concat) to be spliced into a channel splicing feature map with a size of (2C, 1, 1), and the channel splicing feature map is subjected to convolution processing (Conv 1*1) through a 1*1 convolution kernel to realize channel compression, and is subjected to normalization processing through a sigmoid function to obtain the feature weight of the channel dimension with a size of (C, 1, 1). Then the feature weight of the channel dimension is applied to the transformed feature map to obtain a channel dimension feature map with a size of (C, H, W).

[0075] Finally, the spatial dimension feature map and the channel dimension feature map are superimposed, and are sequentially subjected to 1*1 convolution processing (Conv), batch normalization processing (BN) and correction processing (ReLU) to output a global feature map.

[0076] With reference to Figure 3 , step 340, image segmentation processing is performed on the to-be-processed image according to the global feature map to obtain a target segmentation region of the to-be-processed image.

[0077] Specifically, the image segmentation processing is to identify image regions with different attributes or categories in the to-be-processed image, for example, to identify working regions and non-working regions, so as to determine the target segmentation region in the to-be-processed image. Since the global feature map includes multi-dimensional global information, the image segmentation according to the global feature map enables the image segmentation to consider the overall size of the image, and thus is beneficial to the segmentation of large-scale regions in the image to some extent, and increases the accuracy of the segmentation of large-scale regions.

[0078] In an embodiment of the present application, the process of the image segmentation processing specifically includes: performing convolution sampling processing on the global feature map for a preset number of times to obtain a third feature map; performing convolution processing on the third feature map to obtain a fourth feature map; and performing region segmentation processing on the to-be-processed image according to the fourth feature map to obtain a target segmentation region of the to-be-processed image.

[0079] Specifically, the convolution sampling processing includes a convolution processing and an up-sampling processing, and the convolution processing can be a deep convolution processing or other convolution processing. The convolution sampling processing of the preset number of times is in a serial processing manner, that is, the output data of the previous convolution sampling processing is the input data of the next convolution sampling processing. After the convolution sampling processing, a convolution processing is further performed on the third feature map to obtain a fourth feature map. Finally, region segmentation is performed according to the fourth feature map to determine the target segmentation region.

[0080] An exemplary, Figure 7 An output process diagram of the fourth feature map provided by an embodiment of the present application is schematically shown. As Figure 7 shown, the convolution sampling processing includes a deep convolution (DW) with a convolution kernel size of 3*3 and an up-sampling processing (UpSample), and the global feature map has a size of (C, H, W). After one convolution sampling processing, the obtained feature map has a size of (C / 4, H*4, W*4). After another convolution sampling processing (that is, after two convolution sampling processings), the obtained feature map has a size of (C / 16, H*16, W*16). After one convolution sampling processing (that is, after three convolution sampling processings), the third feature map has a size of (C / 32, H*32, W*32). Finally, a 1*1 convolution kernel is used to perform a convolution operation (Conv1*1) on the third feature map to output the fourth feature map.

[0081] In an embodiment of the present application, the process of performing region segmentation according to the fourth feature map includes: determining position information of the target segmentation region according to the fourth feature map; and performing mask processing on the to-be-processed image based on the position information to obtain the target segmentation region after mask processing.

[0082] Specifically, region recognition is performed according to the fourth feature map to detect which different types of regions are contained in the to-be-recognized image, and then the position information of the target segmentation region is determined. The position information is the position of the target segmentation region in the to-be-processed image. The mask processing is to cover the target segmentation region with a mask, so as to distinguish the target segmentation region from other regions, thereby recognizing the target segmentation region. An exemplary, Figure 2 as shown in the to-be-processed image 230, the working region 231 is covered with a mask, and the non-working region 232 is not covered with a mask.

[0083] Continuing to refer to Figure 3 , step 350, the to-be-processed feature map and the global feature map are cross-fused to obtain a detection result of each target object in the target segmentation region.

[0084] Specifically, the cross fusion of the to-be-processed feature map and the global feature map includes a top-down fusion process and a bottom-up fusion process, wherein the bottom-up fusion process is used to transmit semantic information of high-level features, and the bottom-up fusion process is used to transmit position information of low-level features. Through this cross fusion manner, the target object detection fuses high-level semantic information and low-level position information, which is beneficial to improve the detection accuracy. For example, as shown in Figure 2 The to-be-processed image 230 is identified as the target object 233 through cross fusion.

[0085] In the technical scheme provided in the embodiments of the present application, the feature extraction is performed on the to-be-processed image to obtain a to-be-processed feature map; then, the dimension feature maps corresponding to each attention weight are extracted according to the multi-dimensional attention weights, and each dimension feature map is fused to obtain a global feature map; since the dimension feature maps corresponding to each attention weight are extracted from each dimension to obtain the global information of the to-be-processed image, the global feature map contains the global information of multiple dimensions of the to-be-processed image, and the extracted global information is more comprehensive and detailed, which is beneficial to image segmentation of large-scale objects. After obtaining the global feature map, image segmentation is performed according to the global feature map to obtain a target segmentation region, and the to-be-processed feature map and the global feature map are cross fused to obtain detection results of multiple target objects. Thus, the image segmentation and the multi-target detection are simultaneously performed, compared with the processing manner of serially performing multi-target detection in the related art, the technical scheme of the present application can effectively shorten the time consumed by target detection and image segmentation, thereby improving the multi-target detection efficiency.

[0086] In an embodiment of the present application, the process of obtaining the detection results of the target objects based on the cross fusion includes: cross fusing the to-be-processed feature map and the global feature map to obtain multiple target detection feature maps, wherein each target detection feature map includes multiple original detection boxes corresponding to each target object in the target segmentation region; and screening the multiple original detection boxes corresponding to each target object to obtain a target detection box corresponding to each target object.

[0087] Specifically, the multiple target detection feature maps output after cross fusion have different scales. The forming process of the multiple target detection feature maps can be: obtaining a first target detection feature map according to a to-be-processed feature map; obtaining a second target detection feature map according to a fusion result of the to-be-processed feature map and a global feature map; and obtaining a third target detection feature map according to a fusion result of the second target detection feature map and the global feature map. The fusion result of the to-be-processed feature map and the global feature map is a first fusion result, and the fusion result of the second target detection feature map and the global feature map is equivalent to a second fusion result of the to-be-processed feature map and the global feature map. The fusion of the to-be-processed feature map and the global feature map includes up-sampling processing, convolution processing, non-linear transformation, channel superposition and the like of the two, and the difference between the first fusion result and the second fusion result lies in the difference in the order and number of the specific fusion processes.

[0088] The original detection box is a detection box for bounding a target object in the target detection feature map, and each target detection feature map includes multiple original detection boxes. Each target detection feature map can include original detection boxes of different sizes. When multiple original detection boxes corresponding to each target object are screened, the original detection boxes can be screened according to whether the coverage of the area framed by each original detection box on the boundary of the target object is appropriate, for example, the original detection boxes with too low coverage of the target object boundary or the original detection boxes with too much distance beyond the boundary range of the target object are unqualified detection boxes. In an embodiment of the present application, the multiple original detection boxes can be screened by a Non-Maximum Suppression (NMS) method to obtain a target detection box.

[0089] Exemplarily, Figure 8 An exemplary structural schematic diagram of a network model for implementing the technical solution of the present application is shown.

[0090] As Figure 8 shown, the network model includes a backbone network Backbone, an intermediate network Neck and a head network Head. The backbone network Backbone includes two non-linear transformation groups GhoostBlock and an attention module DAM. The two non-linear transformation groups GhoostBlock in the backbone network Backbone are used for feature extraction of a to-be-processed image, and output a to-be-processed feature map to the attention module DAM. The specific structure of the non-linear transformation group GhoostBlock can be referred to in Figure 4 . The attention module DAM is used for extracting a dimensional feature map corresponding to each attention weight in the to-be-processed feature map according to a multi-dimensional attention weight, and fusing the dimensional feature maps to obtain a global feature map. The specific structure of the attention module DAM can be referred to in Figure 6 .

[0091] The intermediate network Neck is used to cross-fuse the to-be-processed feature map and the global feature map, and output multiple target detection feature maps. Specifically, the to-be-processed image is processed by the nonlinear transformation group GhoostBlock1 in the backbone network Backbone and the nonlinear transformation group GhoostBlock3 in the intermediate network Neck, and is converted into a to-be-processed feature map, and then the intermediate network Neck outputs a first target detection feature map according to the to-be-processed feature map. The bottom-up fusion process in the intermediate network Neck is that: the data output by the up-sampling processing UpSample of the global feature map is channel-stacked with the to-be-processed feature map to obtain first stacked data; the stacked data is processed by the nonlinear transformation group GhoostBlock4 and the convolution processing (Conv 3*3), and an upward fusion result is output. The top-down fusion process in the intermediate network Neck is that: after the to-be-processed feature map is processed by the convolution processing (Conv 3*3S2), the to-be-processed feature map is channel-stacked with the upward fusion result to obtain second stacked data; the to-be-processed feature map, the first stacked data and the second stacked data are input into the nonlinear transformation group GhoostBlock5 together, and a second target detection feature map is output; further, after the second target detection feature map is processed by the convolution processing (Conv 3*3S2), the second target detection feature map is channel-stacked with the global feature map and processed by the nonlinear transformation group GhoostBlock6, and a third target detection feature map is output.

[0092] The head network Head includes an image segmentation part H1 and a target detection part H2. The image segmentation part H1 performs convolution sampling processing and convolution processing on the global feature map through a simple sampling module UMP, and outputs a fourth feature map. The specific structure of the simple sampling module UMP can be referred to in Figure 7 ; then the image segmentation part H1 performs image segmentation according to the fourth feature map, and outputs a target segmentation region in the to-be-processed image. The target detection part H2 respectively screens the original detection boxes in the first target detection feature map, the second target detection feature map and the third target detection feature map, obtains the target detection results corresponding to each target detection feature map, and finally the target detection part H2 selects the target detection result corresponding to one of the target detection feature maps as the detection result of each target object in the target segmentation region in the to-be-processed image.

[0093] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps, etc.

[0094] The device embodiment of the present application is introduced below, which can be used to execute the image processing method in the above-mentioned embodiments of the present application. Figure 9 The structural block diagram of the image processing device provided by the embodiments of the present application is schematically shown. As shown in Figure 9 The image processing device provided by the embodiments of the present application comprises:

[0095] The image acquisition module 910 is configured to acquire a to-be-processed image.

[0096] The feature extraction module 920 is configured to perform feature extraction on the to-be-processed image to obtain a to-be-processed feature map.

[0097] The attention module 930 is configured to extract, according to the multi-dimensional attention weight, a dimension feature map corresponding to each attention weight in the to-be-processed feature map, and fuse each dimension feature map to obtain a global feature map.

[0098] The image segmentation module 940 is configured to perform image segmentation processing on the to-be-processed image according to the global feature map to obtain a target segmentation region of the to-be-processed image.

[0099] The target detection module 950 is configured to cross-fuse the to-be-processed feature map and the global feature map to obtain a detection result of each target object in the target segmentation region.

[0100] In an embodiment of the present application, the multi-dimensional attention weight comprises a feature weight of a spatial dimension and a feature weight of a channel dimension, and the device further comprises:

[0101] The nonlinear transformation module is configured to perform nonlinear transformation processing on the to-be-processed feature map to obtain a transformed feature map.

[0102] The spatial pooling module is configured to perform spatial dimension pooling processing on the transformed feature map to obtain a spatial feature map.

[0103] The channel pooling module is configured to perform channel dimension pooling processing on the transformed feature map to obtain a channel feature map.

[0104] The spatial weight acquisition module is configured to perform channel compression and normalization processing on the spatial feature map to obtain the feature weight of the spatial dimension.

[0105] The channel weight acquisition module is configured to perform channel compression and normalization processing on the channel feature map to obtain the feature weight of the channel dimension.

[0106] In an embodiment of the present application, the attention module 930 is specifically configured to:

[0107] perform convolution processing on the transformed feature map according to the feature weight of the spatial dimension to obtain a spatial dimension feature map;

[0108] The transformed feature map is obtained by performing convolution processing on the channel-dimension feature map according to a characteristic weight of the channel dimension.

[0109] The global feature map is obtained by fusing the spatial-dimension feature map and the channel-dimension feature map.

[0110] In an embodiment of the present application, the nonlinear transformation module is specifically configured to:

[0111] The first feature map is obtained by performing convolution processing on the to-be-processed feature map according to a preset convolution kernel;

[0112] The second feature map is obtained by performing batch normalization processing on the first feature map;

[0113] The transformed feature map is obtained by performing correction processing on the second feature map.

[0114] In an embodiment of the present application, the image segmentation module 940 includes:

[0115] The sampling unit is configured to perform convolution sampling processing on the global feature map for a preset number of times to obtain a third feature map; the convolution sampling processing includes convolution processing and up-sampling processing;

[0116] The convolution unit is configured to perform convolution processing on the third feature map to obtain a fourth feature map;

[0117] The region segmentation unit is configured to perform region segmentation processing on the to-be-processed image according to the fourth feature map to obtain a target segmentation region of the to-be-processed image.

[0118] In an embodiment of the present application, the region segmentation unit is specifically configured to:

[0119] The position information of the target segmentation region is determined according to the fourth feature map;

[0120] The to-be-processed image is subjected to mask processing based on the position information to obtain a target segmentation region after mask processing.

[0121] In an embodiment of the present application, the target detection module 950 includes:

[0122] The cross-fusion unit is configured to perform cross-fusion on the to-be-processed feature map and the global feature map to obtain a plurality of target detection feature maps; each target detection feature map includes a plurality of original detection boxes corresponding to each target object in the target segmentation region;

[0123] The screening unit is configured to screen the plurality of original detection boxes corresponding to each target object to obtain a target detection box corresponding to each target object.

[0124] In an embodiment of the present application, the cross-fusion unit is specifically configured to:

[0125] obtain a first target detection feature map based on the to-be-processed feature map;

[0126] perform fusion processing on the to-be-processed feature map and the global feature map to obtain a second target detection feature map;

[0127] perform fusion processing on the global feature map and the second target detection feature map to obtain a third target detection feature map.

[0128] The specific details of the image processing apparatus provided in the embodiments of the present application have been described in detail in the corresponding method embodiments, and will not be repeated here.

[0129] Figure 10 A computer system structure block diagram of an electronic device for implementing the embodiments of the present application is schematically shown.

[0130] It should be noted that, Figure 10 The computer system 1000 of the electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0131] As Figure 10 shown, the computer system 1000 includes a central processor 1001 (CPU), which can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 (ROM) or loaded into a random access memory 1003 (RAM) from a storage portion 1008. In the random access memory 1003, various programs and data required by system processing are also stored. The central processor 1001, the read-only memory 1002, and the random access memory 1003 are connected to each other through a bus 1004. An input / output interface 1005 (I / O interface) is also connected to the bus 1004.

[0132] The following components are connected to the input / output interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a Local Area Network card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output interface 1005 as necessary. A removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1010 as necessary, so that a computer program read therefrom is installed in the storage portion 1008 as necessary.

[0133] In particular, according to embodiments of the present application, the processes described in the various method flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 1009, and / or installed from the removable media 1011. When the computer program is executed by the central processing unit 1001, various functions defined in the system of the present application are performed.

[0134] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0135] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and processes of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the involved functions. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or processes, or can be implemented by a combination of dedicated hardware and computer instructions.

[0136] It should be noted that, although several modules or units for a device for action execution are mentioned in the above detailed description, such a division is not mandatory. Indeed, according to an embodiment of the application, the features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functionalities of one module or unit described above can be further divided into several modules or units embodied.

[0137] Those skilled in the art can easily understand, through the above description of the embodiments, that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes several instructions to make a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the methods according to the embodiments of the present application.

[0138] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the present application that come within the scope of the claims and that the terms describe and of the specification be interpreted to cover such variations. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0139] It should be understood that the application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should be limited only by the appended claims.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a to-be-processed image; performing feature extraction on the to-be-processed image to obtain a to-be-processed feature map; extracting dimension feature maps corresponding to each attention weight in the to-be-processed feature map according to the multi-dimensional attention weights, and fusing each dimension feature map to obtain a global feature map; performing image segmentation processing on the to-be-processed image according to the global feature map to obtain a target segmentation region of the to-be-processed image; obtaining a first target detection feature map based on the to-be-processed feature map; performing channel superposition on data output by the global feature map after upsampling processing and the to-be-processed feature map to obtain first superimposed data; performing convolution processing on the first superimposed data through a nonlinear transformation group to output an upward fusion result; performing channel superposition on the to-be-processed feature map after convolution processing and the upward fusion result to obtain second superimposed data; inputting the to-be-processed feature map, the first superimposed data and the second superimposed data into a nonlinear transformation group to output a second target detection feature map; fusing the global feature map and the second target detection feature map to obtain a third target detection feature map; wherein each target object in the target segmentation region corresponds to a plurality of original detection boxes in the first target detection feature map, the second target detection feature map and the third target detection feature map; screening the original detection boxes in the first target detection feature map, the second target detection feature map and the third target detection feature map respectively to obtain a target detection box corresponding to each target object.

2. The image processing method of claim 1, wherein, The multi-dimensional attention weights comprise feature weights of a spatial dimension and feature weights of a channel dimension, and before the extracting of the dimension feature maps corresponding to each attention weight in the to-be-processed feature map according to the multi-dimensional attention weights, the method further comprises: performing nonlinear transformation processing on the to-be-processed feature map to obtain a transformed feature map; performing pooling processing of a spatial dimension on the transformed feature map to obtain a spatial feature map; performing pooling processing of a channel dimension on the transformed feature map to obtain a channel feature map; performing channel compression and normalization processing on the spatial feature map to obtain the feature weights of the spatial dimension; performing the channel compression and the normalization processing on the channel feature map to obtain the feature weights of the channel dimension.

3. The image processing method of claim 2, wherein, The extracting of the dimension feature maps corresponding to each attention weight in the to-be-processed feature map according to the multi-dimensional attention weights and the fusing of each dimension feature map to obtain a global feature map comprise: performing convolution processing on the transformed feature map according to the feature weights of the spatial dimension to obtain a spatial dimension feature map; performing convolution processing on the transformed feature map according to the feature weights of the channel dimension to obtain a channel dimension feature map; fusing the spatial dimension feature map and the channel dimension feature map to obtain the global feature map.

4. The image processing method of claim 2, wherein, The nonlinear transformation processing on the to-be-processed feature map to obtain a transformed feature map comprises: performing convolution processing on the to-be-processed feature map according to a preset convolution kernel to obtain a first feature map; The first feature map is subjected to batch normalization processing to obtain a second feature map; The second feature map is subjected to correction processing to obtain the transformed feature map.

5. The image processing method of claim 1, wherein, The image segmentation processing of the to-be-processed image according to the global feature map comprises: The global feature map is subjected to a preset number of convolution sampling processes to obtain a third feature map; the convolution sampling process comprises convolution processing and up-sampling processing; The third feature map is subjected to convolution processing to obtain a fourth feature map; The region segmentation processing of the to-be-processed image according to the fourth feature map obtains a target segmentation region of the to-be-processed image.

6. The image processing method of claim 5, wherein, The region segmentation processing of the to-be-processed image according to the fourth feature map obtains a target segmentation region of the to-be-processed image, comprising: The position information of the target segmentation region is determined according to the fourth feature map; The to-be-processed image is subjected to mask processing based on the position information to obtain the target segmentation region after the mask processing.

7. A computer readable medium having stored thereon a computer program, characterized in that The computer program is executed by a processor to implement the image processing method of any one of claims 1 to 6.

8. An electronic device, comprising: Comprise: a processor; and a memory for storing executable instructions of the processor; wherein the processor executes the executable instructions to enable the electronic device to perform the image processing method of any one of claims 1 to 6.

Images