Image processing method, device, electronic device and computer readable medium

Abstract

The present invention provides an image processing method, device, electronic device and computer-readable medium, including: extracting shallow features of a target object in an image to be processed to obtain a shallow feature map of the target object; obtaining a first instance segmentation feature map obtained when an Anchor Based instance segmentation method is used to perform instance segmentation on the target object in the image to be processed; performing a guided filtering operation on the first instance segmentation feature map based on the shallow feature map of the target object to obtain a second instance segmentation feature map; and determining a mask of the target object in the image to be processed based on the first instance segmentation feature map and the second instance segmentation feature map. The second instance segmentation feature map of the present invention carries edge information of the target object, and the edge effect of the mask of the target object in the image to be processed is finally determined to be better, thereby improving the performance and accuracy of instance segmentation, and the method of the present invention has a small amount of calculation and high image processing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method, device, electronic device and computer-readable medium. Background Art

[0002] The target instance segmentation task refers to the network's ability to accurately segment different categories of targets on the image and different individuals of the same category of targets. The current instance segmentation framework design methods mainly include two categories: Anchor Free method and Anchor Based (anchor-based instance segmentation) method. The Anchor Based method first generates the target box and then performs target segmentation; the Anchor Free method directly predicts the Mask and category and combines them to obtain the final segmentation result; the Anchor Free method is faster than the Anchor Based method, but the accuracy is relatively poor. However, unlike the semantic segmentation task with a larger resolution input, the input of the instance segmentation branch is often the feature (7x7) of the target area after ROI Pooling or ROI Alignment, which has a smaller resolution, which results in limited performance of edge detail segmentation. The current common methods to improve the performance of the instance segmentation branch include increasing the ROI size, adding additional supervision Loss, cascading multiple Stages, etc.

[0003] The current instance segmentation framework design methods mainly include: (1) Mask R-CNN, based on Faster R-CNN, changes the ROI size to 14x14 and adds an FCN branch to achieve instance mask segmentation; (2) BMask R-CNN, based on Mask R-CNN, adds a boundary branch, fuses the features with the mask branch and performs edge supervision to achieve the effect of enhancing the edge; (3) SOLO, grids the image, and performs semantic category prediction and instance mask generation for each small grid.

[0004] The problems with these instance segmentation frameworks are: (1) there is no trade-off between ROI size, accuracy, and speed; (2) additional supervision information is required, i.e., creating new multi-task learning labels; and (3) the additional cascade stage increases the amount of computation.

[0005] In summary, existing instance segmentation methods have technical problems such as large amount of calculation and poor segmentation effect when performing instance segmentation on images. Summary of the invention

[0006] In view of this, an object of the present invention is to provide an image processing method, device, electronic device and computer-readable medium to alleviate the technical problems of large computational complexity and poor segmentation effect in existing instance segmentation methods when performing instance segmentation on images.

[0007] In a first aspect, an embodiment of the present invention provides an image processing method, comprising: obtaining an image to be processed, and performing shallow feature extraction on a target object in the image to be processed to obtain a shallow feature map of the target object, wherein the shallow feature map of the target object contains edge information of the target object; obtaining a first instance segmentation feature map obtained when an Anchor Based instance segmentation method is used to perform instance segmentation on the target object in the image to be processed; performing a guided filtering operation on the first instance segmentation feature map based on the shallow feature map of the target object to obtain a second instance segmentation feature map, wherein the second instance segmentation feature map carries edge information of the target object; and determining a mask of the target object in the image to be processed based on the first instance segmentation feature map and the second instance segmentation feature map.

[0008] Furthermore, performing shallow feature extraction on the target object in the image to be processed includes: performing multi-scale scaling processing on the image to be processed to obtain a multi-scale image to be processed; determining the image to be processed of the target scale in the multi-scale image to be processed according to the target frame coordinates of the target object predicted in the Anchor Based instance segmentation method; inputting the image to be processed of the target scale into a shallow feature extraction network to obtain a shallow feature map of the target object, wherein the shallow feature extraction network includes two first target convolutional layers connected in series.

[0009] Furthermore, performing a guided filtering operation on the first instance segmentation feature map based on the shallow feature map of the target object includes: performing a ROI Align operation on the shallow feature map of the target object according to the target frame coordinates to obtain a shallow feature map of the target object at a target scale; upsampling the first instance segmentation feature map to obtain an upsampled first instance segmentation feature map; performing a guided filtering operation on the upsampled first instance segmentation feature map based on the shallow feature map of the target object at the target scale to obtain the second instance segmentation feature map.

[0010] Furthermore, performing a guided filtering operation on the upsampled first instance segmentation feature map based on the shallow feature map of the target object at the target scale includes: performing a mean filtering operation on the shallow feature map of the target object at the target scale and the upsampled first instance segmentation feature map, respectively, to obtain a mean filtering operation result; calculating a first correlation coefficient matrix between the shallow feature map of the target object at the target scale and the shallow feature map of the target object at the target scale, and calculating a second correlation coefficient matrix between the shallow feature map of the target object at the target scale and the upsampled first instance segmentation feature map; and determining the second instance segmentation feature map based on the mean filtering operation result, the first correlation coefficient matrix, and the second correlation coefficient matrix.

[0011] Further, determining the second instance segmentation feature map based on the mean filtering operation result, the first correlation coefficient matrix and the second correlation coefficient matrix includes: calculating a first variance of a shallow feature map of the target object at the target scale and a shallow feature map of the target object at the target scale according to the mean filtering operation result and the first correlation coefficient matrix, and calculating a second variance of the shallow feature map of the target object at the target scale and the upsampled first instance segmentation feature map according to the mean filtering operation result and the second correlation coefficient matrix; calculating a first coefficient matrix according to the first variance and the second variance, and calculating a second coefficient matrix according to the first coefficient matrix and the mean filtering operation result; linearly combining the first coefficient matrix, the second coefficient matrix and the shallow feature map of the target object at the target scale to obtain the second instance segmentation feature map.

[0012] Furthermore, determining the mask of the target object in the image to be processed based on the first instance segmentation feature map and the second instance segmentation feature map includes: inputting the second instance segmentation feature map into a Laplacian convolutional network to obtain an edge feature map of the target object; inputting the first instance segmentation feature map into a preset number of second target convolutional layers to obtain a third instance segmentation feature map; adding the edge feature map of the target object to the third instance segmentation feature map, and determining the mask of the target object in the image to be processed based on the result of the addition operation.

[0013] Furthermore, determining the mask of the target object in the image to be processed according to the result obtained by the addition operation includes: inputting the result obtained by the addition operation into a third target convolution layer to obtain the mask of the target object in the image to be processed.

[0014] Furthermore, the first instance segmentation feature map is a feature obtained after the Anchor Based instance segmentation method performs a ROI Align operation on the image to be processed.

[0015] In the second aspect, an embodiment of the present invention also provides an image processing device, comprising: a shallow feature extraction unit, used to obtain an image to be processed, and perform shallow feature extraction on a target object in the image to be processed to obtain a shallow feature map of the target object, wherein the shallow feature map of the target object contains edge information of the target object; an acquisition unit, used to obtain a first instance segmentation feature map obtained when an Anchor Based instance segmentation method is used to perform instance segmentation on the target object in the image to be processed; a guided filtering unit, used to perform a guided filtering operation on the first instance segmentation feature map based on the shallow feature map of the target object to obtain a second instance segmentation feature map, wherein the second instance segmentation feature map carries edge information of the target object; a determination unit, used to determine a mask of the target object in the image to be processed based on the first instance segmentation feature map and the second instance segmentation feature map.

[0016] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the steps of the method described in any one of the first aspects above.

[0018] In an embodiment of the present invention, first, an image to be processed is obtained, and shallow feature extraction is performed on the target object in the image to be processed to obtain a shallow feature map of the target object, wherein the shallow feature map of the target object contains the edge information of the target object; then, a first instance segmentation feature map is obtained when the Anchor Based instance segmentation method performs instance segmentation on the target object in the image to be processed; then, a guided filtering operation is performed on the first instance segmentation feature map based on the shallow feature map of the target object to obtain a second instance segmentation feature map, wherein the second instance segmentation feature map carries the edge information of the target object; finally, a mask of the target object in the image to be processed is determined based on the first instance segmentation feature map and the second instance segmentation feature map. It can be seen from the above description that since the second instance segmentation feature map carries the edge information of the target object, the edge effect of the mask of the target object in the image to be processed determined based on the first instance segmentation feature map and the second instance segmentation feature map is better, which improves the performance and accuracy of instance segmentation. In addition, the image processing method of the present invention does not need to use additional supervision, only shallow feature extraction and guided filtering operations are required, the amount of calculation is small, and the image processing efficiency is high, which alleviates the technical problems of large amount of calculation and poor segmentation effect when the existing instance segmentation method performs instance segmentation on the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram of an electronic device provided by an embodiment of the present invention;

[0021] Figure 2 A flowchart of an image processing method provided by an embodiment of the present invention;

[0022] Figure 3 A framework diagram of an image processing method provided by an embodiment of the present invention;

[0023] Figure 4 A flow chart of a method for extracting shallow features of a target object in an image to be processed provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of an image to be processed provided by an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of a shallow feature map of a target object provided by an embodiment of the present invention;

[0026] Figure 7 A flow chart of a method for performing a guided filtering operation on a first instance segmentation feature map based on a shallow feature map of a target object provided by an embodiment of the present invention;

[0027] Figure 8 A schematic diagram of a first example segmentation feature map provided by an embodiment of the present invention;

[0028] Fig. 9 A flow chart of a method for performing a guided filtering operation on an upsampled first instance segmentation feature map based on a shallow feature map of a target object at a target scale provided by an embodiment of the present invention;

[0029] Fig.10 A schematic diagram of a second example segmentation feature map provided by an embodiment of the present invention;

[0030] Fig.11 A flowchart of determining a mask of a target object in an image to be processed based on a first instance segmentation feature map and a second instance segmentation feature map provided by an embodiment of the present invention;

[0031] Fig.12 The embodiment of the present invention provides Figure 5 A schematic diagram of a mask of a target object corresponding to the image to be processed;

[0032] Fig.13 A schematic diagram of an image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Embodiment 1:

[0035] First, refer to Figure 1 An electronic device 100 for implementing an embodiment of the present invention is described. The electronic device can be used to execute the image processing method of each embodiment of the present invention.

[0036] like Figure 1 As shown, the electronic device 100 includes one or more processors 102, one or more memories 104, an input device 106, an output device 108, and a camera 110, and these components are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as required.

[0037] The processor 102 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic array (PLA) and an ASIC (Application Specific Integrated Circuit). The processor 102 can be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 100 to perform desired functions.

[0038] The memory 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the client functions (implemented by the processor) and / or other desired functions in the embodiments of the present invention described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0039] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0040] The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0041] The camera 110 is used to acquire an image to be processed, wherein the image to be processed acquired by the camera is processed by the image processing method to obtain a mask of the target object in the image to be processed. For example, the camera can shoot an image desired by a user (such as a photo, video, etc.), and then, the image is processed by the image processing method to obtain a mask of the target object in the image to be processed. The camera can also store the captured image in the memory 104 for use by other components.

[0042] Exemplarily, the electronic device for implementing the image processing method according to the embodiment of the present invention may be implemented as an intelligent mobile terminal such as a smart phone, a tablet computer, etc.

[0043] Embodiment 2:

[0044] According to an embodiment of the present invention, an embodiment of an image processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Figure 2 is a flowchart of an image processing method according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0046] Step S102, obtaining an image to be processed, and performing shallow feature extraction on a target object in the image to be processed to obtain a shallow feature map of the target object, wherein the shallow feature map of the target object contains edge information of the target object;

[0047] In the embodiment of the present invention, the target object may be any object of interest. For example, if there are multiple dog images in the image to be processed, the target object may be any image of a dog in the image to be processed. For another example, if there are multiple human images in the image to be processed, the target object may be any image of a human in the image to be processed. The embodiment of the present invention does not impose any specific limitation on the target object.

[0048] The input of the traditional instance segmentation branch is often the features of the target area (for example, the area of ​​the target object) after ROIPooling or ROI Align, which has a low resolution, resulting in limited performance of edge detail segmentation, and the final target object mask has poor accuracy. Based on this, the inventors consider that the shallow features of the CNN network have rich high-frequency information such as edges, which plays an important role in guiding high-level features to restore edge detail information. If it is applied to instance segmentation, the effect of instance segmentation can be greatly improved, so the scheme of the present invention is designed. After obtaining the image to be processed, the shallow feature map of the target object in the image to be processed is extracted.

[0049] Step S104, obtaining a first instance segmentation feature map obtained when the Anchor Based instance segmentation method is used to perform instance segmentation on the target object in the image to be processed;

[0050] In the Anchor Based instance segmentation method, the target box of the target object is first generated, and then the target is segmented one by one. Figure 3 , which includes a feature extraction network (backbone network), a feature pyramid network (FPN, Feature Pyramid Networks), a ROI Align operation and a preset number of convolution operations (not shown in the figure). The above-mentioned first instance segmentation feature map is a feature obtained after the Anchor Based instance segmentation method performs ROI Align operation on the processed image.

[0051] The above-mentioned Anchor Based instance segmentation method may specifically be a Mask RCNN instance segmentation method. The embodiment of the present invention does not specifically limit the above-mentioned Anchor Based instance segmentation method.

[0052] Step S106, performing a guided filtering operation on the first instance segmentation feature map based on the shallow feature map of the target object to obtain a second instance segmentation feature map, wherein the second instance segmentation feature map carries edge information of the target object;

[0053] After obtaining the shallow feature map of the target object and the first instance segmentation feature map, a guided filtering operation is performed on the first instance segmentation feature map based on the shallow feature map of the target object, so that the edge detail features on the obtained second instance segmentation feature map are retained and some noise features are smoothed.

[0054] Step S108: determining a mask of the target object in the image to be processed based on the first instance segmentation feature map and the second instance segmentation feature map.

[0055] The process is described in detail below and will not be repeated here.

[0056] In an embodiment of the present invention, first, an image to be processed is obtained, and shallow feature extraction is performed on the target object in the image to be processed to obtain a shallow feature map of the target object, wherein the shallow feature map of the target object contains the edge information of the target object; then, a first instance segmentation feature map is obtained when the Anchor Based instance segmentation method performs instance segmentation on the target object in the image to be processed; then, a guided filtering operation is performed on the first instance segmentation feature map based on the shallow feature map of the target object to obtain a second instance segmentation feature map, wherein the second instance segmentation feature map carries the edge information of the target object; finally, a mask of the target object in the image to be processed is determined based on the first instance segmentation feature map and the second instance segmentation feature map. It can be seen from the above description that since the second instance segmentation feature map carries the edge information of the target object, the edge effect of the mask of the target object in the image to be processed determined based on the first instance segmentation feature map and the second instance segmentation feature map is better, which improves the performance and accuracy of instance segmentation. In addition, the image processing method of the present invention does not need to use additional supervision, only shallow feature extraction and guided filtering operations are required, the amount of calculation is small, and the image processing efficiency is high, which alleviates the technical problems of large amount of calculation and poor segmentation effect when the existing instance segmentation method performs instance segmentation on the image.

[0057] The above content briefly introduces the image processing method of the present invention. The specific contents involved therein are described in detail below.

[0058] In an alternative embodiment of the present invention, reference Figure 4 Step S102, the process of extracting shallow features of the target object in the image to be processed includes the following steps:

[0059] Step S401, performing multi-scale scaling processing on the image to be processed to obtain a multi-scale image to be processed;

[0060] Specifically, when performing multi-scale scaling on the image to be processed X, the downsampling multiples of the feature extraction network (backbone network) in the Anchor Based instance segmentation method (such as Figure 3 As shown in the figure, the downsampling multiples are 4, 8, 16, and 32 respectively) to perform multi-scale scaling processing on the image to be processed to obtain a multi-scale image to be processed (such as Figure 3 As shown, the images to be processed are 1 / 4, 1 / 8, 1 / 16 and 1 / 32), thus forming an image pyramid, which is represented by {X 1 / 4 ,X 1 / 8 ,X 1 / 16 ,X 1 / 32}.

[0061] Step S402, determining an image to be processed of a target scale from among the images to be processed of multiple scales according to the target frame coordinates of the target object predicted in the Anchor Based instance segmentation method;

[0062] Specifically, in the Anchor Based instance segmentation method, only one target box can be predicted at a time, that is, only the target box coordinates of one target object can be predicted. After obtaining the target box coordinates of the target object, the image to be processed of the target scale can be determined in the multi-scale image to be processed according to the target box coordinates.

[0063] Below Figure 5 The image to be processed shown in FIG. 1 is taken as an example to introduce the image processing method of the present invention. Figure 5 In the image to be processed shown, if the big dog is the target object, the Anchor Based instance segmentation method predicts the target frame coordinates of the big dog. According to the results, the scale corresponding to the big dog is 1 / 32, so the image to be processed with a scale of 1 / 32 is used as the image to be processed with the target scale in the multi-scale image to be processed; if the puppy is the target object, the Anchor Based instance segmentation method predicts the target frame coordinates of the puppy. According to the results, the scale corresponding to the puppy is 1 / 4, so the image to be processed with a scale of 1 / 4 is used as the image to be processed with the target scale in the multi-scale image to be processed.

[0064] Step S403: input the to-be-processed image of the target scale into a shallow feature extraction network to obtain a shallow feature map of the target object, wherein the shallow feature extraction network includes two first target convolutional layers connected in series.

[0065] In the embodiment of the present invention, the shallow feature extraction network is actually two 1x1 convolutional layers connected in series and sharing weights. As described in step S402, if the image to be processed at a 1 / 4 scale is input to the shallow feature extraction network, the shallow feature map of the target object obtained is represented by DGF 1 / 4 ; If the 1 / 32 scale image to be processed is input into the shallow feature extraction network, the shallow feature map of the target object is represented by DGF 1 / 32 . The convolution channels of the above two series-connected 1x1 convolution layers are (3, 64) and (64, Num_Classes), respectively. In (3, 64), 3 represents the input convolution channel of the first 1x1 convolution layer, and 64 represents the output convolution channel of the first 1x1 convolution layer. In (64, Num_Classes), 64 represents the input convolution channel of the second 1x1 convolution layer, and Num_Classes represents the output convolution channel of the second 1x1 convolution layer. In addition, Num_Classes represents the category of the instance, and its size can be adjusted according to different data sets. For example, Num_Classes of the COCO data set is 80, while Num_Classes of the Citycapes data set is 19.

[0066] The above-mentioned shared weights refer to that the images to be processed of different target scales share a set of parameters. As mentioned above, the images to be processed of 1 / 4 scale and the images to be processed of 1 / 32 scale share the same set of parameters.

[0067] and Figure 5 The shallow feature map of the target object corresponding to the medium-sized dog is as follows Figure 6 As shown, it can be seen that the edge information of the big dog is outlined.

[0068] The above content introduces the process of shallow feature extraction in detail. The following content describes the process of guided filtering operation in detail.

[0069] In an alternative embodiment of the present invention, reference Figure 7 , step S106, the process of performing a guided filtering operation on the first instance segmentation feature map based on the shallow feature map of the target object comprises the following steps:

[0070] Step S701, performing ROI Align operation on the shallow feature map of the target object according to the target frame coordinates to obtain the shallow feature map of the target object at the target scale;

[0071] Specifically, if the target object is Figure 5 The big dog shown in FIG. 1 is subjected to the shallow feature map (as described in step S402 above, which is a shallow feature map DGF at a scale of 1 / 32) 1 / 32) performs ROI Align operation, and the shape of the shallow feature map of the target object of the target scale is (256, 28, 28), where 256 represents the number of channels and 28*28 represents the size of the shallow feature map. At this time, the shallow feature map of the target object of the target scale is expressed as I=DGFFt 1 / 32 ; If the target object is Figure 5 The dog shown in FIG. 1 has a shallow feature map (as described in step S402 above, which is a shallow feature map DGF at a scale of 1 / 4) 1 / 4 ) performs ROI Align operation, and the shape of the shallow feature map of the target object of the target scale is (256, 28, 28), where 256 represents the number of channels and 28*28 represents the size of the shallow feature map. At this time, the shallow feature map of the target object of the target scale is expressed as I=DGFFt 1 / 4 .

[0072] Step S702, upsampling the first instance segmentation feature map to obtain an upsampled first instance segmentation feature map;

[0073] As mentioned above, the first instance segmentation feature map is the feature obtained by the Anchor Based instance segmentation method after performing ROI Align operation on the processed image. Figure 5 The first instance segmentation feature map corresponding to the medium-large dog is Figure 8 (It can be seen that there are noise features besides the target object of the big dog, and the edge information of the big dog is not prominent enough), denoted as Ft 1 / 32 , the shape is (256, 14, 14), where 256 represents the number of channels and 14*14 represents the size of the first instance segmentation feature map; the corresponding Figure 5 The first instance segmentation feature map corresponding to the puppy is denoted as Ft 1 / 4 , the shape is also (256, 14, 14).

[0074] The first instance segmentation feature map is upsampled, and the shape after upsampling becomes (256, 28, 28), and the corresponding upsampled first instance segmentation feature maps are I p =UFt 1 / 32 ,I p =UFt 1 / 4 .

[0075] Step S703 , performing a guided filtering operation on the upsampled first instance segmentation feature map based on the shallow feature map of the target object at the target scale to obtain a second instance segmentation feature map.

[0076] refer to Fig. 9 , specifically including the following steps:

[0077] Step S901, performing mean filtering operation on the shallow feature map of the target object at the target scale and the upsampled first instance segmentation feature map, respectively, to obtain a mean filtering operation result;

[0078] Specifically, for the shallow feature map I (such as DGFFt 1 / 32 ) to perform mean filtering operation M I =F m (I), and segment the feature map I of the first upsampled instance p (such as UFt 1 / 32 ) to perform mean filtering operation M P =F m (I p ), where F m Represents the mean filter operation.

[0079] Step S902, calculating a first correlation coefficient matrix between a shallow feature map of a target object at a target scale and a shallow feature map of a target object at a target scale, and calculating a second correlation coefficient matrix between a shallow feature map of a target object at a target scale and an upsampled first instance segmentation feature map;

[0080] Specifically, the shallow feature map I (such as DGFFt 1 / 32 ) and the shallow feature map I (such as DGFFt 1 / 32 )’s first correlation coefficient matrix C I =F m (I*I), where C I Represents the first correlation coefficient matrix and calculates the shallow feature map I (such as DGFFt 1 / 32 ) and the upsampled first instance segmentation feature map I p (such as UFt 1 / 32 )’s second correlation coefficient matrix C P =F m (I*I P ), where C P represents the second correlation coefficient matrix.

[0081] Step S903, calculating a first variance between a shallow feature map of the target object at the target scale and a shallow feature map of the target object at the target scale according to the mean filtering operation result and the first correlation coefficient matrix, and calculating a second variance between the shallow feature map of the target object at the target scale and an upsampled first instance segmentation feature map according to the mean filtering operation result and the second correlation coefficient matrix;

[0082] Specifically, the first variance V between the shallow feature map of the target object at the target scale and the shallow feature map of the target object at the target scale is calculated according to the mean filtering operation result and the first correlation coefficient matrix.I =C I -M I *M I , where V I represents the first variance, C I represents the first correlation coefficient matrix, M I Represents the shallow feature map I (such as DGFFt 1 / 32 ) performs a mean filtering operation, and obtains a result, and calculates a second variance V between a shallow feature map of the target object at the target scale and the upsampled first instance segmentation feature map according to the mean filtering operation result and the second correlation coefficient matrix P =C p -M I *M P , where V P represents the second variance, C p represents the second correlation coefficient matrix, M I Represents the shallow feature map I (such as DGFFt 1 / 32 ) to perform mean filtering operation, and the result obtained is M P Represents the first instance segmentation feature map I of the upsampled p (such as UFt 1 / 32 ) performs mean filtering operation to obtain the result.

[0083] Step S904, calculating a first coefficient matrix according to the first variance and the second variance, and calculating a second coefficient matrix according to the first coefficient matrix and a mean filter operation result;

[0084] Specifically, the first coefficient matrix M is calculated according to the first variance and the second variance. a =V P / (V I +ε), where M a represents the first coefficient matrix, V P represents the second variance, V I Represents the first variance, and calculates the second coefficient matrix M based on the first coefficient matrix and the mean filter operation result b =M P -M a *M I , where M b represents the second coefficient matrix, M P Represents the first instance segmentation feature map I of the upsampled p (such as UFt 1 / 32 ) to perform mean filtering operation, and the result obtained is M a represents the first coefficient matrix, M I Represents the shallow feature map I (such as DGFFt 1 / 32) performs mean filtering operation to obtain the result.

[0085] Step S905 , linearly combine the first coefficient matrix, the second coefficient matrix, and the shallow feature map of the target object at the target scale to obtain a second instance segmentation feature map.

[0086] Specifically, the linear combination method is Q = M a *I+M b , where Q represents the second instance segmentation feature map, M a represents the first coefficient matrix, I represents the shallow feature map of the target object at the target scale (such as DGFFt 1 / 32 ), M b represents the second coefficient matrix, * represents the product operation of the corresponding elements in the matrix, and the second instance segmentation feature map corresponding to the big dog is represented by QFt 1 / 32 The second instance segmentation feature map corresponding to the puppy is represented by QFt 1 / 4 .

[0087] based on Figure 6 The shallow feature map of the target object is Figure 7 After the guided filtering operation is performed on the first instance segmentation feature map, the second instance segmentation feature map obtained is as follows Fig.10 As shown, it can be seen that the edge detail features of the big dog are retained and the noise features are smoothly erased.

[0088] The above content has provided a detailed introduction to the guided filtering operation process of the present invention. The following is a detailed introduction to the process of determining the mask of the target object in the image to be processed.

[0089] In an alternative embodiment of the present invention, reference Fig.11 , step S108, the process of determining the mask of the target object in the image to be processed based on the first instance segmentation feature map and the second instance segmentation feature map comprises the following steps:

[0090] Step S111, inputting the second instance segmentation feature map into the Laplacian convolutional network to obtain an edge feature map of the target object;

[0091] The convolution kernel parameter of the above Laplace convolutional network is K 3x3 =[[-1,-1,-1],[-1,-8,-8],[-1,-1,-1]], the second instance segmentation feature map QFt 1 / 32 The corresponding edge feature map of the target object is represented as LFt 1 / 32 , the second instance segmentation feature map QFt 1 / 4 The corresponding edge feature map of the target object is represented as LFt 1 / 4 The Laplacian convolutional network can further enhance the saliency of edge information.

[0092] Step S112, inputting the first instance segmentation feature map into a preset number of second target convolutional layers to obtain a third instance segmentation feature map;

[0093] The preset number of second target convolutional layers may be four 3x3 convolutional layers.

[0094] Step S113, performing an addition operation on the edge feature map of the target object and the third instance segmentation feature map;

[0095] Step S114, inputting the result obtained by the addition operation into the third target convolution layer to obtain a mask of the target object in the image to be processed.

[0096] Specifically, the third target convolution layer may be a 1x1 convolution layer. Fig.12 As shown, Figure 5 The corresponding mask of the big dog.

[0097] Combine the following Figure 3 The image processing method of the present invention is generally introduced. Figure 3 , obtain a first instance segmentation feature map (the first instance segmentation feature map is a feature obtained after the Anchor Based instance segmentation method performs ROIAlign operation on the image to be processed), and obtain the image to be processed, then perform shallow feature extraction on the target object in the image to be processed to obtain a shallow feature map of the target object, and then perform ROI Align operation on the shallow feature map of the target object according to the target frame coordinates of the target object to obtain a shallow feature map of the target object at the target scale, and then upsample the first instance segmentation feature map to obtain an upsampled first instance segmentation feature map, and then perform a guided filtering operation on the upsampled first instance segmentation feature map based on the shallow feature map of the target object at the target scale to obtain a second instance segmentation feature map, and then input the second instance segmentation feature map into a Laplacian convolutional network to obtain an edge feature map of the target object, and input the first instance segmentation feature map into four 3x3 convolutional layers to obtain a third instance segmentation feature map, and then perform an addition operation on the edge feature map of the target object and the third instance segmentation feature map, and input the result of the addition operation into a 1x1 convolutional layer to obtain a mask of the target object in the image to be processed.

[0098] In the instance segmentation task, the present invention has a significant improvement over the known public algorithms under the same Backbone settings. Compared with other instance segmentation models, the shallow feature extraction network, guided filtering operation and Laplacian network of the present invention do not require additional supervision, can be plug-and-play, and can be expanded to other modules such as feature pyramid FPN. A comparative experiment was conducted on the COCO dataset. The method of the present invention has an improvement in mAP compared to the best previous method, and also an improvement over the BMask RCNN model with edge supervision, an improvement of 1.5% over Mask RCC, and an improvement of 0.1% over BMask RCNN at the same speed. The specific results are as follows:

[0099]

[0100] It can be seen from the above experimental results that the present invention can achieve the best segmentation effect on the instance segmentation task under the same settings.

[0101] The image processing method of the present invention has the following characteristics:

[0102] 1) Taking full advantage of the rich edge characteristics of shallow features, a learnable guided filtering module is designed to perform guided filtering on the features after ROIAlign, eliminating the noise in the non-target object area on the first instance segmentation feature map;

[0103] 2) The Laplace convolutional network is used to strengthen the edge, and the edge feature significance is enhanced without introducing edge supervision loss;

[0104] 3) The enhanced edge features are fused with the features of the original FCN branch to strengthen the edges, ultimately improving the instance segmentation performance;

[0105] 4) The image pyramid generation method is adopted in the shallow feature extraction process, and the weight of the shallow feature extraction network is shared. Therefore, a large amount of calculation will not be introduced due to excessive resolution.

[0106] Embodiment 3:

[0107] The embodiment of the present invention further provides an image processing device, which is mainly used to execute the image processing method provided by the above content of the embodiment of the present invention. The image processing device provided by the embodiment of the present invention is specifically introduced below.

[0108] Fig.13 is a schematic diagram of an image processing device according to an embodiment of the present invention. Fig.13 As shown, the image processing device mainly includes: a shallow feature extraction unit 10, an acquisition unit 20, a guided filtering unit 30 and a determination unit 40, wherein:

[0109] A shallow feature extraction unit is used to obtain an image to be processed, and perform shallow feature extraction on a target object in the image to be processed to obtain a shallow feature map of the target object, wherein the shallow feature map of the target object contains edge information of the target object;

[0110] An acquisition unit, used for acquiring a first instance segmentation feature map obtained when the Anchor Based instance segmentation method performs instance segmentation on the target object in the image to be processed;

[0111] A guided filtering unit is used to perform a guided filtering operation on the first instance segmentation feature map based on the shallow feature map of the target object to obtain a second instance segmentation feature map, wherein the second instance segmentation feature map carries edge information of the target object;

[0112] The determining unit is used to determine the mask of the target object in the image to be processed based on the first instance segmentation feature map and the second instance segmentation feature map.

[0113] In an embodiment of the present invention, first, an image to be processed is obtained, and shallow feature extraction is performed on the target object in the image to be processed to obtain a shallow feature map of the target object, wherein the shallow feature map of the target object contains the edge information of the target object; then, a first instance segmentation feature map is obtained when the Anchor Based instance segmentation method performs instance segmentation on the target object in the image to be processed; then, a guided filtering operation is performed on the first instance segmentation feature map based on the shallow feature map of the target object to obtain a second instance segmentation feature map, wherein the second instance segmentation feature map carries the edge information of the target object; finally, a mask of the target object in the image to be processed is determined based on the first instance segmentation feature map and the second instance segmentation feature map. It can be seen from the above description that since the second instance segmentation feature map carries the edge information of the target object, the edge effect of the mask of the target object in the image to be processed determined based on the first instance segmentation feature map and the second instance segmentation feature map is better, which improves the performance and accuracy of instance segmentation. In addition, the image processing method of the present invention does not need to use additional supervision, only shallow feature extraction and guided filtering operations are required, the amount of calculation is small, and the image processing efficiency is high, which alleviates the technical problems of large amount of calculation and poor segmentation effect when the existing instance segmentation method performs instance segmentation on the image.

[0114] Optionally, the shallow feature extraction unit is also used to: perform multi-scale scaling processing on the image to be processed to obtain a multi-scale image to be processed; determine the image to be processed of the target scale in the multi-scale image to be processed according to the target frame coordinates of the target object predicted in the Anchor Based instance segmentation method; input the image to be processed of the target scale into the shallow feature extraction network to obtain a shallow feature map of the target object, wherein the shallow feature extraction network includes two first target convolution layers connected in series.

[0115] Optionally, the guided filtering unit is further used to: perform ROIAlign operation on the shallow feature map of the target object according to the target frame coordinates to obtain a shallow feature map of the target object at the target scale; upsample the first instance segmentation feature map to obtain an upsampled first instance segmentation feature map; perform a guided filtering operation on the upsampled first instance segmentation feature map based on the shallow feature map of the target object at the target scale to obtain a second instance segmentation feature map.

[0116] Optionally, the guided filtering unit is also used to: perform mean filtering operations on the shallow feature map of the target object at the target scale and the upsampled first instance segmentation feature map, respectively, to obtain a mean filtering operation result; calculate a first correlation coefficient matrix between the shallow feature map of the target object at the target scale and the shallow feature map of the target object at the target scale, and calculate a second correlation coefficient matrix between the shallow feature map of the target object at the target scale and the upsampled first instance segmentation feature map; determine the second instance segmentation feature map based on the mean filtering operation result, the first correlation coefficient matrix and the second correlation coefficient matrix.

[0117] Optionally, the guided filtering unit is also used to: calculate a first variance between a shallow feature map of a target object at a target scale and a shallow feature map of a target object at a target scale according to a mean filtering operation result and a first correlation coefficient matrix, and calculate a second variance between a shallow feature map of a target object at a target scale and an upsampled first instance segmentation feature map according to a mean filtering operation result and a second correlation coefficient matrix; calculate a first coefficient matrix according to the first variance and the second variance, and calculate a second coefficient matrix according to the first coefficient matrix and the mean filtering operation result; linearly combine the first coefficient matrix, the second coefficient matrix, and the shallow feature map of the target object at the target scale to obtain a second instance segmentation feature map.

[0118] Optionally, the determination unit is also used to: input the second instance segmentation feature map into the Laplacian convolutional network to obtain an edge feature map of the target object; input the first instance segmentation feature map into a preset number of second target convolutional layers to obtain a third instance segmentation feature map; add the edge feature map of the target object to the third instance segmentation feature map, and determine the mask of the target object in the image to be processed based on the result of the addition operation.

[0119] Optionally, the determination unit is further used to: input a result obtained by the addition operation into a third target convolution layer to obtain a mask of the target object in the image to be processed.

[0120] Optionally, the first instance segmentation feature map is a feature obtained after performing a ROI Align operation on the image to be processed using an Anchor Based instance segmentation method.

[0121] The image processing device provided in the embodiment of the present invention has the same implementation principle and technical effects as the method embodiment in the aforementioned embodiment 2. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0122] In another embodiment, a computer-readable medium having a non-volatile program code executable by a processor is provided, wherein the program code enables the processor to execute the steps of the method described in any embodiment of the second embodiment.

[0123] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0124] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0126] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0129] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0130] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be processed, and perform shallow feature extraction on a target object in the image to be processed to obtain a shallow feature map of the target object, wherein the shallow feature map of the target object contains edge information of the target object; Obtaining a first instance segmentation feature map obtained when performing instance segmentation on a target object in the image to be processed using an Anchor Based instance segmentation method; Performing a guided filtering operation on the first instance segmentation feature map based on the shallow feature map of the target object to obtain a second instance segmentation feature map, wherein the second instance segmentation feature map carries edge information of the target object; Determine a mask of a target object in the image to be processed based on the first instance segmentation feature map and the second instance segmentation feature map; The step of extracting shallow features of the target object in the image to be processed includes: Performing multi-scale scaling processing on the image to be processed to obtain a multi-scale image to be processed; Determine an image to be processed of a target scale in the multi-scale images to be processed according to the target frame coordinates of the target object predicted in the Anchor Based instance segmentation method; Inputting the to-be-processed image of the target scale into a shallow feature extraction network to obtain a shallow feature map of the target object, wherein the shallow feature extraction network includes two first target convolutional layers connected in series; The step of performing a guided filtering operation on the first instance segmentation feature map based on the shallow feature map of the target object includes: Performing ROI Align operation on the shallow feature map of the target object according to the target frame coordinates to obtain a shallow feature map of the target object at a target scale; Upsampling the first instance segmentation feature map to obtain an upsampled first instance segmentation feature map; Performing a guided filtering operation on the upsampled first instance segmentation feature map based on a shallow feature map of the target object at the target scale to obtain the second instance segmentation feature map; Wherein, determining the mask of the target object in the image to be processed based on the first instance segmentation feature map and the second instance segmentation feature map comprises: Inputting the second instance segmentation feature map into a Laplacian convolutional network to obtain an edge feature map of the target object; Inputting the first instance segmentation feature map into a preset number of second target convolutional layers to obtain a third instance segmentation feature map; An addition operation is performed on the edge feature map of the target object and the third instance segmentation feature map, and a mask of the target object in the image to be processed is determined according to a result of the addition operation.

2. The method according to claim 1, characterized in that Performing a guided filtering operation on the upsampled first instance segmentation feature map based on the shallow feature map of the target object at the target scale includes: Performing a mean filtering operation on the shallow feature map of the target object at the target scale and the upsampled first instance segmentation feature map respectively to obtain a mean filtering operation result; Calculating a first correlation coefficient matrix between a shallow feature map of the target object at the target scale and a shallow feature map of the target object at the target scale, and calculating a second correlation coefficient matrix between the shallow feature map of the target object at the target scale and the upsampled first instance segmentation feature map; The second instance segmentation feature map is determined based on the mean filtering operation result, the first correlation coefficient matrix and the second correlation coefficient matrix.

3. The method according to claim 2, characterized in that Determining the second instance segmentation feature map based on the mean filtering operation result, the first correlation coefficient matrix, and the second correlation coefficient matrix includes: Calculating a first variance between a shallow feature map of the target object at the target scale and a shallow feature map of the target object at the target scale according to the mean filtering operation result and the first correlation coefficient matrix, and calculating a second variance between the shallow feature map of the target object at the target scale and the upsampled first instance segmentation feature map according to the mean filtering operation result and the second correlation coefficient matrix; Calculating a first coefficient matrix according to the first variance and the second variance, and calculating a second coefficient matrix according to the first coefficient matrix and the mean filtering operation result; The first coefficient matrix, the second coefficient matrix, and a shallow feature map of the target object at the target scale are linearly combined to obtain the second instance segmentation feature map.

4. The method according to claim 1, characterized in that Determining the mask of the target object in the image to be processed according to the result obtained by the addition operation includes: The result obtained by the addition operation is input into the third target convolution layer to obtain a mask of the target object in the image to be processed.

5. The method according to claim 1, characterized in that The first instance segmentation feature map is a feature obtained after the AnchorBased instance segmentation method performs a ROIAlign operation on the image to be processed.

6. An image processing device, implementing the steps of the method according to any one of claims 1 to 5, characterized in that: include: A shallow feature extraction unit is used to obtain an image to be processed, and perform shallow feature extraction on a target object in the image to be processed to obtain a shallow feature map of the target object, wherein the shallow feature map of the target object contains edge information of the target object; An acquisition unit, used for acquiring a first instance segmentation feature map obtained when an Anchor Based instance segmentation method is used to perform instance segmentation on a target object in the image to be processed; A guided filtering unit, configured to perform a guided filtering operation on the first instance segmentation feature map based on the shallow feature map of the target object to obtain a second instance segmentation feature map, wherein the second instance segmentation feature map carries edge information of the target object; A determining unit is used to determine a mask of a target object in the image to be processed based on the first instance segmentation feature map and the second instance segmentation feature map.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the steps of the method according to any one of claims 1 to 5.

Images