Image segmentation method, electronic device, and computer-readable storage medium

Abstract

The present application discloses an image segmentation method, an electronic device, and a computer-readable storage medium, which relate to the field of image processing. The image segmentation method comprises: obtaining an original image to be segmented, wherein the original image comprises an RGB image and a depth image; extracting features from the original image to obtain RGB image features and depth image features; fusing the RGB image features and the depth image features to obtain final fused features; inputting the final fused features into a preset image segmentation network for segmentation processing to obtain a target mask. According to the image segmentation method of the present invention, it is possible to make full use of the depth information of the scene, increase the accuracy of segmentation, and improve the segmentation effect.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image segmentation method, an electronic device, and a computer-readable storage medium. Background Art

[0002] The so-called target segmentation, assuming that an RGB image is input, can output the mask of the target object through certain technologies and means. This is a long-standing task in the field of computer vision. Most existing target segmentation methods use RGB images as input, extract the color and texture information of the objects in the image as features through deep convolutional neural networks, and operate on them to achieve the purpose of segmentation. However, for objects with less color and texture information such as industrial parts, it is difficult to achieve a good segmentation effect using only the features of RGB images; at the same time, in industrial scenarios, parts are stacked and occluded severely, and there are many instances of the same parts, which has a greater impact on segmentation tasks that require higher precision. Summary of the invention

[0003] The purpose of the present application is to solve at least one of the technical problems existing in the prior art and to provide an image segmentation method that can fully utilize the depth information of the scene and increase the accuracy of segmentation.

[0004] According to the first aspect of the present application, the image segmentation method includes:

[0005] Acquire an original image to be segmented, wherein the original image includes an RGB image and a depth image;

[0006] Extracting features from the original image to obtain RGB image features and depth image features;

[0007] Perform feature fusion on the RGB image features and the depth image features to obtain a final fusion feature;

[0008] The final fusion feature is input into a preset image segmentation network for segmentation processing to obtain a target mask.

[0009] According to the image segmentation method of the embodiment of the present application, at least the following technical effects are achieved: when the RGB image and the processed depth image are input into the deep neural network, a large number of data features can be obtained. Through the feature fusion method based on the attention mechanism, high-value information can be quickly screened out from these features and other useless information can be suppressed; the depth information of the image is fully utilized to make up for the deficiency of the deep neural network effect when the color and texture information of the object is not strong in special scenes, the depth information is introduced into the image features, the feature information is increased, and the accuracy of segmentation is improved.

[0010] According to some embodiments of the present application, extracting features from the original image to obtain RGB image features and depth image features includes:

[0011] Acquire the depth image and the RGB image;

[0012] Converting the depth image into a three-channel image;

[0013] Feature extraction is performed on the RGB image to obtain the RGB image feature, and extraction is performed on the three-channel image to obtain the depth image feature.

[0014] According to some embodiments of the present application, converting the depth image into a three-channel image includes:

[0015] The depth of the depth image is converted into three-dimensional coordinates in a world coordinate system according to the camera intrinsic parameter matrix, the rotation matrix and the translation matrix;

[0016] Performing min-max normalization processing on the three-dimensional coordinates;

[0017] The three-channel image is obtained according to the processed three-dimensional coordinate values.

[0018] According to some embodiments of the present application, obtaining the three-channel image according to the processed three-dimensional coordinate values ​​includes:

[0019] Obtaining original pixel positions of the depth image;

[0020] The three-dimensional coordinate values ​​are used as the pixel values ​​of the three channels R, G, and B;

[0021] Fill the original pixel position with the three-dimensional coordinate value to obtain the three-channel image.

[0022] According to some embodiments of the present application, extracting features from the RGB image to obtain RGB image features, and extracting depth image features from the three-channel image includes:

[0023] Inputting the RGB image into the image segmentation network to obtain RGB image features;

[0024] The three-channel image is input into the image segmentation network to obtain deep image features.

[0025] According to some embodiments of the present application, the fusing the RGB image features and the depth image features to obtain the final fused features includes:

[0026] Multiplying the RGB image feature and the depth image feature according to matrix multiplication to obtain a first fusion feature;

[0027] Normalizing the first fusion feature to obtain an attention mechanism coefficient;

[0028] According to the attention mechanism coefficient, the final fusion feature is obtained.

[0029] According to some embodiments of the present application, obtaining the final fusion feature according to the attention mechanism coefficient includes:

[0030] Multiplying the attention mechanism coefficient with the corresponding RGB image feature and the corresponding depth image feature to obtain the corresponding fusion feature value;

[0031] The fused feature values ​​are integrated to obtain the final fused feature.

[0032] According to some embodiments of the present application, the fusion feature value is calculated by the following formula:

[0033] h ijk (y,z)=γ ijk [f ijk (y)+ωg ijk (z)]

[0034] Among them, h ijk (y,z) is the eigenvalue at position (i,j) in the kth channel after fusion, γ ijk is the attention mechanism coefficient of the feature value at position (i, j) in the kth channel, f ijk (y) is the feature value of the RGB image feature at the position (i, j) in the kth channel, g ijk (z) is the eigenvalue of the depth image feature at the position (i, j) in the kth channel, and ω is the balancing weight coefficient.

[0035] An electronic device according to an embodiment of the second aspect of the present application includes:

[0036] at least one processor, and

[0037] a memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions, and the instructions are executed by the at least one processor, so that the at least one processor implements the image segmentation method described in the first aspect when executing the instructions.

[0039] According to the computer-readable storage medium of the embodiment of the third aspect of the present application, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the image segmentation method described in the first aspect above.

[0040] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present application is further described below with reference to the accompanying drawings and embodiments;

[0042] Figure 1 A schematic diagram of the process of an image segmentation method according to an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a process for obtaining RGB image features and depth image features according to an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a process of converting a depth image into a three-channel image according to an embodiment of the present application;

[0045] Figure 4 A schematic diagram of a process for obtaining the final fusion features in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of a process for obtaining the final fusion feature according to the attention mechanism coefficient in an embodiment of the present application;

[0047] Figure 6 It is a schematic diagram of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0048] This section will describe in detail the specific embodiments of the present application. The preferred embodiments of the present application are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present application, but it cannot be understood as a limitation on the scope of protection of the present application.

[0049] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0050] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0051] The image segmentation method according to an embodiment of the present application is described below with reference to the accompanying drawings.

[0052] like Figure 1 As shown, the image segmentation method according to the embodiment of the present application includes:

[0053] S100: Acquire an original image to be segmented, where the original image includes an RGB image and a depth image;

[0054] S200: extracting features from the original image to obtain RGB image features and depth image features;

[0055] S300: Fusing the RGB image features and the depth image features to obtain the final fusion features;

[0056] S400: Input the final fusion feature into a preset image segmentation network for segmentation processing to obtain a target mask.

[0057] Among them, in this embodiment, the image segmentation network includes a backbone network and a head network. The depth image is also called a distance image, which refers to an image that uses the distance from the image collector to each point in the scene, that is, the depth, as the pixel value, which directly reflects the geometric shape of the visible surface of the scene.

[0058] Specifically, first determine the original image that needs to be segmented; obtain the RGB image and depth image of the original image, and preprocess the depth image; input the RGB image and the preprocessed depth image into the backbone network to obtain the corresponding RGB image features and the corresponding depth image features; fuse the RBG image features and the depth image features for the first time and normalize them to obtain the attention mechanism coefficient; fuse the attention mechanism coefficient with the corresponding RGB image features and the corresponding depth image features for the second time to obtain the final fused features; input the final fused features into the head network to obtain the mask of the original image; extract the region of interest in the original image through the mask, and segment the target area in the original image.

[0059] Depth images can be calculated as point cloud data after coordinate transformation, and point cloud data with rules and necessary information can also be inversely calculated as depth image data. Through depth images, the spatial relationship between stacked objects can be obtained more accurately, and spatial information that does not exist in RGB images can be provided to achieve better segmentation effects. In digital image processing, masks are mainly used for: extracting regions of interest, multiplying the pre-made region of interest mask with the image to be processed to obtain the region of interest image, the image value in the region of interest remains unchanged, and the image value outside the region is 0; shielding, using masks to shield certain areas on the image so that they do not participate in processing or do not participate in the calculation of processing parameters, or only processing or statistics on the shielded area; structural feature extraction, using similarity variables or image matching methods to detect and extract structural features in the image that are similar to the mask; production of special shape images.

[0060] According to the image segmentation method of the embodiment of the present application, when the RGB image and the processed depth image are input into the deep neural network, a large number of data features can be obtained. Through the feature fusion method based on the attention mechanism, high-value information can be quickly screened out from these features and other useless information can be suppressed. The depth information of the image is fully utilized to make up for the shortcomings of the deep neural network effect when the color and texture information of the object in special scenes is not strong, and the depth information is introduced into the image features to increase the feature information and improve the accuracy of segmentation.

[0061] like Figure 2 As shown, in some specific embodiments of the present application, feature extraction is performed on the original image to obtain RGB image features and depth image features including:

[0062] S210: Acquire a depth image and an RGB image;

[0063] S220: Convert the depth image into a three-channel image;

[0064] S230: Extract features from the RGB image to obtain RGB image features, and extract features from the three-channel image to obtain depth image features.

[0065] Specifically, the depth image of the original image is obtained, and the single-channel depth image is converted into a three-channel image similar to the RGB image, so that the converted depth image can be preprocessed and data increased in the same way as the RGB image before being input into the neural network, without the need to perform a separate data increase operation with a larger amount of computation. The depth image can more accurately obtain the spatial relationship between stacked objects, and can provide spatial information that does not exist in the RGB image, achieving better segmentation accuracy.

[0066] like Figure 3 As shown, in some specific embodiments of the present application, converting the depth image into a three-channel image includes:

[0067] S221: Convert the depth of the depth image into three-dimensional coordinates in a world coordinate system according to the camera intrinsic parameter matrix, the rotation matrix and the translation matrix;

[0068] S222: performing min-max standardization processing on the three-dimensional coordinates;

[0069] S223: Obtain a three-channel image according to the processed three-dimensional coordinate values.

[0070] Specifically, firstly according to the camera intrinsic parameter matrix M 3×3 , rotation matrix R 3×3 and the translation matrix T 3×1 , the depth image is located at [u,v] T The depth d of the position is converted into the three-dimensional coordinates [x w ,y w ,z w ] T :

[0071]

[0072] Among them, u and v are the position coordinates of the depth image, and d is the position [u,v] in the depth image. T The depth of the position, M is the intrinsic parameter matrix, R is the rotation matrix, and T is the translation matrix.

[0073] The intrinsic parameter matrix M is determined by the camera structure and can be obtained through camera calibration. The rotation matrix R and translation matrix T are the position of the camera relative to the origin of the world coordinate system. When the camera coordinate system coincides with the world coordinate system, the corresponding rotation matrix R is a unit matrix and the translation matrix T is [0,0,0] T ; The internal parameter matrix M at this time is:

[0074]

[0075] Where f is the focal length, that is, the distance from the image plane to the origin of the camera coordinate system; d x ,d y is the physical size of each pixel in the x and y directions of the image plane; u0 and v0 are the coordinates of the origin of the image coordinate system in the pixel coordinate system.

[0076] Combining the above two equations, we can get the depth image point [u,v] T Convert to world coordinate point [x w ,y w ,z w ] T The transformation formula is:

[0077]

[0078] Then, according to the depth image, the x of all three-dimensional coordinates is obtained according to the transformation formula. w ,y w 、z w The coordinates are normalized to the min-max range, and all coordinate values ​​are mapped to the range [0-255], as shown in the following formula:

[0079] In some specific embodiments of the present application, obtaining a three-channel image according to the processed three-dimensional coordinate values ​​includes:

[0080] Get the original pixel position of the depth image;

[0081] The three-dimensional coordinate values ​​are used as the pixel values ​​of the three channels R, G, and B;

[0082] Fill the three-dimensional coordinate values ​​into the original pixel positions to obtain a three-channel image.

[0083] The depth image processed by this method can retain the spatial information of the original image to the greatest extent, and at the same time can be subjected to the same preprocessing and data enhancement as the RGB image before being input into the neural network, thereby reducing the amount of computational effort of the data enhancement operation and improving the efficiency of the algorithm.

[0084] In some specific embodiments of the present application, extracting features from an RGB image to obtain RGB image features and extracting depth image features from a three-channel image include:

[0085] Input the RGB image into the image segmentation network to obtain the RGB image features;

[0086] The three-channel image is input into the image segmentation network to obtain deep image features.

[0087] The image segmentation network includes a backbone network and a head network. The backbone network is a VoVNet network, which is a real-time and efficient target detection backbone network. Specifically, the RGB image and the three-channel image are input into the backbone network of the image segmentation network. The VoVNet network can extract image features more quickly and accurately.

[0088] like Figure 4 As shown, in some specific embodiments of the present application, the RGB image features and the depth image features are fused to obtain the final fused features including:

[0089] S310: Multiply the RGB image feature and the depth image feature according to matrix multiplication to obtain a first fusion feature;

[0090] S320: normalize the first fusion feature to obtain an attention mechanism coefficient;

[0091] S330: According to the attention mechanism coefficient, the final fusion feature is obtained.

[0092] Specifically, the RGB image is input into the VoVNet network to obtain the corresponding RGB image feature f(y); the three-channel image, i.e. the processed depth image, is input into the VoVNet network to obtain the depth image feature g(z); the RGB image feature f(y) and the depth image feature g(z) are multiplied by matrix multiplication to obtain the first fusion feature μ, i.e. μ=f(y) T g(z).

[0093] The first fusion feature is normalized, that is, μ is processed using the softmax function to obtain the attention mechanism coefficient; the softmax function is shown in the following formula:

[0094]

[0095] Among them, μ ijk is the eigenvalue of the first fusion feature at position (i, j) in the kth channel, γ ijk is the attention mechanism coefficient of the feature value at position (i, j) in the kth channel, W is the width of the fused feature map, and H is the height of the fused feature map. According to the attention mechanism coefficient γ obtained by the softmax function ijk , and get the attention mechanism coefficient γ ijk The corresponding final fusion features.

[0096] like Figure 5 As shown, in some specific embodiments of the present application, according to the attention mechanism coefficient, the final fusion feature is obtained including:

[0097] S331: multiply the attention mechanism coefficient with the corresponding RGB image feature and the corresponding depth image feature to obtain the corresponding fusion feature value;

[0098] S332: Integrate the fusion feature values ​​to obtain the final fusion feature.

[0099] The attention mechanism coefficient γ obtained by normalizing the first fusion feature ijk Respectively with the corresponding RGB image feature value f ijk (y) and the corresponding depth image eigenvalue g ijk (z) and perform a second fusion to obtain the fused feature value at the (i, j) position in the k-th channel after fusion. All fused feature values ​​are integrated to obtain the final fused feature.

[0100] When the RGB image and the processed depth image are input into the deep neural network, a large number of data features can be obtained. Through the feature fusion method based on the attention mechanism, high-value information can be quickly screened out from these features and other useless information can be suppressed.

[0101] In some specific embodiments of the present application, the fusion feature value is calculated by the following formula:

[0102] h ijk (y,z)=γ ijk [f ijk (y)+ωg ijk (z)]

[0103] Among them, h ijk (y,z) is the eigenvalue at position (i,j) in the kth channel after fusion, γ ijk is the attention mechanism coefficient of the feature value at position (i, j) in the kth channel, f ijk (y) is the feature value of the RGB image feature at the (i, j) position in the kth channel, g ijk (z) is the eigenvalue of the depth image feature at position (i, j) in the kth channel, and ω is the balancing weight coefficient.

[0104] Specifically, ω is the balancing weight coefficient for balancing the two-level features, namely, RGB image features and depth image features. This parameter can participate in the backward propagation of the network and can be optimized during the network model training process. ω will first set an initial value of 1, and then continuously adjust it during the entire network training process until the entire network converges.

[0105] All fusion feature values ​​h ijk (y,z) is integrated to obtain the final fusion feature h(y,z), h(y,z) is a three-dimensional tensor, h ijk (y,z) represents the value at (i,j) in the kth channel of this three-dimensional tensor; tensor is a tensor.

[0106] The first fusion feature h(y, z) is input into the subsequent neural network for processing, that is, input into the head network for processing to obtain the required mask; wherein the head network includes a detection head network and a segmentation head network, the detection head network is a FOCS network, and the segmentation head network is a SAG-mask network.

[0107] According to a second aspect of an embodiment of the present application, an electronic device is provided. The electronic device 600 may be any type of smart terminal, such as a mobile phone, a tablet computer, a personal computer, etc.

[0108] like Figure 6As shown, according to some embodiments of the present application, the electronic device 600 includes: one or more processors 601 and a memory 602, Figure 6 A processor 601 is taken as an example.

[0109] The processor 601 and the memory 602 may be communicatively connected via a bus or other means. Figure 6 Take the example of connecting via a bus.

[0110] The memory 602 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and units, such as program instructions / units corresponding to the electronic device 600 in the embodiment of the present application. The processor 601 executes various functional applications and data processing by running the non-transitory software programs, instructions and units stored in the memory 602, that is, the image segmentation method of the above method embodiment is implemented.

[0111] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to program instructions / units, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely arranged relative to the processor 601, and these remote memories may be connected to the electronic device 600 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0112] One or more units are stored in the memory 602, and when executed by one or more processors 601, the image segmentation method in any of the above method embodiments is executed. Figure 1 The method steps S100 to S400 in Figure 2 In the method steps S210 to S230, Figure 3 In the method steps S221 to S223, Figure 4 In the method steps S310 to S330, Figure 5 Method steps S331 to S332.

[0113] In a third aspect of the embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors 601, for example, Figure 6The one or more processors 601 may execute the image segmentation method in the above method embodiment, for example, executing the above described Figure 1 The method steps S100 to S400 in Figure 2 In the method steps S210 to S230, Figure 3 In the method steps S221 to S223, Figure 4 In the method steps S310 to S330, Figure 5 Method steps S331 to S332.

[0114] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., may be located in one place, or may be distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0116] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0117] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations that may be made to the embodiments without departing from the principles and purpose of the present application should all be included within the scope of protection of the present application.

Claims

1. An image segmentation method, characterized in that: include: Acquire an original image to be segmented, wherein the original image includes an RGB image and a depth image; The depth of the depth image is converted into three-dimensional coordinates in a world coordinate system according to the camera intrinsic parameter matrix, the rotation matrix and the translation matrix; Performing min-max normalization processing on the three-dimensional coordinates to obtain processed three-dimensional coordinate values; obtaining the original pixel position of the depth image; The processed three-dimensional coordinate values ​​are used as the pixel values ​​of the three channels R, G, and B; Filling the processed three-dimensional coordinate values ​​into the original pixel positions to obtain a three-channel image; Performing feature extraction on the RGB image and the three-channel image to obtain RGB image features and depth image features; Multiplying the RGB image feature and the depth image feature according to matrix multiplication to obtain a first fusion feature; Normalizing the first fusion feature to obtain an attention mechanism coefficient; Multiplying the attention mechanism coefficient with the corresponding RGB image feature and the corresponding depth image feature to obtain the corresponding fusion feature value; Integrate the fusion feature values ​​to obtain a final fusion feature; Inputting the final fusion feature into a preset image segmentation network for segmentation processing to obtain a target mask; The fusion feature value is calculated by the following formula: h ijk (y,z)=γ ijk [f ijk (y)+ωg ijk (z)] Among them, h ijk (y,z) is the eigenvalue at position (i,j) in the kth channel after fusion, γ ijk is the attention mechanism coefficient of the feature value at position (i, j) in the kth channel, f ijk (y) is the feature value of the RGB image feature at the position (i, j) in the kth channel, g ijk (z) is the eigenvalue of the depth image feature at the position (i, j) in the kth channel, and ω is the balancing weight coefficient.

2. The image segmentation method according to claim 1, characterized in that: The extracting features of the RGB image and the three-channel image to obtain RGB image features and depth image features comprises: Feature extraction is performed on the RGB image to obtain the RGB image feature, and extraction is performed on the three-channel image to obtain the depth image feature.

3. The image segmentation method according to claim 2, characterized in that: The extracting features from the RGB image to obtain RGB image features and extracting the three-channel image to obtain depth image features comprises: Inputting the RGB image into the image segmentation network to obtain RGB image features; The three-channel image is input into the image segmentation network to obtain deep image features.

4. An electronic device, characterized in that: include: at least one processor, and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions, and the instructions are executed by the at least one processor, so that the at least one processor implements the image segmentation method according to any one of claims 1 to 3 when executing the instructions.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the image segmentation method according to any one of claims 1 to 3.

Images