Method and device for segmenting image based on convolutional neural network, and storage medium

Abstract

The invention discloses a method and a device for segmenting an image based on a convolutional neural network, and a storage medium. The method comprises the following steps: carrying out high-angle shooting on a material pile by using a depth camera with gray scale arranged above the material pile to obtain an original gray scale image and an original depth image of a material at the top layer of the material pile; inputting the original grayscale image and the original depth image into a convolutional neural network, and performing multiple rounds of convolutional fusion operation on the original grayscale image and the original depth image to obtain a final fusion feature map; and performing image segmentation on the top layer material according to the final fusion feature map, and determining the stacking position of each material on the top layer of the material pile according to the image segmentation result. According to the scheme, the depth image information and the grayscale image information are effectively fused through the convolutional neural network, so that the accuracy of material segmentation is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method, device and storage medium for segmenting an image based on a convolutional neural network. Background Art

[0002] Magnesium alloys are widely used in aviation parts due to their light weight and high strength. During the magnesium alloy die-casting melting process, the magnesium ingot can be automatically grabbed by a robotic arm.

[0003] However, during the placement of the magnesium alloy ingot stack, the magnesium ingot may be offset (e.g., tilted), resulting in a deviation from the grabbing coordinates of the robot arm. Therefore, in order to achieve accurate grabbing of the robot arm, the placement position of the top magnesium ingot of the ingot stack needs to be detected. Summary of the invention

[0004] The present application embodiment provides a method for image segmentation based on a convolutional neural network, comprising:

[0005] Using a grayscale depth camera disposed above the material pile to take a bird's-eye view of the material pile to obtain an original grayscale image and an original depth image of the material on the top layer of the material pile;

[0006] Inputting the original grayscale image and the original depth image into a convolutional neural network, performing multiple rounds of convolution fusion operations on the original grayscale image and the original depth image to obtain a final fusion feature map;

[0007] The top material is segmented according to the final fusion feature map, and the stacking position of each material on the top layer of the material pile is determined according to the result of the image segmentation.

[0008] An embodiment of the present application provides an image segmentation device based on a convolutional neural network, comprising: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned method for image segmentation based on a convolutional neural network are implemented.

[0009] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for image segmentation based on a convolutional neural network.

[0010] The method, device and storage medium for image segmentation based on a convolutional neural network provided in the embodiments of the present application use a depth camera with grayscale disposed above a material pile to shoot the material pile from above, and obtain an original grayscale image and an original depth image of the material on the top layer of the material pile; the original grayscale image and the original depth image are input into a convolutional neural network, and multiple rounds of convolution fusion operations are performed on the original grayscale image and the original depth image to obtain a final fusion feature map; image segmentation is performed on the top layer of the material according to the final fusion feature map, and the stacking position of each material on the top layer of the material pile is determined according to the result of the image segmentation. The above scheme can effectively fuse the depth image and grayscale image information through a convolutional neural network, thereby improving the accuracy of material segmentation.

[0011] Other aspects will be apparent upon reading and understanding the drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0013] Figure 1 A flowchart of a method for image segmentation based on a convolutional neural network according to an embodiment of the present application;

[0014] Figure 2-1 A schematic diagram of N rounds of convolution fusion operations in an embodiment of the present application;

[0015] Figure 2-2 A schematic diagram of another N-round convolution fusion operation according to an embodiment of the present application;

[0016] Figure 3 A schematic diagram of the structure of a convolution module used in a first round of convolution operation in an embodiment of the present application;

[0017] Figure 4 A schematic diagram of the structure of a convolution module used in an i-th (1≤i≤N) round of convolution operation in an embodiment of the present application;

[0018] Figure 5 A schematic diagram of the structure of an inverted bottleneck residual attention module according to an embodiment of the present application;

[0019] Figure 6 A schematic diagram of the structure of a segmentation module according to an embodiment of the present application;

[0020] Figure 7 A schematic diagram of the structure of a segmentation unit according to an embodiment of the present application;

[0021] Figure 8-1 A schematic diagram of a target circle according to an embodiment of the present application (point B is not lower than the center of the sphere);

[0022] Figure 8-2 A schematic diagram of another target circle of an embodiment of the present application (point B is below the center of the sphere);

[0023] Fig. 9 A schematic diagram of a device for segmenting images based on a convolutional neural network according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The present application describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present application. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0025] The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements disclosed in the present application may also be combined with any conventional features or elements to form a unique invention scheme defined by the appended claims. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme defined by the appended claims. Therefore, it should be understood that any feature shown and / or discussed in the present application may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the appended claims and their equivalents, the embodiments are not subject to other restrictions. In addition, various modifications and changes may be made within the scope of protection of the appended claims.

[0026] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps of the specific order described. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the appended claims. In addition, the claims for the method and / or process should not be limited to the steps of performing them in the order written, and those skilled in the art can easily understand that these sequences can be changed and still remain within the spirit and scope of the embodiments of the present application.

[0027] like Figure 1As shown, the embodiment of the present application provides a method for segmenting an image based on a convolutional neural network, comprising:

[0028] Step S10, using a grayscale depth camera disposed above the material pile to take a bird's-eye view of the material pile to obtain an original grayscale image and an original depth image of the material on the top layer of the material pile;

[0029] Step S20, inputting the original grayscale image and the original depth image into a convolutional neural network, performing multiple rounds of convolution fusion operations on the original grayscale image and the original depth image to obtain a final fusion feature map;

[0030] Step S30, performing image segmentation on the top material according to the final fusion feature map, and determining the stacking position of each material on the top layer of the material pile according to the result of the image segmentation.

[0031] The method for image segmentation based on a convolutional neural network provided in the above embodiment uses a depth camera with grayscale set above the material pile to shoot the material pile from above, and obtains the original grayscale image and original depth image of the material on the top layer of the material pile; the original grayscale image and the original depth image are input into the convolutional neural network, and multiple rounds of convolution fusion operations are performed on the original grayscale image and the original depth image to obtain the final fusion feature map; the top layer material is image segmented according to the final fusion feature map, and the stacking position of each material on the top layer of the material pile is determined according to the result of the image segmentation. The above method can effectively fuse the depth image and grayscale image information through the convolutional neural network, thereby improving the accuracy of material segmentation.

[0032] In some exemplary embodiments, the material includes: a magnesium alloy ingot. In other embodiments, the material may be an ingot of other materials, such as: an aluminum alloy ingot, etc. The difference in the type of material has no effect on the implementation of the method.

[0033] In some exemplary embodiments, the depth camera includes: a ToF (Time of Flight) camera.

[0034] In some exemplary embodiments, the depth camera comprises a wide-angle depth camera.

[0035] In some exemplary embodiments, performing multiple rounds of convolution fusion operations on the original grayscale image and the original depth image to obtain a final fusion feature map includes:

[0036] Perform the first round of convolution fusion operation: perform the first round of convolution operation on the original grayscale image to obtain a grayscale feature map and output it, perform the first round of convolution operation on the original depth image to obtain a depth feature map and output it, perform fusion operation on the grayscale feature map and the depth feature map to obtain a fusion feature map and output it;

[0037] Execute the second to Nth rounds of convolution fusion operations, wherein any round of convolution fusion operations includes: performing this round of convolution operations on the fusion feature map output by the previous round of convolution fusion operations to obtain a fusion feature map to be processed, performing this round of convolution operations on the grayscale feature map output by the previous round of convolution fusion operations to obtain a new grayscale feature map and output it, performing this round of convolution operations on the depth feature map output by the previous round of convolution fusion operations to obtain a new depth feature map and output it, performing fusion operations on the fusion feature map to be processed, the new grayscale feature map and the new depth feature map to obtain a new fusion feature map and output it;

[0038] After the Nth round of convolution fusion operation is completed, the convolution fusion operation is ended, and the fused feature map output by the Nth round of convolution fusion operation is used as the final fused feature map.

[0039] In some exemplary embodiments, when performing the first round of convolution fusion operation, the original grayscale image and the original depth image are respectively input into the same convolution module to perform the first round of convolution operation;

[0040] When executing the 2nd to Nth rounds of convolution fusion operations, for any round of convolution fusion operations, the fused feature map output by the previous round of convolution fusion operations, the grayscale feature map output by the previous round of convolution fusion operations, and the depth feature map output by the previous round of convolution fusion operations are respectively input into the same convolution module for this round of convolution operations.

[0041] Figure 2-1 A schematic diagram of N rounds of convolution fusion operations in an embodiment of the present application is provided. Figure 2-1 As shown, when performing the first round of convolution fusion operation, the original grayscale image and the original depth image are respectively input into the same convolution module (convolution 1) for the first round of convolution operation;

[0042] When executing the 2nd to Nth rounds of convolution fusion operations, for any round (i-th round) of convolution fusion operations, the fused feature map output by the previous round (i-1th round) of convolution fusion operations, the grayscale feature map output by the previous round (i-th round) of convolution fusion operations, and the depth feature map output by the previous round (i-th round) of convolution fusion operations are respectively input into the same convolution module (convolution i) for this round of convolution operations; 2≤i≤N.

[0043] In some exemplary embodiments, when performing the first round of convolution fusion operation, the original grayscale image and the original depth image are respectively input into different convolution modules to perform the first round of convolution operation;

[0044] When executing the 2nd to Nth rounds of convolution fusion operations, for any round of convolution fusion operations, the fused feature map output by the previous round of convolution fusion operations, the grayscale feature map output by the previous round of convolution fusion operations, and the depth feature map output by the previous round of convolution fusion operations are respectively input into different convolution modules for this round of convolution operations.

[0045] Figure 2-2 A schematic diagram of N rounds of convolution fusion operations in an embodiment of the present application is provided. Figure 2-2 As shown, when performing the first round of convolution fusion operation, the original grayscale image is input into the first first convolution module (convolution A1) for the first round of convolution operation, and the original depth image is input into the first second convolution module (convolution B1) for the first round of convolution operation;

[0046] When executing the 2nd to Nth rounds of convolution fusion operations, for any round (i-th round) of convolution fusion operations, the grayscale feature map output by the previous round (i-1th round) of convolution fusion operation is input into the i-th first convolution module (convolution Ai) for the i-th round of convolution operation, the depth feature map output by the previous round (i-1th round) of convolution fusion operation is input into the i-th second convolution module (convolution Bi) for the i-th round of convolution operation, and the fused feature map output by the previous round (i-1th round) of convolution fusion operation is input into the i-th third convolution module (convolution Ci) for the i-th round of convolution operation.

[0047] In some exemplary embodiments, the fusion operation includes: an addition operation.

[0048] Figure 3 FIG. 1 is a schematic diagram of the structure of a convolution module used in a first round of convolution operation in an embodiment of the present application. Figure 3 As shown, in some exemplary embodiments, the convolution module used in the first round of convolution operation includes a two-dimensional convolution layer, a batch normalization layer, and an activation layer connected in sequence.

[0049] In some exemplary embodiments, the two-dimensional convolutional neural network layer used in the first round of convolution operation includes M1 convolution kernels with a kernel size of K1, and the step size of the convolution kernel is S1.

[0050] In some exemplary embodiments, M1=16, K1=3, and S1=1.

[0051] In some exemplary embodiments, the activation layer of the convolution module used in the first round of convolution operation may use the HSwish activation function.

[0052] Figure 4 Schematic diagram of the structure of a convolution module used in the i-th round (2≤i≤N) of convolution operation in an embodiment of the present application. Figure 4As shown, in some exemplary embodiments, the convolution module used in the i-th round of convolution operation includes at least one inverted bottleneck residual attention module; 2≤i≤N.

[0053] Figure 5 Schematic diagram of the structure of a reverse bottleneck residual attention module in an embodiment of the present application. Figure 5 As shown, in some exemplary embodiments, the inverted bottleneck residual attention module includes a first two-dimensional convolutional layer, a first batch normalization BN layer, a first activation layer, a second two-dimensional convolutional layer, a second BN layer, a second activation layer, a third two-dimensional convolutional layer, a third BN layer, an attention mechanism layer, a fourth two-dimensional convolutional layer and a residual connection layer, which are connected in sequence.

[0054] In some exemplary embodiments, convolution kernel parameters of a first two-dimensional convolution layer, a second two-dimensional convolution layer, a third two-dimensional convolution layer, and a fourth two-dimensional convolution layer of any inverted bottleneck residual attention module are set independently;

[0055] The convolution kernel parameters include: convolution kernel size, number of convolution kernels and convolution kernel step size.

[0056] In some exemplary embodiments, the parameters of each inverted bottleneck residual attention module of each round of convolution operation are set independently.

[0057] In some exemplary embodiments, the first activation layer, the second activation layer, and the third activation layer may adopt the HSwish activation function.

[0058] In some exemplary embodiments, the attention mechanism layer may adopt a channel attention mechanism (Channel Squeeze-and-Excitation, CSE for short).

[0059] In some exemplary embodiments, performing image segmentation on the top material according to the final fusion feature map includes:

[0060] The final fused feature map of the top material is input into the segmentation module for image segmentation;

[0061] The segmentation module includes a plurality of segmentation units connected in sequence; any segmentation unit includes a deconvolution layer, a convolution layer, a batch normalization (BN) layer and an activation layer connected in sequence.

[0062] In some exemplary embodiments, the activation layer of the segmentation unit may adopt a HSwish activation function or a ReLU activation function.

[0063] Figure 6 FIG. 1 is a schematic diagram of a structure of a segmentation module in an embodiment of the present application. Figure 6As shown, the segmentation module includes a plurality of segmentation units connected in sequence.

[0064] Figure 7 FIG. 1 is a schematic diagram of a structure of a segmentation unit in an embodiment of the present application. Figure 7 As shown, the segmentation unit includes a deconvolution layer, a convolution layer, a batch normalization (BN) layer and an activation layer connected in sequence.

[0065] In some exemplary embodiments, before inputting the original depth image into the convolutional neural network, the method further includes:

[0066] Performing 2D distortion correction on the original depth image taken by the depth camera to obtain a first depth image;

[0067] The Hough circle transform method is used to reconstruct a three-dimensional point cloud of the first depth image to achieve spherical distortion correction.

[0068] In some exemplary embodiments, the 2D distortion correction includes: using OpenCV dedistortion function to perform 2D distortion correction. There is a lens in front of the camera lens. Due to the shape of the lens, when light passes through the lens, the refraction of light close to the optical axis is smaller than that of light far from the optical axis, which will produce radial distortion. At this time, straight lines in the real world will be bent in the image. Using OpenCV dedistortion function to perform 2D distortion correction belongs to the prior art and will not be repeated here.

[0069] In some exemplary embodiments, using the Hough circle transform method to perform three-dimensional point cloud reconstruction on the first depth image to achieve spherical distortion correction includes:

[0070] Establish an XYZ three-dimensional rectangular coordinate system based on the X axis, Y axis and Z axis;

[0071] Fitting a circular contour of the three-dimensional point cloud of the first depth image by using Hough circle transform in each first detection plane parallel to the XOY plane, and determining the center of the circular contour; averaging the first coordinates and the second coordinates of the center of the circle in all first detection planes to obtain a first coordinate mean x11 and a second coordinate mean y11;

[0072] Fitting the circular contour of the three-dimensional point cloud of the first depth image by Hough circle transform in each second detection plane parallel to the XOZ plane, and determining the center of the circular contour; averaging the third coordinates and fourth coordinates of the center of the circle in all second detection planes to obtain the third coordinate mean x12 and the fourth coordinate mean z01;

[0073] Fitting the circular contour of the three-dimensional point cloud of the first depth image by using the Hough circle transform in each third detection plane parallel to the YOZ plane, and determining the center of the circular contour; averaging the fifth coordinates and sixth coordinates of the center of the circle in all third detection planes to obtain the fifth coordinate mean y12 and the sixth coordinate mean z02;

[0074] The first coordinate mean x11 and the third coordinate mean x12 are averaged to obtain X1, the second coordinate mean y11 and the fifth coordinate mean y12 are averaged to obtain Y1, and the fourth coordinate mean z01 and the sixth coordinate mean z02 are averaged to obtain Z0;

[0075] Establishing a distorted spherical surface cluster based on the three-dimensional point cloud of the first depth image in the XYZ three-dimensional rectangular coordinate system, where all distorted spherical surfaces have the same spherical center O, the coordinates of the spherical center O are (X1, Y1, Z0), and the radii of any two distorted spherical surfaces are different;

[0076] All points in the original three-dimensional point cloud of the first depth image are processed by position transformation to generate a new three-dimensional point cloud; wherein, the position transformation processing of any point B in the three-dimensional point cloud of the first depth image includes: determining the distorted sphere where point B is located, and determining the highest point C of the distorted sphere; taking the plane determined by point B, point C and the center of the sphere O as the longitudinal section; taking the first detection plane passing through point C and parallel to the XOY plane as the target plane; taking the circle intersecting the longitudinal section and the distorted sphere as the target circle; when the Z-axis coordinate of point B is greater than or equal to the Z-axis coordinate of the center of the sphere O, mapping point B to point D on the intersection of the target plane and the longitudinal section, and the distance from point D to point C is equal to the arc length in the target circle When the Z-axis coordinate of point B is less than the Z-axis coordinate of the sphere center O, a second detection plane passing through the sphere center O and parallel to the XOY plane is used as a reference plane, and a symmetric point B' of point B is determined on the target circle. Point B and its symmetric point B' are symmetric about the intersection line of the reference plane and the longitudinal section, and point B is mapped to point D on the intersection line of the target plane and the longitudinal section. The distance from point D to point C is equal to the arc length in the target circle.

[0077] Figure 8-1 Schematic diagram of a target circle of an embodiment of the present application (point B is not lower than the center of the sphere). Figure 8-1As shown, point B (X2, Y2, Z1) is any point in the three-dimensional point cloud of the first depth image, point O is the center O (X1, Y1, Z0) of the distorted sphere cluster, Z1>Z0, that is, the Z-axis coordinate of point B is greater than the Z-axis coordinate of the center O. Determine the distorted sphere where point B is located. Point C (X1, Y1, Z2) is the highest point (with the largest Z-axis coordinate value) of the distorted sphere where point B (X2, Y2, Z1) is located. The plane determined by point B, point C and the center O is taken as the longitudinal section; the first detection plane passing through point C and parallel to the XOY plane is taken as the target plane; the circle where the longitudinal section intersects with the distorted sphere is taken as the target circle; point B is mapped to point D (X3, Y3, Z2) on the intersection of the target plane and the longitudinal section, and the distance from point D to point C is equal to the arc length within the target circle.

[0078] like Figure 8-1 As shown, within the target circle, according to the center O (X1, Y1, Z0) and point B (X2, Y2, Z1), the radius R of the target circle can be determined as the vector Length The coordinates of point C are (X1, Y1, Z0+R), and the central angle ∠BOC is the vector and vector Assuming ∠BOC = α, the arc length α is in radians, then L can be calculated using the following formula (1):

[0079] L=R*α; (1)

[0080] Inside the target circle, draw a perpendicular line from point B to radius OC, with the foot of the perpendicular being A(X1, Y1, Z1).

[0081] Considering that the Z-axis variable does not affect the vector The calculation result of vector It can be expressed as (X3-X1, Y3-Y1). The distance from point D to point C is equal to the arc length within the target circle. So we get the following relation (3):

[0082]

[0083] Considering that the Z-axis variable does not affect the vector The calculation result is, then the vector It can be expressed as (X2-X1,Y2-Y1). Vector and vector Parallel, the cross product of the two is a 0 vector, given by The following relationship (4) can be obtained, where “×” is the cross product symbol;

[0084] (X2 - X1)(Y3 - Y1) - (X3 - X1)(Y2 - Y1) = 0 (4)

[0085] According to relational expressions (3) and (4), the Y-axis coordinate Y3 of point D can be obtained, and Y3 can be expressed as the following formula (5);

[0086]

[0087] Substituting the Y-axis coordinate Y3 of point D into the following formula (6) can obtain the X-axis coordinate X3 of point D;

[0088]

[0089] The Z-axis coordinate values of point D and point C are equal. Therefore, the Z-axis coordinate value Z2 of point D = Z0 + R.

[0090] Figure 8-2 Schematic diagram of a target circle in an embodiment of the present application (point B is lower than the center of the sphere). As Figure 8-2 shown, point B(X2, Y2, Z1) is an arbitrary point in the three-dimensional point cloud of the first depth image, and point O is the center of the sphere O(X1, Y1, Z0) of the distorted spherical cluster. Z1 < Z0, that is, the Z-axis coordinate of point B is less than the Z-axis coordinate of the center of the sphere O. Determine the distorted sphere where point B is located. Point C(X1, Y1, Z2) is the highest point (the largest Z-axis coordinate value) of the distorted sphere where point B(X2, Y2, Z1) is located. The plane determined by point B, point C, and the center of the sphere O is used as the longitudinal section; the first detection plane passing through point C and parallel to the XOY plane is used as the target plane; the circle formed by the intersection of the longitudinal section and the distorted sphere is used as the target circle; the second detection plane passing through the center of the sphere O and parallel to the XOY plane is used as the reference plane. On the target circle, determine the symmetric point B'(X2, Y2, Z3) of point B. Point B and its symmetric point B' are symmetric about the intersection line of the reference plane and the longitudinal section, that is, Z3 > Z0 > Z1, Z3 = 2Z0 - Z1. Map point B to point D(X3, Y3, Z2) on the intersection line of the target plane and the longitudinal section. The distance from point D to point C is equal to the arc length within the target circle After determining the coordinates (X2, Y2, Z3) of the symmetric point B' of point B, the symmetric point B' in Figure 8-2 can be regarded as Figure 8-1 the point B in Figure 8-1 , and then the same method of mapping point B shown in Figure 8-1 is used to obtain the coordinates of point D, which will not be elaborated here.

[0091] In some exemplary embodiments, the method further includes: performing supervised training on the convolutional neural network.

[0092] For example, 350 magnesium ingot image data can be annotated. After the annotation is completed, all magnesium ingot image data are divided into 280 training sets, 35 validation sets and 35 test sets according to the ratio of 8:1:1. After 50 to 100 rounds of training, the final model is selected based on the Dice value obtained by the model in the validation set. If the Dice value is greater than 0.95 when the training is completed, the model corresponding to the maximum Dice value is selected as the magnesium ingot segmentation model. If the Dice value is less than 0.95 when the training is completed, the training can be postponed for 20 to 30 rounds; if the Dice value is still less than 0.95 after the postponed training, the model corresponding to the maximum Dice value can be directly selected.

[0093] like Fig. 9 As shown, an embodiment of the present disclosure provides a device for image segmentation based on a convolutional neural network, comprising: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned method for image segmentation based on a convolutional neural network are implemented.

[0094] An embodiment of the present disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for image segmentation based on a convolutional neural network.

[0095] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A method for segmenting an image based on a convolutional neural network, comprising: Using a grayscale depth camera disposed above the material pile to take a bird's-eye view of the material pile to obtain an original grayscale image and an original depth image of the material on the top layer of the material pile; Inputting the original grayscale image and the original depth image into a convolutional neural network, performing multiple rounds of convolution fusion operations on the original grayscale image and the original depth image to obtain a final fusion feature map; The top material is segmented according to the final fusion feature map, and the stacking position of each material on the top layer of the material pile is determined according to the result of the image segmentation.

2. The method according to claim 1, characterized in that: The performing multiple rounds of convolution fusion operations on the original grayscale image and the original depth image to obtain a final fusion feature map includes: Perform the first round of convolution fusion operation: perform the first round of convolution operation on the original grayscale image to obtain a grayscale feature map and output it, perform the first round of convolution operation on the original depth image to obtain a depth feature map and output it, perform fusion operation on the grayscale feature map and the depth feature map to obtain a fusion feature map and output it; Execute the second to Nth rounds of convolution fusion operations, wherein any round of convolution fusion operations includes: performing this round of convolution operations on the fusion feature map output by the previous round of convolution fusion operations to obtain a fusion feature map to be processed, performing this round of convolution operations on the grayscale feature map output by the previous round of convolution fusion operations to obtain a new grayscale feature map and output it, performing this round of convolution operations on the depth feature map output by the previous round of convolution fusion operations to obtain a new depth feature map and output it, performing fusion operations on the fusion feature map to be processed, the new grayscale feature map and the new depth feature map to obtain a new fusion feature map and output it; After the Nth round of convolution fusion operation is completed, the convolution fusion operation is ended, and the fused feature map output by the Nth round of convolution fusion operation is used as the final fused feature map.

3. The method according to claim 2, characterized in that: When performing the first round of convolution fusion operation, the original grayscale image and the original depth image are respectively input into the same convolution module for the first round of convolution operation; when performing the second to Nth rounds of convolution fusion operation, for any round of convolution fusion operation, the fusion feature map output by the previous round of convolution fusion operation, the grayscale feature map output by the previous round of convolution fusion operation, and the depth feature map output by the previous round of convolution fusion operation are respectively input into the same convolution module for the current round of convolution operation; or When performing the first round of convolution fusion operation, the original grayscale image and the original depth image are respectively input into different convolution modules for the first round of convolution operation; When executing the 2nd to Nth rounds of convolution fusion operations, for any round of convolution fusion operations, the fused feature map output by the previous round of convolution fusion operations, the grayscale feature map output by the previous round of convolution fusion operations, and the depth feature map output by the previous round of convolution fusion operations are respectively input into different convolution modules for this round of convolution operations.

4. The method according to claim 3, characterized in that: The convolution module used in the first round of convolution operation includes a two-dimensional convolution layer, a batch normalization layer and an activation layer connected in sequence; The convolution module used in the i-th round of convolution operation includes at least one inverted bottleneck residual attention module; 2≤i≤N; The inverted bottleneck residual attention module includes a first two-dimensional convolutional layer, a first batch normalization BN layer, a first activation layer, a second two-dimensional convolutional layer, a second BN layer, a second activation layer, a third two-dimensional convolutional layer, a third BN layer, an attention mechanism layer, a fourth two-dimensional convolutional layer and a residual connection layer, which are connected in sequence.

5. The method according to claim 2, characterized in that: The fusion operation includes: addition operation.

6. The method according to claim 1, characterized in that: The image segmentation of the top material according to the final fusion feature map includes: The final fused feature map of the top material is input into the segmentation module for image segmentation; The segmentation module includes a plurality of segmentation units connected in sequence; any segmentation unit includes a deconvolution layer, a convolution layer, a batch normalization (BN) layer and an activation layer connected in sequence.

7. The method according to claim 1, characterized in that: Before inputting the original depth image into the convolutional neural network, the method further includes: Performing 2D distortion correction on the original depth image taken by the depth camera to obtain a first depth image; The Hough circle transform method is used to reconstruct a three-dimensional point cloud of the first depth image to achieve spherical distortion correction.

8. The method according to claim 7, characterized in that: Using the Hough circle transform method to reconstruct a three-dimensional point cloud of the first depth image to achieve spherical distortion correction, including: Establish an XYZ three-dimensional rectangular coordinate system based on the X axis, Y axis and Z axis; Fitting a circular contour of the three-dimensional point cloud of the first depth image by using Hough circle transform in each first detection plane parallel to the XOY plane, and determining the center of the circular contour; averaging the first coordinates and the second coordinates of the center of the circle in all first detection planes to obtain a first coordinate mean x11 and a second coordinate mean y11; Fitting the circular contour of the three-dimensional point cloud of the first depth image by Hough circle transform in each second detection plane parallel to the XOZ plane, and determining the center of the circular contour; averaging the third coordinates and fourth coordinates of the center of the circle in all second detection planes to obtain the third coordinate mean x12 and the fourth coordinate mean z01; Fitting the circular contour of the three-dimensional point cloud of the first depth image by using the Hough circle transform in each third detection plane parallel to the YOZ plane, and determining the center of the circular contour; averaging the fifth coordinates and sixth coordinates of the center of the circle in all third detection planes to obtain the fifth coordinate mean y12 and the sixth coordinate mean z02; The first coordinate mean x11 and the third coordinate mean x12 are averaged to obtain X1, the second coordinate mean y11 and the fifth coordinate mean y12 are averaged to obtain Y1, and the fourth coordinate mean z01 and the sixth coordinate mean z02 are averaged to obtain Z0; Establishing a distorted spherical surface cluster based on the three-dimensional point cloud of the first depth image in the XYZ three-dimensional rectangular coordinate system, where all distorted spherical surfaces have the same spherical center O, the coordinates of the spherical center O are (X1, Y1, Z0), and the radii of any two distorted spherical surfaces are different; All points in the original three-dimensional point cloud of the first depth image are processed by position transformation to generate a new three-dimensional point cloud; wherein, the position transformation processing of any point B in the three-dimensional point cloud of the first depth image includes: determining the distorted sphere where point B is located, and determining the highest point C of the distorted sphere; taking the plane determined by point B, point C and the center of the sphere O as the longitudinal section; taking the first detection plane passing through point C and parallel to the XOY plane as the target plane; taking the circle intersecting the longitudinal section and the distorted sphere as the target circle; when the Z-axis coordinate of point B is greater than or equal to the Z-axis coordinate of the center of the sphere O, mapping point B to point D on the intersection of the target plane and the longitudinal section, and the distance from point D to point C is equal to the arc length in the target circle When the Z-axis coordinate of point B is less than the Z-axis coordinate of the sphere center O, a second detection plane passing through the sphere center O and parallel to the XOY plane is used as a reference plane, and a symmetric point B' of point B is determined on the target circle. Point B and its symmetric point B' are symmetric about the intersection line of the reference plane and the longitudinal section, and point B is mapped to point D on the intersection line of the target plane and the longitudinal section. The distance from point D to point C is equal to the arc length in the target circle.

9. A device for segmenting an image based on a convolutional neural network, comprising: A memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the method for image segmentation based on a convolutional neural network described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of the method for image segmentation based on a convolutional neural network described in any one of claims 1 to 8 are implemented.

Images