Image segmentation method, device, electronic device and storage medium

Abstract

The present application relates to an image segmentation method, device, electronic device and storage medium, and proposes a deformable convolution kernel idea. The deformable convolution kernel is guided to deform by determining the gradient information of the image, and convolution processing is performed on the image based on the deformed convolution kernel. Since the gradient information of the image pixels can reflect the change status of the image pixels, for irregular objects, the pixel change status information of the irregular area is utilized to guide the deformable convolution kernel to perform adaptive deformation, so as to fully extract the image feature information of the irregular object, thereby achieving the purpose of accurately and efficiently completing the semantic segmentation task of the irregular object image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an image segmentation method, device, electronic device and storage medium. Background Art

[0002] Image segmentation is the technology and process of dividing an image into several specific regions with unique properties and proposing objects of interest.

[0003] Traditional image segmentation algorithms, such as threshold-based, edge-based, or region-based segmentation algorithms, extract limited feature information, have poor generalization performance, and have simple segmentation logic and difficulty in achieving semantic segmentation of images.

[0004] Image semantic segmentation is to add semantic labels to the segmented images based on image segmentation. The image segmentation algorithm based on deep learning can extract rich feature information and solve the semantic segmentation problem of images better than traditional segmentation algorithms. However, most image segmentation networks based on deep learning are built based on standard convolution kernels. For segmentation tasks, it is not only necessary to complete the semantic segmentation of regular objects. In real scenes, there are also many tasks that require the semantic segmentation of irregular objects. However, it is difficult for standard convolution kernels to fully utilize the convolution kernel to extract the feature information of images of irregular objects (especially the edges of irregular objects) in image semantic segmentation tasks. Summary of the Invention

[0005] In view of this, the present application provides an image segmentation method, device, electronic device and storage medium, which achieves the purpose of fully extracting image feature information of irregular objects by utilizing a gradient-based deformable convolution kernel for image semantic segmentation.

[0006] The specific technical solutions are as follows:

[0007] An image segmentation method, comprising:

[0008] Obtaining a target image;

[0009] Determining gradient information of the target image;

[0010] Based on the gradient information, deforming the first convolution kernel of the current convolution layer of the target image to be processed in the deep segmentation network model to obtain a deformed second convolution kernel;

[0011] The target image is convolved using the second convolution kernel, and a semantic segmentation result of the target image is obtained based on the convolution processing of the target image.

[0012] Optionally, obtaining the target image includes:

[0013] If the current convolutional layer is the first convolutional layer in the deep segmentation network model, obtaining an original image to be segmented;

[0014] If the current convolutional layer is not the first convolutional layer in the depth segmentation network model, obtain a corresponding feature image of the original image output by the previous convolutional layer adjacent to the current convolutional layer.

[0015] Optionally, determining the gradient information of the target image includes:

[0016] Calculating the gradient vectors of all pixels of the target image to obtain a gradient image;

[0017] or,

[0018] Calculate the gradient vector of each pixel point of the current pixel area to be scanned corresponding to the first convolution kernel in the target image during the convolution operation.

[0019] Optionally, calculating the gradient vectors of all pixels of the target image to obtain a gradient image includes:

[0020] The gradient calculation network layer provided above the current convolution layer in the depth segmentation network model is used to calculate the gradient vectors of all pixels of the target image to obtain a gradient image.

[0021] Optionally, the deforming the first convolution kernel of the current convolution layer of the target image to be processed in the deep segmentation network model based on the gradient information includes:

[0022] Extracting a gradient vector corresponding to each pixel point of the pixel area to be scanned from the gradient image, or obtaining a gradient vector of each pixel point of the determined pixel area to be scanned; wherein the gradient vector includes a gradient direction and a gradient magnitude;

[0023] Based on the gradient direction of each pixel point in the pixel area to be scanned, controlling the moving direction of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel;

[0024] Based on the gradient size of each pixel point in the pixel area to be scanned, the moving distance of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel is controlled.

[0025] Optionally, based on the gradient direction of each pixel point in the pixel area to be scanned, controlling the movement direction of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel, and based on the gradient size of each pixel point in the pixel area to be scanned, controlling the movement distance of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel includes:

[0026] Controlling each sub-window in the sliding window corresponding to the first convolution kernel to move along the gradient direction of the corresponding pixel point in the pixel area to be scanned, and moving along the gradient direction by a distance equal to the gradient magnitude of the corresponding pixel point;

[0027] The sub-windows that have completed the movement constitute the second convolution kernel.

[0028] Optionally, the first convolution kernel is a standard convolution kernel of a predetermined size;

[0029] The method further comprises: deforming a first convolution kernel of a current convolution layer of the target image to be processed in a deep segmentation network model based on the gradient information to obtain a deformed second convolution kernel, including:

[0030] During a complete convolution operation on the target image, the standard convolution kernel is used to slide on the target image according to a set scanning mode and sliding step size;

[0031] Moreover, whenever sliding to a pixel area to be scanned, the standard convolution kernel sliding to the pixel area to be scanned is deformed based on the gradient vector of each pixel point of the pixel area to be scanned to obtain a second convolution kernel.

[0032] Optionally, performing convolution processing on the target image using the second convolution kernel includes:

[0033] Convolution processing is performed on the target image based on the current pixel positions of each sub-window in the second convolution kernel in the target image.

[0034] An image segmentation device, comprising:

[0035] an acquisition unit, configured to obtain a target image;

[0036] a determining unit, configured to determine gradient information of the target image;

[0037] A deformation processing unit, configured to perform deformation processing on a first convolution kernel of a current convolution layer of the target image to be processed in a deep segmentation network model based on the gradient information to obtain a deformed second convolution kernel;

[0038] A convolution processing unit is used to perform convolution processing on the target image using the second convolution kernel, and obtain a semantic segmentation result of the target image based on the convolution processing on the target image.

[0039] An electronic device, comprising:

[0040] Memory, used to store computer instruction sets;

[0041] A processor, configured to implement the image segmentation method according to any one of claims 1 to 8 by executing an instruction set stored in the memory.

[0042] A computer-readable storage medium stores a computer instruction set, wherein the computer instruction set, when executed by a processor, implements the image segmentation method according to any one of claims 1 to 8.

[0043] The image segmentation method, device, electronic device and storage medium provided in the embodiments of the present application propose a deformable convolution kernel idea, which guides the deformation of the deformable convolution kernel by determining the gradient information of the image, and performs convolution processing on the image based on the deformed convolution kernel. Since the gradient information of the image pixels can reflect the change status of the image pixels, for irregular objects, it is possible to utilize the pixel change status information of its irregular area to guide the deformable convolution kernel to perform adaptive deformation, so as to fully extract the image feature information of the irregular object, thereby achieving the purpose of accurately and efficiently completing the semantic segmentation task of the irregular object image. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0045] Figure 1 This is a flow chart of the image segmentation method provided in an embodiment of the present application;

[0046] Figure 2 This is a schematic diagram of standard convolution kernels of different sizes;

[0047] Figure 3 This is another flowchart of the image segmentation method provided in an embodiment of the present application;

[0048] FIG4( a ) is a schematic diagram of gradient vectors of all pixel points of a target image provided by an embodiment of the present application;

[0049] FIG4( b ) is a schematic diagram of the gradient vectors of each pixel point in the current pixel area to be scanned in the target image provided by an embodiment of the present application;

[0050] Figure 5 is a schematic diagram of calculating pixel gradient vectors provided in an embodiment of the present application;

[0051] Figure 6 Schematic diagram of deforming the convolution kernel based on the size and direction of the image gradient provided by an embodiment of the present application;

[0052] Figure 7 is a schematic diagram of sliding a convolution kernel in a target image provided by an embodiment of the present application;

[0053] Figure 8 Schematic diagram of the structure of the image segmentation device provided in an embodiment of the present application;

[0054] Figure 9 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0056] In order to fully extract the image feature information of irregular objects and complete the semantic segmentation task of their images as accurately and efficiently as possible, the present application discloses an image segmentation method, device, electronic device and storage medium. The image segmentation method, device, electronic device and storage medium of the present application will be explained through multiple embodiments below.

[0057] In an optional embodiment, an image segmentation method is provided. The method is suitable for scenarios where image segmentation (such as image semantic segmentation) is performed using an electronic device. The electronic device performing image segmentation can be, but is not limited to, a personal PC such as a laptop, desktop, or all-in-one computer, or can also be a cloud server, a local area network server, or other device, which is not limited here.

[0058] like Figure 1 As shown, the image segmentation method may include the following processing steps:

[0059] Step 101: Obtain a target image.

[0060] The embodiment of the present application uses deep learning-based image segmentation technology to complete the semantic segmentation of the image.

[0061] Specifically, in view of the defect that traditional deep learning-based image segmentation networks all use standard convolution kernels to extract image feature information of objects in image semantic segmentation tasks, and the corresponding defect that it is difficult to fully extract image feature information of irregular objects, this application proposes the following technical ideas: based on the gradient information of the image to be segmented, the deformable convolution kernel is guided to deform, and the deformed convolution kernel is used to perform convolution operations on the image, so as to achieve the purpose of fully extracting the image feature information of irregular objects, and then realize the use of feature information of the region of interest (such as the edge area of the irregular object) to complete the semantic segmentation task of the image as accurately and efficiently as possible.

[0062] The traditional standard convolution kernel is generally rectangular, refer to Figure 2 , for example, it can be a rectangular convolution kernel with a size of 3x3, 5x5, 7x7, etc., and the deformable convolution kernel in the embodiment of the present application means that the shape of the kernel can be changed arbitrarily and is no longer limited to a rectangle.

[0063] Figure 1 The illustrated steps 101-104 aim to provide an implementation process for performing convolution processing on an image using a deformable convolution kernel based on the above technical ideas to complete the image semantic segmentation task.

[0064] The deep segmentation network model constructed based on deep learning image segmentation technology usually includes multiple convolutional layers with convolution kernels. If the current convolutional layer of the target image to be processed in the deep segmentation network model is the first convolutional layer in the model, then the target image is obtained in this step 101, specifically, the original image to be segmented is obtained; if the current convolutional layer of the target image to be processed in the deep segmentation network model is not the first convolutional layer in the model, then the target image is obtained in this step 101, specifically, the corresponding feature image of the original image to be segmented output by the previous convolutional layer adjacent to the current convolutional layer is obtained.

[0065] Step 102: Determine the gradient information of the target image.

[0066] An image can be considered as a two-dimensional discrete function, which can be expressed as a functional relationship between the pixel value of each pixel in the image and the coordinates of the pixel point in the image. Specifically, for example, it can be expressed as a functional relationship between the RGB value or grayscale value of the image pixel and the two-dimensional position coordinates (x, y) of the pixel.

[0067] The gradient is a vector information with direction and magnitude. The gradient specifically indicates that the directional derivative of a function at a certain point on the function curve can achieve the maximum value along a certain direction. That is, the function changes fastest along this direction at this point and the rate of change is the largest. The direction of the fastest change is the gradient direction, and the rate of change is the magnitude of the gradient (or the modulus of the gradient).

[0068] The gradient of an image is the result of differentiating the two-dimensional discrete function of the image. In an image, the image pixels at the edge are quite different, especially the image edges of irregular objects, which have larger image pixel differences and correspondingly larger gradients. In flat areas, the gradient values are generally smaller.

[0069] It is precisely based on this feature that the present application reflects the pixel changes of the image area based on the gradient information of the image area, and accordingly uses the gradient information of the image area to guide the dynamic deformation of the convolution kernel during the convolution process, so that the shape change of the convolution kernel is consistent with the pixel change of the image (for example, the image edge shape formed by the edge pixels of an irregular object is irregular, and may show arc-shaped, wavy, sawtooth-shaped, sharp-angled and other changing characteristics), and accordingly can extract its feature information more accurately and efficiently.

[0070] In this step, the gradient information of the target image is determined, which correspondingly includes determining the gradient magnitude and gradient direction of the pixels in the target image.

[0071] Step 103: Based on the gradient information, deform the first convolution kernel of the current convolution layer of the target image to be processed in the deep segmentation network model to obtain a deformed second convolution kernel.

[0072] After determining the gradient magnitude and gradient direction of the pixels in the target image, the gradient information of the gradient magnitude and gradient direction of the pixels in the target image is further used to guide the offset of the first convolution kernel (i.e., the current convolution kernel of the current convolution layer of the target image to be processed in the deep segmentation network model) to perform deformation processing on it, and accordingly obtain the deformed second convolution kernel.

[0073] Step 104: Perform convolution processing on the target image using the second convolution kernel, and obtain a semantic segmentation result of the target image based on the convolution processing on the target image.

[0074] Finally, the target image is convolved with the deformed second convolution kernel. Because gradient information can reflect the pixel variations of an image, guiding the convolution kernel's deformation based on this gradient information allows the kernel's shape to align with the pixel variations of the image. This is particularly true for irregular image edges, where the kernel's deformation matches the pixel variations of the image edge. This allows the kernel to utilize the feature information of the region of interest to complete the semantic segmentation task as accurately and efficiently as possible.

[0075] The image segmentation method of this embodiment proposes a deformable convolution kernel. By determining the image's gradient information, the deformable convolution kernel is guided to deform, and the image is then convolved with the deformed convolution kernel. This allows the deformable convolution kernel to adaptively deform irregular objects by leveraging pixel variation information in irregular regions to fully extract the object's image features, thereby achieving accurate and efficient semantic segmentation of the image.

[0076] In an alternative embodiment, see Figure 3 , the image segmentation method disclosed in this application can be further implemented as the following processing process:

[0077] Step 301: Obtain a target image.

[0078] Among them, if the current convolution layer of the target image to be processed in the deep segmentation network model is the first convolution layer in the model, the original image to be segmented is obtained.

[0079] If the current convolutional layer of the target image to be processed in the deep segmentation network model is not the first convolutional layer in the model, the corresponding feature image of the original image to be segmented output by the previous convolutional layer adjacent to the current convolutional layer is obtained.

[0080] Step 302: Calculate the gradient vectors of all pixels of the target image to obtain a gradient image; or calculate the gradient vectors of each pixel of the current pixel area to be scanned in the target image corresponding to the first convolution kernel during the convolution operation.

[0081] In an optional embodiment, referring to FIG. 4( a ), the gradient information of the target image can be determined by calculating the gradient vectors of all pixels of the target image to obtain a gradient image. That is, the gradient image includes gradient vectors corresponding to all pixels of the target image. In FIG. 4( a ), the ellipsis in each box indicates that the gradient vector of the pixel corresponding to the position of the box has also been calculated, but the calculated gradient vector is not specifically shown in the figure.

[0082] In addition, optionally, in combination with reference to FIG4(b), the gradient information of the target image can be determined by calculating the gradient vectors of each pixel point in the current pixel area to be scanned in the target image corresponding to the first convolution kernel (the current convolution kernel when processing the image) during the convolution operation. The blank boxes in FIG4(b) indicate that the gradient vectors of the pixel points corresponding to the positions of the blank boxes have not been calculated.

[0083] The gradient vector calculated based on any of the methods shown in Figure 4(a) or Figure 4(b) includes the gradient magnitude and direction.

[0084] In practice, considering the size of the convolution kernel, it is preferred to use two filter kernels with the same size as the first convolution kernel: the filter kernel S in the x direction and x and the filter kernel S in the y direction y , and perform convolution operations with the target image respectively, that is, (where f represents the target image). Then, the results obtained in the two directions are combined to obtain the gradient vector of the target image at a certain pixel point. Specifically, the gradient vector of the pixel point can be expressed as: For easier understanding, please refer to Figure 5 As shown, the vector in the x direction represents the S x The result of the convolution operation with the target image, the vector in the y direction represents the S y The result of the convolution operation with the target image, the sum of the two vectors is the gradient vector of the pixel.

[0085] Step 303: extracting the gradient vector corresponding to each pixel point of the pixel area to be scanned from the gradient image, or obtaining the gradient vector of each pixel point of the determined pixel area to be scanned; wherein the gradient vector includes a gradient direction and a gradient magnitude.

[0086] The convolution operation is performed on the image (original image or feature image) using the convolution kernel. In essence, the sliding window corresponding to the convolution kernel is used to slide on the image according to a predetermined scanning method and sliding step size. For example, a 3x3, 5x5 or 7x7 window corresponding to the convolution kernel is used to slide on the image from left to right combined with a top-to-bottom scanning method and based on a sliding step size of one pixel, etc., and when sliding to each position, each pixel point of the image currently at the convolution kernel window position (each pixel point at the window position can be called a pixel point of the area to be scanned) is subjected to dot multiplication (dot multiplication of the pixel value with the weight of the corresponding position in the convolution kernel) and summation (summing the dot product results of each pixel) operations to extract the image features at the window position in the image, i.e., the area to be scanned. For a more detailed processing process, please refer to the convolution processing process of the image in the prior art, which will not be described in detail.

[0087] A complete convolution operation performed on an image using a convolution kernel is to use the convolution kernel to complete a scanning process (including the above-mentioned dot multiplication and summation) based on the sliding of the convolution kernel window on the entire image, and accordingly complete the feature extraction process of the entire image. In this embodiment, the deformation of the convolution kernel specifically refers to that in the process of using the convolution kernel to perform a convolution operation on the target image, each time the convolution kernel window is slid to a certain position to be scanned for the pixel area corresponding to the position, the convolution kernel is first deformed based on the gradient information of each pixel in the area to be scanned corresponding to the position, and then a scanning process based on dot multiplication and summation is performed based on the deformed convolution kernel. Therefore, in this application, in the process of using a convolution kernel to perform a complete convolution operation on the image, the convolution kernel is continuously and dynamically deformed, wherein, when the window based on the convolution kernel slides and each time a new window position needs to be scanned, the convolution kernel is first deformed based on the gradient information of each pixel corresponding to the window position before scanning.

[0088] On this basis, for the situation shown in Figure 4(a), the gradient vectors of each pixel point in the current pixel area to be scanned of the first convolution kernel can be extracted from the above-mentioned gradient image including the gradient information of all pixel points of the target image. The extracted gradient vectors of each pixel point in the pixel area to be scanned can specifically be the gradient vector information shown in Figure 4(b). For the situation shown in Figure 4(b), the gradient vectors of each pixel point in the pixel area to be scanned as shown in 4(b) can be directly read.

[0089] Step 304: Based on the gradient direction of each pixel point in the pixel area to be scanned, control the moving direction of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel.

[0090] Step 305: Based on the gradient size of each pixel point in the pixel area to be scanned, control the movement distance of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel; the sub-windows that have completed the movement constitute the second convolution kernel.

[0091] Afterwards, the offset of the first convolution kernel to be subjected to the convolution operation can be guided based on the gradient vector of each pixel point in the pixel area to be scanned that is extracted or read, and the first convolution kernel can be deformed.

[0092] The gradient vector includes direction and size. When the gradient vector is used to guide the deformation of the first convolution kernel, the movement direction of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel can be guided based on the gradient direction of each pixel point in the pixel area to be scanned; and the movement distance of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel can be guided based on the gradient size of each pixel point in the pixel area to be scanned. The deformation of the first convolution kernel is achieved by moving each sub-window in the sliding window corresponding to the first convolution kernel.

[0093] Optionally, in implementation, see Figure 6 (a)- Figure 6 (b) When guiding the moving distance and direction of the first convolution kernel based on the image gradient size and direction respectively, each sub-window in the sliding window corresponding to the first convolution kernel can be specifically controlled to move along the gradient direction of the corresponding pixel point in the pixel area to be scanned, and move along the gradient direction by a distance corresponding to the gradient size of the corresponding pixel point.

[0094] It should be noted that the first convolution kernel is a standard convolution kernel of a predetermined size, such as a rectangular convolution kernel of 3x3, 5x5 or 7x7, and the second convolution kernel is a non-standard convolution kernel of the same size. The same size here specifically means that the window corresponding to the second convolution kernel has the same number of sub-windows as the window corresponding to the first convolution kernel before deformation. For example, assuming that for the standard first convolution kernel of 3x3, its window shape is a rectangle with a length and width of 3 respectively, including a total of 9 sub-windows. After deformation to obtain the second convolution kernel, the second convolution kernel also has 9 sub-windows, but the shape is no longer a rectangle. Its specific shape depends on the deformation processing based on gradient information.

[0095] like Figure 6 In (a), sub-windows 1-9 form a 3x3 standard convolution kernel based on their respective positions, which is the first convolution kernel in this embodiment. Each sub-window in the convolution kernel moves along the gradient direction of each corresponding pixel by a distance equal to the gradient size ( Figure 6 The gradient vectors (vectors) of pixels 1-9 in (a) respectively reflect both direction and magnitude, and then they are transformed into Figure 6 The second convolution kernel in (b) is composed of sub-windows 1-9 at corresponding positions.

[0096] Step 306: Perform convolution processing on the target image based on the current pixel position of each sub-window in the second convolution kernel in the target image.

[0097] Afterwards, the second convolution kernel is used, based on the corresponding area position of each sub-window in the second convolution kernel in the target image, such as Figure 6The area positions corresponding to the sub-windows 1-9 in (b) are subjected to scanning operations including dot multiplication and summation on the area positions of the target image.

[0098] After the scanning operation is completed, the first convolution kernel can be slid from the current position to the next scanning position based on the predetermined scanning mode and sliding step size. For example, see Figure 7 (a)-7(b), and further based on the gradient information of each pixel corresponding to the new sliding position (i.e., the new area to be scanned), guide the first convolution kernel at the new position to deform, and then perform a scanning operation based on the deformed second convolution kernel until a complete convolution operation process of the target image is completed (based on the sliding scanning of the convolution kernel) and the corresponding feature map of the target image is obtained.

[0099] Subsequently, the next processing step of image segmentation can be continued based on the deep segmentation network model, such as using the next convolution kernel (such as the corresponding convolution kernel of the next convolution layer) to continue the convolution operation until the entire image segmentation process is completed.

[0100] Among them, for different convolution kernels of the same or different convolution layers of the deep segmentation network model, the convolution operation based on the deformation of the convolution kernel is performed on the processed image based on the above processing process. During the convolution operation, whenever the window of the convolution kernel is slid to a certain position, preferably, the gradient information of each pixel corresponding to the current window position can be used to guide the deformation of each sub-window in the window corresponding to the convolution kernel, and then the scanning operation of the position can be performed based on the deformed convolution kernel.

[0101] Since the objects to be segmented are generally irregular, there will be problems with image segmentation using standard convolution kernels such as rectangular convolution kernels, such as the presence of jagged edges on the segmented image. However, the irregularly shaped convolution kernels of the embodiments of the present application can extract feature information that is more consistent with the edges of the object. Furthermore, since the gradient information can reflect the pixel changes of the image, the convolution kernel is deformed based on the gradient information, so that the shape change of the convolution kernel can be consistent with the pixel changes of the image. In particular, for the image edges of irregular objects, the convolution kernel can produce a deformation that is consistent with the changes in the edge pixels of the irregular object image (such as the irregular shape of the image edge formed by the edge pixels of the irregular object image, which may present arc-shaped, wavy, jagged, sharp-angled changes, etc.). In this way, the feature information of the region of interest is utilized to complete the semantic segmentation task of the image as accurately and efficiently as possible.

[0102] In an optional implementation, the model structure of the existing deep segmentation network model (such as FCN, U-Net, PSPNet or DeepLab, etc.) can be improved, and a gradient calculation network layer can be added on the basis of the structure of these network models. Specifically, a gradient calculation network layer can be added on top of each convolutional layer.

[0103] Before each convolution operation, in order to weaken the problem of regularly selecting the receptive field in the standard convolution and limiting the accurate extraction of image features, first, the corresponding gradient calculation network layer can be used to perform gradient calculation on the image to be processed to obtain a gradient image including the gradient vectors of all pixels of the image to be processed. Then, the position offset of the next layer of convolution kernel is guided based on the gradient vector in the gradient image to deform the convolution kernel. The actual increase in the amount of calculation is not large, but the dynamic deformation of the convolution kernel during the scanning process of the convolution operation is realized. In the image segmentation task, the deformed convolution kernel that matches the edge features of the image can fully play its role at the edge position of the image, thereby achieving better image segmentation effect.

[0104] Corresponding to the above-mentioned image segmentation method, the embodiment of the present application further discloses an image segmentation device, such as Figure 8 As shown, the image segmentation device includes:

[0105] An acquisition unit 801 is configured to acquire a target image;

[0106] A determining unit 802 is configured to determine gradient information of the target image;

[0107] A deformation processing unit 803 is configured to perform deformation processing on a first convolution kernel of a current convolution layer of the target image to be processed in a deep segmentation network model based on the gradient information to obtain a deformed second convolution kernel;

[0108] The convolution processing unit 804 is configured to perform convolution processing on the target image using the second convolution kernel, and obtain a semantic segmentation result of the target image based on the convolution processing on the target image.

[0109] In an optional implementation of the embodiment of the present application, the acquiring unit 801 is specifically configured to:

[0110] In a case where the current convolutional layer is the first convolutional layer in the deep segmentation network model, obtaining an original image to be segmented;

[0111] In the case that the current convolutional layer is not the first convolutional layer in the deep segmentation network model, a corresponding feature image of the original image output by the previous convolutional layer adjacent to the current convolutional layer is obtained.

[0112] In an optional implementation of the embodiment of the present application, the determining unit 802 is specifically configured to:

[0113] Calculating the gradient vectors of all pixels of the target image to obtain a gradient image;

[0114] or,

[0115] Calculate the gradient vector of each pixel point of the current pixel area to be scanned corresponding to the first convolution kernel in the target image during the convolution operation.

[0116] In an optional implementation of the embodiment of the present application, the determining unit 802 calculates the gradient vectors of all pixels of the target image to obtain a gradient image, including:

[0117] The gradient calculation network layer provided above the current convolution layer in the depth segmentation network model is used to calculate the gradient vectors of all pixels of the target image to obtain a gradient image.

[0118] In an optional implementation of the embodiment of the present application, the deformation processing unit 803 is specifically configured to:

[0119] Extracting a gradient vector corresponding to each pixel point of the pixel area to be scanned from the gradient image, or obtaining a gradient vector of each pixel point of the determined pixel area to be scanned; wherein the gradient vector includes a gradient direction and a gradient magnitude;

[0120] Based on the gradient direction of each pixel point in the pixel area to be scanned, controlling the moving direction of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel;

[0121] Based on the gradient size of each pixel point in the pixel area to be scanned, controlling the movement distance of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel;

[0122] The first convolution kernel is deformed by moving each sub-window in the sliding window corresponding to the first convolution kernel.

[0123] In an optional implementation of the embodiment of the present application, the deformation processing unit 803 controls the movement direction of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel based on the gradient direction of each pixel point in the pixel area to be scanned, and controls the movement distance of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel based on the gradient size of each pixel point in the pixel area to be scanned, including:

[0124] Controlling each sub-window in the sliding window corresponding to the first convolution kernel to move along the gradient direction of the corresponding pixel point in the pixel area to be scanned, and moving along the gradient direction by a distance equal to the gradient magnitude of the corresponding pixel point;

[0125] The sub-windows that have completed the movement constitute the second convolution kernel.

[0126] In an optional implementation of the embodiment of the present application, the first convolution kernel is a standard convolution kernel of a predetermined size;

[0127] The deformation processing unit 803 is specifically used for:

[0128] During a complete convolution operation on the target image, the standard convolution kernel is used to slide on the target image according to a set scanning mode and sliding step size;

[0129] Moreover, whenever sliding to a pixel area to be scanned, the standard convolution kernel sliding to the pixel area to be scanned is deformed based on the gradient vector of each pixel point in the pixel area to be scanned to obtain a second convolution kernel.

[0130] In an optional implementation of the embodiment of the present application, the convolution processing unit 804 is specifically configured to:

[0131] Convolution processing is performed on the target image based on the current pixel positions of each sub-window in the second convolution kernel in the target image.

[0132] As for the image segmentation device disclosed in the embodiment of the present application, since it corresponds to the image segmentation method disclosed in the above embodiments, the description is relatively simple. For relevant similarities, please refer to the description of the image segmentation method part in the above embodiments, and no further details will be given here.

[0133] In addition, an embodiment of the present application also discloses an electronic device, which may be, but is not limited to, a personal PC such as a notebook, desktop, or all-in-one computer, or may also be a cloud server, a local area network server, or other device, which is not limited here.

[0134] like Figure 9 As shown, the electronic device at least includes:

[0135] Memory 901, used for storing computer instruction sets;

[0136] The computer instruction set may be implemented in the form of a computer program.

[0137] The memory 901 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0138] The processor 902 is configured to implement the image segmentation method provided in any of the above embodiments by executing the instruction set stored in the memory.

[0139] The processor 902 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices.

[0140] In addition, the above electronic device may further include components such as a communication interface and a communication bus. The memory, processor and communication interface communicate with each other via the communication bus.

[0141] Communication interfaces are used to connect electronic devices to other devices, such as other PCs or servers. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This bus can be divided into an address bus, a data bus, and a control bus.

[0142] In this embodiment, the processor in the electronic device proposes a deformable convolution kernel by executing a computer instruction set stored in the memory. The deformable convolution kernel is guided to deform by determining the gradient information of the image, and convolution processing is performed on the image based on the deformed convolution kernel. Since the gradient information of the image pixels can reflect the change status of the image pixels, for irregular objects, the pixel change status information of the irregular area is utilized to guide the deformable convolution kernel to perform adaptive deformation, so as to fully extract the image feature information of the irregular object, thereby achieving the purpose of accurately and efficiently completing the semantic segmentation task of the irregular object image.

[0143] At the same time, an embodiment of the present application also discloses a computer-readable storage medium, which stores a computer instruction set. When the computer instruction set is executed by a processor, the image segmentation method provided in any of the above embodiments is implemented.

[0144] When the instruction set stored in the above-mentioned computer-readable storage medium is run, a deformable convolution kernel idea is proposed. The deformable convolution kernel is guided to deform by determining the gradient information of the image, and the image is convolved based on the deformed convolution kernel. Since the gradient information of the image pixels can reflect the change status of the image pixels, for irregular objects, the pixel change status information of the irregular area is utilized to guide the deformable convolution kernel to perform adaptive deformation, so as to fully extract the image feature information of the irregular object, thereby achieving the purpose of accurately and efficiently completing the semantic segmentation task of the irregular object image.

[0145] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.

[0146] For the convenience of description, the above systems or devices are described as being divided into various modules or units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0147] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0148] Finally, it should be noted that, in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0149] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image segmentation method, characterized in that: include: Obtaining a target image; Determining gradient information of the target image; Based on the gradient information, deforming the first convolution kernel of the current convolution layer of the target image to be processed in the deep segmentation network model to obtain a deformed second convolution kernel; Performing convolution processing on the target image using the second convolution kernel, and obtaining a semantic segmentation result of the target image based on the convolution processing on the target image; Wherein, determining the gradient information of the target image includes: Calculating the gradient vectors of all pixels of the target image to obtain a gradient image; or, Calculating the gradient vector of each pixel point of the current pixel area to be scanned corresponding to the first convolution kernel in the target image during the convolution operation; The step of deforming the first convolution kernel of the current convolution layer of the target image to be processed in the deep segmentation network model based on the gradient information includes: Extracting a gradient vector corresponding to each pixel point of the pixel area to be scanned from the gradient image, or obtaining a gradient vector of each pixel point of the determined pixel area to be scanned; wherein the gradient vector includes a gradient direction and a gradient magnitude; Based on the gradient direction of each pixel point in the pixel area to be scanned, controlling the moving direction of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel; Based on the gradient size of each pixel point in the pixel area to be scanned, the moving distance of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel is controlled.

2. The method according to claim 1, characterized in that The obtaining of the target image comprises: If the current convolutional layer is the first convolutional layer in the deep segmentation network model, obtaining an original image to be segmented; If the current convolutional layer is not the first convolutional layer in the depth segmentation network model, obtain a corresponding feature image of the original image output by the previous convolutional layer adjacent to the current convolutional layer.

3. The method according to claim 1, characterized in that Calculating the gradient vectors of all pixels of the target image to obtain a gradient image includes: The gradient calculation network layer provided above the current convolution layer in the depth segmentation network model is used to calculate the gradient vectors of all pixels of the target image to obtain a gradient image.

4. The method according to claim 1, wherein The method includes: controlling the movement direction of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel based on the gradient direction of each pixel point in the pixel area to be scanned; and controlling the movement distance of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel based on the gradient size of each pixel point in the pixel area to be scanned, including: Controlling each sub-window in the sliding window corresponding to the first convolution kernel to move along the gradient direction of the corresponding pixel point in the pixel area to be scanned, and moving along the gradient direction by a distance equal to the gradient magnitude of the corresponding pixel point; The sub-windows that have completed the movement constitute the second convolution kernel.

5. The method according to claim 1, wherein The first convolution kernel is a standard convolution kernel of a predetermined size; The method further comprises: deforming a first convolution kernel of a current convolution layer of the target image to be processed in a deep segmentation network model based on the gradient information to obtain a deformed second convolution kernel, including: During a complete convolution operation on the target image, the standard convolution kernel is used to slide on the target image according to a set scanning mode and sliding step size; Moreover, whenever sliding to a pixel area to be scanned, the standard convolution kernel sliding to the pixel area to be scanned is deformed based on the gradient vector of each pixel point of the pixel area to be scanned to obtain a second convolution kernel.

6. The method according to claim 5, characterized in that The performing convolution processing on the target image using the second convolution kernel includes: Convolution processing is performed on the target image based on the current pixel positions of each sub-window in the second convolution kernel in the target image.

7. An image segmentation device, characterized in that: include: an acquisition unit, configured to obtain a target image; a determining unit, configured to determine gradient information of the target image; A deformation processing unit, configured to perform deformation processing on a first convolution kernel of a current convolution layer of the target image to be processed in a deep segmentation network model based on the gradient information to obtain a deformed second convolution kernel; a convolution processing unit, configured to perform convolution processing on the target image using the second convolution kernel, and obtain a semantic segmentation result of the target image based on the convolution processing on the target image; The determining unit is used to determine the gradient information of the target image, specifically including: Calculating the gradient vectors of all pixels of the target image to obtain a gradient image; or, Calculating the gradient vector of each pixel point of the current pixel area to be scanned corresponding to the first convolution kernel in the target image during the convolution operation; The deformation processing unit is used to perform deformation processing on the first convolution kernel of the current convolution layer of the target image to be processed in the deep segmentation network model based on the gradient information, specifically including: Extracting a gradient vector corresponding to each pixel point of the pixel area to be scanned from the gradient image, or obtaining a gradient vector of each pixel point of the determined pixel area to be scanned; wherein the gradient vector includes a gradient direction and a gradient magnitude; Based on the gradient direction of each pixel point in the pixel area to be scanned, controlling the moving direction of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel; Based on the gradient size of each pixel point in the pixel area to be scanned, the moving distance of each sub-window corresponding to the position of each pixel point in the sliding window corresponding to the first convolution kernel is controlled.

8. An electronic device, characterized in that: include: Memory, used to store computer instruction sets; A processor, configured to implement the image segmentation method according to any one of claims 1 to 6 by executing an instruction set stored in the memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer instruction set, and when the computer instruction set is executed by a processor, the image segmentation method according to any one of claims 1 to 6 is implemented.

Images