Image segmentation method, device and server

Abstract

The present invention provides an image segmentation method, device and server, wherein the method comprises: obtaining an image to be processed and auxiliary information of the image to be processed; wherein the auxiliary information comprises edge information and / or corner information of the image to be processed; inputting the image to be processed and the auxiliary information into a pre-trained image segmentation model, and outputting a segmentation result of the image to be processed; wherein the image segmentation model is trained based on a preset sample image and the auxiliary information of the sample image. In this method, when the image to be processed is segmented by the image segmentation model, the information input into the image segmentation model includes not only the image to be processed, but also auxiliary information such as edge information and / or corner information of the image to be processed. Similarly, the image segmentation model is also trained based on the sample image and the auxiliary information of the sample image during the training process; since the auxiliary information of the image is taken into consideration, the segmentation accuracy of the image segmentation can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image segmentation method, device and server. Background Art

[0002] In the related art, the image segmentation method usually segments the image to be segmented by threshold comparison. This method is simple to operate but cannot obtain the semantic information of the image to be segmented, resulting in poor accuracy of the image segmentation result. In order to obtain the semantic information of the image to be segmented, the related art also adopts a method of using a neural network to segment the image to be segmented. This method requires training the neural network with images in the data set so as to use the trained neural network to segment the image to be segmented. Since the texture information of the image is relatively complex, this method is difficult to perform fine segmentation of the image, which affects the accuracy of image segmentation. Summary of the invention

[0003] The object of the present invention is to provide an image segmentation method, device and server to improve the accuracy of image segmentation.

[0004] In a first aspect, an embodiment of the present invention provides an image segmentation method, the method comprising: obtaining an image to be processed and auxiliary information of the image to be processed; wherein the auxiliary information comprises edge information and / or corner point information of the image to be processed; inputting the image to be processed and the auxiliary information into a pre-trained image segmentation model, and outputting a segmentation result of the image to be processed; wherein the image segmentation model is trained based on a preset sample image and the auxiliary information of the sample image.

[0005] In an optional embodiment, the above-mentioned image segmentation model is specifically trained in the following manner: determining a target sample based on a preset sample set; the sample set includes multiple samples, each of which includes a sample image, and auxiliary information corresponding to the sample image and a standard segmentation result; inputting the sample image in the target sample and the auxiliary information of the sample image into a preset initial model; wherein the initial model includes a feature extraction layer and an overlay layer; through the feature extraction layer, extracting features from the sum image composed of the sample image and the auxiliary information of the sample image to obtain a first feature map; sampling the auxiliary information of the sample image through the overlay layer, and obtaining a second feature map corresponding to the auxiliary information of the sample image based on the sampling result; superimposing the first feature map and the second feature map to obtain a superimposed feature map; calculating a loss value based on the superimposed feature map and the standard segmentation result, and training the initial model based on the loss value; continuing to execute the step of determining the sample image based on the preset sample set until the loss value converges to obtain the image segmentation model.

[0006] In an optional embodiment, the step of sampling the auxiliary information of the sample image through an overlay layer and obtaining a second feature map corresponding to the auxiliary information of the sample image based on the sampling result includes: downsampling the auxiliary information of the sample image through an overlay layer according to the scale of the first feature map to obtain a sampling result; copying multiple copies of the sampling results, overlaying the multiple sampling results, and obtaining a second feature map corresponding to the auxiliary information of the sample image; wherein the second feature map has the same scale as the first feature map.

[0007] In an optional embodiment, if the auxiliary information of the above-mentioned image to be processed includes edge information of the image to be processed, the edge information of the image to be processed is obtained in the following manner: calculating the gradient value of each pixel point in the image to be processed; for each pixel point, determining whether the gradient value of the pixel point meets a preset condition; if the preset condition is met, determining the pixel point as an edge point; and combining each edge point to obtain edge information.

[0008] In an optional embodiment, if the auxiliary information of the image to be processed includes corner point information of the image to be processed, the corner point information of the image to be processed is obtained in the following manner: calculating the first derivative of the image to be processed in the horizontal direction and the second derivative in the vertical direction; obtaining an outer product image based on the first derivative and the second derivative; calculating the corner point response value of each pixel in the outer product image through a preset corner point response function; performing non-maximum suppression on the corner point response values ​​to obtain the corner point values ​​of the image to be processed.

[0009] In a second aspect, an embodiment of the present invention provides an image segmentation device, which includes: an information acquisition module, used to acquire an image to be processed, and auxiliary information of the image to be processed; wherein the auxiliary information includes edge information and / or corner point information of the image to be processed; an image processing module, used to input the image to be processed and the auxiliary information into a pre-trained image segmentation model, and output a segmentation result of the image to be processed; wherein the image segmentation model is trained based on a preset sample image and the auxiliary information of the sample image.

[0010] In an optional embodiment, the above-mentioned device also includes a model training module, including: a sample determination module, which is used to determine the target sample based on a preset sample set; the sample set includes multiple samples, each sample contains a sample image, and auxiliary information corresponding to the sample image and a standard segmentation result; a sample input module, which is used to input the sample image and the auxiliary information of the sample image in the target sample into a preset initial model; wherein the initial model includes a feature extraction layer and an overlay layer; the feature extraction module, which is used to extract features from the sum image composed of the sample image and the auxiliary information of the sample image through the feature extraction layer to obtain a first feature map; a feature overlay module, which is used to sample the auxiliary information of the sample image through the overlay layer, and obtain a second feature map corresponding to the auxiliary information of the sample image based on the sampling result; the first feature map and the second feature map are superimposed to obtain a superimposed feature map; a model determination module, which is used to calculate the loss value according to the superimposed feature map and the standard segmentation result, and train the initial model according to the loss value; continue to perform the step of determining the sample image based on the preset sample set until the loss value converges to obtain the image segmentation model.

[0011] In an optional embodiment, the feature overlay module is further used to: downsample the auxiliary information of the sample image through an overlay layer according to the scale of the first feature map to obtain a sampling result; copy multiple copies of the sampling results, overlay the multiple copies of the sampling results to obtain a second feature map corresponding to the auxiliary information of the sample image; wherein the second feature map has the same scale as the first feature map.

[0012] In an optional embodiment, if the auxiliary information of the image to be processed includes edge information of the image to be processed, the above-mentioned device also includes an edge information determination module, which is used to: calculate the gradient value of each pixel point in the image to be processed; for each pixel point, determine whether the gradient value of the pixel point meets a preset condition; if the preset condition is met, determine the pixel point as an edge point; and combine each edge point to obtain edge information.

[0013] In an optional embodiment, if the auxiliary information of the image to be processed includes corner point information of the image to be processed, the above-mentioned device also includes a corner point information determination module, which is used to: calculate the first derivative of the image to be processed in the horizontal direction and the second derivative in the vertical direction; obtain an outer product image based on the first derivative and the second derivative; calculate the corner point response value of each pixel point in the outer product image through a preset corner point response function; perform non-maximum suppression on the corner point response value to obtain the corner point of the image to be processed.

[0014] In a third aspect, an embodiment of the present invention provides a server, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the image segmentation method described in any one of the aforementioned embodiments.

[0015] In a fourth aspect, an embodiment of the present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the image segmentation method described in any one of the aforementioned implementations.

[0016] The embodiments of the present invention bring the following beneficial effects:

[0017] An image segmentation method, device and server provided by an embodiment of the present invention first obtain an image to be processed and auxiliary information of the image to be processed; wherein the auxiliary information includes edge information and / or corner information of the image to be processed; the image to be processed and the auxiliary information are input into a pre-trained image segmentation model, and a segmentation result of the image to be processed is output; wherein the image segmentation model is trained based on a preset sample image and the auxiliary information of the sample image. In this method, when the image to be processed is segmented by the image segmentation model, the information input into the image segmentation model includes not only the image to be processed, but also auxiliary information such as edge information and / or corner information of the image to be processed. Similarly, the image segmentation model is also trained based on the sample image and the auxiliary information of the sample image during the training process; since the auxiliary information of the image is taken into consideration, the segmentation accuracy of the image segmentation can be improved.

[0018] Other features and advantages of the present invention will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present invention.

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are specifically listed below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flowchart of an image segmentation method provided by an embodiment of the present invention;

[0022] Figure 2 A flowchart of another image segmentation method provided by an embodiment of the present invention;

[0023] Figure 3A flowchart of another image segmentation method provided by an embodiment of the present invention;

[0024] Figure 4 A flowchart of another image segmentation method provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of the structure of an image segmentation device provided by an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Taking into account the problem of poor image segmentation accuracy of image segmentation methods in related technologies, an embodiment of the present invention provides an image segmentation method, device and server, which can be applied to image segmentation of various images and target contour recognition scenarios in images.

[0030] In order to facilitate understanding of the embodiment of the present invention, an image segmentation method disclosed in the embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following specific steps:

[0031] Step S102, obtaining an image to be processed and auxiliary information of the image to be processed; wherein the auxiliary information includes edge information and / or corner point information of the image to be processed.

[0032] The above-mentioned image to be processed can be an image taken by a camera, or a certain video frame in a video. The auxiliary information of the image to be processed can include only edge information, only corner information, or both edge information and corner information. The edge information is also the contour information of the image, which can refer to a set of points where the grayscale value of the image is discontinuous or the grayscale value changes dramatically, and can also be understood as the boundary between two uniform image areas with different grayscales. Usually, the grayscale changes along the edge direction are relatively gentle, while the grayscale changes in the normal direction of the edge are relatively drastic; the corner point information is also the inflection point information of the image contour, which can also be called a corner point. Usually, at the corner point, the first-order derivative is the largest, the second-order derivative is zero, and the gradient value and gradient direction at the corner point in the image are very fast.

[0033] In specific implementation, if the auxiliary information includes edge information, edge detection can be performed on the image to be processed to obtain edge information of the image to be processed, for example, edge detection can be performed on the image to be processed using a canny edge detection operator, a Laplace operator, a second-order differential operator, etc.; if the auxiliary information includes corner point information, corner point detection can be performed on the image to be processed to obtain corner point information of the image to be processed, for example, using a Harris corner point detection algorithm, a Moravec corner point detection algorithm, etc. If the auxiliary information includes both edge information and corner point information, edge detection and corner point detection can be performed on the image to be processed respectively to obtain edge information and corner point information.

[0034] Step S104, inputting the above-mentioned image to be processed and auxiliary information into a pre-trained image segmentation model, and outputting the segmentation result of the image to be processed; wherein the image segmentation model is trained based on a preset sample image and the auxiliary information of the sample image.

[0035] The above-mentioned image segmentation model is usually a deep learning model, a neural network model, etc. In the process of training the image segmentation model, it is necessary to select a sample image from a preset sample set, and perform edge detection or corner detection on the sample image to obtain auxiliary information corresponding to the sample image, and then input the sample image and its auxiliary information into the image segmentation model, output the segmentation result of the sample image, and then obtain the loss value of the segmentation result based on the loss function, and continue to perform the step of selecting a sample image from the preset sample set until the loss value converges to obtain the trained image segmentation model; after the training is completed, if the image to be processed and the auxiliary information of the image to be processed are obtained, the image to be processed and the auxiliary information are input into the trained image segmentation model to obtain the segmentation result of the image to be processed.

[0036] An image segmentation method provided by an embodiment of the present invention first obtains an image to be processed and auxiliary information of the image to be processed; wherein the auxiliary information includes edge information and / or corner information of the image to be processed; the image to be processed and the auxiliary information are input into a pre-trained image segmentation model, and a segmentation result of the image to be processed is output; wherein the image segmentation model is trained based on a preset sample image and the auxiliary information of the sample image. In this method, when the image to be processed is segmented by the image segmentation model, the information input into the image segmentation model includes not only the image to be processed, but also auxiliary information such as edge information and / or corner information of the image to be processed. Similarly, the image segmentation model is also trained based on the sample image and the auxiliary information of the sample image during the training process; since the auxiliary information of the image is taken into consideration, the segmentation accuracy of the image segmentation can be improved.

[0037] The embodiment of the present invention also provides another image segmentation method, which is implemented on the basis of the method of the above embodiment; the method focuses on describing the specific process of training the image segmentation model before obtaining the image to be processed (implemented by the following steps S202-S210); Figure 2 As shown, the method comprises the following specific steps:

[0038] Step S202, determining a target sample based on a preset sample set; the sample set includes a plurality of samples, each sample includes a sample image, and auxiliary information corresponding to the sample image and a standard segmentation result.

[0039] The preset sample set may include a large number of samples, each of which includes a sample image, as well as auxiliary information corresponding to the sample image and a standard segmentation result, and the auxiliary information is usually in the form of a picture. The sample image may be a pre-collected image of various scenes, which may include people, scenery, animals, vehicles, buildings, etc.; wherein the auxiliary information corresponding to each sample image may include only edge information or corner information, or may include both edge information and corner information. The edge information may be in the form of an edge image, which includes the edge information in the sample image; the corner information may be in the form of a corner image, which includes the corner information in the sample image. The target sample may be a sample randomly determined from multiple samples in the sample set.

[0040] Step S204, inputting the sample image in the target sample and the auxiliary information of the sample image into a preset initial model; wherein the initial model includes a feature extraction layer and an overlay layer.

[0041] The above initial model includes a feature extraction layer and an overlay layer; the feature extraction layer is used to extract features from the sample image and the image with its auxiliary information overlaid thereon, and the overlay layer is used to overlay the auxiliary information with the image after feature extraction.

[0042] In a specific implementation, the initial model may include multiple feature extraction layers and overlay layers connected in sequence to achieve better network training results, that is, the output result of the previous overlay layer will be input into the next feature extraction layer, and then processed by the next overlay layer, and the cycle will continue until there is no feature extraction layer connected to the next overlay layer.

[0043] Step S206: Perform feature extraction processing on the sum image consisting of the sample image and the auxiliary information of the sample image through the feature extraction layer to obtain a first feature map.

[0044] When the feature extraction layer receives a sample image and auxiliary information of the sample image, the auxiliary information of the sample image is first superimposed on the sample image to obtain a sum image, and then the image features of the sum image are extracted to obtain high-level semantic information of the image features. In order to improve the performance of the feature extraction layer, the feature extraction layer usually includes a pooling layer and multiple groups of sequentially connected convolutional layers, batch normalization layers, and activation function layers.

[0045] The convolution layer in the feature extraction layer is used to perform convolution processing on the sample image and the sum image of the auxiliary information combination of the sample image to obtain the feature image; the batch normalization layer is used to normalize the feature image output by the convolution layer. This process can speed up the convergence speed of the feature extraction layer and the model, and can alleviate the problem of gradient diffusion in the multi-layer convolution network, making the feature extraction layer more stable. The activation function layer in the feature extraction layer can perform function transformation on the normalized image features. This transformation process can break the linear combination of the convolution layer input. The activation function layer can specifically be a Sigmoid function, a tanh function, a Relu function, etc.; the pooling layer can perform a global average pooling operation on the image features obtained by the activation function layer to obtain the global features of the input image (equivalent to the first feature map mentioned above).

[0046] Step S208, sampling the auxiliary information of the sample image through the superposition layer, and obtaining a second feature map corresponding to the auxiliary information of the sample image based on the sampling result; superimposing the first feature map and the second feature map to obtain a superimposed feature map.

[0047] In a specific implementation, in order to make the size of the edge image or corner image corresponding to the auxiliary information of the sample image consistent with the size of the first feature map output by the feature extraction layer, it is necessary to perform sampling processing on the auxiliary information to obtain a sampling result, and then set a preset number of sampling results to obtain a second feature map, and the second feature map may store at least one sampling result. Specifically, the above step S208 can be implemented by the following steps 10-11:

[0048] Step 10: Down-sample the auxiliary information of the sample image through an overlay layer according to the scale of the first feature map to obtain a sampling result.

[0049] In a specific implementation, since the size (equivalent to the scale) of the feature image will be halved after being processed by the pooling layer, it is necessary to downsample the auxiliary information so that the obtained sampling result has the same size as the first feature map.

[0050] Step 11, copy multiple copies of the above sampling results, and perform superposition processing on the multiple sampling results to obtain a second feature map corresponding to the auxiliary information of the sample image; wherein the second feature map has the same scale as the first feature map.

[0051] Copying multiple sampling results can also be understood as copying a sampling result on each added channel or directly copying multiple sampling results to obtain multiple copies (the number can be set according to research and development needs, such as half the number of images output by the feature extraction layer) of the sampling results, and then superimposing the multiple sampling results to obtain a second feature map, and then superimposing the first feature map and the second feature map to obtain a superimposed feature map.

[0052] In specific implementation, the concat feature fusion method can be used to obtain multiple sampling results by copying, and the multiple sampling results are feature fused (also called superposition) with the second feature map. The concat feature fusion method usually increases the number of channels, that is, the number of channels describing the image itself (each channel has sampling results) increases, while the feature information under each channel does not increase. The increased number of channels, that is, the multiple copied sampling results, are superimposed to obtain the first feature map, and then the first feature map is fused with the second feature map to obtain the superimposed feature map.

[0053] Step S210, calculate the loss value according to the above-mentioned superimposed feature map and the standard segmentation result, and train the initial model according to the loss value; continue to execute the step of determining the sample image based on the preset sample set until the loss value converges to obtain the image segmentation model.

[0054] According to the difference between the superimposed feature map and the standard segmentation result, the loss value can be obtained. Generally, the larger the difference, the larger the loss value, and the smaller the difference, the smaller the loss value. In specific implementation, the loss value can be calculated by a preset loss function, or the difference between the superimposed feature map and the standard segmentation result can be directly calculated. The preset loss function can be a cross entropy loss function, a mean loss function, a square difference loss function, etc.

[0055] In the specific implementation, the image segmentation model can be obtained through the following steps 20-23:

[0056] Step 20: Calculate the derivative of the loss value with respect to the parameters to be updated in the initial model Where L1 is the loss value; W is the parameter to be updated; the parameter to be updated can be all the parameters in the initial model, or some parameters randomly determined from the initial model; the updated whole parameter can also be called the weight of each layer of the network in the model. Usually, the derivative of the parameter to be updated can be solved according to the back propagation algorithm; if the loss value is large, it means that the output of the current initial model does not match the expected output result, then the derivative of the above loss value to the parameter to be updated in the attribute recognition model is calculated, and the derivative can be used as the basis for updating the parameter to be updated.

[0057] Step 21: Update the parameters to be updated to obtain the updated parameters to be updated Among them, α is the preset coefficient. This process can also be called the stochastic gradient descent algorithm; the derivative of each parameter to be updated can also be understood as the direction in which the loss value decreases fastest relative to the current parameter. By adjusting the parameter in this direction, the loss value can be quickly reduced and the parameter can be converged.

[0058] Step 22, determining whether the parameters of the updated initial model have all converged, if so, executing the step of determining a sample image based on a preset sample set; otherwise, executing step S23.

[0059] If the parameters of the updated initial model are not all converged, the step of determining the sample image based on the preset sample set is continued until the parameters of the updated initial model are all converged, that is, the loss value converges.

[0060] Step 23, determining the initial model after parameter update as the trained image segmentation model.

[0061] Step S212, obtaining the image to be processed and auxiliary information of the image to be processed.

[0062] Step S214: input the above-mentioned image to be processed and auxiliary information into the trained image segmentation model, and output the segmentation result of the image to be processed.

[0063] The above-mentioned image segmentation method first determines the target sample based on a preset sample set; then the sample image in the target sample and its auxiliary information are input into the initial model, and the initial model includes a feature extraction layer and an overlay layer; then the feature extraction layer performs feature extraction on the sum image composed of the sample image and its auxiliary information to obtain a first feature map; then the overlay layer performs sampling processing on the auxiliary information of the sample image, and obtains a second feature map corresponding to the auxiliary information of the sample image based on the sampling result; the above-mentioned first feature map and the second feature map are superimposed to obtain a superimposed feature map; then the loss value is calculated based on the superimposed feature map and the standard segmentation result, and the initial model is trained based on the loss value to obtain an image segmentation model; finally, the image to be processed and its auxiliary information are input into the trained image segmentation model, and the segmentation result of the image to be processed is output. The image segmentation model trained in this way can retain auxiliary information as much as possible during the training process, thereby avoiding the loss of image feature information during the image segmentation process, so as to obtain an accurate image segmentation result.

[0064] The embodiment of the present invention further provides another image segmentation method, which is implemented on the basis of the method of the above embodiment; the method focuses on describing the specific process of determining the edge information of the image to be processed if the auxiliary information of the image to be processed includes the edge information of the image to be processed (implemented by the following steps S304-S308); Figure 3 As shown, the method comprises the following specific steps:

[0065] Step S302, obtaining an image to be processed.

[0066] Step S304, calculating the gradient value of each pixel in the above-mentioned image to be processed.

[0067] In a specific implementation, the edge of an image can point to multiple directions. Therefore, the gradient values ​​of each pixel in the image to be processed in the horizontal direction, vertical direction, left diagonal direction, and right diagonal direction can be calculated; then the gradient values ​​of each pixel in the horizontal direction, vertical direction, left diagonal direction, and right diagonal direction are integrated to obtain the final gradient value corresponding to each pixel. For example, the gradient value in each direction is regarded as a vector, and the final gradient value can be obtained by adding the vectors.

[0068] When there is a lot of noise in the image, the result obtained by edge detection will be very poor. Therefore, before step S304, the image to be processed needs to be denoised. There are two common noises in the image: Gaussian noise and salt and pepper noise. Usually, Gaussian filtering and median filtering can be used for image denoising, or the image can be Gaussian blurred by the canny operator to reduce image noise, thereby reducing pseudo edge information in the image.

[0069] Step S306: for each pixel point, determine whether the gradient value of the pixel point meets a preset condition; if the preset condition is met, determine the pixel point as an edge point.

[0070] The above preset condition may be that the gradient value is greater than a preset threshold, or that the gradient value is within a preset range. In specific implementation, for each pixel point, the above step S306 may be implemented by the following steps 30-34:

[0071] Step 30, determining whether the current gradient value is greater than or equal to a first preset threshold; if it is greater than or equal to the first preset threshold, executing step 31; otherwise, executing step 32.

[0072] Step 31: determine the pixel point corresponding to the current gradient value as an edge point.

[0073] Step 32, determining whether the current gradient value is greater than or equal to a second preset threshold; if it is greater than or equal to the second preset threshold, executing step 33; otherwise, executing step 34.

[0074] Step 33, detecting whether the connected domain of the candidate node corresponding to the current node contains edge points; if it contains edge points, executing step 31; otherwise, executing step 34.

[0075] Step 34: Determine the pixel point corresponding to the current gradient value as a non-edge point. Usually, it is necessary to suppress the non-edge points and retain the edge points.

[0076] The first preset threshold is usually greater than the second preset threshold. Usually, the pixel points greater than or equal to the first preset threshold can be marked as strong edge points, which are also true edge points; the pixel points less than the first preset threshold and greater than the second preset threshold are marked as weak edge points; the pixel points less than the second preset threshold are suppressed; the weak edge point may be an edge point, or it may be a weak edge point caused by noise or color change. In order to obtain accurate results, it is necessary to remove the weak edge points caused by the latter. Usually, the weak edge points and strong edge points caused by the real edge are connected, so the hysteresis boundary tracking algorithm can be used to check whether there is a strong edge point in the 8 connected domains of a weak edge point. If there is a strong edge point, the weak edge point is determined to be a true edge point and the pixel point is retained.

[0077] In specific implementation, since the edge gradient values ​​of the image are relatively coarse and bright, it is usually necessary to refine the edges of the gradient values ​​in the image before determining whether the gradient values ​​of the pixel points meet the preset conditions. For example, the non-maximum suppression method can be used to retain the local maximum gradient value and suppress other gradient values, so as to better implement subsequent operations.

[0078] Step S308: Combine each of the above edge points to obtain edge information.

[0079] Step S310: input the above-mentioned image to be processed and the edge information of the image to be processed into a pre-trained image segmentation model, and output the segmentation result of the image to be processed.

[0080] The above-mentioned image segmentation method first performs edge detection on the acquired image to be processed to obtain edge information of the image to be processed, and then inputs the image to be processed and its edge information into a pre-trained image segmentation model to obtain an image segmentation result. This method can protect the edges in the image to be processed from loss to a great extent and filter non-edge information to a great extent, thereby improving the accuracy of image segmentation.

[0081] The embodiment of the present invention further provides another image segmentation method, which is implemented on the basis of the method of the above embodiment; the method focuses on describing the specific process of determining the corner point information of the image to be processed if the auxiliary information of the image to be processed includes the edge corner point information of the image to be processed (implemented by the following steps S404-S410); Figure 4 As shown, the method comprises the following specific steps:

[0082] Step S402, obtaining an image to be processed.

[0083] Step S404, calculating the first derivative in the horizontal direction and the second derivative in the vertical direction of the above-mentioned image to be processed.

[0084] In a specific implementation, the image to be processed can be convolved with the derivative of the Gaussian function to obtain the first derivative Ix in the horizontal direction and the second derivative Iy in the vertical direction of the image, wherein the mean and variance in the Gaussian function can be set manually. The setting of these two values ​​will affect the accuracy of corner point detection and the number of corner points detected. The mean is generally set to zero. Generally, the larger the variance, the flatter the function and the less accurate the detected corner points. The smaller the variance, the steeper the function and the fewer the detected corner points.

[0085] Step S406: Obtain an outer product image according to the first derivative and the second derivative.

[0086] By calculating the product of the first derivative in the horizontal direction and the second derivative in the vertical direction, an outer product image can be obtained. In a specific implementation, the outer product image can be realized by the following steps 40-42:

[0087] Step 40, calculate the product of the first derivative and itself to obtain a first image; calculate the product of the second derivative and itself to obtain a second image; calculate the product of the first derivative and the second derivative to obtain a third image.

[0088] Step 41: Perform convolution filtering on the first image, the second image and the third image using a Gaussian function.

[0089] Step 42, weighted summing is performed on the first image, the second image and the third image after convolution filtering to obtain an outer product image, wherein the weight corresponding to each image can be set according to research and development requirements.

[0090] Step S408: Calculate the corner point response value of each pixel in the outer product image by using a preset corner point response function.

[0091] The above preset corner point response function can be set according to research and development requirements. Usually, the corner point response value at the corner point is usually a large positive value, the corner point response value in the flat area of ​​the image is very small, and the corner point response value in the edge area is a large negative value.

[0092] Step S410: performing non-maximum suppression on the corner point response values ​​to obtain corner points of the image to be processed.

[0093] By performing non-maximum suppression on the above corner point response values, a larger corner point response value can be obtained. The pixel point corresponding to the corner point response value is the corner point. By combining all the corner points, the corner point information of the image to be processed can be obtained.

[0094] Step S412: input the above-mentioned image to be processed and the corner point information of the image to be processed into a pre-trained image segmentation model, and output the segmentation result of the image to be processed.

[0095] The above-mentioned image segmentation method first performs corner point detection on the acquired image to be processed to obtain the corner point information of the image to be processed, and then inputs the image to be processed and its corner point information into a pre-trained image segmentation model to obtain the image segmentation result. This method can greatly protect the edges in the image to be processed from loss, thereby improving the accuracy of image segmentation.

[0096] Corresponding to the above method embodiment, the embodiment of the present invention provides an image segmentation device, such as Figure 5 As shown, the device comprises:

[0097] The information acquisition module 50 is used to acquire the image to be processed and auxiliary information of the image to be processed; wherein the auxiliary information includes edge information and / or corner point information of the image to be processed.

[0098] The image processing module 51 is used to input the above-mentioned image to be processed and auxiliary information into a pre-trained image segmentation model, and output the segmentation result of the image to be processed; wherein the image segmentation model is trained based on a preset sample image and the auxiliary information of the sample image.

[0099] The above-mentioned image segmentation device first obtains the image to be processed and the auxiliary information of the image to be processed; wherein the auxiliary information includes the edge information and / or corner information of the image to be processed; the image to be processed and the auxiliary information are input into a pre-trained image segmentation model, and the segmentation result of the image to be processed is output; wherein the image segmentation model is trained based on a preset sample image and the auxiliary information of the sample image. In this method, when the image to be processed is segmented by the image segmentation model, the information input into the image segmentation model includes not only the image to be processed, but also the auxiliary information such as the edge information and / or corner information of the image to be processed. Similarly, the image segmentation model is also trained based on the sample image and the auxiliary information of the sample image during the training process; since the auxiliary information of the image is taken into consideration, the segmentation accuracy of the image segmentation can be improved.

[0100] Furthermore, the above-mentioned device also includes a model training module, including: a sample determination module, which is used to determine the target sample based on a preset sample set; the sample set includes multiple samples, each sample contains a sample image, and auxiliary information corresponding to the sample image and a standard segmentation result; a sample input module, which is used to input the sample image in the target sample and the auxiliary information of the sample image into a preset initial model; wherein the initial model includes a feature extraction layer and an overlay layer; the feature extraction module, which is used to extract features from the sum image composed of the sample image and the auxiliary information of the sample image through the feature extraction layer to obtain a first feature map; the feature overlay module, which is used to sample the auxiliary information of the sample image through the overlay layer, and obtain a second feature map corresponding to the auxiliary information of the sample image based on the sampling result; the first feature map and the second feature map are superimposed to obtain a superimposed feature map; the model determination module, which is used to calculate the loss value according to the superimposed feature map and the standard segmentation result, and train the initial model according to the loss value; continue to execute the step of determining the sample image based on the preset sample set until the loss value converges to obtain the image segmentation model.

[0101] Furthermore, the feature superposition module is also used to: downsample the auxiliary information of the sample image through the superposition layer according to the scale of the first feature map to obtain a sampling result; copy multiple sampling results, superimpose the multiple sampling results to obtain a second feature map corresponding to the auxiliary information of the sample image; wherein the second feature map has the same scale as the first feature map.

[0102] In a specific implementation, if the auxiliary information of the image to be processed includes edge information of the image to be processed, the device also includes an edge information determination module, which is used to: calculate the gradient value of each pixel in the image to be processed; for each pixel, determine whether the gradient value of the pixel meets a preset condition; if the preset condition is met, determine the pixel as an edge point; and combine each edge point to obtain edge information.

[0103] If the auxiliary information of the above-mentioned image to be processed includes the corner point information of the image to be processed, the above-mentioned device also includes a corner point information determination module, which is used to: calculate the first derivative of the image to be processed in the horizontal direction and the second derivative in the vertical direction; obtain the outer product image according to the first derivative and the second derivative; calculate the corner point response value of each pixel point in the outer product image through a preset corner point response function; perform non-maximum suppression on the corner point response value to obtain the corner point value of the image to be processed.

[0104] The image segmentation device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0105] The embodiment of the present invention also provides a server for running the above image segmentation method, see Figure 6 As shown, the server includes a processor 101 and a memory 100. The memory 100 stores machine executable instructions that can be executed by the processor 101. The processor 101 executes the machine executable instructions to implement the above-mentioned image segmentation method.

[0106] Further, Figure 6 The server shown also includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .

[0107] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0108] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and completes the steps of the method of the above embodiment in combination with its hardware.

[0109] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned image segmentation method. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0110] The computer program products of the image segmentation method, device and server provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.

[0111] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0112] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An image segmentation method, characterized in that: The method comprises: Acquire an image to be processed and auxiliary information of the image to be processed; wherein the auxiliary information includes edge information and / or corner point information of the image to be processed; Inputting the image to be processed and the auxiliary information into a pre-trained image segmentation model, and outputting a segmentation result of the image to be processed; wherein the image segmentation model is trained based on a preset sample image and the auxiliary information of the sample image; The image segmentation model is specifically trained in the following manner: determining a target sample based on a preset sample set; the sample set includes a plurality of samples, each of which includes a sample image, and auxiliary information corresponding to the sample image and a standard segmentation result; inputting the sample image in the target sample and the auxiliary information of the sample image into a preset initial model; wherein the initial model includes a feature extraction layer and an overlay layer; performing feature extraction on a sum image composed of the sample image and the auxiliary information of the sample image through the feature extraction layer to obtain a first feature map; performing sampling processing on the auxiliary information of the sample image through the overlay layer, and obtaining a second feature map corresponding to the auxiliary information of the sample image based on the sampling result; overlaying the first feature map and the second feature map to obtain a superimposed feature map; calculating a loss value based on the superimposed feature map and the standard segmentation result, and training the initial model based on the loss value; continuing to perform the step of determining the sample image based on the preset sample set until the loss value converges to obtain the image segmentation model; The steps of sampling the auxiliary information of the sample image through the superimposed layer and obtaining a second feature map corresponding to the auxiliary information of the sample image based on the sampling result include: down-sampling the auxiliary information of the sample image through the superimposed layer according to the scale of the first feature map to obtain a sampling result; copying multiple copies of the sampling result, superimposing the multiple copies of the sampling results, and obtaining a second feature map corresponding to the auxiliary information of the sample image; wherein the second feature map has the same scale as the first feature map.

2. The method according to claim 1, characterized in that If the auxiliary information of the image to be processed includes edge information of the image to be processed, the edge information of the image to be processed is obtained in the following manner: Calculating the gradient value of each pixel in the image to be processed; For each pixel point, determining whether the gradient value of the pixel point meets a preset condition; if the preset condition is met, determining the pixel point as an edge point; Each of the edge points is combined to obtain edge information.

3. The method according to claim 1, characterized in that If the auxiliary information of the image to be processed includes the corner point information of the image to be processed, the corner point information of the image to be processed is obtained by: Calculating a first derivative in the horizontal direction and a second derivative in the vertical direction of the image to be processed; Obtaining an outer product image according to the first derivative and the second derivative; Calculate the corner point response value of each pixel in the outer product image through the preset corner point response function; Non-maximum suppression is performed on the corner point response values ​​to obtain corner points of the image to be processed.

4. An image segmentation device, characterized in that: The device comprises: An information acquisition module, used to acquire an image to be processed and auxiliary information of the image to be processed; wherein the auxiliary information includes edge information and / or corner point information of the image to be processed; An image processing module, used for inputting the image to be processed and the auxiliary information into a pre-trained image segmentation model, and outputting a segmentation result of the image to be processed; wherein the image segmentation model is trained based on a preset sample image and the auxiliary information of the sample image; The device also includes a model training module, including: a sample determination module, used to determine a target sample based on a preset sample set; the sample set includes a plurality of samples, each of which includes a sample image, and auxiliary information corresponding to the sample image and a standard segmentation result; a sample input module, used to input the sample image in the target sample and the auxiliary information of the sample image into a preset initial model; wherein the initial model includes a feature extraction layer and an overlay layer; a feature extraction module, used to perform feature extraction on a sum image composed of the sample image and the auxiliary information of the sample image through the feature extraction layer to obtain a first feature map; a feature overlay module, used to perform sampling processing on the auxiliary information of the sample image through the overlay layer, and obtain a second feature map corresponding to the auxiliary information of the sample image based on the sampling result; the first feature map and the second feature map are overlaid to obtain a superimposed feature map; a model determination module, used to calculate a loss value based on the superimposed feature map and the standard segmentation result, and train the initial model based on the loss value; continue to perform the step of determining the sample image based on the preset sample set until the loss value converges to obtain the image segmentation model; The feature superposition module is also used to: downsample the auxiliary information of the sample image through the superposition layer according to the scale of the first feature map to obtain a sampling result; copy multiple copies of the sampling results, and superimpose the multiple copies of the sampling results to obtain a second feature map corresponding to the auxiliary information of the sample image; wherein the second feature map has the same scale as the first feature map.

5. The device according to claim 4, characterized in that If the auxiliary information of the image to be processed includes edge information of the image to be processed, the device further includes an edge information determining module, which is used to: Calculating the gradient value of each pixel in the image to be processed; For each pixel point, determining whether the gradient value of the pixel point meets a preset condition; if the preset condition is met, determining the pixel point as an edge point; Each of the edge points is combined to obtain edge information.

6. The device according to claim 4, characterized in that If the auxiliary information of the image to be processed includes corner point information of the image to be processed, the device further includes a corner point information determining module, which is used to: Calculating a first derivative in the horizontal direction and a second derivative in the vertical direction of the image to be processed; Obtaining an outer product image according to the first derivative and the second derivative; Calculate the corner point response value of each pixel in the outer product image through the preset corner point response function; Non-maximum suppression is performed on the corner point response values ​​to obtain corner points of the image to be processed.

7. A server, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the image segmentation method according to any one of claims 1 to 3.

8. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the image segmentation method according to any one of claims 1 to 3.

Images