Skeleton extraction method and device of image, electronic equipment and storage medium

Abstract

The application discloses a skeleton extraction method and device of an image, electronic equipment and a storage medium. The skeleton extraction method of the image comprises the following steps: acquiring a graphic image, and generating input data based on the graphic image, wherein the input data comprises edge information of the graphic image; inputting the input data into a pre-trained neural network model to obtain a first skeleton pixel point prediction image and a second skeleton pixel point prediction image of a target in the graphic image, wherein the neural network model comprises a first neural network and a second neural network, the residual block in the first neural network is not less than 3 layers, and the second neural network comprises a plurality of dense blocks; and obtaining the final skeleton pixel point of the target in the graphic image based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image. The embodiment of the application can effectively improve the recognition accuracy of the skeleton pixel point, and further improve the accuracy of the obtained skeleton pixel point.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a skeleton extraction method and device of an image, an electronic device and a storage medium. BACKGROUND

[0002] Skeleton pixel points in a graphic image can be used for feature extraction and target topology representation. Skeleton extraction is actually to extract the center pixel contour on the graphic image and keep the connectivity of the skeleton and satisfy single-pixel.

[0003] In related technologies, skeleton pixel point extraction of a graphic image, that is, skeleton extraction of an image, can be implemented by using a U-Net convolutional neural network. The class of each pixel point in the graphic image is segmented, and skeleton pixel point extraction is performed. The class of the pixel point is classified into two categories. However, in the technology of skeleton pixel point recognition by using the algorithm of the U-Net convolutional neural network, the skeleton pixel point recognition accuracy is poor, noise points appear in the skeleton, and the connectivity of the skeleton cannot be guaranteed, that is, the skeleton pixel point extraction effect is poor. SUMMARY

[0004] In view of the above defects or deficiencies in the prior art, it is expected to provide a skeleton extraction method and device of an image, an electronic device and a storage medium, which can effectively improve the recognition accuracy of skeleton pixel points, and further improve the accuracy of obtained skeleton pixel points.

[0005] In a first aspect, an embodiment of the present application provides a skeleton extraction method of an image, comprising:

[0006] obtaining a graphic image, and generating input data based on the graphic image, wherein the input data comprises edge information of the graphic image;

[0007] inputting the input data into a pre-trained neural network model to obtain a first skeleton pixel point prediction image and a second skeleton pixel point prediction image of a target in the graphic image, wherein the neural network model comprises a first neural network and a second neural network, a residual block in the first neural network is not less than 3 layers, and the second neural network comprises a plurality of dense blocks;

[0008] obtaining a final skeleton pixel point of the target in the graphic image based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image.

[0009] In some examples, the first neural network comprises an input layer, a hidden layer and an output layer, the hidden layer of the first neural network comprises a plurality of residual blocks, and each residual block comprises at least a first convolutional layer, a pooling layer and a second convolutional layer.

[0010] In some examples, the second neural network includes an input layer, a hidden layer, and an output layer, the hidden layer of the second neural network includes a plurality of dense blocks, each dense block includes a plurality of dense layers, and the plurality of dense layers are tightly connected.

[0011] In some examples, the predicting, based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image, a final skeleton pixel point of a target in the graphic image includes:

[0012] Obtaining a pixel value of each pixel point in the first skeleton pixel point prediction image;

[0013] Obtaining a pixel value of each pixel point in the second skeleton pixel point prediction image;

[0014] Obtaining a pixel average value of corresponding pixel points in the first skeleton pixel point prediction image and the second skeleton pixel point prediction image;

[0015] Based on the pixel average value of the corresponding pixel points, obtaining a skeleton pixel point of a target in the graphic image.

[0016] In some examples, the obtaining, based on the pixel average value of the corresponding pixel points, a skeleton pixel point of a target in the graphic image includes:

[0017] The pixel average value of the corresponding pixel points is greater than the first preset value, and the pixel point is taken as the final skeleton pixel point of the target in the graphic image.

[0018] In some examples, before obtaining the graphic image, the method further includes:

[0019] By the inflation and / or corrosion method, the graphic image is smoothed and denoised to eliminate the edge noise and isolated noise of the graphic image.

[0020] In some examples, the method further includes:

[0021] Extracting edge points of the graphic image to obtain the edge information based on the edge points, specifically including:

[0022] Using a convolution kernel to traverse the pixel points in the graphic image;

[0023] If the pixel value of any pixel point is greater than a second preset value, then taking the any pixel point as the center of the convolution kernel, if there is a pixel point with the second preset value in the convolution kernel coverage area, then the pixel point with the second preset value in the convolution kernel coverage area is an edge point, and the edge information is obtained based on the edge point.

[0024] In some examples, the input data further includes distance information of the graphic image.

[0025] In some examples, the method further comprises:

[0026] obtaining a non-edge point in the graph image, and calculating distances from the non-edge point to center pixel points of all edge points;

[0027] obtaining a distance weight from the non-edge point to the center pixel points based on the distances from the non-edge point to the center pixel points of all edge points and a minimum distance to all edge points;

[0028] obtaining a pixel value of the non-edge point based on the distance weight and the distances from the non-edge point to the center pixel points of all edge points;

[0029] obtaining the distance information based on the pixel value of the non-edge point.

[0030] In some examples, the distance weight from the non-edge point to the center pixel points is obtained by the following formula:

[0031] wight = 2 * (1 - norm.cdf(t0 / sigma)),

[0032] wherein the wight is the distance weight from the non-edge point to the center pixel points, the norm.cdf is a normal distribution function-cumulative probability density function, the sigma is a constant, and the t0 is a difference between the distances from the non-edge point to the center pixel points of all edge points and the minimum distance;

[0033] the pixel value of the non-edge point is obtained by the following formula:

[0034] PointValue = sum(weight * dis) / sum(weight),

[0035] wherein the PointValue is the pixel value of the non-edge point.

[0036] In a second aspect, an embodiment of the present application provides a skeleton extraction device of an image, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and characterized in that the processor executes the following method:

[0037] obtaining a graph image, and generating input data based on the graph image, wherein the input data comprises edge information of the graph image;

[0038] inputting the input data into a pre-trained neural network model to obtain a first skeleton pixel point prediction image and a second skeleton pixel point prediction image of a target in the graphic image, wherein the neural network model comprises a first neural network and a second neural network, a residual block in the first neural network is not less than 3 layers, and the second neural network comprises a plurality of dense blocks.

[0039] Based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image, the final skeleton pixel point of the target in the graphic image is obtained.

[0040] In some examples, the first neural network includes an input layer, a hidden layer and an output layer, and the hidden layer of the first neural network includes a plurality of residual blocks, each of which includes at least a first convolutional layer, a pooling layer and a second convolutional layer.

[0041] In some examples, the second neural network includes an input layer, a hidden layer and an output layer, and the hidden layer of the second neural network includes a plurality of dense blocks, each of which includes a plurality of dense layers, and the plurality of dense layers are tightly connected

[0042] In some examples, the final skeleton pixel point of the target in the graphic image is obtained based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image, comprising:

[0043] Obtain the pixel value of each pixel point in the first skeleton pixel point prediction image;

[0044] Obtain the pixel value of each pixel point in the second skeleton pixel point prediction image;

[0045] Obtain the pixel average value of the corresponding pixel points in the first skeleton pixel point prediction image and the second skeleton pixel point prediction image;

[0046] Based on the pixel average value of the corresponding pixel points, the skeleton pixel point of the target in the graphic image is obtained.

[0047] In some examples, before obtaining the graphic image, the processor is further configured to perform:

[0048] Smooth and denoise the graphic image by inflation and / or erosion method to eliminate edge noise and isolated noise of the graphic image.

[0049] In some examples, the processor is configured to perform:

[0050] Extract the edge points of the graphic image to obtain the edge information based on the edge points, specifically comprising:

[0051] Traverse the pixel points in the graphic image using a convolution kernel;

[0052] If the pixel value of any pixel point is greater than the second preset value, the any pixel point is taken as the center of the convolution kernel, and if there is a pixel point with the second preset value in the convolution kernel coverage area, the pixel point with the second preset value in the convolution kernel coverage area is an edge point, and the edge information is obtained based on the edge point.

[0053] In some examples, the input data further includes distance information of the graphic image, and the processor is configured to perform:

[0054] obtain non-edge points in the graphic image, and calculate distances from the non-edge points to center pixel points of all edge points;

[0055] obtain distance weights from the non-edge points to the center pixel points of all edge points based on distances from the non-edge points to the center pixel points of all edge points and minimum distances to all edge points;

[0056] obtain pixel values of the non-edge points based on the distance weights and the distances from the non-edge points to the center pixel points of all edge points;

[0057] obtain the distance information based on the pixel values of the non-edge points.

[0058] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the skeleton extraction method of the image according to the first aspect.

[0059] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program for implementing the skeleton extraction method of the image according to the first aspect.

[0060] In a fifth aspect, an embodiment of the present application provides a computer program product, which stores a computer program for implementing the skeleton extraction method of the image according to the first aspect.

[0061] The skeleton extraction method, device, electronic equipment and storage medium of the image provided by the embodiment of the application input data into the first neural network and the second neural network improved from the traditional neural network such as the U-Net neural network to respectively predict the skeleton pixel points, and the residual block in the first neural network is not less than 3 layers, and the residual block is replaced by the dense block in the second neural network, so that the accuracy of the prediction results of the skeleton pixel points respectively performed by the first neural network and the second neural network is effectively improved, and the edge information of the graphic image is contained in the input data, so that the accuracy of the prediction results of the skeleton pixel points of the two neural networks is further improved. In addition, on this basis, the prediction results of the two improved neural networks are further fused, so that the recognition accuracy of the skeleton pixel points of the target in the final graphic image is further improved, and the recognition accuracy of the skeleton pixel points is improved. BRIEF DESCRIPTION OF DRAWINGS

[0062] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0063] Figure 1 The flowchart of the skeleton extraction method of the image of the embodiment of the application is shown in the figure;

[0064] Figure 2 The model structure of the traditional U-Net neural network is shown in the figure;

[0065] Figure 3 The schematic diagram of the first neural network in the skeleton extraction method of the image of the embodiment of the application is shown in the figure;

[0066] Figure 4 The model structure of the dense connection network DenseNet in the second neural network in the skeleton extraction method of the image of the embodiment of the application is shown in the figure;

[0067] Figure 5 The method flowchart of obtaining the skeleton pixel points of the target in the final graphic image in the skeleton extraction method of the image of the embodiment of the application is shown in the figure;

[0068] Figure 6 The schematic diagram of the skeleton extraction method of the image of the embodiment of the application is shown in the figure;

[0069] Figure 7 The effect schematic diagram of the skeleton extraction method of the image of the embodiment of the application before and after the graphic image is preprocessed is shown in the figure;

[0070] Figure 8 The schematic diagram of the edge information Edge and the distance information SDE obtained after the graphic image is processed in the skeleton extraction method of the image of the embodiment of the application is shown in the figure;

[0071] Figure 9 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0072] The present application will be further described below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended for the purpose of interpretation of the related disclosure, and not for the purpose of limiting the disclosure. In addition, it should be noted that only parts related to the disclosure are shown in the drawings for the purpose of description.

[0073] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0074] The skeleton extraction method and device of an image, the electronic device and the storage medium according to the embodiments of the present application will be described below in conjunction with the drawings.

[0075] Before describing the skeleton extraction method and device of an image, the electronic device and the storage medium according to the embodiments of the present application, the skeleton pixel extraction will be described first. The skeleton pixel extraction is to extract the skeleton pixel points of a figure, i.e. a target, in a figure image, which is actually to extract the center pixel contour on the figure image, and to maintain the connectivity of the skeleton and satisfy the single pixel. It is usually used for feature extraction and target topology representation.

[0076] In the related art, the skeleton extraction of a figure image can be realized by using a U-Net convolutional neural network, which can segment the category of each pixel point in the figure image. The skeleton pixel points can be extracted by performing binary classification on the category of the pixel points.

[0077] Figure 1 FIG. 1 is a flowchart of the skeleton extraction method of an image according to an embodiment of the present application, as shown in the figure, the skeleton extraction method of an image according to an embodiment of the present application includes the following steps: Figure 1

[0078] S101: Obtain a figure image, and generate input data based on the figure image. That is, the input data includes edge information and the like of the figure image.

[0079] S102: Input the input data into a pre-trained neural network model to obtain a first skeleton pixel point prediction image and a second skeleton pixel point prediction image of a target in the figure image, wherein the neural network model includes a first neural network and a second neural network, the residual block in the first neural network is not less than 3 layers, and the second neural network includes a plurality of dense blocks.

[0080] ​The first skeleton pixel point prediction image can also be referred to as a first skeleton pixel point prediction result, and similarly, the second skeleton pixel point prediction image can also be referred to as a second skeleton pixel point prediction result.

[0081] In the above example, the first neural network and the second neural network are respectively improvements on a traditional neural network such as a traditional U-Net, wherein a model structure of the traditional U-Net neural network is as shown in Figure 2 As can be seen from the figure, the residual block ResNet Block in the residual network ResNet in the traditional U-Net neural network includes two layers, i.e., a convolution layer and a pooling layer.

[0082] On this basis, embodiments of the present application improve the first neural network, i.e., the residual block ResNet Block in the first neural network is not less than three layers, as shown in Figure 3 As shown in the figure, a first neural network including a residual block ResNet Block including three layers is shown. Specifically, the first neural network includes an input layer, a hidden layer, and an output layer, and in the hidden layer, a plurality of residual blocks are included, and each residual block is different from the structure including two layers in the traditional ResNet Block, and the residual block in the embodiments of the present application includes at least a first convolution layer, a pooling layer, and a second convolution layer. As shown in Figure 3 As shown in the figure, a residual block including three layers, i.e., a first convolution layer, a pooling layer, and a second convolution layer is shown.

[0083] As for the improvement of the second neural network, i.e., there is no residual block in the second neural network, but a plurality of dense blocks are included. As shown in Figure 4 As shown in the figure, a neural network structure including a plurality of dense blocks is shown, and from Figure 4 As can be seen from the figure, a plurality of dense layers are included in each dense block, and the plurality of dense layers are tightly connected. Specifically, the input of the second neural network is: a graph image and edge information of the graph image, then convolution is performed through a convolution layer, then enters a first dense block Dense Block, the Dense Block has a dense layer Dense Layer, and each Dense Layer is tightly connected. After passing through a transition block Transition Block composed of a convolution layer and a pooling layer, a second Dense Block and a Transition Block are entered, and then a third Dense Block is entered. Next, a classification block Classification Block composed of a pooling layer and a linear layer is passed through, and finally an activation function is passed through to obtain a final prediction result.

[0084] Through experiments, it is known that the first neural network improves the recognition accuracy of the first skeleton pixel point prediction result output after recognizing the input data by about 2%. The second neural network improves the recognition accuracy of the second skeleton pixel point prediction result output after recognizing the input data by about 0.1%.

[0085] S103: Obtain the skeleton pixel point of the target in the final graphical image based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image.

[0086] In an embodiment of the present application, as shown in Figure 5 obtaining the skeleton pixel point of the target in the final graphical image based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image, includes:

[0087] S501: Obtain the pixel value of each pixel point in the first skeleton pixel point prediction image.

[0088] S502: Obtain the pixel value of each pixel point in the second skeleton pixel point prediction image.

[0089] S503: Obtain the skeleton pixel point of the target in the final graphical image based on the pixel value of each pixel point in the first skeleton pixel point prediction image and the pixel value of each pixel point in the second skeleton pixel point prediction image.

[0090] In the above example, S503 can be implemented in the following way: obtain the pixel average value of the corresponding pixel points in the first skeleton pixel point prediction image and the second skeleton pixel point prediction image; and obtain the skeleton pixel point of the target in the graphical image based on the pixel average value of the corresponding pixel points.

[0091] In this example, obtaining the skeleton pixel point of the target in the graphical image based on the pixel average value of the corresponding pixel points in the first skeleton pixel point prediction image and the second skeleton pixel point prediction image includes: regarding the pixel point whose pixel average value is greater than a first preset value as the skeleton pixel point of the target in the final graphical image.

[0092] That is, the pixel value of each pixel point in the first skeleton pixel point prediction result of the first neural network is a value between 0 and 1, and similarly, the pixel value of each pixel point in the second skeleton pixel point prediction result of the second neural network is a value between 0 and 1. In this way, the first skeleton pixel point prediction result of the first neural network and the second skeleton pixel point prediction result of the second neural network are first summed for the pixel values of the corresponding pixel points, and then the average is taken, and when the average is greater than a first preset value such as 0.9, the pixel value of the pixel point is set to 1, that is, the pixel point is a skeleton pixel point. The pixel values of the remaining pixel points are set to 0. Through the fusion of the skeleton pixel point prediction results of the first neural network and the second neural network, the recognition accuracy of the skeleton pixel points can be effectively improved, and through experiments, the recognition accuracy can reach 80.71% or more. That is, the input data is respectively input into the two improved U-Net neural networks to obtain the corresponding prediction results after inference, and then the two prediction results are summed for the pixel points and averaged to obtain the fused output result, that is, the final skeleton pixel points of the target in the image are obtained.

[0093] The skeleton extraction method and device of an image, electronic equipment and storage medium provided by the embodiments of the present application input input data into a first neural network and a second neural network improved from a traditional neural network such as a U-Net neural network for skeleton pixel point prediction, and the residual block in the first neural network is not less than 3 layers, and the residual block in the second neural network is replaced by a dense block, so that the accuracy of the skeleton pixel point prediction results of the first neural network and the second neural network is effectively improved, and the edge information of the image is included in the input data, so that the accuracy of the skeleton pixel point prediction results of the two neural networks is further improved. In addition, the prediction results of the two improved neural networks are further fused, so that the recognition accuracy of the skeleton pixel points of the target in the final image is further improved.

[0094] Before obtaining the image, the skeleton extraction method of an image of the embodiments of the present application further includes: performing smoothing and noise reduction processing on the image to eliminate edge noise and isolated noise of the image.

[0095] Specifically, the smoothing and noise reduction processing on the image to eliminate edge noise and isolated noise of the image includes: performing smoothing and noise reduction processing on the image by an inflation and / or corrosion method. For example, Figure 6As shown, the graphic image is preprocessed such as smoothing and denoising, that is, the graphic image is processed by using expansion and corrosion methods, so that the edge noise and isolated noise points in the input data can be solved, and the subsequent judgment of the center pixel point of the graphic image is avoided. The expansion and corrosion principle is, for example, using a 3*3 convolution kernel, and the effects before and after processing are as shown in Figure 7 As shown. Therefore, before the skeleton pixel points are recognized by the neural network, the graphic image is preprocessed, the edge noise and isolated noise points in the input data can be removed, so that the center pixel point judgment accuracy of the graphic image can be improved, and further, the recognition accuracy of the skeleton pixel points recognized by the neural network is further improved.

[0096] As shown Figure 6 In an embodiment of the present application, the input data further includes edge information Edge and distance information SDE, wherein the edge information and the distance information are obtained based on the graphic image.

[0097] Specifically, the edge information Edge is obtained in the following manner:

[0098] The edge points of the graphic image are extracted to obtain the edge information Edge based on the edge points. In this example, the convolution kernel is used to traverse the pixel points in the graphic image; if the pixel value of any pixel point is greater than the second preset value, then the center of the convolution kernel is any pixel point, and when the convolution kernel covers the region with a pixel point with a pixel value of the second preset value, the pixel point with a pixel value of the second preset value in the convolution kernel covering region is an edge point, and the edge information Edge is obtained based on the edge point.

[0099] That is, the edge information Edge is extracted as new input data to provide the edge information of the graphic image, which can improve the correctness of the judgment of the center pixel point of the target, because the position of the center pixel point is only related to the position of the edge point.

[0100] In this example, the principle of extracting edge points is, for example, using a 3*3 convolution kernel to traverse all points in the graphic image, if the pixel value of a point is greater than 0, that is, the second preset value, and the point is the center of the 3*3 convolution kernel, if the convolution kernel covering region has a pixel point with a value of 0, then the point is an edge point. Traverse all points in the graphic image to find all edge points, and only display and save the pixels with edge points as edge information Edge.

[0101] The distance information SDE is obtained in the following manner:

[0102] obtain non-edge points in the graphic image, and calculate distances from the non-edge points to all edge points; obtain a distance weight from a non-edge point to a center pixel point based on distances from the non-edge point to center pixel points of all edge points and a minimum distance to all edge points; obtain a pixel value of the non-edge point based on the distance weight and distances from the non-edge point to all edge points; and obtain edge information based on the pixel value of the non-edge point.

[0103] Specifically, the distance weight from the non-edge point to the center pixel point is obtained by the following formula:

[0104] wight = 2 * (1 - norm.cdf(t0 / sigma)),

[0105] wherein wight is the distance weight from the non-edge point to the center pixel point, norm.cdf is a normal distribution function-cumulative probability density function, sigma is a constant, and t0 is a difference between the distance from the non-edge point to the center pixel point of all edge points and the minimum distance.

[0106] The pixel value of the non-edge point is obtained by the following formula:

[0107] PointValue = sum(weight * dis) / sum(weight),

[0108] wherein PointValue is the pixel value of the non-edge point.

[0109] That is, the distance information SDE is added as an input data to provide distance information from a pixel point to a center pixel point, to help the target center point to judge, and thus to improve the recognition accuracy of the neural network for the skeleton pixel point.

[0110] wherein the distance information SDE (SDE: smooth distance estimation) is obtained by finding a pixel point of a non-edge point in the graphic image, calculating a distance dis from the point to a center pixel point in an edge point list, and then obtaining the distance information SDE by the above formula.

[0111] As shown in FIG. 1, an edge information Edge and a distance information SDE obtained after processing a graphic image are shown. Figure 8

[0112] ​According to the skeleton extraction method of the image provided in the embodiment of the present application, the algorithm such as expansion and corrosion is used to pre-process the graphic image, so as to reduce the burr noise of the graphic image, then the graphic image, the edge information Edge and the distance information SDE are combined together as the input of the neural network model, and the neural network model comprises two improved neural networks, the skeleton pixel points are predicted by the two improved neural networks respectively, the accuracy of the two prediction results is improved respectively, and the two prediction results are further fused to obtain the final recognition result of the skeleton pixel points, so that the recognition accuracy of the skeleton pixel points can be effectively improved.

[0113] In another aspect, the embodiment of the present application provides a skeleton extraction device of an image, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the following method: obtaining a graphic image, and generating input data based on the graphic image, wherein the input data comprises edge information of the graphic image; inputting the input data into a pre-trained neural network model to obtain a first skeleton pixel point prediction image and a second skeleton pixel point prediction image of a target in the graphic image, wherein the neural network model comprises a first neural network and a second neural network, the residual block in the first neural network is not less than 3 layers, and the second neural network comprises a plurality of dense blocks; based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image, obtaining the final skeleton pixel point of the target in the graphic image.

[0114] In an embodiment of the present application, the first neural network comprises an input layer, a hidden layer and an output layer, and the hidden layer of the first neural network comprises a plurality of residual blocks, each residual block comprising at least a first convolutional layer, a pooling layer and a second convolutional layer.

[0115] In an embodiment of the present application, the second neural network comprises an input layer, a hidden layer and an output layer, and the hidden layer of the second neural network comprises a plurality of dense blocks, each dense block comprising a plurality of dense layers, and the plurality of dense layers are tightly connected

[0116] In an embodiment of the present application, based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image, the final skeleton pixel point of the target in the graphic image is obtained, comprising: obtaining the pixel value of each pixel point in the first skeleton pixel point prediction image; obtaining the pixel value of each pixel point in the second skeleton pixel point prediction image; based on the pixel value of each pixel point in the first skeleton pixel point prediction image and the pixel value of each pixel point in the second skeleton pixel point prediction image, the final skeleton pixel point of the target in the graphic image is obtained.

[0117] In an embodiment of the present application, before the graphic image is acquired, the processor is further configured to perform: smoothing and denoising processing on the graphic image to eliminate edge noise and isolated noise of the graphic image.

[0118] In an embodiment of the present application, the input data further comprises edge information and distance information, wherein the edge information and the distance information are both based on the graphic image.

[0119] In an embodiment of the present application, the processor is configured to perform: extracting edge points of the graphic image to obtain the edge information based on the edge points.

[0120] In an embodiment of the present application, the processor is configured to perform: obtaining non-edge points in the graphic image and calculating distances from the non-edge points to center pixel points of all edge points; obtaining distance weights from the non-edge points to the center pixel points based on distances from the non-edge points to center pixel points of all edge points and minimum distances to all edge points; obtaining pixel values of the non-edge points based on the distance weights and distances from the non-edge points to center pixel points of all edge points; and obtaining the edge information based on the pixel values of the non-edge points.

[0121] The skeleton extraction method and device for images, the electronic device and the storage medium provided by the embodiments of the present application input input data into a first neural network and a second neural network improved from a traditional neural network such as a U-Net neural network to respectively predict skeleton pixel points, and the residual block in the first neural network is not less than 3 layers, and the second neural network replaces the residual block with a dense block, so that the accuracy of the prediction results of the skeleton pixel points respectively performed by the first neural network and the second neural network is effectively improved, and the edge information of the graphic image is included in the input data, so that the accuracy of the prediction results of the skeleton pixel points by the two neural networks is further improved. In addition, the prediction results of the two improved neural networks are further fused, so that the recognition accuracy of the skeleton pixel points of the target in the final graphic image is further improved, and the recognition accuracy of the skeleton pixel points is improved.

[0122] It should be noted that the specific implementation of the skeleton extraction device for images in the embodiments of the present application is similar to the specific implementation of the skeleton extraction method for images in the embodiments of the present application. For details, please refer to the description in the method part, which will not be repeated here.

[0123] Figure 9 The structure of the electronic device in the embodiments of the present application is shown in the schematic diagram.

[0124] As Figure 9As shown, the electronic device 600 includes a central processing unit (CPU) 601 which can perform various appropriate actions and processes in accordance with a program stored in a read only memory (ROM) 602 or a program loaded from the storage section 602 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0125] Connected to the I / O interface 605 are an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 610 as necessary, so that a computer program read therefrom is installed into the storage section 608 as necessary.

[0126] In particular, in accordance with embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a machine-readable medium, the computer program comprising program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable media 611. When the computer program is executed by the central processing unit (CPU) 601, the above-described functions defined in the electronic device of the present application are performed.

[0127] It should be noted that the computer-readable medium shown in the application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor electronic device, device or apparatus, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution electronic device, device or apparatus. In this application, the computer-readable signal medium can include a data signal carried in a baseband or as a carrier wave part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit programs for use by or in conjunction with an instruction execution electronic device, device or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0128] The flowcharts and block diagrams in the drawings illustrate the possible implementation architecture, function and operation of the receiving device, method and computer program product according to various embodiments of the application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the figure. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the involved functions. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based electronic device that performs the specified function or operation, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0129] The units or modules described in the embodiments of the present application can be implemented in software or hardware. The described units or modules can also be arranged in a processor for executing the program to implement the skeleton extraction method of the image: obtaining a graphic image, and generating input data based on the graphic image, wherein the input data includes edge information of the graphic image; inputting the input data into a pre-trained neural network model to obtain a first skeleton pixel point prediction image and a second skeleton pixel point prediction image of a target in the graphic image, wherein the neural network model includes a first neural network and a second neural network, the residual block in the first neural network is not less than 3 layers, and the second neural network includes a plurality of dense blocks; and obtaining the final skeleton pixel point of the target in the graphic image based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image.

[0130] As another aspect, the present application also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The computer readable storage medium stores one or more programs, when the programs are used by one or more processors to execute the skeleton extraction method of the image described in the present application: obtaining a graphic image, and generating input data based on the graphic image, wherein the input data includes edge information of the graphic image; inputting the input data into a pre-trained neural network model to obtain a first skeleton pixel point prediction image and a second skeleton pixel point prediction image of a target in the graphic image, wherein the neural network model includes a first neural network and a second neural network, the residual block in the first neural network is not less than 3 layers, and the second neural network includes a plurality of dense blocks; and obtaining the final skeleton pixel point of the target in the graphic image based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image.

[0131] As another aspect, the present application also provides a computer program product, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer program product stores one or more programs, and when the programs are used by one or more processors to execute the skeleton extraction method of the image described in the present application, the programs are used by one or more processors to execute the skeleton extraction method of the image described in the present application: obtaining a graphic image, and generating input data based on the graphic image, wherein the input data includes edge information of the graphic image; inputting the input data into a pre-trained neural network model to obtain a first skeleton pixel point prediction image and a second skeleton pixel point prediction image of a target in the graphic image, wherein the neural network model includes a first neural network and a second neural network, the residual block in the first neural network is not less than 3 layers, and the second neural network includes a plurality of dense blocks; and obtaining the final skeleton pixel point of the target in the graphic image based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image.

[0132] The above description is merely the preferred embodiments of the present application and the explanation of the applied technical principles. It should be understood by those skilled in the art that the disclosed range of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the disclosed concept. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

Claims

1. A skeleton extraction method of an image, characterized by, The method comprises: obtaining a graphic image, and generating input data based on the graphic image, wherein the input data comprises edge information of the graphic image and distance information of the graphic image; inputting the input data into a pre-trained neural network model to obtain a first skeleton pixel point prediction image and a second skeleton pixel point prediction image of a target in the graphic image, wherein the neural network model comprises a first neural network and a second neural network, a residual block in the first neural network has no less than 3 layers, and the second neural network comprises a plurality of dense blocks; obtaining a final skeleton pixel point of the target in the graphic image based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image; wherein, further comprising: obtaining non-edge points in the graphic image, and calculating distances from the non-edge points to center pixel points of all edge points; obtaining a distance weight from the non-edge points to the center pixel points of all edge points based on the distances from the non-edge points to the center pixel points of all edge points and the smallest distance to all edge points; obtaining pixel values of the non-edge points based on the distance weight and the distances from the non-edge points to the center pixel points of all edge points; obtaining the distance information based on the pixel values of the non-edge points.

2. The skeleton extraction method of an image according to claim 1, characterized by, The first neural network comprises an input layer, a hidden layer and an output layer, and the hidden layer of the first neural network comprises a plurality of residual blocks, each of which comprises at least a first convolutional layer, a pooling layer and a second convolutional layer.

3. The skeleton extraction method of an image according to claim 1, characterized by, The second neural network comprises an input layer, a hidden layer and an output layer, and the hidden layer of the second neural network comprises a plurality of dense blocks, each of which comprises a plurality of dense layers which are tightly connected.

4. The skeleton extraction method of an image according to claim 1, characterized by, The method of obtaining a final skeleton pixel point of the target in the graphic image based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image comprises: obtaining pixel values of each pixel point in the first skeleton pixel point prediction image; obtaining pixel values of each pixel point in the second skeleton pixel point prediction image; obtaining pixel average values of corresponding pixel points in the first skeleton pixel point prediction image and the second skeleton pixel point prediction image; obtaining the skeleton pixel point of the target in the graphic image based on the pixel average values of the corresponding pixel points.

5. The skeleton extraction method of an image according to claim 4, characterized by, The method of obtaining the skeleton pixel point of the target in the graphic image based on the pixel average values of the corresponding pixel points comprises: regarding pixel points with pixel average values greater than a first preset value as the final skeleton pixel point of the target in the graphic image.

6. The skeleton extraction method of an image according to claim 1, characterized by, Before obtaining the graphic image, the method further comprises: performing smoothing and noise reduction processing on the graphic image by an expansion and / or erosion method to eliminate edge noise and isolated noise of the graphic image.

7. The skeleton extraction method of an image according to claim 1, characterized by, The method further comprises: extracting edge points of the graphic image to obtain the edge information based on the edge points, specifically comprising: traversing pixel points in the graphic image using a convolution kernel; If a pixel value of any pixel point is greater than a second preset value, the any pixel point is taken as a center of the convolution kernel, when the convolution kernel covers a pixel point with the second preset value, the pixel point with the second preset value in the convolution kernel covering area is an edge point, and the edge information is obtained based on the edge point.

8. The skeleton extraction method of an image according to claim 1, wherein, the distance weight of the non-edge point to the center pixel point is obtained by the following formula: weight = 2 * (1 - norm.cdf (t0 / sigma) Wherein, the non-edge point to the center pixel point distance weight of the weight is the distance weight of the non-edge point to the center pixel point, and the norm.cdf is the normal distribution function-cumulative probability density function, and the sigma is a constant, and the t0 is the difference between the distance of the non-edge point to the center pixel point of all edge points and the minimum distance. the pixel value of the non-edge point is obtained by the following formula: ， Wherein, the PointValue is the pixel value of the non-edge point, sum is a summation function, and dis is the distance from the pixel point of the non-edge point to the center pixel point in the edge point list.

9. A skeleton extraction apparatus of an image, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the following method: obtain a graphic image, and generate input data based on the graphic image, wherein the input data includes edge information of the graphic image and distance information of the graphic image; input the input data into a pre-trained neural network model to obtain a first skeleton pixel point prediction image and a second skeleton pixel point prediction image of a target in the graphic image, wherein the neural network model includes a first neural network and a second neural network, the residual block in the first neural network is not less than 3 layers, and the second neural network includes a plurality of dense blocks; obtain the final skeleton pixel point of the target in the graphic image based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image; wherein the processor is further configured to execute: obtain non-edge points in the graphic image and calculate distances of the non-edge points to center pixel points of all edge points; obtain the distance weight of the non-edge point to the center pixel point based on the distance of the non-edge point to the center pixel point of all edge points and the minimum distance to all edge points; obtain the pixel value of the non-edge point based on the distance weight and the distance of the non-edge point to the center pixel point of all edge points; obtain the distance information based on the pixel value of the non-edge point.

10. The skeleton extraction apparatus of the image according to claim 9, characterized by, The first neural network includes an input layer, a hidden layer and an output layer, and the hidden layer of the first neural network includes a plurality of residual blocks, each of which includes at least a first convolution layer, a pooling layer and a second convolution layer.

11. The skeleton extraction apparatus of the image according to claim 9, characterized by, The second neural network includes an input layer, a hidden layer and an output layer, and the hidden layer of the second neural network includes a plurality of dense blocks, each of which includes a plurality of dense layers which are tightly connected.

12. The skeleton extraction apparatus of the image according to claim 9, characterized by, The final skeleton pixel point of the target in the graphic image is obtained based on the first skeleton pixel point prediction image and the second skeleton pixel point prediction image, including: obtaining a pixel value of each pixel point in the first skeleton pixel point prediction image; obtaining a pixel value of each pixel point in the second skeleton pixel point prediction image; obtaining a pixel average value of corresponding pixel points in the first skeleton pixel point prediction image and the second skeleton pixel point prediction image; obtaining the skeleton pixel point of the target in the graphic image based on the pixel average value of the corresponding pixel points.

13. The skeleton extraction apparatus of the image according to claim 9, characterized by, Before acquiring the graphic image, the processor is further configured to perform: performing smoothing and noise reduction on the graphic image by an inflation and / or erosion method to eliminate edge noise and isolated noise of the graphic image.

14. The skeleton extraction apparatus of the image according to claim 9, characterized by, The processor is configured to perform: extracting edge points of the graphic image to obtain the edge information based on the edge points, specifically comprising: traversing pixel points in the graphic image by using a convolution kernel; if a pixel value of any pixel point is greater than a second preset value, taking the any pixel point as a center of the convolution kernel, if there is a pixel point with the second preset value in a region covered by the convolution kernel, the pixel point with the second preset value in the region covered by the convolution kernel is an edge point, and the edge information is obtained based on the edge point.

15. An electronic device, comprising: A computer readable storage medium having stored thereon a computer program for implementing the skeleton extraction method of the image according to any one of claims 1-8.

16. A computer readable storage medium having stored thereon a computer program for implementing the skeleton extraction method of the image according to any one of claims 1-8.

17. A computer program product, characterised in that, A computer readable storage medium having stored thereon a computer program for implementing the skeleton extraction method of the image according to any one of claims 1-8.

Images