Method for identifying image key points, image segmentation method, device and medium

Abstract

The present invention relates to the technical field of image processing, and specifically provides a method for identifying image key points, an image segmentation method, a device, and a medium, aiming to solve the problem of poor key point recognition effect. The method provided by the present invention includes: obtaining image features, randomly generating a first feature vector for forward diffusion of key points in the image in the feature space, where the feature space is formed by singular value decomposition of the key point feature vector, and the key point feature vector is constructed from the coordinates of the key points in the image coordinate space; using a diffusion model to reverse-diffuse the first feature vector according to the image features to obtain a second feature vector; converting the second feature vector to the image coordinate space based on the singular value decomposition method to obtain the actual coordinates of the key points; obtaining the recognition result according to the actual coordinates; the diffusion model is obtained by training for forward diffusion and reverse diffusion of the key points, and the key points include edge and / or non-edge key points. By diffusing the key points, the correlation of the key points is ensured, and the recognition effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to a method for identifying image key points, an image segmentation method, a device, and a medium. Background Art

[0002] Image key points refer to points in an image with specific features and positions. After obtaining the image key points, filling the key points can segment the segmentation object of interest from the image. In many tasks of image key point recognition and segmentation, deep learning often achieves the best results. For example, in the medical field, it is often necessary to segment lesions from the acquired images to facilitate doctors' diagnosis and treatment. In the security field, it is often necessary to identify human face key points to facilitate the identification of users' identity information.

[0003] Currently, methods based on deep learning are generally used to segment images. For example, the UNet model is used for image segmentation. The input of this model is the original image, and the output is the segmentation result. By training the model through methods such as gradient descent, the model can predict its segmentation result under the condition of a given input image. However, when using existing methods to segment images, it cannot be guaranteed that the segmentation result must contain all segmentation objects of interest. For key point recognition, currently, methods based on coordinate regression or methods based on heatmap regression are generally used for key point recognition. However, when using the method based on coordinate regression for key point recognition, although it is fast, has a small computational amount, and can ensure that the recognition result must contain all these determined key points, its accuracy is not as high as that of the method based on heatmap regression for key point recognition. When using the method based on heatmap regression for key point recognition, it cannot be guaranteed that the recognized key points must contain all fixed segmentation objects.

[0004] Correspondingly, a new solution is needed in this field to solve the above problems. Summary of the Invention

[0005] In order to overcome the above defects, the present invention is proposed to solve or at least partially solve the technical problem of poor key point recognition effect.

[0006] In a first aspect, a method for identifying image key points is provided, and the method includes:

[0007] Obtain the image features of the image to be recognized;

[0008] Randomly generate a first feature vector obtained by forward diffusion of the key points of the target object in the image to be recognized in a preset feature space. The preset feature space is a singular value decomposition space formed by performing singular value decomposition on the key point feature vectors of the target object. The key point feature vectors are feature vectors constructed according to the coordinates of the key points in the image coordinate space corresponding to the image to be recognized;

[0009] Use a preset diffusion model and perform reverse diffusion on the first feature vector according to the image features to obtain the second feature vector of the key points in the preset feature space;

[0010] Based on the singular value decomposition method, convert the second feature vector of the key points from the preset feature space to the image coordinate space to obtain the actual coordinates of the key points in the image coordinate space, and obtain the recognition result of the key points according to the actual coordinates of the key points;

[0011] Among them, the preset diffusion model is a model obtained by training the key points of the target object in the image sample in the preset feature space by performing forward diffusion and reverse diffusion in sequence;

[0012] The key points of the target object include the key points located at the edge and / or non-edge of the target object.

[0013] In a technical solution of the above method for identifying image key points, the step of "performing reverse diffusion on the first feature vector to obtain the second feature vector of the key points in the preset feature space" specifically includes: performing reverse diffusion on the first feature vector multiple times, and respectively obtaining the second feature vector obtained by each reverse diffusion;

[0014] The step of "obtaining the actual coordinates of the key points in the image coordinate space" specifically includes: respectively converting the second feature vector obtained by each reverse diffusion from the preset feature space to the image coordinate space to respectively obtain the actual coordinates of the key points corresponding to the second feature vector obtained by each reverse diffusion;

[0015] The step of "obtaining the recognition result of the key points according to the actual coordinates of the key points" specifically includes: obtaining the average value of the actual coordinates of the key points corresponding to the second feature vector obtained by each reverse diffusion, and obtaining the recognition result of the key points according to the average value.

[0016] In a technical solution of the above method for identifying image key points, the step of "obtaining the image features of the image to be recognized" specifically includes:

[0017] If the image to be recognized is a 3D image, intercept multiple consecutive 2D images with the same distance interval from the image to be recognized and respectively obtain the image features of each 2D image, so as to be able to respectively obtain the actual coordinates of the key points of the target object on each 2D image in the image coordinate space according to the image features of each 2D image;

[0018] The step of "obtaining the recognition result of the key points according to the actual coordinates of the key points" specifically includes:

[0019] For every two adjacent 2D images, match the key points on the two 2D images according to the actual coordinates of the key points of the target object on the two 2D images to determine the one-to-one correspondence between the key points on the two 2D images;

[0020] According to the correspondence, connect the corresponding key points on the two 2D images respectively, obtain the intersection points of the connection lines and the image to be recognized, and use the intersection points as the key points of the target object on the image area between the two 2D images;

[0021] According to the actual coordinates of the key points of the target object on each 2D image and the key points on the image area between intervals in the image coordinate space, obtain the recognition results of the key points.

[0022] In a technical solution of the above method for recognizing image key points, before the step of "matching the key points on the two 2D images according to the actual coordinates of the key points of the target object on the two 2D images to determine the one-to-one correspondence between the key points on the two 2D images", the method further includes:

[0023] For each 2D image, calculate the average value of the Z-axis coordinates of the key points of the target object on the 2D image according to the actual coordinates of each key point of the target object on the 2D image, and

[0024] Replace the Z-axis coordinates in the actual coordinates of each key point with the average value to form new actual coordinates.

[0025] In a technical solution of the above method for recognizing image key points, the preset diffusion model is trained through the following steps:

[0026] Obtain the key points of the target object in the image sample;

[0027] According to the coordinates of each key point in each coordinate dimension of the image coordinate space corresponding to the image sample, construct the initial key point feature vector of the target object;

[0028] Respectively obtain the coordinate means of the key points on each coordinate dimension in the image coordinate space;

[0029] Subtract the coordinate of each key point in each coordinate dimension of the initial key feature vector from the corresponding coordinate mean of each coordinate dimension to obtain the final key point feature vector;

[0030] Perform singular value decomposition on the final key point feature vector to obtain the initial feature vector of the key points in the singular value decomposition space;

[0031] The initial feature vector is trained by forward diffusion using a diffusion model to obtain the first feature vector of the key points in the singular value decomposition space, and the first feature vector is trained by reverse diffusion using a preset diffusion model according to the image features of the image sample to obtain the second feature vector of the key points in the singular value decomposition space, thereby obtaining a trained diffusion model.

[0032] In a technical solution of the above method for identifying image key points, the step of "according to the key points of the target object in the image sample" specifically includes that if the key point is a key point located at the edge of the target object, the key points of the target object are obtained by the following method:

[0033] Obtain all the edge points of the target object;

[0034] Calculate the coordinate mean of all the edge points in the image coordinate space corresponding to the image sample, and use the point at the coordinate mean position as the center of the target object;

[0035] Traverse each edge point in turn in the preset direction of rotation around the center to sort each edge point;

[0036] From the sorted edge points, obtain multiple edge points that are consecutive and have the same number of edge points as the interval as the key points of the target object.

[0037] In a technical solution of the above method for identifying image key points, the step of "obtaining all the edge points of the target object" specifically includes:

[0038] If the image sample is a 3D image, intercept multiple 2D images that are consecutive and have the same distance interval from the image sample, and respectively obtain all the edge points corresponding to the target object on each 2D image.

[0039] In a second aspect, an image segmentation method is provided, and the method includes: obtaining an image to be segmented;

[0040] Adopt the method for identifying image key points described in any one of the first aspect to identify the key points located at the edge of the target object in the image to be segmented;

[0041] Segment the target object according to the key points located at the edge of the target object.

[0042] In a third aspect, a computer device is provided, which includes a processor and a storage device. The storage device is adapted to store multiple program codes, and the program codes are adapted to be loaded and run by the processor to execute the method for identifying image key points described in any one of the technical solutions of the above method for identifying image key points or to be loaded and run by the processor to execute the image segmentation method described in any one of the technical solutions of the above image segmentation method.

[0043] In a fourth aspect, a computer-readable storage medium is provided, which stores multiple pieces of program code. The program code is adapted to be loaded and run by a processor to execute the method for identifying image key points according to any one of the technical solutions of the above-mentioned method for identifying image key points, or to be loaded and run by a processor to execute the method for image segmentation according to any one of the technical solutions of the above-mentioned method for image segmentation.

[0044] One or more of the above technical solutions of the present invention have at least one or more of the following beneficial effects:

[0045] In implementing the technical solution of the present invention, first, image features to be recognized are obtained. Then, after randomly generating a first feature vector obtained by forward diffusion of key points of a target object in a to-be-recognized image in a preset feature space, the first feature vector is reversely diffused in the preset feature space by using a preset diffusion model according to the image features to obtain key points. The key points are recognized according to the key points of the target object and the image features, which can ensure that all key points of the target object are included in the recognition result, and further enable all target objects in the to-be-recognized image to be accurately recognized and segmented. There is a correlation between key points. For example, the key points at both ends of the bones of a person's forearm can form a straight line segment, and the correlation between the two key points will not change no matter how the person moves. When the prior art diffuses key points, it often uses the method of directly adding random noise to key points in the image coordinate space, which will destroy the correlation between key points. Singular value decomposition of key points can determine the feature vector closest to the key points in the image coordinate space in the singular value decomposition space. Therefore, diffusing in the singular value decomposition space preserves the correlation between key points compared with directly adding random noise to key points in the image coordinate space. By performing singular value decomposition on the key point feature vector of the target object and then diffusing, and finally converting the diffused feature vector from the singular value decomposition space to the image coordinate space to finally determine the key points of the target object, the accuracy of image key point recognition can be further ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Referring to the accompanying drawings, the disclosure of the present invention will become more understandable. It is easily understood by those skilled in the art that these drawings are only for illustrative purposes and are not intended to limit the protection scope of the present invention. Among them:

[0047] Figure 1 is a schematic flowchart of the main steps of the method for identifying image key points according to an embodiment of the present invention;

[0048] Figure 2It is a schematic diagram of the main steps of a diffusion model training method according to an embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram of the main steps of an image segmentation method according to an embodiment of the present invention;

[0050] Figure 4 It is a schematic diagram of the main structure of a computer device according to an embodiment of the present invention. Detailed implementation manners

[0051] The following describes some implementation manners of the present invention with reference to the accompanying drawings. Those skilled in the art should understand that these implementation manners are only used to explain the technical principle of the present invention and are not intended to limit the protection scope of the present invention.

[0052] In the description of the present invention, a "processor" may include hardware, software, or a combination of both. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, in hardware, or in a combination of both. A non-transitory computer-readable storage medium includes any suitable medium that can store program code, such as a magnetic disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, and so on. The term "A and / or B" represents all possible combinations of A and B, such as only A, only B, or A and B. The terms "first", "second", etc. are used to distinguish similar objects and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here.

[0053] The following describes an embodiment of the method for identifying image key points provided by the present invention.

[0054] Refer to the attached Figure 1 , Figure 1 It is a schematic diagram of the main steps of the method for identifying image key points according to an embodiment of the present invention. As Figure 1 shown, the method for identifying image key points in the embodiment of the present invention mainly includes the following steps S101 to step S103.

[0055] Step S101: Obtain the image features of the image to be recognized.

[0056] The image features of the image to be recognized can be obtained by inputting the image to be recognized into a deep neural network for extracting image features. The deep neural network for extracting image features can select commonly used deep neural networks such as Resnet, MaskAE, 3DResnet, etc.

[0057] Step S102: Randomly generate a first feature vector obtained by forward diffusion of the key points of the target object in the to-be-recognized image in a preset feature space. The preset feature space is a singular value decomposition space formed by performing singular value decomposition on the key point feature vectors of the target object. The key point feature vectors are feature vectors constructed according to the coordinates of the key points in the image coordinate space corresponding to the to-be-recognized image. The key points of the target object include key points located at the edge and / or non-edge of the target object.

[0058] The key points refer to the points that play a key role in recognizing the target object in the image and can cover the features of the target object. Generally, when determining the key points, they can be extracted at the edge of the target object or from the inside of the target object. For example, when the target object is a human face, the key points can be the points at the edge of the human face or the points on the facial features. The first feature vector is a randomly generated unit vector, and the first feature vector can be expressed as Z t , where t represents the number of steps of forward diffusion of the first feature vector. Each first feature vector corresponds to each target object one by one. If there are multiple target objects in the to-be-recognized image, multiple first feature vectors can be randomly generated.

[0059] Step S103: Use a preset diffusion model and perform reverse diffusion on the first feature vector according to the image features to obtain a second feature vector of the key points in the preset feature space, where the preset diffusion model is a model obtained by training the key points of the target object in the image samples in the preset feature space by performing forward diffusion and reverse diffusion in sequence.

[0060] Reverse diffusion is the reverse process of forward diffusion. The number of steps of reverse diffusion is the same as that of forward diffusion. Input the first feature vector after t steps of forward diffusion and the image features into the preset diffusion model, and after t steps of reverse diffusion, a second feature vector can be obtained, where t>0. The preset diffusion model can be selected as a fusion neural network composed of multiple perceptrons. It should be noted that in order to improve the efficiency of obtaining the second feature vector of the key points in the preset feature space by reverse diffusion, multiple first feature vectors can be combined into a feature vector matrix, and then reverse diffusion is performed on this feature vector matrix to obtain the feature vector matrix after reverse diffusion. Finally, the feature vector matrix after reverse diffusion is split, and all the second feature vectors can be obtained at one time. The method of performing reverse diffusion on the feature vector matrix is the same as that of performing reverse diffusion on a single feature vector, which will not be elaborated here. Compared with performing reverse diffusion on all the first feature vectors one by one, combining all the first feature vectors into a feature vector matrix can improve the computational efficiency of reverse diffusion of all the first feature vectors.

[0061] Step S104: Based on the singular value decomposition method, transform the second eigenvector of the key point from the preset feature space to the image coordinate space to obtain the actual coordinates of the key point in the image coordinate space, and obtain the recognition result of the key point according to the actual coordinates of the key point.

[0062] Among them, the following formula can be used to transform the second eigenvector of the key point from the preset feature space to the image coordinate space, so as to determine the actual coordinates of the key point in the image coordinate space:

[0063] Point = Z 0 ΣV T

[0064] Z 0 is the second eigenvector of the key point, the Σ matrix is the Sigma matrix, and the V T matrix is the transpose matrix of the V matrix obtained by singular value decomposition. The above Σ matrix and V matrix can be obtained by the singular value decomposition method. The V matrix is a unitary matrix, VV T = I. Except for the elements on the main diagonal of the Σ matrix, all other elements are 0, and each element on the main diagonal is a singular value.

[0065] Based on the method described in the above steps S101 to S104, first obtain the image features to be recognized, then randomly generate the first eigenvector obtained by forward diffusion of the key points of the target object in the to-be-recognized image in the preset feature space, and then use the preset diffusion model in the preset feature space and perform reverse diffusion on the first eigenvector according to the image features to obtain the key points. Recognize the key points according to the key points of the target object and the image features, which can ensure that all the key points of the target object are included in the recognition result, and further enable all the target objects in the to-be-recognized image to be accurately recognized and segmented. There is a correlation between key points. For example, the key points at both ends of the bones of a person's forearm can form a straight line segment. No matter how this person moves, the correlation between the two key points will not change. In the prior art, when performing diffusion on key points, the method of directly adding random noise to the key points in the image coordinate space is often used, which will destroy the correlation between key points. Performing singular value decomposition on key points can determine the eigenvector closest to the key points in the image coordinate space in the singular value decomposition space. Therefore, performing diffusion in the singular value decomposition space compared to directly adding random noise to the key points in the image coordinate space ensures the correlation between key points. By performing singular value decomposition on the key point eigenvector of the target object and then performing diffusion, and finally transforming the diffused eigenvector from the singular value decomposition space to the image coordinate space to determine the key points of the target object, the accuracy of image key point recognition can be further ensured.

[0066] The following further explains the above steps S101, S103, and S104.

[0067] I. Further explanation of step S101.

[0068] The image to be recognized may be a 2D planar image or a 3D stereoscopic image.

[0069] In a possible implementation manner according to an embodiment of the present invention, the step of "acquiring the image features of the image to be recognized (the foregoing step S101)" specifically includes:

[0070] If the image to be recognized is a 2D image, the image features of the image to be recognized can be obtained by inputting the image to be recognized into a deep neural network for extracting image features. Commonly used deep neural networks for extracting image features can include Resnet, MaskAE, 3DResnet, etc.

[0071] If the image to be recognized is a 3D image, multiple consecutive 2D images with the same interval distance are intercepted from the image to be recognized, and the image features of each 2D image are respectively obtained, so as to be able to respectively obtain the actual coordinates of the key points of the target object on each 2D image in the image coordinate space. The size of the interval distance can be set by those skilled in the art according to actual needs, and the specific numerical value of the interval distance is not limited in the embodiments of the present invention. It should be noted that if the image to be recognized is a 3D image, the actual coordinates of the key points on the 2D image intercepted from the image to be recognized are still 3D coordinates.

[0072] By intercepting multiple consecutive 2D images with the same interval distance on the 3D image to be recognized and respectively obtaining the image features of each 2D image, the problem of obtaining the image features of the 3D image to be recognized can be converted into the problem of obtaining the image features of multiple 2D images to be recognized, so that the image features of the 3D image to be recognized can be obtained simply and quickly.

[0073] II. Further explanation of step S103.

[0074] In order to improve the recognition accuracy of image key points, image key point recognition can be performed multiple times, and then the results obtained from multiple recognitions are processed to determine the final recognition result.

[0075] In a possible implementation manner according to an embodiment of the present invention, the step of "performing reverse diffusion on the first feature vector to obtain the second feature vector of the key point in the preset feature space (the foregoing step S103)" specifically includes:

[0076] Perform reverse diffusion on the first feature vector multiple times, and respectively obtain the second feature vectors obtained by each reverse diffusion.

[0077] By performing reverse diffusion on the first feature vector multiple times, multiple different second feature vectors can be obtained, which facilitates subsequent further identification of image key points.

[0078] Refer to the appendix Figure 2 , Figure 2 is a schematic flow chart of the main steps of a diffusion model training method according to an embodiment of the present invention. The method for identifying image key points in the embodiment of the present invention further includes training a preset diffusion model through the following steps S201 to S206:

[0079] Step S201: Obtain the key points of the target object in the image sample.

[0080] When training the diffusion model, it is necessary to obtain the key points of the target object in the image sample. In the training set for the task of identifying image key points, each image sample is labeled with the key points of the target object.

[0081] Step S202: Construct an initial key point feature vector of the target object according to the coordinates of each key point in each coordinate dimension of the image coordinate space corresponding to the image sample.

[0082] As an example, if the coordinates of a key point in the image coordinate space corresponding to the image sample are (1, 2, 3), where 1 represents that in the three-dimensional coordinate system, the coordinate of this key point on the X-axis is 1, 2 represents that the coordinate of this key point on the Y-axis is 2, and 3 represents that the coordinate of this key point on the Z-axis is 3. The initial key point feature vector of this key point is {1, 2, 3}.

[0083] Step S203: Respectively obtain the coordinate means of the key points on each coordinate dimension in the image coordinate space.

[0084] As an example, if the coordinates of a key point in the image coordinate space corresponding to the image sample are (1, 2, 3), then the value of this key point on the X-axis coordinate dimension in the three-dimensional coordinate system is 1, the value on the Y-axis coordinate dimension is 2, and the value on the Z-axis coordinate dimension is 3. From this, it can be obtained that the coordinate mean on each coordinate dimension in the image coordinate space is 2.

[0085] Step S204: Subtract the coordinates of each key point in each coordinate dimension of the initial key feature vector from the respective coordinate means corresponding to each coordinate dimension to obtain the final key point feature vector.

[0086] As an example, if the feature vector of a key point is {1, 2, 3}, and the coordinate mean of this key point in each coordinate dimension in the image coordinate space is 2, after subtracting the coordinate mean corresponding to each coordinate dimension from the coordinates of each key point in each coordinate dimension of the initial key feature vector, the final key point feature vector obtained is {-1, 0, 1}.

[0087] Step S205: Perform singular value decomposition on the final key point feature vector to obtain the initial feature vector of the key point in the singular value decomposition space.

[0088] Among them, the method of performing singular value decomposition on the final key point feature vector is similar to the method of singular value decomposition described in the foregoing step S102, and will not be elaborated here.

[0089] Step S206: Use a diffusion model to perform forward diffusion training on the initial feature vector to obtain the first feature vector of the key point in the singular value decomposition space, and use a preset diffusion model to perform reverse diffusion training on the first feature vector according to the image features of the image sample to obtain the second feature vector of the key point in the singular value decomposition space, so as to obtain a trained diffusion model.

[0090] Among them, the method of forward diffusion training can be carried out using the following formula:

[0091]

[0092] where t is the number of diffusion steps, t ≥ 1, Z t is the set of feature vectors after t steps of diffusion, N is the normal distribution, and Normalize() can normalize the diffused feature vectors. Singular value decomposition refers to an important matrix decomposition method in linear algebra. The embodiments of the present invention do not limit the specific method of singular value decomposition. After the vector is subjected to singular value decomposition, a U matrix, a Sigma matrix, and a V T matrix can be obtained, where both the U matrix and the V matrix are unitary matrices, UU T = I, VV T = I, and except for the elements on the main diagonal of the Sigma matrix, all other elements are 0, and each element on the main diagonal is a singular value.

[0093] Reverse diffusion is the inverse process of forward diffusion. When training a diffusion model, the number of steps for forward diffusion needs to be determined, and the number of steps for reverse diffusion is the same as that for forward diffusion. The method of performing reverse diffusion on the model is the same as that in step S103 described above and will not be elaborated here. The preset diffusion model can be selected as a fusion neural network composed of multiple perceptrons. Training the preset diffusion model can adopt conventional diffusion model training methods in the art. As an example, Adam (lr = 0.001) can be selected as the optimizer. Adam refers to Adaptive Moment Estimation, that is, the adaptive moment estimation algorithm, and lr refers to the learning rate, that is, the neural network learning rate. The optimizer refers to the algorithm that guides the neural network to update parameters.

[0094] Through the methods of step S201 to step S206 above, the coordinate mean of the key points on each coordinate dimension in the image coordinate space is obtained. Then, the coordinates of each key point on each coordinate dimension in the initial key feature vector are respectively subtracted by the corresponding coordinate mean of each coordinate dimension to obtain the final key point feature vector, which can make the mean of the final key point feature vector be 0, thereby reducing the subsequent calculation amount and improving the training efficiency of the diffusion model.

[0095] In the image segmentation task, only the edges of the target object are labeled in the training set of image segmentation, and the key points of the target object are not labeled. Therefore, it is necessary to extract key points from the edges of the target object.

[0096] In a possible implementation manner according to an embodiment of the present invention, the step of "obtaining the key points of the target object in the image sample (step S201 above)" specifically includes that if the key point is a key point located at the edge of the target object, then the key points of the target object are obtained through the following steps 11 to 14:

[0097] Step 11: Obtain all the edge points of the target object.

[0098] The edge point refers to the point on the boundary line of the target object in the image. There are various methods to determine whether a point belongs to an edge point, which is not limited in the embodiment of the present invention. For example, it can be determined by whether the surrounding points of the point are inside the target object. If there are points that do not belong to the inside of the target object among the 8 surrounding points centered on a certain point, then this point is an edge point.

[0099] Step 12: Calculate the coordinate mean of all the edge points in the image coordinate space corresponding to the image sample, and use the point at the coordinate mean position as the center of the target object.

[0100] Calculate the coordinate mean of all edge points in the image coordinate space corresponding to the image sample, that is, calculate the center position of the image sample. As an example, if the edge point coordinates of an image are (1, 0, 0), (1, 2, 3), (4, 1, 3), then the coordinate mean in the image coordinate space corresponding to this image sample is (2, 1, 2), and the center of the target object is (2, 1, 2).

[0101] Step 13: Traverse each edge point in turn according to the preset direction of rotation around the center to sort each edge point.

[0102] Among them, the direction of rotation can be clockwise or counterclockwise.

[0103] Step 14: From the sorted edge points, obtain multiple edge points that are consecutive and have the same number of edge points as the interval as the key points of the target object.

[0104] Suppose there are a total of N sorted edge points and a total of K key points are required. Then, one point can be selected as a key point every s = (N / K) points, and finally K key points are obtained, where N ≥ K ≥ 1. s represents the interval size for sampling edge points. The value of N is the same as the number of all edge points of the obtained target object. The value of K can be set by those skilled in the art, and K is inversely proportional to the interval s.

[0105] By the method of steps 11 to 14 above, select a point around the center, and then traverse the edge points by rotating around the center, the sorting of all edge points can be achieved. After sorting all the edge points, sample all the edge points at the same interval, and use the sampled edge points as the key points of the target object, which can ensure that the key points of the target object are evenly distributed on the edge of the target object.

[0106] If the image sample is a 3D image, it is necessary to intercept a 2D image from the 3D image and obtain edge points for the 2D image.

[0107] In a possible implementation manner according to an embodiment of the present invention, the step of "obtaining all edge points of the target object" specifically includes:

[0108] If the image sample is a 3D image, intercept multiple 2D images that are consecutive and have the same interval distance from the image sample, and respectively obtain all the edge points corresponding to the target object on each 2D image.

[0109] As an example, assume that a total of K key points need to be obtained, and assume that K = h * w, where both h and w are integers. K, h, and w are all pre-set parameters. K represents the total number of key points to be obtained, h represents the number of 2D images intercepted, and w represents the number of key points to be obtained for each intercepted 2D image. Assume that the height of a 3D image is H. The 3D image is intercepted at intervals of H / h in a certain direction (such as the Z direction) to obtain a 2D image, and finally h 2D images are obtained. Then, each 2D image is processed according to the methods of steps 11 to 14 above, and each 2D image can obtain w key points.

[0110] By continuously intercepting multiple 2D images from the image samples of the 3D image at the same interval distance and obtaining the edge points of the 2D images, the problem of obtaining the edge points of the 3D image can be transformed into the problem of obtaining the edge points of the 2D image, thereby reducing the difficulty of obtaining the edge points of the 3D image and making the acquisition of the edge points of the 3D image more convenient.

[0111] III. Further explanation of step S104.

[0112] To improve the recognition result of the key points, the image key points can be recognized multiple times, and then the recognition result of the final key points can be determined by synthesizing the results of multiple recognitions. After performing multiple reverse diffusions on the first feature vector in the foregoing step S103 and respectively obtaining the second feature vectors obtained by each reverse diffusion, the second feature vectors obtained by each reverse diffusion can be further processed.

[0113] In a possible implementation manner according to an embodiment of the present invention, the step of "obtaining the actual coordinates of the key points in the image coordinate space (the foregoing step S104)" specifically includes: respectively converting the second feature vectors obtained by each reverse diffusion from the preset feature space to the image coordinate space to respectively obtain the actual coordinates of the key points corresponding to the second feature vectors obtained by each reverse diffusion.

[0114] The method of converting the second feature vector from the preset feature space to the image coordinate space is the same as the foregoing method based on singular value decomposition, and will not be elaborated here.

[0115] In a possible implementation manner according to an embodiment of the present invention, the step of "obtaining the recognition result of the key points according to the actual coordinates of the key points (the foregoing step S104)" specifically includes: obtaining the average value of the actual coordinates of the key points corresponding to the second feature vectors obtained by each reverse diffusion, and obtaining the recognition result of the key points according to the average value.

[0116] As an example, assume that key point recognition is performed 3 times. The coordinates of the key points obtained in the first recognition are (1, 2, 2), the coordinates of the key points obtained in the second recognition are (1, 2, 3), and the coordinates of the key points obtained in the third recognition are (1, 2, 4). After averaging the coordinates of the key points obtained in the three recognitions, the recognition result of the key points is (1, 2, 3).

[0117] By performing key point recognition multiple times and then taking the average of the coordinates of the recognized key points, the error in the process of recognizing key points can be reduced, and the accuracy of recognizing key points can be improved.

[0118] As described in the foregoing step S101, if the image to be recognized is a 3D image, then multiple consecutive 2D images at the same interval distance can be intercepted on the 3D image to be recognized, so as to convert the recognition of the key points of the 3D image into the recognition of the key points of multiple 2D images.

[0119] In a possible implementation manner according to an embodiment of the present invention, the step of "obtaining the recognition result of the key points according to the actual coordinates of the key points (the foregoing step S104)" specifically includes the following steps 21 to 23:

[0120] Step 21: For every two adjacent 2D images, match the key points on the two 2D images according to the actual coordinates of the key points of the target object on the two 2D images to determine the one-to-one correspondence between the key points on the two 2D images.

[0121] The method of matching key points on every two adjacent 2D images can adopt the method of calculating the minimum sum of the connection distances after pairing. For example, when pairing the key points on the i-th 2D image and the (i + 1)-th 2D image, the key points on the i-th 2D image can be sorted in the order of traversal. Suppose there are x key points in total, then after sorting, the key points on the i-th 2D image are the first key point, the second key point, the third key point,..... the x-th key point. Similarly, the (i + 1)-th 2D image also has x key points, and they can also be sorted in the order of traversal as the first key point, the second key point, the third key point,..... the x-th key point. Select the q-th key point on the i-th 2D image (q is less than or equal to x), and then select the s-th key point on the (i + 1)-th 2D image and connect them (s is less than or equal to x). Connect the q-th key point on the i-th 2D image with the s-th key point on the (i + 1)-th 2D image, connect the (q + 1)-th key point on the i-th 2D image with the (s + 1)-th key point on the (i + 1)-th 2D image, and so on until all the key points on the i-th 2D image are connected with all the key points on the (i + 1)-th 2D image. Then calculate the total sum of the connection distances between all the key points on the i-th 2D image and all the key points on the (i + 1)-th 2D image. Then connect the q-th key point on the i-th 2D image with the (s + 1)-th key point on the (i + 1)-th 2D image, connect the (q + 1)-th key point on the i-th 2D image with the (s + 2)-th key point on the (i + 1)-th 2D image, until all the key points on the i-th 2D image are connected with all the key points on the (i + 1)-th 2D image, and then count the total sum of the connection distances between all the key points on the i-th 2D image and all the key points on the (i + 1)-th 2D image again. After x traversals, the connection method with the minimum total sum of the connection distances between all the key points on the i-th 2D image and all the key points on the (i + 1)-th 2D image is the matching method of the key points on the two 2D images. Using this matching method, the one-to-one correspondence of the key points on the two 2D images can be obtained.

[0122] Step 22: According to the correspondence, connect the corresponding key points on the two 2D images respectively, obtain the intersection points of the connections and the image to be recognized, and use the intersection points as the key points of the target object on the image area between the two 2D images.

[0123] Step 23: According to the actual coordinates of the key points of the target object on each 2D image and the key points on the image area between them in the image coordinate space, obtain the recognition results of the key points.

[0124] By the method of steps 21 to 23 above, adjacent 2D images are matched. After the adjacent 2D images are matched, the corresponding relationship of key points of the adjacent 2D images can be obtained. According to the corresponding relationship of the key points, the corresponding 2D image key points are connected by lines, and the key points of the images in the spaced image region are determined according to the intersection points of the lines and the spaced image region, without the need to identify the key points of the spaced image region, reducing the workload of identifying key points and improving the efficiency of identifying the key points of 3D images. In addition, since it is relatively complex to directly obtain the key points of 3D images, by continuously intercepting multiple 2D images of the image samples of 3D images at the same interval distance, and obtaining the key points of each 2D image and the key points on the spaced image region, the problem of obtaining the key points of 3D images is transformed into the problem of obtaining the key points of 2D images and the key points on the spaced image region, reducing the difficulty of obtaining the edge points of 3D images and making the acquisition of the edge points of 3D images more convenient.

[0125] In a possible implementation manner according to an embodiment of the present invention, before the step of "matching the key points on two 2D images according to the actual coordinates of the key points of the target object on the two 2D images to determine the one-to-one correspondence between the key points on the two 2D images (the foregoing step 21)", the method further includes:

[0126] For each 2D image, according to the actual coordinates of each key point of the target object on the 2D image, calculate the average value of the Z-axis coordinates of the key points of the target object on the 2D image, and replace the Z-axis coordinates in the actual coordinates of each key point with the average value to form new actual coordinates.

[0127] As an example, if the actual coordinates of three key points of the target object on the 2D image are (1, 1, 5), (2, 2, 4), and (3, 3, 0), the Z-axis coordinate of the first key point is 5, the Z-axis coordinate of the second key point is 4, the Z-axis coordinate of the third key point is 0, and the average value of the Z-axis coordinates of the three key points is 3. After replacing the Z-axis coordinates of the actual coordinates of the three key points with 3, the new actual coordinates of the three key points are (1, 1, 3), (2, 2, 3), and (3, 3, 3).

[0128] The 2D image is obtained by continuously intercepting the image samples of the 3D image at equal intervals in the Z-axis direction. The Z-axis coordinates of the key points on one 2D image should be the same. However, due to various reasons, the Z-axis coordinates of the key points on a certain 2D image may have errors, resulting in different Z-axis coordinates of the key points on this 2D image. For example, there is an error in the interception direction of the 3D image, resulting in non-horizontal cutting of the 3D image. In this case, the height of a 2D image obtained by interception is also different in the Z-axis. By replacing the Z-axis coordinates of the actual coordinates of the key points of the 2D image with the average value of the Z-axis coordinates of all the key points of this 2D image, it can be ensured that the Z-axis coordinate values of the 2D image are the same, thus avoiding the influence of the error in the Z-axis coordinates of the 2D image on subsequent key point matching.

[0129] The embodiments of the image segmentation method provided by the present invention will be described below.

[0130] After obtaining the key points of the image, the image can be segmented according to the positions of the key points of the image.

[0131] Refer to the appendix Figure 3 , Figure 3 is a schematic diagram of the main step flow of the image segmentation method according to an embodiment of the present invention. As Figure 3 shown, the image segmentation method in the embodiments of the present invention mainly includes the following steps S301 to step S303.

[0132] Step S301: Obtain the image to be segmented.

[0133] Step S302: Use the image key point recognition method in any of the foregoing embodiments of the image key point recognition method to recognize the key points located at the edge of the target object in the image to be segmented.

[0134] Step S303: Segment the target object according to the key points located at the edge of the target object.

[0135] Segmenting the target object according to the key points at the edge of the target object can be performed using a polygon filling algorithm. For example, skimage.draw.polygon2mask can be used to generate a polygon shape containing the key points according to the key points, so as to realize segmenting the target object according to the key points located at the edge of the target object. Of course, other methods can also be used to segment the target object according to the key points located at the edge of the target object, and the present invention does not make any limitations.

[0136] After obtaining the key points located at the edge of the image to be segmented, image segmentation can be performed based on these key points located at the edge. Since the key points of the image to be segmented are identified by using the image key point recognition method of any of the foregoing embodiments of the image key point recognition method, the accuracy of the identified key points is high, and the result of image segmentation based on the key points is also more accurate. Segmenting the image to be segmented according to the identified key points can accurately segment the target object without omission.

[0137] It should be noted that although the above embodiments describe the various steps in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the protection scope of the present invention.

[0138] Those skilled in the art can understand that all or part of the processes in the method of the above embodiments of the present invention can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable storage medium can include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal, and software distribution medium that can carry the computer program code. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0139] Furthermore, the present invention also provides a computer device.

[0140] Refer to the appendix Figure 4 , Figure 4 which is a schematic diagram of the main structure of a computer device embodiment according to the present invention. As Figure 4As shown in the figure, the computer device in the embodiment of the present invention mainly includes a storage device 41 and a processor 42. The storage device 41 can be configured to store a program for executing the method for identifying image key points in the above method embodiment or a program for executing the method for image segmentation in the above method embodiment. The processor 42 can be configured to execute the program in the storage device, and the program includes, but is not limited to, a program for executing the method for identifying image key points in the above method embodiment or a program for executing the method for image segmentation in the above method embodiment. For the sake of convenience of description, only the parts related to the embodiment of the present invention are shown. For the specific technical details not disclosed, please refer to the method part of the embodiment of the present invention.

[0141] In the embodiment of the present invention, the computer device can be a control device including various electronic devices. In some possible implementation manners, the computer device can include multiple storage devices 41 and multiple processors 42. The programs for executing the method for identifying image key points and the method for image segmentation in the above method embodiment can be divided into multiple sub-programs, and each sub-program can be loaded and run by the processor respectively to execute different steps of the method for identifying image key points or the method for image segmentation in the above method embodiment. Specifically, each sub-program can be stored in different storage devices 41 respectively, and each processor 42 can be configured to execute the programs in one or more storage devices 41 to jointly implement the method for identifying image key points or the method for image segmentation in the above method embodiment, that is, each processor 42 respectively executes different steps of the method for identifying image key points or the method for image segmentation in the above method embodiment to jointly implement the method for identifying image key points or the method for image segmentation in the above method embodiment.

[0142] The above-mentioned multiple processors 42 can be processors deployed on the same device. For example, the above-mentioned computer device can be a high-performance device composed of multiple processors, and the above-mentioned multiple processors 42 can be the processors configured on the high-performance device. In addition, the above-mentioned multiple processors 42 can also be processors deployed on different devices. For example, the above-mentioned computer device can be a server cluster, and the above-mentioned multiple processors 42 can be the processors on different servers in the server cluster.

[0143] Furthermore, the present invention also provides a computer-readable storage medium.

[0144] In an embodiment of a computer-readable storage medium according to the present invention, the computer-readable storage medium may be configured to store a program for executing the method for identifying image key points in the above method embodiment or store a program for executing the image segmentation method in the above method embodiment. This program can be loaded and run by a processor to implement the above method for identifying image key points or loaded and run to implement the above image segmentation method. For the sake of convenience, only the parts related to the embodiments of the present invention are shown. For the specific technical details not disclosed, please refer to the method part of the embodiments of the present invention. The computer-readable storage medium may be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiments of the present invention is a non-transitory computer-readable storage medium.

[0145] So far, the technical solution of the present invention has been described in conjunction with one embodiment shown in the drawings. However, those skilled in the art can easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims

1. A method for identifying key points of an image, characterized in that, The method includes: Obtaining the image features of the image to be recognized; Randomly generating a first feature vector obtained by forward diffusion of the key points of the target object in the image to be recognized in a preset feature space, where the preset feature space is a singular value decomposition space formed by performing singular value decomposition on the key point feature vectors of the target object, and the key point feature vectors are feature vectors constructed based on the coordinates of the key points in the image coordinate space corresponding to the image to be recognized; Using a preset diffusion model and performing reverse diffusion on the first feature vector according to the image features to obtain a second feature vector of the key points in the preset feature space; Converting the second feature vector of the key points from the preset feature space to the image coordinate space based on the method of singular value decomposition to obtain the actual coordinates of the key points in the image coordinate space, and obtaining the recognition result of the key points according to the actual coordinates of the key points; Among them, the preset diffusion model is a model obtained by training the key points of the target object in the image sample in the preset feature space by performing forward diffusion and reverse diffusion in sequence; The key points of the target object include key points located at the edge and / or non-edge of the target object.

2. The method for recognizing image key points according to claim 1, wherein: The step of "performing reverse diffusion on the first feature vector to obtain a second feature vector of the key points in the preset feature space" specifically includes: performing multiple reverse diffusions on the first feature vector, and respectively obtaining the second feature vector obtained by each reverse diffusion; The step of "obtaining the actual coordinates of the key points in the image coordinate space" specifically includes: respectively converting the second feature vector obtained by each reverse diffusion from the preset feature space to the image coordinate space to respectively obtain the actual coordinates of the key points corresponding to each second feature vector obtained by each reverse diffusion; The step of "obtaining the recognition result of the key points according to the actual coordinates of the key points" specifically includes: obtaining the average value of the actual coordinates of the key points corresponding to each second feature vector obtained by each reverse diffusion, and obtaining the recognition result of the key points according to the average value.

3. The method for recognizing image key points according to claim 1, wherein: The step of "obtaining the image features of the image to be recognized" specifically includes: If the image to be recognized is a 3D image, then intercepting multiple consecutive 2D images with the same interval distance from the image to be recognized and respectively obtaining the image features of each 2D image, so as to be able to respectively obtain the actual coordinates of the key points of the target object on each 2D image in the image coordinate space; The step of "obtaining the recognition result of the key points according to the actual coordinates of the key points" specifically includes: For every two adjacent 2D images, matching the key points on the two 2D images according to the actual coordinates of the key points of the target object on the two 2D images to determine the one-to-one correspondence between the key points on the two 2D images; According to the corresponding relationship, connecting the corresponding key points on the two 2D images respectively, obtaining the intersection points of the connection lines and the image to be recognized, and taking the intersection points as the key points of the target object on the image area between the two 2D images; Obtain the recognition result of the key points according to the actual coordinates of the key points of the target object on each 2D image and the key points on the spaced image area in the image coordinate space.

4. The method for recognizing image key points according to claim 3, characterized in that, Before the step of "matching the key points on two 2D images according to the actual coordinates of the key points of the target object on the two 2D images to determine the one-to-one correspondence between the key points on the two 2D images", the method further includes: For each 2D image, calculate the average value of the Z-axis coordinates of the key points of the target object on the 2D image according to the actual coordinates of each key point of the target object on the 2D image, and Replace the Z-axis coordinate in the actual coordinates of each key point with the average value to form new actual coordinates.

5. The method for identifying image key points according to claim 1, characterized in that, The preset diffusion model is trained through the following steps: Obtain the key points of the target object in the image sample; Construct an initial key point feature vector of the target object according to the coordinates of each key point in each coordinate dimension of the image coordinate space corresponding to the image sample; Respectively obtain the coordinate means of the key points on each coordinate dimension in the image coordinate space; Subtract the coordinate of each key point in each coordinate dimension of the initial key feature vector from the respective coordinate means corresponding to each coordinate dimension to obtain the final key point feature vector; Perform singular value decomposition on the final key point feature vector to obtain the initial feature vector of the key points in the singular value decomposition space; Adopt the diffusion model to perform forward diffusion training on the initial feature vector to obtain the first feature vector of the key points in the singular value decomposition space, and adopt the preset diffusion model to perform reverse diffusion training on the first feature vector according to the image features of the image sample to obtain the second feature vector of the key points in the singular value decomposition space, so as to obtain the trained diffusion model.

6. The method for identifying image key points according to claim 5, characterized in that, The step of "obtaining the key points of the target object in the image sample" specifically includes that if the key points are the key points located at the edge of the target object, then the key points of the target object are obtained in the following manner: Obtain all the edge points of the target object; Calculate the coordinate mean of all the edge points in the image coordinate space corresponding to the image sample, and use the point at the coordinate mean position as the center of the target object; Traverse each edge point in turn according to the preset direction of rotation around the center to sort each edge point; From the sorted edge points, obtain multiple consecutive edge points with the same number of edge points spaced apart as the key points of the target object.

7. The method for identifying key points of an image according to claim 6, wherein The step of "obtaining all the edge points of the target object" specifically includes: If the image sample is a 3D image, then intercept multiple consecutive 2D images with the same distance interval from the image sample and respectively obtain all the edge points corresponding to the target object on each 2D image.

8. An image segmentation method, characterized in that, The method includes: Obtain the image to be segmented; Adopt the method for recognizing image key points according to any one of claims 1 to 7 to recognize the key points located at the edge of the target object in the image to be segmented; Perform image segmentation on the target object according to the key points located at the edge of the target object.

9. A computer device, comprising a processor and a storage device, the storage device being adapted to store a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by the processor to execute the method for identifying image key points according to any one of claims 1 to 7, or to execute the image segmentation method according to claim 8.

10. A computer-readable storage medium storing multiple program codes, characterized in that, The program code is adapted to be loaded and run by a processor to execute the method for identifying image key points according to any one of claims 1 to 7, or to execute the image segmentation method according to claim 8.

Images