Key point detection method and device and storage medium
By using the diffusion model to perform reverse diffusion denoising based on the predicted coordinates of the key point detector, the problem of insufficient key point positioning accuracy in the existing technology is solved, higher-precision key point detection is achieved, the cost is reduced, and the efficiency is improved Accuracy and stability for industrial inspection and robotic manufacturing.
Patent Information
- Application Number
- CN202411320023.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-01-24
AI Technical Summary
In the prior art, the key point positioning accuracy is insufficient and the cost is high, making it difficult to accurately detect the key point coordinates of the workpiece in complex scenes and noise-free images.
The predicted coordinates are obtained through the key point detector, a predicted Gaussian distribution is constructed, and the pre-trained diffusion model is used for reverse diffusion denoising to obtain the precise coordinates of the key points.
The detection accuracy of key points coordinates has been improved, and the problem of insufficient accuracy of traditional computer vision methods in complex scenes and noise images has been overcome, reducing costs, and improving the accuracy and stability of industrial testing and robotic manufacturing.
Smart Images
Figure CN119359800A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a key point detection method, device and storage medium. Background Art
[0002] In manufacturing, it is often necessary to use visual positioning to guide the robot to process the key points of the workpiece, especially in precision manufacturing. The visual positioning accuracy of the key points is crucial. Once the key points are not positioned accurately enough, the robot may make mistakes when performing related operations.
[0003] Taking vision-guided automatic welding as an example, it uses sensors such as cameras to obtain images of the workpieces to be welded, and then uses computer vision algorithms to identify and locate the coordinates of key points in the image. Then, based on the position calibration relationship between the robot coordinate system and the camera coordinate system, the position of the key point coordinates in the robot coordinate system is calculated, and then the robot is guided to carry the welding tools to the desired position to carry out related welding work. The typical process is as follows: Figure 1 shown.
[0004] However, in practical applications, some workpieces have complex and changeable shapes, blurred edges, unclear features, etc. It is difficult to detect accurate key point coordinates by relying on traditional computer vision methods. Although there are also detectors that use key point detection methods based on deep learning for key point detection, such as the YOLO system detector, such as Ultralytics, "YOLOv8," 2024, [Online], Available: ]https: / / github.com / ultralytics / ultralytics, or its variants, such as ZhaoS, Ma Y, Fan J, et al. DeepKP: A Robust and Accurate Framework for Weld Seam Keypoint Extraction in Welding Robots[J].IEEE Transactions on Instrumentation and Measurement, 2024.
[0005] However, the above detectors are also difficult to achieve the expected accuracy and stability in actual application. One of the reasons is that the input of this type of key point detector is a scaled image, which makes it difficult for the detector to integrate all local features for accurate detection. Therefore, the detection accuracy is limited, especially in complex scenes and noisy image scenes. This type of detector lacks good detection capabilities. Figure 2As shown in the figure, the red circles represent the key point coordinates detected by the existing key point detector, and the green circles represent the key point coordinates manually marked. It can be seen that there is a certain deviation between the key point coordinates detected by the key point detector based on deep learning and the true values marked manually.
[0006] In order to improve the accuracy and quality of imaging, the current practice is to use sensors with higher accuracy and larger data volumes, such as depth cameras and radars. However, this does not solve the problem of insufficient key point positioning accuracy of the detector itself, and it will also bring a greater cost burden.
[0007] Therefore, there is an urgent need for a detection method that can improve the accuracy of key point coordinate positioning, so as to use existing sensors to perform more accurate key point positioning. Summary of the invention
[0008] In view of the problems of insufficient key point positioning accuracy and high cost in the prior art, an object of the present invention is to provide a key point detection method, device and storage medium to at least partially solve the above problems.
[0009] To achieve the above object, the technical solution of the present invention is: In a first aspect, the present invention provides a key point detection method, the method comprising the following steps: The image to be tested is processed by a key point detector to obtain the predicted coordinates of the key points; Constructing a predicted Gaussian distribution with the predicted coordinates of the key point as the center; intercepting a local image containing the predicted coordinates of the key points from the image to be tested; The predicted Gaussian distribution is processed by a pre-trained diffusion model, and the local image is used as context information to guide the pre-trained diffusion model to perform reverse diffusion so as to remove noise in the predicted Gaussian distribution and obtain a denoised Gaussian distribution, the center of which is the precise coordinate of the key point.
[0010] In some preferred embodiments, the pre-trained diffusion model is obtained by the following steps: The training images are processed by a key point detector to obtain the predicted coordinates of the key points, and the training images are processed by manual annotation to obtain the annotated coordinates of the key points; The predicted Gaussian distribution is constructed by the predicted coordinates of the key points h k , construct the labeled Gaussian distribution through the labeled coordinates of the key points h 0 ; Cutting out a local image including the predicted coordinates of the key points from the training image; The Gaussian distribution of the annotations is simulated by the diffusion model h 0 Processing is performed, and the local image is used as context information to guide the diffusion model to perform forward diffusion, and a noise Gaussian distribution with noise is obtained. h 0 ; The noise Gaussian distribution is calculated by the loss function ^ h 0 With the predicted Gaussian distribution h k The deviation between them is calculated, and the parameters of the diffusion model are optimized until the model converges to obtain the pre-trained diffusion model.
[0011] In some preferred embodiments, the local image is obtained by the following steps: taking the predicted coordinates of the key point as the center, cutting out a region of size m×n as the local image, wherein m≥1, n≥1.
[0012] In some preferred embodiments, the predicted Gaussian distribution has a set variance.
[0013] In some preferred embodiments, the parameters of the diffusion model are optimized by a gradient descent method.
[0014] In some preferred embodiments, the local images are first stacked and then feature extracted through a convolutional neural network model to obtain local spatial context semantic embedding, and the local spatial context semantic embedding is concatenated with the diffusion step encoding to form a context vector for the diffusion model prompt information.
[0015] In some preferred embodiments, the partial images are scaled to the same size before being stacked.
[0016] In a second aspect, the present invention further provides an electronic device, comprising a memory storing executable program code and a processor coupled to the memory; wherein the processor calls the executable program code stored in the memory to execute the method as described above.
[0017] In a third aspect, the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program executes the method described above when executed by a processor.
[0018] By adopting the above technical scheme, the beneficial effect of the present invention is that: based on the results detected by the currently commonly used deep learning-based key point detector and the local information in the sensor information, the present invention removes the noise in the current detection result through a diffusion model, thereby realizing the refinement of the current detection result, and further improving the detection accuracy of the key point coordinates in the workpiece. It can overcome the problem of insufficient positioning accuracy of the key point coordinates of the workpiece by the current detection method based on computer vision, thereby improving the accuracy and stability of industrial inspection and robot intelligent manufacturing processes guided by vision and other sensors. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a schematic diagram of the welding system process based on vision guidance in the prior art.
[0020] Figure 2 Schematic diagram of the comparison between the key point coordinates detected by the existing key point detector and the manually marked coordinates.
[0021] Figure 3 It is a schematic diagram of the flow of the key point detection method in the present invention.
[0022] Figure 4 Schematic diagram of the predicted Gaussian distribution constructed with the predicted coordinates of the key point as the center.
[0023] Figure 5 Schematic diagram of the training process of the diffusion model.
[0024] Figure 6 The predicted coordinates of the key points obtained by detecting the workpiece terminal through the existing key point detector.
[0025] Figure 7 Schematic diagram of the process of constructing the local spatial contextual semantic embedding of the diffusion model using the predicted coordinates of key points.
[0026] Figure 8 Schematic diagram of the workflow of the diffusion model in the present invention. DETAILED DESCRIPTION
[0027] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0028] Example 1 A key point detection method is used to detect the coordinates of key points in a workpiece image, such as Figure 3 As shown, the method comprises the following steps: S1. Process the image to be tested through the key point detector to obtain the predicted coordinates of the key points; S2. Construct a predicted Gaussian distribution with the predicted coordinates of the key points as the center; S3. Extracting a local image containing the predicted coordinates of the key points from the image to be tested; S4. The predicted Gaussian distribution is processed by a pre-trained diffusion model, and the local image is used as context information to guide the pre-trained diffusion model to perform reverse diffusion so as to remove the noise in the predicted Gaussian distribution and obtain a denoised Gaussian distribution. The center of the denoised Gaussian distribution is the precise coordinate of the key point.
[0029] In step S1, the image to be tested is an image of a workpiece to be processed. The workpiece usually has more than one key point (i.e., the point to be processed). Therefore, after the image to be tested is processed by the key point detector, more than one predicted coordinates of the key points are obtained, usually four predicted coordinates of the key points, such as Figure 2 At the four corner points of the workpiece shown.
[0030] In step S2, when constructing a predicted Gaussian distribution with the predicted coordinates of the key point as the center, the variance is also set for the predicted Gaussian distribution, such as Figure 4 As shown, a set of four predicted Gaussian distributions are obtained based on the predicted coordinates of four key points in the image to be tested of the same workpiece.
[0031] In step S3, the local image is obtained by the following steps: taking the predicted coordinates of the key point as the center, cutting out a region of size m×n in the image to be tested as the local image, where m≥1 and n≥1. m and n can be the same, such as 80×80, 60×60 or 40×40, etc.; m and n can also be different, such as 60×80, 80×60, 60×40, etc.
[0032] It is easy to understand that the contents of step S2 and step S3 can be executed sequentially, in reverse order, or simultaneously.
[0033] In step S4, the pre-trained diffusion model is a model based on deep learning. Specifically, Figure 5 As shown, the diffusion model is trained through the following steps: S11. Process the training image through the key point detector and obtain the predicted coordinates of the key points, and process the training image through manual annotation and obtain the annotated coordinates of the key points.
[0034] The training images are acquired by taking photos of the workpiece to be processed by the image acquisition device (sensor), for example Figure 2 The workpiece in FIG. 4 has 4 points to be processed, and there are usually multiple training images, and the content of each training image is different.
[0035] The training image is processed by the key point detector to obtain the predicted coordinates of the key points. The process is the same as step S1 above. The key point detection results are as follows: Figure 2 As shown in the red circle, the coordinates of the red circle are the predicted coordinates of the key points. The training images are processed by manual annotation to obtain the labeled coordinates of the key points. The key point annotation results are as follows Figure 2 As shown in the green circle in the figure, the coordinates of the green circle are the marked coordinates of the key points.
[0036] S12. Constructing predicted Gaussian distribution through predicted coordinates of key points h k , construct the labeled Gaussian distribution through the labeled coordinates of the key points h 0 .
[0037] For each training image, the predicted coordinates and the labeled coordinates of the key points therein are constructed with reference to the contents of step S2 above for Gaussian distribution. During the construction, the key point coordinates are also taken as the center, and after the variance is set, the required Gaussian distribution can be constructed.
[0038] S13. Extract a local image containing the predicted coordinates of the key points from the training image.
[0039] Referring to the above step S3, the local image in this step is also centered on the predicted coordinates of the key point, and an area of size m×n is cut out from the training image as the local image, where m≥1 and n≥1. m and n can be the same, such as 80×80, 60×60 or 40×40, etc.; m and n can also be different, such as 60×80, 80×60, 60×40, etc.
[0040] It is also easy to understand that step S13 and step S12 can be executed sequentially, in reverse order, or synchronously.
[0041] S14. Annotating Gaussian distribution through diffusion model h 0 The local image is used as context information to guide the diffusion model to perform forward diffusion and obtain a noisy Gaussian distribution with noise. h 0 .
[0042] Among them, a group of several local images cut out from the training image are first stacked, and then feature extraction is performed through the convolutional neural network model to obtain local spatial context semantic embedding. The local spatial context semantic embedding is concatenated with the diffusion step number encoding to form a context vector, which is used as the prompt information of the diffusion model to further label the Gaussian distribution.h 0 Perform forward diffusion and add noise to obtain a noisy Gaussian distribution with noise^ h 0 .
[0043] Among them, a group of several local images cut out from the training image are first scaled to the same size before stacking, usually reduced to the same size, because image reduction helps to increase the speed of model training, for example, unified to 16×16, of course, it can also be other sizes, depending on the requirements of the diffusion model for input.
[0044] S15. Calculate the noise Gaussian distribution through the loss function^ h 0 With the predicted Gaussian distribution h k The deviation between them is calculated and the parameters of the diffusion model are optimized until the model converges to obtain the pre-trained diffusion model.
[0045] During training, the parameters of the diffusion model are optimized by the gradient descent method.
[0046] The final pre-trained diffusion model has the ability to reversely diffuse and denoise the predicted Gaussian distribution composed of the predicted coordinates of the key points detected by the existing key point detector under the context prompt condition, thereby obtaining the denoised Gaussian distribution h , the denoised Gaussian distribution h The center of is the exact coordinate of the key point.
[0047] This embodiment takes the welding of a workpiece (hereinafter referred to as a terminal) by a vision-guided welding robot as an example to introduce the implementation process of the method of the present invention: 1. Get the image to be tested, and you can see that there are multiple terminals in it. Then use the existing key point detector (Base pins and key points detector) to process the image to be tested, so that all the terminals and their key points in the image to be tested can be detected, such as Figure 6 shown.
[0048] 2. For each terminal, the predicted coordinates of the key points detected by the existing key point detector are taken as the center, and then the surrounding area (e.g., 80×80 size) is cut out, and then the image (16×16 size) is obtained by image scaling, and then they are stacked in order to form a module, and then a convolutional neural network model is used to extract features, so as to obtain local spatial context semantic embedding. The process is as follows Figure 7 shown.
[0049] 3. The predicted coordinates of the key points detected by the existing key point detector constitute a predicted Gaussian distribution h k , and concatenate the diffusion step encoding and local spatial context semantic embedding into a context vector as the hint information of the diffusion model to predict the Gaussian distribution h k Perform reverse diffusion and denoising to finally obtain the denoised Gaussian distribution after denoising h , the denoised Gaussian distribution h The center of is the exact coordinate of the key point. Figure 8 shown.
[0050] 4. The precise coordinates of the key points are used to calculate their positions in the robot coordinate system, and the robot is guided to carry the welding tools to the corresponding positions to carry out welding work.
[0051] Example 2 An electronic device includes a memory storing executable program code and a processor coupled to the memory; wherein the processor calls the executable program code stored in the memory to execute the method steps disclosed in the above embodiment.
[0052] Example 3 A computer storage medium stores a computer program, and when the computer program is executed by a processor, the method steps disclosed in the above embodiment are executed.
[0053] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0054] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0055] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0056] It should be noted that, in the description of the present invention, the directions or positional relationships indicated by the terms "up", "down", "left", "right", "front", "back", etc. are descriptions of the structure of the present invention based on the accompanying drawings, and are only for the convenience of describing the present invention, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.
[0057] The "first" and "second" in this technical solution are only used to distinguish the names of the same or similar structures, or corresponding structures with similar functions, and are not an arrangement of the importance of these structures, nor do they have a ranking, comparison of size, or other meanings.
[0058] In addition, unless otherwise clearly specified and limited, the terms "installation" and "connection" should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be a connection between the two structures. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood based on the overall idea of the present invention and the specific context of the present solution.
[0059] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.
Claims
1. A key point detection method, characterized in that: The method comprises the following steps: The image to be tested is processed by a key point detector to obtain the predicted coordinates of the key points; Constructing a predicted Gaussian distribution with the predicted coordinates of the key point as the center; intercepting a local image containing the predicted coordinates of the key points from the image to be tested; The predicted Gaussian distribution is processed by a pre-trained diffusion model, and the local image is used as context information to guide the pre-trained diffusion model to perform reverse diffusion so as to remove noise in the predicted Gaussian distribution and obtain a denoised Gaussian distribution, the center of which is the precise coordinate of the key point.
2. The key point detection method according to claim 1, characterized in that: The pre-trained diffusion model is obtained by the following steps: The training images are processed by a key point detector to obtain the predicted coordinates of the key points, and the training images are processed by manual annotation to obtain the annotated coordinates of the key points; The predicted Gaussian distribution is constructed by the predicted coordinates of the key points h k , construct the labeled Gaussian distribution through the labeled coordinates of the key points h 0 ; Cutting out a local image including the predicted coordinates of the key points from the training image; The Gaussian distribution of the annotations is simulated by the diffusion model h 0 Processing is performed, and the local image is used as context information to guide the diffusion model to perform forward diffusion, and a noise Gaussian distribution with noise is obtained. h 0 ; The noise Gaussian distribution is calculated by the loss function ^ h 0 With the predicted Gaussian distribution h k The deviation between them is calculated, and the parameters of the diffusion model are optimized until the model converges to obtain the pre-trained diffusion model.
3. The key point detection method according to claim 1, characterized in that: The local image is obtained by the following steps: taking the predicted coordinates of the key point as the center, cutting out a region of size m×n as the local image, wherein m≥1, n≥1.
4. The key point detection method according to claim 1, characterized in that: The predicted Gaussian distribution has a set variance.
5. The key point detection method according to claim 2, characterized in that: The parameters of the diffusion model are optimized by gradient descent method.
6. The key point detection method according to claim 1, characterized in that: The local images are first stacked, and then feature extracted through a convolutional neural network model to obtain local spatial context semantic embedding, and the local spatial context semantic embedding is spliced with the diffusion step encoding to form a context vector for the diffusion model prompt information.
7. The key point detection method according to claim 6, characterized in that: Before the partial images are stacked, they are scaled to the same size.
8. An electronic device, characterized in that: It comprises a memory storing executable program code and a processor coupled to the memory; wherein the processor calls the executable program code stored in the memory to execute the method according to any one of claims 1-7.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is performed.