A method, apparatus, equipment, and medium for automatically determining the registration points of an operational object.

By training a model using a combination of registration point supervision loss and coordinate regression loss function based on 3D image data and semantic mask image data, the problem of accuracy in determining registration points in medical images is solved, and more efficient registration point prediction is achieved.

CN120894294BActive Publication Date: 2026-11-17GUANGZHOU AIMUYI TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510959591.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-07-11
Publication Date
2026-11-17
Estimated Expiration
2045-07-11

AI Technical Summary

Technical Problem

Existing technologies struggle to accurately determine registration points in medical images due to interference from factors such as lesions, deformation, and noise, resulting in low accuracy in registration point determination.

Method used

Registration points are determined based on the 3D image data of the operation object and its semantic mask image data. A joint loss function of registration point supervision loss and coordinate regression loss is used to train the registration point prediction model, eliminating the influence of interference factors and improving the accuracy and robustness of registration point prediction.

Benefits of technology

This improves the accuracy and robustness of registration point determination, ensuring the precision of surgical navigation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120894294B_ABST
    Figure CN120894294B_ABST
Patent Text Reader

Abstract

This application discloses a method, apparatus, device, and medium for automatically determining registration points of an object, belonging to the field of image processing technology. The method includes: acquiring three-dimensional image data of the object; performing semantic segmentation on the three-dimensional image data to obtain semantic mask image data; generating input image data based on the three-dimensional image data and the semantic mask image data; inputting the input image data into a pre-trained registration point prediction model to obtain probability distribution image data of registration points; wherein the registration point prediction model is trained based on a joint loss function of registration point supervision loss and coordinate regression loss; and determining registration points in the three-dimensional image data according to the probability distribution image data. This technical solution provides important prior knowledge for determining registration points to improve accuracy and eliminates the influence of interference factors in the three-dimensional image data, improving the robustness and accuracy of the registration point prediction model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to a method, apparatus, device, and medium for automatically determining the registration points of an operation object. Background Technology

[0002] Registration points can accurately match the patient's preoperative medical images with the actual surgical structures during the operation. Therefore, the accuracy of the registration point determination directly determines the accuracy of surgical navigation and the surgical outcome.

[0003] Currently, automatic registration point determination in medical images is mainly achieved through image processing techniques such as edge detection and template matching. However, due to the complex factors often present in medical images, including lesions, deformations, and noise, these image processing techniques struggle to accurately extract features, leading to significant deviations in the determined registration points. Furthermore, while deep learning models can also be used to determine registration points, the various interfering factors in medical images prevent deep learning models from being properly trained, resulting in registration points output by these models being affected by these interferences and exhibiting low accuracy.

[0004] Therefore, how to eliminate the influence of interfering factors in medical images and improve the accuracy of registration point determination is a problem that urgently needs to be solved by those in this field. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for automatically determining registration points of an object. The purpose is to determine registration points based on the three-dimensional image data and semantic mask image data of the object, which can provide important prior knowledge for the determination of registration points, thereby improving the accuracy of registration point determination. Furthermore, by training the registration point prediction model through a joint loss function of registration point supervision loss and coordinate regression loss, the influence of interference factors in the three-dimensional image data can be eliminated, thereby improving the robustness and accuracy of the registration point prediction model.

[0006] In a first aspect, embodiments of this application provide a method for automatically determining the registration points of an operational object, the method comprising: Acquire the 3D image data of the object being manipulated; The semantic mask image data is obtained by performing semantic segmentation on the three-dimensional image data. Input image data is generated based on the three-dimensional image data and the semantic mask image data; The input image data is fed into a pre-trained registration point prediction model to obtain the probability distribution image data of registration points; wherein, the registration point prediction model is trained based on a joint loss function of registration point supervision loss and coordinate regression loss; Based on the probability distribution image data, registration points are determined in the three-dimensional image data.

[0007] Furthermore, the registration point prediction model includes a feature extraction module, a channel mapping module, an upsampling module, a skip connection module, and a probability mapping module; the feature extraction module is connected to the channel mapping module and the skip connection module respectively, the channel mapping module is connected to the upsampling module, the skip connection module is connected to the upsampling module, and the upsampling module is connected to the probability mapping module. Accordingly, the input image data is input into a pre-trained registration point prediction model to obtain the probability distribution image data of registration points through the registration point prediction model, including: The feature extraction module extracts features from the input image data to obtain feature image data. The feature image data is mapped to intermediate image data through the channel mapping module; wherein, the number of channels in the intermediate image data is equal to the preset number of registration points; The intermediate image data is upsampled by the upsampling module, and the upsampling result is connected by the skip connection module to obtain the reconstructed image data. The probability mapping module maps the resolved and restored image data into probability distribution image data of registration points.

[0008] Furthermore, the training process of the registration point prediction model includes: Acquire training sample image data and corresponding real registration points; Based on the training sample image data and the real registration points, Gaussian distribution image data is constructed; The training sample image data is input into a pre-built registration point prediction model to obtain the probability distribution image data of the predicted registration points of the training sample image data through the registration point prediction model. Based on the joint loss function of registration point supervision loss and coordinate regression loss, the loss error of the registration point prediction model is determined according to the Gaussian distribution image data and the probability distribution image data. The model parameters of the registration point prediction model are optimized based on the loss error. Repeat the above steps until the loss error meets the early training stop condition or the number of executions reaches the preset total number of training iterations.

[0009] Furthermore, based on the joint loss function of registration point supervision loss and coordinate regression loss, and according to the Gaussian distribution image data and the probability distribution image data, the loss error of the registration point prediction model is determined, including: The registration point supervision loss is determined based on the Gaussian distribution image data and the probability distribution image data, and the coordinate regression loss is determined based on the probability distribution image data; The registration point supervision loss is multiplied by a first preset weight to obtain a first loss calculation factor, and the coordinate regression loss is multiplied by a second preset weight to obtain a second loss calculation factor. The loss error of the registration point prediction model is obtained by adding the first loss calculation factor and the second loss calculation factor.

[0010] Furthermore, determining the registration point supervision loss based on the Gaussian distribution image data and the probability distribution image data includes: The mean square error between the Gaussian distribution image data and the probability distribution image data is calculated as the registration point supervision loss.

[0011] Furthermore, determining the coordinate regression loss based on the probability distribution image data includes: The continuous coordinate positions of the predicted registration points are determined based on the probability distribution image data. The Euclidean distance between the predicted coordinate position of the predicted registration point and the continuous coordinate position is calculated as the coordinate regression loss.

[0012] Furthermore, generating input image data based on the 3D image data and the semantic mask image data includes: Based on preset cropping rules, the semantic mask image data is cropped to obtain the first interest image data; The three-dimensional image data is cropped based on the first image data of interest to obtain the second image data of interest; The first image data of interest and the second image data of interest are merged to obtain the input image data.

[0013] Secondly, embodiments of this application provide an automatic registration point determination device for an operational object, the device comprising: The 3D image acquisition module is used to acquire 3D image data of the object being manipulated. A semantic segmentation module is used to perform semantic segmentation on the three-dimensional image data to obtain the semantic mask image data; An input image generation module is used to generate input image data based on the three-dimensional image data and the semantic mask image data; The probability distribution prediction module is used to input the input image data into a pre-trained registration point prediction model to obtain the probability distribution image data of the registration points through the registration point prediction model; wherein, the registration point prediction model is trained based on a joint loss function of registration point supervision loss and coordinate regression loss; The registration point determination module is used to determine registration points in the three-dimensional image data based on the probability distribution image data.

[0014] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0015] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0016] In this embodiment, three-dimensional image data of the object to be operated on is acquired; semantic segmentation is performed on the three-dimensional image data to obtain semantic mask image data; input image data is generated based on the three-dimensional image data and the semantic mask image data; the input image data is input into a pre-trained registration point prediction model to obtain probability distribution image data of registration points through the registration point prediction model; wherein, the registration point prediction model is trained based on a joint loss function of registration point supervision loss and coordinate regression loss; and registration points are determined in the three-dimensional image data according to the probability distribution image data. The above-described automatic registration point determination method for the object to be operated on, by determining registration points based on the three-dimensional image data and semantic mask image data of the object to be operated on, can provide important prior knowledge for the determination of registration points, thereby improving the accuracy of registration point determination. Furthermore, training the registration point prediction model using a joint loss function of registration point supervision loss and coordinate regression loss can eliminate the influence of interference factors in the three-dimensional image data, improving the robustness and accuracy of the registration point prediction model. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the automatic determination method for registration points of the operation object provided in Embodiment 1 of this application; Figure 2 This is a schematic diagram of the registration point prediction model provided in Embodiment 2 of this application; Figure 3 This is a flowchart illustrating the automatic determination method for registration points of the operation object provided in Embodiment 2 of this application; Figure 4This is an example diagram illustrating the processing procedure of the registration point prediction model provided in Embodiment 2 of this application; Figure 5 This is a flowchart illustrating the training process of the registration point prediction model provided in Embodiment 3 of this application; Figure 6 This is a schematic diagram of the automatic registration point determination device for the operation object provided in Embodiment 4 of this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in Embodiment 5 of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] The following description, in conjunction with the accompanying drawings, details the automatic registration point determination method, apparatus, equipment, and medium for the operational object provided in this application, through specific embodiments and application scenarios.

[0022] Example 1 Figure 1 This is a flowchart illustrating the automatic determination method for registration points of the operation object provided in Embodiment 1 of this application. Figure 1 As shown, the specific steps include the following: S101, Obtain the three-dimensional image data of the object being operated on; First, this application applies to scenarios involving spatial registration based on the object being operated on, specifically, for example, in the context of surgical navigation registration based on the patient's surgical site before hip replacement surgery. Spatial registration refers to the process of precisely aligning spatial data acquired from different sources, at different times, or from different perspectives in terms of spatial location.

[0023] Registration points are feature points used to establish mapping relationships between different coordinate systems during spatial registration. In the context of hip replacement surgery, registration points have anatomical significance.

[0024] It is understood that the executing entity of this application can be a terminal device with image processing capabilities. Specifically, the generation of each image data and the determination of registration points can be performed by the terminal device, and the registration between the three-dimensional image space of the object being operated on and the actual operating space can be completed based on the determined registration points.

[0025] The object of manipulation can refer to the physical entity that requires specific manipulation. In the context of hip replacement surgery, the object of manipulation is the patient's surgical site, namely the hip.

[0026] Three-dimensional image data is an image that can describe the three-dimensional spatial structure of an object being manipulated. Three-dimensional image data of an object can be acquired using equipment such as laser scanning instruments, optical photogrammetry instruments, structured light scanning instruments, or tomographic scanning instruments.

[0027] In the context of hip replacement surgery, three-dimensional image data can refer to CT (Computed Tomography) images. Specifically, CT images are tomographic images generated using computed tomography technology, which can present the internal structure of an object in three dimensions; CT images can be acquired by a CT scanner.

[0028] S102, perform semantic segmentation on the three-dimensional image data to obtain the semantic mask image data; Semantic segmentation can classify voxels in 3D image data into predefined semantic categories. In semantic mask image data, the specific value of each voxel corresponds to a semantic category. A voxel is the basic unit used to represent volume data in 3D space.

[0029] In the context of hip replacement surgery, predefined semantic categories can include the acetabulum and the femoral head. Accordingly, in the semantic mask image data, a voxel with a specific value of 1 can represent the acetabulum, a voxel with a specific value of 2 can represent the femoral head, and a voxel with a specific value of 0 can represent background tissue.

[0030] Three-dimensional image data is input into a semantic segmentation model, which can automatically output semantic mask image data. The semantic segmentation model can be a deep learning model pre-trained based on a large amount of 3D image data and its corresponding semantic category annotation data.

[0031] S103, Generate input image data based on the three-dimensional image data and the semantic mask image data; Input image data can refer to images that can be accurately read by the registration point prediction model.

[0032] The method of generating input image data based on 3D image data and semantic mask image data can be achieved by directly merging the 3D image data and semantic mask image data to obtain the input image data; alternatively, the semantic mask image data can be cropped based on a preset cropping rule to obtain the first image data of interest, and the 3D image data can be cropped based on the first image data of interest to obtain the second image data of interest, and the first image data of interest and the second image data of interest can be merged to obtain the input image data.

[0033] Correspondingly, the input image data is in the form of... The four-dimensional tensor. Among them, It is the standard symbol for the set of real numbers in mathematics, used to describe the range of values ​​that a variable's elements can take; This represents the input image data. H represents the number of channels in the input image data, W represents the three-dimensional length of the input image data, and D represents the three-dimensional width of the input image data.

[0034] In this technical solution, optionally, generating input image data based on the three-dimensional image data and the semantic mask image data includes: Based on preset cropping rules, the semantic mask image data is cropped to obtain the first interest image data; The three-dimensional image data is cropped based on the first image data of interest to obtain the second image data of interest; The first image data of interest and the second image data of interest are merged to obtain the input image data.

[0035] Predictive cropping rules can be pre-defined rules used to crop regions of interest from semantic mask image data. Understandably, the first image data of interest is the region of interest within the semantic mask image data. The region of interest can refer to a key region related to the current task (i.e., determining registration points), which is the area that needs to be focused on and analyzed in subsequent processing.

[0036] Specifically, as an example, in the scenario of hip replacement surgery, the prediction clipping rule can be: according to a preset fixed size, select the region with the most voxels representing the acetabulum and femoral head in the semantic mask image data as the region of interest, i.e., the first focus image data.

[0037] The second image data of interest is the region in the 3D image data corresponding to the positions of each voxel between it and the first image data of interest. The second image data of interest can be obtained by cropping the 3D image data based on the first image data of interest. This can be achieved by determining the position coordinates of each voxel in the first image data of interest within the semantic mask image data, and then using the voxels corresponding to these position coordinates in the 3D image data as the second image data of interest.

[0038] The method of merging the first and second focus image data to obtain the input image data can be achieved by directly concatenating the first and second focus image data along the channel dimension to form multi-channel input image data. Multi-channel can refer to multiple parallel information layers contained in a voxel, where each channel independently carries a portion of feature information, collectively constituting the complete representation information of that voxel.

[0039] The advantage of this approach is that by cropping the semantic mask image data to obtain the first image data of interest, and cropping the 3D image data to obtain the second image data of interest, and merging the first and second image data of interest to obtain the input image data, the key structural semantic information of the object being operated on can be preserved and the focus can be placed on the registration point-related region, reducing interference from irrelevant backgrounds.

[0040] S104, The input image data is input into a pre-trained registration point prediction model to obtain the probability distribution image data of registration points through the registration point prediction model; wherein, the registration point prediction model is trained based on a joint loss function of registration point supervision loss and coordinate regression loss; The number of channels in the probability distribution image data of registration points is equal to the preset number of registration points. In the probability distribution image of registration points, the specific value of a voxel in each channel represents the probability that the voxel is a registration point.

[0041] A registration point prediction model can be a neural network model that automatically outputs the probability distribution image data of the corresponding registration points based on the input image data. The registration point prediction model can be represented as a program function that takes the input image data as input parameters and outputs the probability distribution image data of the registration points as the return value.

[0042] The joint loss function of registration point supervision loss and coordinate regression loss is a loss function that incorporates both registration point supervision loss and coordinate regression loss. Specifically, the loss function is used in machine learning and deep learning to measure the difference between the model's prediction results and the actual results, that is, to quantify the degree of error in the model's prediction; the registration point supervision loss refers to the difference between the predicted probability distribution of the registration point by the registration point prediction model and the actual probability distribution of the registration point; the coordinate regression loss refers to the difference in the accuracy of the predicted coordinates of the registration point.

[0043] S105, Based on the probability distribution image data, determine the registration point in the three-dimensional image data.

[0044] Based on probability distribution image data, the method for determining registration points in 3D image data can be to determine the position coordinates of the voxel with the largest specific value in each channel of the probability distribution image of the registration point, and then determine the voxel with the position coordinates as the registration point in the 3D image data.

[0045] In this embodiment, three-dimensional image data of the object to be operated on is acquired; semantic segmentation is performed on the three-dimensional image data to obtain semantic mask image data; input image data is generated based on the three-dimensional image data and the semantic mask image data; the input image data is input into a pre-trained registration point prediction model to obtain probability distribution image data of registration points through the registration point prediction model; wherein, the registration point prediction model is trained based on a joint loss function of registration point supervision loss and coordinate regression loss; and registration points are determined in the three-dimensional image data according to the probability distribution image data. The above-described automatic registration point determination method for the object to be operated on, by determining registration points based on the three-dimensional image data and semantic mask image data of the object to be operated on, can provide important prior knowledge for the determination of registration points, thereby improving the accuracy of registration point determination. Furthermore, training the registration point prediction model using a joint loss function of registration point supervision loss and coordinate regression loss can eliminate the influence of interference factors in the three-dimensional image data, improving the robustness and accuracy of the registration point prediction model.

[0046] Example 2 This solution makes a better improvement on the above embodiment, specifically as follows: the registration point prediction model includes a feature extraction module, a channel mapping module, an upsampling module, a skip connection module, and a probability mapping module; the feature extraction module is connected to the channel mapping module and the skip connection module respectively, the channel mapping module is connected to the upsampling module, the skip connection module is connected to the upsampling module, and the upsampling module is connected to the probability mapping module; correspondingly, the input image data is input to the pre-trained registration point prediction model to obtain the probability distribution image data of the registration points through the registration point prediction model, including: extracting features from the input image data through the feature extraction module to obtain feature image data; mapping the feature image data into intermediate image data through the channel mapping module; wherein, the number of channels of the intermediate image data is equal to the preset number of registration points; performing an upsampling operation on the intermediate image data through the upsampling module, and performing a feature connection operation on the upsampling operation result through the skip connection module to obtain the reconstructed image data; and mapping the reconstructed image data into the probability distribution image data of the registration points through the probability mapping module.

[0047] Figure 2 This is a schematic diagram of the registration point prediction model provided in Embodiment 2 of this application. Figure 2 As shown, the registration point prediction model 200 includes a feature extraction module 210, a channel mapping module 220, an upsampling module 230, a skip connection module 240, and a probability mapping module 250. The feature extraction module 210 is connected to the channel mapping module 220 and the skip connection module 240, the channel mapping module 220 is connected to the upsampling module 230, the skip connection module 240 is connected to the upsampling module 230, and the upsampling module 230 is connected to the probability mapping module 250.

[0048] Figure 3 This is a flowchart illustrating the automatic determination method for registration points of the operation object provided in Embodiment 2 of this application. Figure 3 As shown, the specific steps include the following: S301, acquire the 3D image data of the object being operated on; S302, perform semantic segmentation on the three-dimensional image data to obtain the semantic mask image data; S303, Generate input image data based on the three-dimensional image data and the semantic mask image data; S304, The feature extraction module extracts features from the input image data to obtain feature image data; The feature extraction module is used to extract features from the input image data to obtain feature image data. The feature extraction module may include at least three feature extraction units, each of which includes at least one convolutional layer and at least one pooling layer. The image data output by the last feature extraction unit is the feature image data.

[0049] Convolutional layers perform convolution operations, responsible for capturing local detail features from images. Convolutional layers typically change the number of channels in the image data. The number of channels in the output image data of a convolutional layer is equal to the number of convolutional kernels. The convolutional kernel is a matrix with a specific size and number of channels.

[0050] Pooling layers perform pooling operations, which are responsible for reducing the dimensionality and compressing the image to enhance key features and suppress irrelevant details; pooling layers generally change the size of the image data.

[0051] The activation function for both convolutional and pooling layers is the ReLU function.

[0052] Figure 4 This is an example diagram illustrating the processing procedure of the registration point prediction model provided in Embodiment 2 of this application. For example... Figure 4 As shown, the cube labeled ① represents the input image data, the cube labeled ② represents the image data obtained after performing a convolution operation on the previous image data, and the cube labeled ③ represents the image data obtained after performing a pooling operation on the previous image data.

[0053] In this example, the input image data is a 256*256*256*6 ( Image data. Figure 4 The topmost cube represents the processing of the feature extraction module, which includes five feature extraction units, each corresponding to a color. Each feature extraction unit includes two convolutional layers and one pooling layer, and the final feature image data is 8*8*8*4096.

[0054] S305, the feature image data is mapped to intermediate image data through the channel mapping module; wherein, the number of channels in the intermediate image data is equal to the preset number of registration points; The channel mapping module does not change the size of the feature image data, but only maps the number of channels of the feature image data to a preset number of registration points to obtain intermediate image data. The channel mapping module uses the preset number of convolution kernels to perform convolution operations on the feature image data to obtain intermediate image data.

[0055] like Figure 4As shown, the cube labeled ④ represents the intermediate image data obtained after processing by the channel mapping module; the 8*8*8*k intermediate image data has the same size as the 8*8*8*4096 feature image data, but the number of channels is mapped to k. Here, k is the preset number of registration points.

[0056] S306, the intermediate image data is upsampled by the upsampling module, and the upsampling result is connected by the skip connection module to obtain the reconstructed image data. The upsampling module is used to perform upsampling operations. Upsampling is a technique that converts low-resolution feature maps into high-resolution feature maps, with the core purpose of restoring the size of the image data.

[0057] In this scheme, specifically, the upsampling operation can be a bilinear upsampling operation. Bilinear upsampling refers to generating new voxels by weighted averaging of surrounding voxel values. Compared to simple nearest-neighbor interpolation (directly copying voxels), this produces smoother results and reduces jagged artifacts.

[0058] The skip connection module is used to connect intermediate feature image data obtained during the feature extraction module's processing with intermediate upsampled image data obtained during the upsampling module's processing. Specifically, the skip connection module selects intermediate feature image data output from the pooling layer in the intermediate feature extraction unit of the feature extraction module, performs pooling on the intermediate feature image data to obtain intermediate skip image data with a reduced size and a preset number of channels, and finally performs a connection operation on intermediate skip image data of the same size with intermediate upsampled image data to obtain the connected result image data. The connection operation can be to add the specific values ​​of the same channel of the same voxel.

[0059] The reconstructed image data is the image data obtained after upsampling and feature connection operations, which has the same size as the input image data and the number of channels equal to the preset number.

[0060] like Figure 4 As shown, the cube labeled ⑤ represents the image data obtained after performing an upsampling operation on the previous image data, and the last cube labeled ⑤ represents the reconstructed image data; the white cube labeled ③ represents the intermediate skip image data; and the cube labeled ⑥ represents the concatenated image data.

[0061] S307, The probability mapping module maps the resolved and restored image data into probability distribution image data of registration points; The probability mapping module is used to perform convolution operations on the reconstructed image data, so that the reconstructed image data is mapped to the probability value of the voxel in each channel within the interval [0,1], thus obtaining the probability distribution image data of the registration points.

[0062] like Figure 4 As shown, the cube labeled ⑦ represents the probability distribution image data.

[0063] S308, Based on the probability distribution image data, determine the registration point in the three-dimensional image data.

[0064] The advantages of this scheme are that the feature extraction module deeply mines the multi-level semantic features of the input image data, the channel mapping module adjusts the feature dimensions to focus on the key information of the registration point, the upsampling module gradually restores the spatial resolution of the feature image data to match the input image data, the skip connection module fuses high and low level features to avoid the loss of detailed information in feature transmission, and the probability mapping module finally outputs the probability distribution of the registration point, thus achieving efficient and accurate registration point prediction.

[0065] Example 3 Figure 5 This is a flowchart illustrating the training process of the registration point prediction model provided in Embodiment 3 of this application. This solution makes a further improvement on the above embodiment, specifically: the training process of the registration point prediction model includes: acquiring training sample image data and corresponding real registration points; constructing Gaussian distribution image data based on the training sample image data and the real registration points; inputting the training sample image data into the pre-constructed registration point prediction model to obtain probability distribution image data of the predicted registration points corresponding to the training sample image data; determining the loss error of the registration point prediction model based on the joint loss function of registration point supervision loss and coordinate regression loss, according to the Gaussian distribution image data and the probability distribution image data; optimizing the model parameters of the registration point prediction model based on the loss error; repeating the above steps until the loss error meets the early training termination condition or the number of executions reaches the preset total number of training iterations.

[0066] like Figure 5 As shown, the specific steps include the following: S501, acquire training sample image data and corresponding real registration points; Training sample image data refers to the input image data used to train the registration point prediction model. A large amount of 3D image data of different objects is acquired, and semantic segmentation is performed on these 3D image data to obtain semantic mask image data. Input image data is then generated based on the 3D image data and the semantic mask image data as training sample image data.

[0067] True registration points refer to registration points that are manually marked in 3D image data.

[0068] S502, construct Gaussian distribution image data based on the training sample image data and the real registration points; Gaussian distribution image data refers to image data that transforms real registration points into a probability distribution image with a three-dimensional Gaussian distribution. The process of constructing Gaussian distribution image data based on training sample image data and real registration points includes: constructing blank image data of the same size and with a preset number of channels equal to the number of registration points based on the training sample image data; for each real registration point, generating a three-dimensional Gaussian distribution function centered on its position coordinates on the corresponding channel, and assigning specific values ​​to each voxel in that channel according to this three-dimensional Gaussian distribution function. The three-dimensional Gaussian distribution function is used to describe three continuous random variables (…). The joint probability distribution of ).

[0069] S503, the training sample image data is input into a pre-built registration point prediction model to obtain the probability distribution image data of the predicted registration points of the training sample image data through the registration point prediction model; The registration point prediction model can be represented as a program function that takes training sample image data as input parameters and outputs the probability distribution image data of the predicted registration points as the return value.

[0070] S504, Based on the joint loss function of registration point supervision loss and coordinate regression loss, determine the loss error of the registration point prediction model according to the Gaussian distribution image data and the probability distribution image data; The loss error of the registration point prediction model is the quantified difference between the Gaussian distributed image data and the probability distributed image data, calculated based on the joint loss function of the registration point supervision loss and the coordinate regression loss. The method for determining the loss error of the registration point prediction model based on the joint loss function of the registration point supervision loss and the coordinate regression loss, according to the Gaussian distributed image data and the probability distributed image data, can be as follows: determine the registration point supervision loss based on the Gaussian distributed image data and the probability distributed image data; determine the coordinate regression loss based on the probability distributed image data; multiply the registration point supervision loss by a first preset weight to obtain a first loss calculation factor; multiply the coordinate regression loss by a second preset weight to obtain a second loss calculation factor; and add the first loss calculation factor and the second loss calculation factor to obtain the loss error of the registration point prediction model.

[0071] In this technical solution, optionally, based on the joint loss function of registration point supervision loss and coordinate regression loss, the loss error of the registration point prediction model is determined according to the Gaussian distribution image data and the probability distribution image data, including: The registration point supervision loss is determined based on the Gaussian distribution image data and the probability distribution image data, and the coordinate regression loss is determined based on the probability distribution image data; The registration point supervision loss is multiplied by a first preset weight to obtain a first loss calculation factor, and the coordinate regression loss is multiplied by a second preset weight to obtain a second loss calculation factor. The loss error of the registration point prediction model is obtained by adding the first loss calculation factor and the second loss calculation factor.

[0072] The registration point supervision loss can be determined by using Gaussian distributed image data and probability distributed image data. One method is to use the mean square error between the Gaussian distributed image data and the probability distributed image data as the registration point supervision loss.

[0073] In this technical solution, optionally, determining the registration point supervision loss based on the Gaussian distribution image data and the probability distribution image data includes: The mean square error between the Gaussian distribution image data and the probability distribution image data is calculated as the registration point supervision loss.

[0074] Mean squared error is a commonly used statistical indicator used to measure the average squared difference between predicted and actual values.

[0075] The formula for calculating the mean square error of the same channel for Gaussian distribution image data and probability distribution image data is as follows: ; in, This represents the mean square error value of the t-th channel.

[0076] Correspondingly, the formula for calculating the mean square error between Gaussian distribution image data and probability distribution image data is: ; in, This represents the mean square error between Gaussian distributed image data and probability distributed image data.

[0077] The advantage of this approach is that by calculating the mean square error between the Gaussian distribution image data and the probability distribution image data as the registration point supervision loss, the discrete real registration point coordinates can be transformed into a continuously differentiable probability distribution optimization problem. This provides the model with a smooth and robust supervision signal, effectively avoiding gradient fluctuations when directly regressing coordinates, and ultimately improving the accuracy of registration point prediction.

[0078] One way to determine the coordinate regression loss based on probability distribution image data is to determine the continuous coordinate positions of the predicted registration points based on the probability distribution image data, and then calculate the Euclidean distance between the predicted coordinate positions of the predicted registration points and the continuous coordinate positions as the coordinate regression loss.

[0079] In this technical solution, optionally, determining the coordinate regression loss based on the probability distribution image data includes: The continuous coordinate positions of the predicted registration points are determined based on the probability distribution image data. The Euclidean distance between the predicted coordinate position of the predicted registration point and the continuous coordinate position is calculated as the coordinate regression loss.

[0080] Continuous coordinate position refers to the coordinate position with real-number precision obtained by "continuous processing" of probability distribution image data.

[0081] By using the Soft-argmax method, the continuous coordinate positions of the predicted registration points can be determined based on the probability distribution image data.

[0082] The Soft-argmax method is an operation in deep learning used to "continuousize" probability distributions. It is used to extract continuous coordinates or positional information from discrete probability graphs, combining the interpretability of probability distributions with the continuity of regression tasks.

[0083] Specifically, for each channel in the probability distribution image data, the normalized probability of each voxel is determined based on the Softmax operation, and the coordinate position of each voxel is multiplied by the normalized probability. Finally, the results of the multiplications are summed to obtain the continuous coordinate position of the prediction registration point corresponding to that channel.

[0084] The formula for calculating the Euclidean distance between the predicted coordinate position and the continuous coordinate position of the predicted registration point is: ; in, This represents the Euclidean distance between the predicted coordinates of the predicted registration point corresponding to the t-th channel and its continuous coordinates. , , This represents the predicted coordinate position of the predicted registration point corresponding to the t-th channel. , , This represents the continuous coordinate position of the predicted registration point corresponding to the t-th channel.

[0085] Accordingly, the formula for calculating the coordinate regression loss is: ; in, This represents the coordinate regression loss.

[0086] The advantage of this approach is that by determining the continuous coordinate positions of the predicted registration points based on probability distribution image data, and calculating the Euclidean distance between the predicted coordinate positions and the continuous coordinate positions as the coordinate regression loss, the continuous coordinate positions can be used as the supervision target to strengthen the model's learning of the precise positions of the registration points, thus forcing the predicted coordinate positions output by the model to be closer to the true coordinate positions.

[0087] The formula for calculating the loss error of the registration point prediction model is: ; in, This represents the loss error of the registration point prediction model; As the first preset weight, This is the first loss calculation factor; As the second preset weight, This is the second loss calculation factor; .

[0088] The advantage of this scheme is that the registration point supervision loss can guide the model to learn the spatial distribution characteristics of the registration points, and the coordinate regression loss can strengthen the constraint on the precise location of the registration points. After weighted summation, the influence weights of the two losses can be flexibly adjusted according to different scenarios, so that the model can achieve a balance between overall distribution learning and precise positioning.

[0089] S505, Optimize the model parameters of the registration point prediction model based on the loss error; Model parameters refer to the adjustable parameters in the registration point prediction model that can be trained and learned.

[0090] By using gradient descent, the model parameters of the registration point prediction model can be optimized based on the loss error. Gradient descent is an optimization algorithm that continuously adjusts the model parameters "along the direction in which the loss error decreases".

[0091] S506, identify whether the loss error meets the training early termination condition; if yes, end the process; if no, execute S507. As is understandable, S501-S505 constitutes one complete epoch (training round). In machine learning and deep learning, an epoch refers to the process by which a model completes one full training run on the entire training dataset.

[0092] Early stopping of training can refer to a situation where the accuracy of the registration point prediction model has reached its limit before the preset total number of iterations has been achieved, at which point further training of the registration point prediction model can be stopped. As an example, the loss error satisfying the early stopping condition could be that the loss error has not decreased for a consecutive preset number of epochs (e.g., 10 epochs).

[0093] S507: Check if the number of executions has reached the preset total number of training iterations; if yes, end the process; if no, execute S501.

[0094] The preset total number of times refers to the total number of epochs that the plan is scheduled to execute in advance.

[0095] Stopping epochs when the preset total number of training iterations is reached avoids resource waste caused by unlimited training.

[0096] The advantage of this approach is that by training the registration point prediction model using a joint loss function based on registration point supervision loss and coordinate regression loss, the influence of interference factors in 3D image data can be eliminated, thereby improving the robustness and accuracy of the registration point prediction model.

[0097] Example 4 Figure 6 This is a schematic diagram of the automatic registration point determination device for the operation object provided in Embodiment 4 of this application. Figure 6 As shown, the device includes: The 3D image acquisition module 610 is used to acquire 3D image data of the object being operated on. The semantic segmentation module 620 is used to perform semantic segmentation on the three-dimensional image data to obtain the semantic mask image data; The input image generation module 630 is used to generate input image data based on the three-dimensional image data and the semantic mask image data; The probability distribution prediction module 640 is used to input the input image data into a pre-trained registration point prediction model to obtain the probability distribution image data of the registration points through the registration point prediction model; wherein, the registration point prediction model is trained based on a joint loss function of registration point supervision loss and coordinate regression loss; The registration point determination module 650 is used to determine registration points in the three-dimensional image data based on the probability distribution image data.

[0098] Furthermore, the registration point prediction model includes a feature extraction module, a channel mapping module, an upsampling module, a skip connection module, and a probability mapping module; the feature extraction module is connected to the channel mapping module and the skip connection module respectively, the channel mapping module is connected to the upsampling module, the skip connection module is connected to the upsampling module, and the upsampling module is connected to the probability mapping module. Accordingly, the probability distribution prediction module 640 is specifically used for: The feature extraction module extracts features from the input image data to obtain feature image data. The feature image data is mapped to intermediate image data through the channel mapping module; wherein, the number of channels in the intermediate image data is equal to the preset number of registration points; The intermediate image data is upsampled by the upsampling module, and the upsampling result is connected by the skip connection module to obtain the reconstructed image data. The probability mapping module maps the resolved and restored image data into probability distribution image data of registration points.

[0099] Furthermore, the device also includes: a model training module 660, specifically used for: Acquire training sample image data and corresponding real registration points; Based on the training sample image data and the real registration points, Gaussian distribution image data is constructed; The training sample image data is input into a pre-built registration point prediction model to obtain the probability distribution image data of the predicted registration points of the training sample image data through the registration point prediction model. Based on the joint loss function of registration point supervision loss and coordinate regression loss, the loss error of the registration point prediction model is determined according to the Gaussian distribution image data and the probability distribution image data. The model parameters of the registration point prediction model are optimized based on the loss error. Repeat the above steps until the loss error meets the early training stop condition or the number of executions reaches the preset total number of training iterations.

[0100] Furthermore, the model training module 660 is also used for: The registration point supervision loss is determined based on the Gaussian distribution image data and the probability distribution image data, and the coordinate regression loss is determined based on the probability distribution image data; The registration point supervision loss is multiplied by a first preset weight to obtain a first loss calculation factor, and the coordinate regression loss is multiplied by a second preset weight to obtain a second loss calculation factor. The loss error of the registration point prediction model is obtained by adding the first loss calculation factor and the second loss calculation factor.

[0101] Furthermore, the model training module 660 is also used for: Determining the registration point supervision loss based on the Gaussian distribution image data and the probability distribution image data includes: The mean square error between the Gaussian distribution image data and the probability distribution image data is calculated as the registration point supervision loss.

[0102] Furthermore, the model training module 660 is also used for: The continuous coordinate positions of the predicted registration points are determined based on the probability distribution image data. The Euclidean distance between the predicted coordinate position of the predicted registration point and the continuous coordinate position is calculated as the coordinate regression loss.

[0103] Furthermore, the input image generation module 630 is specifically used for: Based on preset cropping rules, the semantic mask image data is cropped to obtain the first interest image data; The three-dimensional image data is cropped based on the first image data of interest to obtain the second image data of interest; The first image data of interest and the second image data of interest are merged to obtain the input image data.

[0104] In this embodiment, a 3D image acquisition module is used to acquire 3D image data of the object to be manipulated; a semantic segmentation module is used to perform semantic segmentation on the 3D image data to obtain the semantic mask image data; an input image generation module is used to generate input image data based on the 3D image data and the semantic mask image data; a probability distribution prediction module is used to input the input image data into a pre-trained registration point prediction model to obtain the probability distribution image data of registration points through the registration point prediction model; wherein, the registration point prediction model is trained based on a joint loss function of registration point supervision loss and coordinate regression loss; and a registration point determination module is used to determine the registration points in the 3D image data according to the probability distribution image data. The aforementioned automatic registration point determination device for the operation object determines the registration points based on the 3D image data and semantic mask image data of the operation object. This provides important prior knowledge for the determination of registration points, thereby improving the accuracy of registration point determination. Furthermore, by training the registration point prediction model through the joint loss function of registration point supervision loss and coordinate regression loss, the influence of interference factors in the 3D image data can be eliminated, improving the robustness and accuracy of the registration point prediction model.

[0105] The automatic registration point determination device for the operation object in the embodiments of this application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. The embodiments of this application do not impose specific limitations.

[0106] The automatic registration point determination device for the operation object in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit it.

[0107] The automatic registration point determination device for the operation object provided in this application embodiment can realize the various processes implemented in embodiments one to three above. To avoid repetition, it will not be described again here.

[0108] Example 5 Figure 7 This is a schematic diagram of the structure of the electronic device provided in Embodiment 5 of this application. Figure 7 As shown, this application provides an electronic device 700, including a processor 701, a memory 702, and a program or instructions stored in the memory 702 and executable on the processor 701. When the program or instructions are executed by the processor 701, they implement the various processes of the above-mentioned automatic determination method for registration points of the operation object and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0109] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0110] Example 6 This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described method embodiment for automatically determining the registration point of the operation object and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0111] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0112] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0114] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0115] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A method of automatically determining a registration point of an object, characterized by, The method includes: Acquire the 3D image data of the object being manipulated; Semantic segmentation is performed on the three-dimensional image data to obtain semantic mask image data; Input image data is generated based on the three-dimensional image data and the semantic mask image data; The input image data is fed into a pre-trained registration point prediction model to obtain the probability distribution image data of registration points; wherein, the registration point prediction model is trained based on a joint loss function of registration point supervision loss and coordinate regression loss; Based on the probability distribution image data, registration points are determined in the three-dimensional image data; The training process of the registration point prediction model includes: acquiring training sample image data and corresponding real registration points; constructing Gaussian distribution image data based on the training sample image data and the real registration points; inputting the training sample image data into the pre-constructed registration point prediction model to obtain probability distribution image data of the predicted registration points of the training sample image data through the registration point prediction model; determining the loss error of the registration point prediction model based on the joint loss function of registration point supervision loss and coordinate regression loss, according to the Gaussian distribution image data and the probability distribution image data; optimizing the model parameters of the registration point prediction model based on the loss error; repeating the above steps until the loss error meets the early training termination condition or the number of executions reaches the preset total number of training iterations.

2. The method for automatically determining the registration points of the operation object according to claim 1, characterized in that, The registration point prediction model includes a feature extraction module, a channel mapping module, an upsampling module, a skip connection module, and a probability mapping module; the feature extraction module is connected to the channel mapping module and the skip connection module respectively, the channel mapping module is connected to the upsampling module, the skip connection module is connected to the upsampling module, and the upsampling module is connected to the probability mapping module. Accordingly, the input image data is input into a pre-trained registration point prediction model to obtain the probability distribution image data of registration points through the registration point prediction model, including: The feature extraction module extracts features from the input image data to obtain feature image data. The feature image data is mapped to intermediate image data through the channel mapping module; wherein, the number of channels in the intermediate image data is equal to the preset number of registration points; The intermediate image data is upsampled by the upsampling module, and the upsampling result is connected by the skip connection module to obtain the reconstructed image data. The probability mapping module maps the resolved and restored image data into probability distribution image data of registration points.

3. The method for automatically determining the registration points of the operation object according to claim 1, characterized in that, Based on the joint loss function of registration point supervision loss and coordinate regression loss, and according to the Gaussian distribution image data and the probability distribution image data, the loss error of the registration point prediction model is determined, including: The registration point supervision loss is determined based on the Gaussian distribution image data and the probability distribution image data, and the coordinate regression loss is determined based on the probability distribution image data; The registration point supervision loss is multiplied by a first preset weight to obtain a first loss calculation factor, and the coordinate regression loss is multiplied by a second preset weight to obtain a second loss calculation factor. The loss error of the registration point prediction model is obtained by adding the first loss calculation factor and the second loss calculation factor.

4. The method for automatically determining the registration points of the operation object according to claim 3, characterized in that, Determining the registration point supervision loss based on the Gaussian distribution image data and the probability distribution image data includes: The mean square error between the Gaussian distribution image data and the probability distribution image data is calculated as the registration point supervision loss.

5. The method for automatically determining the registration points of the operation object according to claim 3, characterized in that, Determining the coordinate regression loss based on the probability distribution image data includes: The continuous coordinate positions of the predicted registration points are determined based on the probability distribution image data. The Euclidean distance between the predicted coordinate position of the predicted registration point and the continuous coordinate position is calculated as the coordinate regression loss.

6. The method for automatically determining the registration points of the operation object according to claim 1, characterized in that, The input image data is generated based on the three-dimensional image data and the semantic mask image data, including: Based on preset cropping rules, the semantic mask image data is cropped to obtain the first interest image data; The three-dimensional image data is cropped based on the first image data of interest to obtain the second image data of interest; The first image data of interest and the second image data of interest are merged to obtain the input image data.

7. An automatic registration point determination device for an operating object, characterized in that, The device includes: The 3D image acquisition module is used to acquire 3D image data of the object being manipulated. The semantic segmentation module is used to perform semantic segmentation on the three-dimensional image data to obtain semantic mask image data; An input image generation module is used to generate input image data based on the three-dimensional image data and the semantic mask image data; The probability distribution prediction module is used to input the input image data into a pre-trained registration point prediction model to obtain the probability distribution image data of the registration points through the registration point prediction model; wherein, the registration point prediction model is trained based on a joint loss function of registration point supervision loss and coordinate regression loss; The registration point determination module is used to determine registration points in the three-dimensional image data based on the probability distribution image data. The device is further configured to: acquire training sample image data and corresponding real registration points; construct Gaussian distribution image data based on the training sample image data and the real registration points; input the training sample image data into a pre-constructed registration point prediction model to obtain probability distribution image data of the predicted registration points of the training sample image data through the registration point prediction model; determine the loss error of the registration point prediction model based on the joint loss function of registration point supervision loss and coordinate regression loss, according to the Gaussian distribution image data and the probability distribution image data; optimize the model parameters of the registration point prediction model based on the loss error; and repeat the above steps until the loss error meets the early training termination condition or the number of executions reaches the preset total number of training iterations.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the automatic determination method for registration points of the operating object as described in any one of claims 1-6.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the automatic determination method for registration points of the operating object as described in any one of claims 1-6.

Citation Information

Patent Citations

  • Feature point matching method and system for three-dimensional reconstruction

    CN113095371A

  • Image registration method and device, computer equipment and storage medium

    CN118570264A