Jaw key point recognition and key point recognition model training method and oral robot system

Abstract

The embodiment of the invention provides a jawbone key point recognition method, a key point recognition model training method and an oral robot system. The jawbone key point recognition method comprises the steps that a to-be-trained key point recognition model predicts the positions of jawbone key points according to jawbone multi-modal data so as to generate multiple task outputs; calling a plurality of configured loss functions, and calculating a plurality of loss values according to the plurality of task outputs; based on fusion weights distributed for different loss values, fusing the plurality of loss values to obtain a fused loss value; adjusting network parameters of the to-be-trained key point recognition model according to the fusion loss value until model training is finished to obtain a jawbone key point recognition model; and on the basis of the jawbone key point identification model, jawbone key points are identified from input jawbone CBCT data.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of robotics technology, and in particular to a method for recognizing jaw key points and a training method for a key point recognition model, and an oral robot system. Background Art

[0002] The jawbone, as an important part of the human oral structure, can be finely divided into the maxilla and the mandible. These two parts of bones and the muscle tissues attached to them together constitute the upper and lower parts of our oral cavity. Usually, the upper part of the oral cavity is called the maxilla, and the lower part is called the mandible. In order to accurately quantitatively analyze the structure of the jawbone, and to more effectively plan and design oral rehabilitation surgery, it is particularly important to accurately identify the key points in the CBCT (cone beam computed tomography) images of the jaw joint. In the traditional operation process, the identification of these key points is usually completed by experienced physicians, who need to manually mark these key points on the image. However, this manual marking method is inefficient and not very intelligent, which to some extent limits the efficiency and accuracy of oral rehabilitation surgery. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method for recognizing jaw key points and a training method for a key point recognition model, and an oral robot system to at least partially solve the above problems.

[0004] According to a first aspect of an embodiment of the present invention, a method for identifying jaw key points is provided, comprising: The key point recognition model to be trained predicts the positions of the key points of the jaw according to the multimodal data of the jaw to generate multiple task outputs; Calling multiple configured loss functions, and calculating multiple loss values ​​according to the multiple task outputs; Based on the fusion weights assigned to different loss values, the multiple loss values ​​are fused to obtain a fusion loss value; Adjusting the network parameters of the key point recognition model to be trained according to the fusion loss value until the model training is completed to obtain the jaw key point recognition model; The jaw key points are identified from the input jaw CBCT data based on the jaw key point identification model.

[0005] Optionally, the mandibular multimodal data includes: mandibular CBCT data samples, maxillary and mandibular segmentation result samples, key point deviation value samples and original key point coordinate samples; The key point recognition model to be trained predicts the position of the jaw key points according to the jaw multimodal data to generate multiple task outputs, including: The key point recognition model to be trained performs feature extraction on the jaw CBCT data samples and the maxillary and mandibular segmentation result samples to generate multiple task outputs.

[0006] Optionally, the key point recognition model to be trained includes: a downsampling network, a feature fusion network, and a feature extraction network. The key point recognition model to be trained performs feature extraction on the jaw CBCT data samples and the maxillary and mandibular segmentation result samples to generate multiple task outputs, including: Based on the downsampling network, downsampling the jaw CBCT data sample to obtain a first downsampling feature; Based on the downsampling network, downsampling the maxillary and mandibular segmentation result samples to obtain second downsampling features; Based on the feature fusion network, the first down-sampled feature and the second down-sampled feature are subjected to feature fusion to obtain a down-sampled fusion feature; Based on the feature extraction network, feature extraction is performed on the downsampled fusion features to generate multiple task outputs.

[0007] Optionally, the feature extraction network includes a plurality of parallel feature extraction sub-networks, and different feature extraction sub-networks have different resolutions; The step of extracting features from the downsampled fusion features based on the feature extraction network to predict the positions of the key points of the jaw on the jaw CBCT data sample includes: Based on each feature extraction sub-network, the down-sampled fusion features are extracted step by step according to the corresponding resolution, and in the process of step-by-step feature extraction, different feature extraction sub-networks perform cross-scale feature interleaving fusion of features to generate multiple task outputs.

[0008] Optionally, the feature fusion network includes: a summation layer, a skip connection layer, and a convolutional layer; The step of fusing the first down-sampled feature and the second down-sampled feature based on the feature fusion network to obtain a down-sampled fusion feature includes: Adding the first down-sampled feature and the second down-sampled feature to obtain a summed feature; The sum feature is added to the first down-sampled feature and the second down-sampled feature respectively through the jump connection layer to obtain a first down-sampled sum feature and a second down-sampled sum feature; The first down-sampled sum feature and the second down-sampled sum feature are projected based on the convolution layer to obtain the down-sampled fusion feature.

[0009] Optionally, the multiple task outputs include: a prediction heat map, a prediction error offset, the prediction heat map is used to calculate the positions of predicted key points, and the prediction error offset is used to calculate the error offset values ​​of the predicted key points.

[0010] Optionally, the calling configured multiple loss functions calculates multiple loss values ​​according to the multiple task outputs, including: Calling the configured thermal loss function, and calculating the predicted thermal loss value according to the probability of the position of the predicted key point represented by the predicted thermal map; Calling the configured error offset loss function to calculate the error offset loss value according to the error offset value; The configured position loss function is called to calculate the predicted position loss value according to the predicted key point position and the original key point coordinate sample.

[0011] Optionally, the method further comprises: based on a jaw segmentation model, segmenting the jaw CBCT data sample to obtain the maxillary and mandibular segmentation result samples.

[0012] Optionally, the jaw segmentation model includes an image encoder, a label decoder, and a prompt encoder; The method of segmenting the jaw CBCT data sample based on the jaw segmentation model to obtain the maxillary and mandibular segmentation result samples includes: Encoding the jaw CBCT data sample based on the image encoder to obtain a corresponding CBCT image vector; Encoding the input segmentation hint based on the hint encoder to obtain a corresponding segmentation hint vector, wherein the segmentation hint is generated based on a box selection and a point selection operation on a segmentation area on the jaw CBCT data sample; The label decoder decodes the CBCT image vector based on the segmentation hint vector to obtain the maxillary and mandibular segmentation result samples.

[0013] According to a second aspect of an embodiment of the present invention, a method for training a key point recognition model is provided, which includes: The key point recognition model to be trained predicts the positions of the key points of the jaw according to the multimodal data of the jaw to generate multiple task outputs; Calling multiple configured loss functions, and calculating multiple loss values ​​according to the multiple task outputs; Based on the fusion weights assigned to different loss values, the multiple loss values ​​are fused to obtain a fusion loss value; The network parameters of the key point recognition model to be trained are adjusted according to the fusion loss value until the model training is completed to obtain the jaw key point recognition model.

[0014] According to a third aspect of an embodiment of the present invention, an oral robot system is provided, which includes: a robotic arm, a main unit, and a trolley, wherein the robotic arm and the main unit are assembled on the trolley, and the main unit is used to execute the key point recognition method and / or the training method described in any one of the embodiments of the present application.

[0015] The solution of the embodiment of the present invention has at least the following technical advantages: 1) This solution uses a mandibular segmentation model including an image encoder, a label decoder, and a prompt encoder to segment the mandibular CBCT data samples to obtain maxillary and mandibular segmentation result samples. In this process, segmentation prompts are generated based on the box selection and point selection operations on the segmentation area on the mandibular CBCT data samples. The mandibular CBCT data samples are first encoded by the image encoder to obtain the corresponding CBCT image vector, and then the prompt encoder is used to encode the input segmentation prompt to obtain the corresponding segmentation prompt vector. Finally, the label decoder decodes the CBCT image vector based on the segmentation prompt vector to obtain maxillary and mandibular segmentation result samples. This method does not require segmentation and annotation of the upper and lower mandibular regions and retraining of the model as in traditional methods, effectively eliminating the cost of manually annotating segmentation data and retraining the model.

[0016] 2) In terms of data input, multimodal data input is adopted, which specifically includes three types of modal data: jaw CBCT data samples, maxillary and mandibular segmentation result samples, key point deviation value samples, and original key point coordinate samples. Through the input of this multimodal data, the key point recognition model to be trained can fully learn the feature information between different modal data, laying a good foundation for the subsequent accurate recognition of jaw key points.

[0017] 3) In the network output link, a multi-task output form is used. Among them, the first task output is a prediction heat map, which can be used to calculate the position of the predicted key points; the second task output is a prediction error offset, which can be used to calculate the error offset value of the predicted key points. Different from the traditional point recognition method, this scheme adds the learning of the key point error offset value on this basis. By mastering this additional information, the final key point recognition error can be effectively reduced, and the accuracy of key point recognition can be further improved.

[0018] 4) In the process of network regression, multiple loss functions are jointly trained. Different from the traditional single loss function network, multiple loss functions are jointly trained in this scheme, so that the key point recognition model to be trained can learn the relationship between multiple tasks more comprehensively. In addition, based on the fusion weights assigned to different loss values, multiple loss values ​​can be fused to obtain a fusion loss value, and then the weights between different loss functions can be adjusted to make the network reach an ideal balance state, thereby improving the key point recognition accuracy. Specifically, the configured thermal loss function is called to calculate the predicted thermal loss value based on the probability of the predicted key point position represented by the predicted thermal map; the configured error offset loss function is called to calculate the error offset loss value based on the error offset value; the configured position loss function is called to calculate the predicted position loss value according to the predicted key point position and the original key point coordinate sample, so as to achieve the purpose of jointly training the network using multiple loss functions.

[0019] Here, it should be noted that the above technical benefits do not need to be achieved simultaneously in the solution of one embodiment, but a certain embodiment may only have some of the technical benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0021] Figure 1A This is a flow chart of a method for identifying jaw key points according to an embodiment of the present application.

[0022] Figure 1B This is a schematic diagram of the structure of the feature fusion network of the embodiment of the present application.

[0023] Figure 1C This is a schematic diagram of the predicted heat map of the embodiment of the present application.

[0024] Figure 1D The diagram is a schematic diagram of the training principle of the key point recognition model to be trained for the above specific structure.

[0025] Figure 1E This is a schematic diagram of the principle of implementing segmentation using a specific jaw segmentation model according to an embodiment of the present application.

[0026] Figure 1F Schematic diagram of a training system composed of the above-mentioned specific jaw segmentation model and the key point recognition model to be trained.

[0027] Figure 2A flowchart of a method for training a key point recognition model according to an embodiment of the present application.

[0028] Figure 3 This is a schematic diagram of an oral surgery robot system according to an embodiment of the present application.

[0029] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0031] It should be understood that the terms "first", "second", and "third" in the claims, specifications, and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0032] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0033] In the following embodiments of the present application, a multimodal data input form is adopted, covering jaw CBCT data samples, maxillary and mandibular segmentation result samples, key point deviation value samples and original key point coordinate samples, so that the key point recognition model to be trained can fully learn the characteristic information of different modal data. In terms of network output, a multi-task output form is used. The first task outputs a predicted thermal map for calculating the key point position, and the second task outputs a predicted error offset for calculating the key point error offset value. The learning of the error offset value is added to reduce the recognition error and improve the accuracy. In network regression, a joint training form of multiple loss functions is adopted. The corresponding loss values ​​are calculated by calling the thermal loss function, the error offset loss function, and the position loss function respectively, and then the fusion loss value is obtained based on the fusion weights assigned to different loss values. The weights are adjusted to balance the network to improve the key point recognition accuracy. Finally, the jaw key points are identified from the input jaw CBCT data based on the trained jaw key point recognition model.

[0034] Figure 1A This is a schematic diagram of a method for identifying key points of the jaw according to an embodiment of the present application. Figure 1A As shown, it includes: The key point recognition model to be trained predicts the positions of the key points of the jaw according to the multimodal data of the jaw to generate multiple task outputs; Calling multiple configured loss functions, and calculating multiple loss values ​​according to the multiple task outputs; Based on the fusion weights assigned to different loss values, the multiple loss values ​​are fused to obtain a fusion loss value; Adjusting the network parameters of the key point recognition model to be trained according to the fusion loss value until the model training is completed to obtain the jaw key point recognition model; The jaw key points are identified from the input jaw CBCT data based on the jaw key point identification model.

[0035] Optionally, in a specific application scenario, a total of 13 jaw key points are identified, which are as follows: Mandible: left condylar point, right condylar point, left medial pole, left lateral pole, right medial pole, right lateral pole; Maxilla: left infraorbital margin point, right infraorbital margin point, left ala nasi point, right ala nasi point, anterior nasal spine point, nasal root point, incisive foramen point.

[0036] It should be noted here that the above-mentioned jaw key points are merely examples and are not exclusive limitations.

[0037] Optionally, the mandibular multimodal data includes: mandibular CBCT data samples, maxillary and mandibular segmentation result samples, key point deviation value samples and original key point coordinate samples; The key point recognition model to be trained predicts the position of the jaw key points according to the jaw multimodal data to generate multiple task outputs, including: The key point recognition model to be trained performs feature extraction on the jaw CBCT data samples and the maxillary and mandibular segmentation result samples to generate multiple task outputs.

[0038] Optionally, the key point recognition model to be trained includes: a downsampling network, a feature fusion network, and a feature extraction network. The key point recognition model to be trained performs feature extraction on the jaw CBCT data samples and the maxillary and mandibular segmentation result samples to generate multiple task outputs, including: Based on the downsampling network, downsampling the jaw CBCT data sample to obtain a first downsampling feature; Based on the downsampling network, downsampling the maxillary and mandibular segmentation result samples to obtain second downsampling features; Based on the feature fusion network, the first down-sampled feature and the second down-sampled feature are subjected to feature fusion to obtain a down-sampled fusion feature; Based on the feature extraction network, feature extraction is performed on the downsampled fusion features to generate multiple task outputs.

[0039] Optionally, the feature extraction network includes a plurality of parallel feature extraction sub-networks, and different feature extraction sub-networks have different resolutions; The step of extracting features from the downsampled fusion features based on the feature extraction network to predict the positions of the key points of the jaw on the jaw CBCT data sample includes: Based on each feature extraction sub-network, the down-sampled fusion features are extracted step by step according to the corresponding resolution, and in the process of step-by-step feature extraction, different feature extraction sub-networks perform cross-scale feature interleaving fusion of features to generate multiple task outputs.

[0040] Optionally, the feature fusion network includes: a summation layer, a skip connection layer, and a convolutional layer; The step of fusing the first down-sampled feature and the second down-sampled feature based on the feature fusion network to obtain a down-sampled fusion feature includes: Adding the first down-sampled feature and the second down-sampled feature to obtain a summed feature; The sum feature is added to the first down-sampled feature and the second down-sampled feature respectively through the jump connection layer to obtain a first down-sampled sum feature and a second down-sampled sum feature; The first down-sampled sum feature and the second down-sampled sum feature are projected based on the convolution layer to obtain the down-sampled fusion feature.

[0041] Figure 1B FIG. 1 is a schematic diagram of the structure of the feature fusion network in the embodiment of the present application. Figure 1B As shown in the figure, ⊕ represents feature addition, ⊙ represents feature multiplication, and W represents the weight factor. For example, W=0.5 is set here.

[0042] The jaw CBCT data and the maxillary and mandibular segmentation results are weighted and summed through the sum layer, and then added to the first down-sampled feature and the second down-sampled feature through a skip connection to obtain the first down-sampled sum feature and the second down-sampled sum feature, so as to alleviate the gradient vanishing problem and simplify the learning process. Finally, the first down-sampled sum feature and the second down-sampled sum feature are projected through a 1x1x1 convolution layer to obtain the down-sampled fusion feature.

[0043] The feature fusion formula is as follows:

[0044] Where F represents the down-sampling fusion feature, C1x1x1 represents the 1x1x1 convolution layer, represents the first down-sampled sum feature, represents the second down-sampled sum feature.

[0045] Here, it should be noted that the 1x1x1 convolutional layer is only an example and is not a unique limitation.

[0046] The mandibular CBCT data and the maxillary and mandibular segmentation results are fused after downsampling. The purpose is to provide the network with bone edge information, facilitate the positioning of key points to the bone edges in the CBCT data, reduce the error of key point recognition, and avoid the problem of key points being far away from the bone or trapped inside the bone.

[0047] Specifically, the following formula is used to project the first down-sampled sum feature and the second down-sampled sum feature based on the convolution layer to obtain the down-sampled fusion feature:

[0048] Optionally, the multiple task outputs include: a prediction heat map, a prediction error offset, the prediction heat map is used to calculate the positions of predicted key points, and the prediction error offset is used to calculate the error offset values ​​of the predicted key points.

[0049] As mentioned above, a multi-task output format is adopted. The first task outputs the predicted heat map (denoted as Heatmap_pre), which is used to calculate the position of each key point; the second task outputs the predicted error offset (denoted as Offset_pre), which is used to calculate the error offset value of each key point.

[0050] Prediction error offset: refers to the original key point coordinates after two downsampling, the value will be reduced by 4 times. Since the pixel coordinates are integer values, the original key point coordinate value will be reduced by 4 times, and there will be a remainder, which is the Offset value, and Offset_pre represents the predicted remainder value, as described below: Original key point coordinate sample data = (Heatmap coordinate value + Offset value) * 4 Predicted original keypoint coordinates = (Heatmap_pre coordinate value + Offset_pre value) * 4 Optionally, the calling configured multiple loss functions calculates multiple loss values ​​according to the multiple task outputs, including: Calling the configured thermal loss function, and calculating the predicted thermal loss value according to the probability of the position of the predicted key point represented by the predicted thermal map; Calling the configured error offset loss function to calculate the error offset loss value according to the error offset value; The configured position loss function is called to calculate the predicted position loss value according to the predicted key point position and the original key point coordinate sample.

[0051] Optionally, in a specific scenario, each loss function is as follows: The thermal loss function (denoted as Loss_1) uses the BCE Focal Loss loss function to calculate the thermal loss value;

[0052] Among them, p is the probability of the predicted key point location, α is the balance factor used to adjust the influence between positive and negative samples. γ is the focus factor used to adjust the weight of difficult and easy samples. y=1 represents a positive sample, and y=0 represents a negative sample. When γ=2 and α=0.25, the effect is best.

[0053] Error offset loss function (denoted as Loss_2). The center point offset loss is added here because the input data is too large and needs to be downsampled twice. After downsampling, the key point coordinates are rounded, and there will be deviations after restoring to the original coordinates. The error offset loss value of the key point is calculated according to the following formula;

[0054] Where N represents the number of key points. represents the center point offset value, pi represents the center point coordinates before scaling, and R represents the scaling factor. Represents the location (in coordinates) of the predicted keypoints.

[0055] The position loss function (denoted as Loss_3) calculates the position loss between the predicted key points and the original key point coordinate samples.

[0056]

[0057] Where N is the number of original key point coordinate samples, yi is the true value of the i-th original key point coordinate sample, is the predicted value of the i-th original key point coordinate sample.

[0058] The final loss function Loss (calculating the fusion loss value) is expressed as:

[0059] Among them, a+b+c=1, here we set a=0.35, b=0.3, c=0.35. This design makes the network pay more attention to the loss calculation of Heatmap and the loss calculation of the original key point coordinates.

[0060] It should be noted here that the above weight settings are only examples and are not exclusive limitations.

[0061] To this end, the above-mentioned multiple loss functions are jointly trained. By adjusting the weights between different loss functions, the network focuses on a certain part of the training. The first loss function (Loss_1) calculates the heat map loss; the second loss function (Loss_2) calculates the error offset loss of the key point; the third loss function (Loss_3) calculates the loss of the original key point coordinates.

[0062] Figure 1C Schematic diagram of the predicted heat map of the embodiment of the present application. As shown in Figure 1, the predicted heat map (Heatmap) is to apply a Gaussian distribution function to the position of each key point to generate a Gaussian heat map centered on the key point to represent the position of the key point. The Gaussian distribution function centered on the key point will reach the maximum value at the point and gradually decrease as the distance from the center point increases.

[0063] Heatmap is equivalent to converting discrete value targets into continuous value targets, which can add directional guidance to the training of the model. The closer to the target point, the greater the activation value, so that the network can reach the target point quickly and in a directional manner.

[0064] For example, if a 3D Heatmap is used, the position of each key point is as follows:

[0065] Among them, (u x ,u y ,u z) represents the coordinates of the key points (i.e., spatial coordinates), and sets , the original coordinate point can be converted into a spatial sphere with a radius of 9. The probability value at the center of the sphere is 1, and the probability value gradually decreases to 0 as it approaches the edge.

[0066] Figure 1D Schematic diagram of the training principle of the key point recognition model to be trained for the above specific structure. Figure 1D As shown, the key point recognition model to be trained includes: a downsampling network, a feature fusion network, a feature extraction network, and multiple task outputs include: predicted thermal map, predicted error offset, multiple loss functions including thermal loss function, error offset loss function, and position loss function. Schematic diagram of the principle, detailed technical description, see the above description.

[0067] Optionally, the method further comprises: based on a jaw segmentation model, segmenting the jaw CBCT data sample to obtain the maxillary and mandibular segmentation result samples.

[0068] The method of segmenting the jaw CBCT data sample based on the jaw segmentation model to obtain the maxillary and mandibular segmentation result samples includes: Encoding the jaw CBCT data sample based on the image encoder to obtain a corresponding CBCT image vector; Encoding the input segmentation hint based on the hint encoder to obtain a corresponding segmentation hint vector, wherein the segmentation hint is generated based on a box selection and a point selection operation on a segmentation area on the jaw CBCT data sample; The label decoder decodes the CBCT image vector based on the segmentation hint vector to obtain the maxillary and mandibular segmentation result samples.

[0069] Optionally, Figure 1E The following is a schematic diagram of the principle of implementing segmentation using a specific jaw segmentation model in an embodiment of the present application. Figure 1E As shown, the jaw segmentation model includes an image encoder, a label decoder, and a prompt encoder. The image encoder can use Vision Transformer as a backbone extraction network, for example. For a detailed technical description, please refer to the above description.

[0070] Figure 1F Schematic diagram of a training system composed of the above-mentioned specific jaw segmentation model and the key point recognition model to be trained. Figure 1FAs shown, the key point recognition model to be trained includes: a downsampling network, a feature fusion network, and a feature extraction network. The feature extraction network includes multiple parallel feature extraction sub-networks, and different feature extraction sub-networks have different resolutions; the feature fusion network includes: a sum layer, a jump connection layer, and a convolution layer; the multiple task outputs include: a predicted thermal map, a predicted error offset, and the loss function includes a thermal loss function (denoted as Loss_1), an error offset loss function (denoted as Loss_2), and a position loss function (denoted as Loss_3).

[0071] Figure 1F The working principle of each structure can be found in the above-mentioned embodiment.

[0072] Figure 2 The following is a flow chart of a training method for a key point recognition model according to an embodiment of the present application. Figure 2 As shown, it includes: The key point recognition model to be trained predicts the positions of the key points of the jaw according to the multimodal data of the jaw to generate multiple task outputs; Calling multiple configured loss functions, and calculating multiple loss values ​​according to the multiple task outputs; Based on the fusion weights assigned to different loss values, the multiple loss values ​​are fused to obtain a fusion loss value; The network parameters of the key point recognition model to be trained are adjusted according to the fusion loss value until the model training is completed to obtain the jaw key point recognition model.

[0073] Figure 3 Schematic diagram of an oral robot system according to an embodiment of the present application. Figure 3 As shown, it includes: a robotic arm, a host, and a trolley, the robotic arm and the host are assembled on the trolley, and the host is used to execute the jaw key point recognition method and / or training method in the above embodiments of the present application.

[0074] In addition, in yet another embodiment of the present application, a method for identifying jaw key points is provided, which includes: Obtain input jaw CBCT data; The jaw key points are identified from the input jaw CBCT data based on the jaw key point recognition model.

[0075] Optionally, the jaw key point recognition model is trained based on the following steps: The key point recognition model to be trained predicts the positions of the key points of the jaw according to the multimodal data of the jaw to generate multiple task outputs; Calling multiple configured loss functions, and calculating multiple loss values ​​according to the multiple task outputs; Based on the fusion weights assigned to different loss values, the multiple loss values ​​are fused to obtain a fusion loss value; The network parameters of the key point recognition model to be trained are adjusted according to the fusion loss value until the model training is completed to obtain the jaw key point recognition model.

[0076] It should be noted here that the above-mentioned model training process is only an example and not a unique limitation.

[0077] In addition, considering the core idea of ​​the above-mentioned embodiment of the present application, it is not limited to the recognition of jaw key points. Therefore, a recognition method applicable to biological features in a broad sense is provided as follows: A method for identifying key points of biological characteristics, comprising: The key point recognition model to be trained predicts the location of the biometric key points based on the biometric multimodal data to generate multiple task outputs; Calling multiple configured loss functions, and calculating multiple loss values ​​according to the multiple task outputs; Based on the fusion weights set for different loss values, the multiple loss values ​​are fused to obtain a fusion loss value; Adjusting the network parameters of the key point recognition model to be trained according to the fusion loss value until the model training is completed to obtain the biometric key point recognition model; Based on the biometric key point recognition model, biometric key points are recognized from input biometric specific modality data.

[0078] Reference Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

[0079] like Figure 4 As shown, the electronic device may include: a processor (processor) 402 , a communication interface (Communications Interface) 404 , a memory (memory) 406 storing a program 410 , and a communication bus 408 .

[0080] The processor, the communication interface, and the memory communicate with each other via a communication bus. The communication interface is used to communicate with other electronic devices or servers. The processor is used to execute a program, and specifically can execute the relevant steps in the above method embodiment. Specifically, the program may include a program code, and the program code includes computer operation instructions.

[0081] The processor may be a CPU, or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0082] Memory is used to store programs. The memory may include high-speed RAM memory and may also include non-volatile memory, such as at least one disk storage.

[0083] The program can be specifically used to enable a processor to perform any operation in the above method.

[0084] The above implementation methods are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims. The systems, devices, modules or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.

[0085] For the convenience of description, the above device is described as being divided into various units according to their functions. Of course, when implementing the present invention, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0086] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0088] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0090] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0091] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0092] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0093] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific transactions or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where transactions are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0095] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A method for identifying jaw key points, characterized in that: include: The key point recognition model to be trained predicts the positions of the key points of the jaw according to the multimodal data of the jaw to generate multiple task outputs; Calling multiple configured loss functions, and calculating multiple loss values ​​according to the multiple task outputs; Based on the fusion weights assigned to different loss values, the multiple loss values ​​are fused to obtain a fusion loss value; Adjusting the network parameters of the key point recognition model to be trained according to the fusion loss value until the model training is completed to obtain the jaw key point recognition model; The jaw key points are identified from the input jaw CBCT data based on the jaw key point identification model.

2. The method for identifying jaw key points according to claim 1, characterized in that: The mandibular multimodal data includes: mandibular CBCT data samples, maxillary and mandibular segmentation result samples, key point deviation value samples and original key point coordinate samples; The key point recognition model to be trained predicts the position of the jaw key points according to the jaw multimodal data to generate multiple task outputs, including: The key point recognition model to be trained performs feature extraction on the jaw CBCT data samples and the maxillary and mandibular segmentation result samples to generate multiple task outputs.

3. The method for identifying jaw key points according to claim 2, characterized in that: The key point recognition model to be trained includes: a downsampling network, a feature fusion network, and a feature extraction network. The key point recognition model to be trained performs feature extraction on the jaw CBCT data samples and the maxillary and mandibular segmentation result samples to generate multiple task outputs, including: Based on the downsampling network, downsampling the jaw CBCT data sample to obtain a first downsampling feature; Based on the downsampling network, downsampling the maxillary and mandibular segmentation result samples to obtain second downsampling features; Based on the feature fusion network, the first down-sampled feature and the second down-sampled feature are subjected to feature fusion to obtain a down-sampled fusion feature; Based on the feature extraction network, feature extraction is performed on the downsampled fusion features to generate multiple task outputs.

4. The method for identifying jaw key points according to claim 3, characterized in that: The feature extraction network includes a plurality of parallel feature extraction sub-networks, and different feature extraction sub-networks have different resolutions; The step of extracting features from the downsampled fusion features based on the feature extraction network to predict the positions of the key points of the jaw on the jaw CBCT data sample includes: Based on each feature extraction sub-network, the down-sampled fusion features are extracted step by step according to the corresponding resolution, and in the process of step-by-step feature extraction, different feature extraction sub-networks perform cross-scale feature interleaving fusion of features to generate multiple task outputs.

5. The method for identifying jaw key points according to claim 3, characterized in that: The feature fusion network includes: a summation layer, a skip connection layer, and a convolutional layer; The step of fusing the first down-sampled feature and the second down-sampled feature based on the feature fusion network to obtain a down-sampled fusion feature includes: Adding the first down-sampled feature and the second down-sampled feature to obtain a summed feature; The sum feature is added to the first down-sampled feature and the second down-sampled feature respectively through the jump connection layer to obtain a first down-sampled sum feature and a second down-sampled sum feature; The first down-sampled sum feature and the second down-sampled sum feature are projected based on the convolution layer to obtain the down-sampled fusion feature.

6. The method for identifying jaw key points according to any one of claims 1 to 5, characterized in that: The multiple task outputs include: a prediction heat map, a prediction error offset, the prediction heat map is used to calculate the position of the predicted key points, and the prediction error offset is used to calculate the error offset value of the predicted key points.

7. The method for identifying jaw key points according to claim 6, characterized in that: The calling configured multiple loss functions calculate multiple loss values ​​according to the multiple task outputs, including: Calling the configured thermal loss function, and calculating the predicted thermal loss value according to the probability of the position of the predicted key point represented by the predicted thermal map; Calling the configured error offset loss function to calculate the error offset loss value according to the error offset value; The configured position loss function is called to calculate the predicted position loss value according to the predicted key point position and the original key point coordinate sample.

8. A method for training a key point recognition model, characterized in that: include: The key point recognition model to be trained predicts the positions of the key points of the jaw according to the multimodal data of the jaw to generate multiple task outputs; Calling multiple configured loss functions, and calculating multiple loss values ​​according to the multiple task outputs; Based on the fusion weights assigned to different loss values, the multiple loss values ​​are fused to obtain a fusion loss value; The network parameters of the key point recognition model to be trained are adjusted according to the fusion loss value until the model training is completed to obtain the jaw key point recognition model.

9. An oral robot system, characterized in that: include: A robotic arm, a host, and a trolley, wherein the robotic arm and the host are assembled on the trolley, and the host is used to execute the jaw key point recognition method described in any one of claims 1 to 7 and / or the training method described in claim 8.

10. A method for identifying jaw key points, characterized in that: include: Obtain input jaw CBCT data; The jaw key points are identified from the input jaw CBCT data based on the jaw key point recognition model.

Images