Correction methods and devices based on grouped case-level fusion classification
By performing group feature extraction and case-level fusion classification on multi-source images of medical cases, the problem of inaccurate adjustment of correction device parameters was solved, achieving higher classification accuracy and device stability.
Patent Information
- Application Number
- CN202611353444.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-02
- Publication Date
- 2026-11-10
AI Technical Summary
The accuracy of grouping and case-level fusion classification based on multi-source images of medical cases in the existing technology is low, which leads to inaccurate adjustment of the parameters of the correction device and increases the damage rate and wear rate of the correction device.
By performing group feature extraction and case-level fusion classification on multi-source images of medical cases, and using the trained group image feature extraction model and case-level classification model, an image classification information set is obtained to determine the adjustment parameters of the correction device.
It improved the accuracy of adjusting the parameters of the correction equipment, reduced the wear and tear and damage rate of the equipment, and improved the accuracy of the classification results.
Smart Images

Figure CN123071731A_ABST
Abstract
Description
Technical Field
[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to a correction device method and apparatus based on grouped case-level fusion classification. Background Technology
[0002] Adjusting the parameters of orthodontic devices is a crucial process for controlling their precise operation and achieving personalized correction. More than 70% of the population nationwide requires orthodontic treatment. However, the low accuracy of case-level classification based on multi-source medical images leads to inaccurate correction, necessitating repeated adjustments and extending treatment time and increasing costs.
[0003] Currently, for grouped case-level fusion classification based on multi-source images of medical cases, the common approach is to directly stitch together the input multi-source image set of medical cases to obtain stitched medical case images. Then, the stitched medical case images are input into a classification model to obtain an image classification information set. Based on the image classification information set, the adjustment parameters of the correction device are determined to control the correction device to perform correction.
[0004] The following technical problems often exist: directly stitching images from multiple sources of medical case images without considering the different groupings of cases included in the images from different sources, and directly inputting the stitched medical case images into the model for classification, fails to fully learn the distinguishing features between different groups and is prone to confusion. The extracted feature information is relatively simple, resulting in low accuracy of classification results based on multiple sources of medical case images. In addition, the image stitching requires a large amount of computing power, resulting in a waste of computing resources. This leads to low accuracy in adjusting the parameters of the correction device, requiring repeated adjustments to the correction device, increasing the damage and wear rate of the correction device.
[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the present disclosure concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.
[0007] Some embodiments of this disclosure propose a correction method and apparatus for correction devices based on grouped case-level fusion classification to solve the technical problems mentioned in the background section above.
[0008] In a first aspect, some embodiments of this disclosure provide a method for corrective device correction based on grouped case-level fusion classification, comprising: in response to determining that there is a correction abnormality in the corrective device, acquiring a multi-source image set of medical cases, a trained set of grouped image feature extraction models, a trained case-level fusion feature extraction model, and a trained case-level classification model; grouping the aforementioned multi-source image set of medical cases to obtain a multi-source image group set of medical cases, wherein the aforementioned multi-source image group set of medical cases and the aforementioned set of grouped image feature extraction models have a one-to-one correspondence; using the aforementioned trained set of grouped image feature extraction models, performing grouped feature extraction and fusion on the aforementioned multi-source image group set of medical cases to obtain a set of grouped image feature vectors; in response to determining that there is a missing target image set in the aforementioned multi-source image group set of medical cases, performing grouped image feature extraction on the target image set. The vector set is masked to obtain a masked feature vector set. The target image set is one of the following: a multi-source image group of medical cases, or an image set within the multi-source image group of medical cases. Using the trained case-level fusion feature extraction model, the target group image feature vector set is fused at the case level to obtain a case-level fusion feature vector. The target group image feature vector set includes one of the following: the masked feature vector set and the group image feature vector set corresponding to the non-missing multi-source image group of medical cases, or the group image feature vector set. The case-level fusion feature vector is input into the trained case-level classification model to obtain an image classification information set for the multi-source image group of medical cases. Based on the image classification information set, the adjustment parameters of the correction device for the multi-source image group of medical cases are determined to control the correction device.
[0009] Secondly, some embodiments of this disclosure provide an orthodontic device correction apparatus based on grouped case-level fusion classification, comprising: an acquisition unit configured to acquire a multi-source image set of medical cases, a trained set of grouped image feature extraction models, a trained case-level fusion feature extraction model, and a trained case-level classification model in response to determining that an orthodontic device has a correction abnormality; a grouping unit configured to group the aforementioned multi-source image set of medical cases to obtain a multi-source image set of medical cases, wherein the aforementioned multi-source image set of medical cases and the aforementioned set of grouped image feature extraction models have a one-to-one correspondence; an extraction and fusion unit configured to perform grouped feature extraction and fusion on the aforementioned multi-source image set of medical cases using the aforementioned trained set of grouped image feature extraction models to obtain a set of grouped image feature vectors; and a masking unit configured to, in response to determining that a target image set is missing in the aforementioned multi-source image set of medical cases, process the corresponding target image set... The grouped image feature vector set is masked to obtain a masked feature vector set, wherein the target image set is one of the following: a multi-source image group of medical cases, or an image set within the multi-source image group of medical cases; a fusion unit is configured to use the trained case-level fusion feature extraction model to perform case-level fusion on the target grouped image feature vector set to obtain a case-level fusion feature vector, wherein the target grouped image feature vector set includes one of the following: the masked feature vector set and the grouped image feature vector set corresponding to the non-missing multi-source image group of medical cases, or the grouped image feature vector set; an input unit is configured to input the case-level fusion feature vector into the trained case-level classification model to obtain an image classification information set for the multi-source image group of medical cases; a determination unit is configured to determine the adjustment parameters of the correction device for the multi-source image group of medical cases based on the image classification information set, so as to control the correction device to perform correction.
[0010] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.
[0011] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.
[0012] The above embodiments of this disclosure have the following beneficial effects: The correction method of the correction device based on grouped case-level fusion classification of some embodiments of this disclosure can effectively distinguish the feature information of grouped image feature vectors under different categories by extracting grouped features from multi-source image sets of medical cases. The case-level fusion feature vector obtained by case fusion can effectively fuse shared feature vectors of different categories, improve classification accuracy, improve the precision of parameter adjustment of the correction device, and reduce the wear and tear of the correction device. Specifically, the reason why the classification results based on multi-source images of medical cases are inaccurate and the image stitching requires a large amount of computing power, resulting in a waste of computing resources and the need for multiple adjustments to the correction device is that: directly stitching images from the multi-source image set of medical cases does not take into account the case information of different groups included in the images from different sources. Instead, the stitched medical case images are directly input into the model for classification. The model does not fully learn the distinguishing features between different groups and is prone to confusion. The extracted feature information is relatively simple, resulting in low accuracy of classification results based on multi-source images of medical cases and a large amount of computing power required for image stitching, resulting in a waste of computing resources. This, in turn, leads to low precision in adjusting the parameters of the correction device, requiring repeated adjustments to the correction device, increasing the damage and wear rate of the correction device. Based on this, some embodiments of the correction device correction method based on grouped case-level fusion classification in this disclosure firstly, in response to determining that there is a correction abnormality in the correction device, the multi-source image set of medical cases, the trained grouped image feature extraction model set, the trained case-level fusion feature extraction model, and the trained case-level classification model are obtained. Therefore, the multi-source image set of medical cases is used as input to the subsequently trained grouped image feature extraction model set. The trained case-level fusion feature extraction model and the trained case-level classification model are used to perform case-level feature fusion and classification on the multi-source image set of medical cases. Next, the above-mentioned multi-source image set of medical cases is grouped to obtain multi-source image sets of medical cases, where there is a one-to-one correspondence between the multi-source image sets of medical cases and the grouped image feature extraction model set. Thus, image grouping facilitates the effective learning of visual information from different categories of multi-source image sets of medical cases, improving the quality of the grouped image feature vector set obtained by grouped feature extraction and fusion. Then, using the trained grouped image feature extraction model set, the multi-source image sets of medical cases are subjected to grouped feature extraction and fusion to obtain a grouped image feature vector set. This allows for the extraction of feature vectors within different categories of groups, effectively distinguishing feature information from different categories, reducing the probability of confusion between feature information from different categories, effectively adapting to the discrimination space of different categories, and preserving the differences between different categories.Next, in response to the determination that the target image set is missing in the aforementioned multi-source image set of medical cases, the feature vector set of the grouped images corresponding to the target image set is masked to obtain a masked feature vector set. The target image set is one of the following: the multi-source image set of medical cases, or an image set within the multi-source image set of medical cases. Thus, masking can handle missing images when there are gaps in the multi-source image set of medical cases, ensuring the stability and comprehensiveness of the masked feature vector set, thereby improving the accuracy of the case-level fusion feature extraction model output and reducing underfitting. Then, using the trained case-level fusion feature extraction model, case-level fusion is performed on the target grouped image feature vector set to obtain a case-level fused feature vector. The target grouped image feature vector set includes one of the following: the masked feature vector set and the grouped image feature vector set corresponding to the non-missing multi-source image set of medical cases, or the aforementioned grouped image feature vector set. Therefore, case-level fusion can fuse feature information from different categories, i.e., different components of the same part, improving the comprehensiveness of the case-level fused feature vector. Secondly, the aforementioned case-level fusion feature vectors are input into the trained case-level classification model to obtain an image classification information set for the multi-source image set of medical cases. Therefore, due to the comprehensiveness of the case-level fusion feature vectors, the case-level classification model can capture more feature information, and the inclusion of multiple classification heads in the case-level classification model can improve the accuracy of the image classification information set, including multi-classification tasks. Finally, based on the aforementioned image classification information set, the adjustment parameters of the correction device for the aforementioned multi-source image set of medical cases are determined to control the correction device during correction. Thus, determining the adjustment parameters and controlling the correction device based on the image classification information set can improve the accuracy of the adjustment parameters, reduce repeated adjustments caused by classification bias, reduce the wear and tear on the correction device, and improve the stability and safety of the device. Therefore, this correction device correction method based on grouped case-level fusion classification, by extracting grouped features from the multi-source image set of medical cases, can effectively distinguish the feature information of grouped image feature vectors under different categories. The case-level fusion feature vectors obtained by case fusion can effectively fuse shared feature vectors from different categories, improving classification accuracy, increasing the precision of parameter adjustment of the correction device, and reducing the wear and tear on the correction device. Attached Figure Description
[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.
[0014] Figure 1This is a flowchart of some embodiments of the orthodontic device correction method based on grouped case-level fusion classification according to this disclosure; Figure 2 This is a schematic diagram of the structure of some embodiments of the orthodontic device based on grouped case-level fusion classification according to the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation
[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0016] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.
[0017] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0018] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0019] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0020] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.
[0021] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of the orthodontic device correction method and apparatus method based on grouped case-level fusion classification according to this disclosure. The orthodontic device correction method based on grouped case-level fusion classification includes the following steps: Step 101: In response to determining that there is a correction abnormality in the correction device, acquire the multi-source image set of medical cases, the set of grouped image feature extraction models that have been trained, the set of case-level fusion feature extraction models that have been trained, and the set of case-level classification models that have been trained.
[0022] In some embodiments, the execution entity of the above-described orthodontic device correction method based on grouped case-level fusion classification can, in response to determining that there is an orthodontic abnormality in the orthodontic device, acquire a multi-source medical case image set, a trained grouped image feature extraction model set, a trained case-level fusion feature extraction model, and a trained case-level classification model via a wired or wireless connection. The aforementioned orthodontic abnormality can be a deviation of the orthodontic device's correction state from its corrective state. The multi-source medical case images in the aforementioned multi-source medical case image set can be images describing the patient's disease (e.g., orthodontic diseases, thyroid nodules). The aforementioned multi-source medical case image set can include medical images of various image types. For example, the aforementioned multi-source medical case image set can include, but is not limited to, at least one of the following: facial image type case images, occlusal image type case images, occlusal surface image type case images, and X-ray image type case images. The aforementioned multi-source medical case image set can be image data collected from different examination devices for the patient's affected area. There is a one-to-one correspondence between the various image types of medical images included in the aforementioned multi-source medical case image set and the trained grouped image feature extraction model set. The grouped image feature extraction model in the aforementioned trained set of model sets can be a deep neural network model that extracts visual features from a multi-source medical case image set corresponding to the model and performs intra-group feature fusion, outputting an intra-group fused feature vector. This trained model can include: a trained intra-group feature encoding network and a trained intra-group feature aggregation layer. The trained intra-group feature encoding network can be a deep neural network model that extracts features from a multi-source medical case image set of the image type corresponding to the model and outputs an initial set of image feature vectors. For example, the trained intra-group feature encoding network can be a ConvNeXt-Tiny model. The trained intra-group feature aggregation layer can be a model that performs intra-group feature fusion on the initial set of image feature vectors and outputs a grouped image feature vector. For example, the intra-group feature aggregation layer can be a single-layer multilayer perceptron. The grouped image feature extraction model trained above is processed through the following steps: First, the multi-source image set of medical cases corresponding to the image type of the grouped image feature extraction model is input into the trained intra-group feature encoding network to obtain an initial image feature vector set. Second, the initial image feature vector set is input into the trained intra-group feature aggregation layer to obtain grouped image feature vectors. The case-level fusion feature extraction model trained above can be a deep neural network model that performs case-level feature fusion on the input target grouped image feature vector set and outputs a case-level fused feature vector. The case-level fused feature vector can represent the visual feature information of different groups describing the patient's disease, and is used to comprehensively represent the feature information describing the patient's overall disease state.The target group image feature vectors in the aforementioned target group image feature vector set can be feature vectors that have undergone linear transformation of the intra-group fused feature vectors, so that the target group image feature vectors lie in the same feature space. For example, the aforementioned trained case-level fusion feature extraction model can include: a multilayer perceptron with two fully connected layers, an activation function ReLU (Linear Rectification Unit), and a Dropout layer. The aforementioned trained case-level fusion feature extraction model can be processed through the following steps: First, the aforementioned target group image feature vector set is concatenated by channel features to obtain a concatenated feature vector. Second, the concatenated feature vector is input into the aforementioned multilayer perceptron with two fully connected layers for feature transformation to obtain a transformed feature vector. Then, the transformed intra-group feature vector is input into the activation function ReLU to obtain an activated feature vector. Finally, the activated feature vector is input into the Dropout layer to obtain a case-level fusion feature vector. The aforementioned trained case-level classification model can be a deep neural network model that performs multi-task classification prediction on the input case-level fusion feature vector and outputs an image classification information set for a multi-source image set of medical cases. The image classification information in the aforementioned image classification information set can be the disease category information corresponding to the predicted multi-source images of medical cases. The trained case-level classification model can include multiple task classification heads. These multiple task classification heads can be fully connected layers, including a Softmax activation function, that simultaneously perform classification predictions for multiple disease types on the input case-level fusion feature vector. The trained case-level classification model is processed through the following steps: First, the case-level fusion feature vector is copied to obtain a case-level fusion feature vector set. There is a one-to-one correspondence between the case-level fusion feature vector set and the multiple task classification heads. Second, the case-level fusion feature vector set is input into the case-level classification model composed of multiple task classification heads to perform classification prediction, obtaining a case image classification prediction result set. Third, the case image classification prediction result set is input into an activation function for classification aggregation, obtaining an image classification information set.
[0023] In some optional implementations of certain embodiments, each trained group image feature extraction model in the aforementioned set of trained group image feature extraction models can be obtained through the following steps: The first step involves acquiring a first group of case image sample sets and a second group of case image sample sets for the initial grouped image feature extraction model. The first group of case image sample sets includes a set of first grouped case images and corresponding first classification label information. The second group of case image sample sets includes a set of second grouped case images and corresponding second classification label information. The first group of case images in the first group of case image sample sets can be samples used for model training of the initial grouped image feature extraction model. The first group of case images in the first group of case image sample sets can be case images describing the patient's disease condition used for model training of the initial grouped image feature extraction model. The first classification label information can be the category information of the first group of case image sets. The second group of case image samples in the second group of case image sample sets can be samples used for model validation of the initial grouped image feature extraction model trained on the first group of case image sample sets. The second group of case images in the second group of case image sample sets can be case images describing the patient's disease condition used for model validation of the initial grouped image feature extraction model trained on the first group of case image sample sets. The aforementioned second category label information can be the category information of the aforementioned second group of case image sets.
[0024] The second step involves performing the following determination steps based on the image sample set of the first group of cases: Sub-step 1 involves inputting at least one first group of case images from the aforementioned first group of case image sample set into the initial group image feature extraction model to obtain a group image sample feature vector. This group image sample feature vector can characterize the visual feature information of at least one first group of case images describing the patient's disease.
[0025] Sub-step 2 involves inputting the aforementioned grouped image sample feature vector set into an initial intra-group classification prediction network to obtain image prediction classification information. This image prediction classification information can be the predicted classification result of at least one first group of case images. The initial intra-group classification prediction network can be a deep neural network model that performs classification prediction on the input grouped image sample feature vector set and outputs image prediction classification information. For example, the initial intra-group classification prediction network can include fully connected layers with a Softmax activation function.
[0026] Sub-step 3 involves determining the within-group classification loss values for the aforementioned image prediction classification information and classification label information. The within-group classification loss value characterizes the degree of difference between the image prediction classification information and the classification label information. In practice, the executing entity can input the image prediction classification information and the classification label information into a preset loss function to obtain the within-group classification loss value. The preset loss function can be a cross-loss function.
[0027] Sub-step 4: In response to determining that the number of times the above determination step has been executed has reached a first preset training count threshold, the initial grouped image feature extraction model corresponding to the number of executions is determined as the trained grouped image feature extraction model. The first preset training count threshold can be a pre-set maximum number of times the determination step can be executed. For example, the preset training count threshold can be 300.
[0028] Sub-step 5: In response to determining that the number of executions has not reached the first preset training threshold, the second group of case images is input into the initial group image feature extraction model corresponding to the number of executions to obtain an initial prediction classification information set. The initial prediction classification information in the initial prediction classification information set may be the predicted category information to which the second group of case images belongs.
[0029] Sub-step 6 involves determining the predicted classification accuracy of the initial predicted classification information set and the second classification label information. The predicted classification accuracy can be the percentage of the number of initial predicted classification information sets containing information identical to the second classification label information out of the total number of initial predicted classification information sets included in the initial predicted classification information set. In practice, the executing entity can first filter out at least one initial predicted classification information set containing information identical to the second classification label information from the initial predicted classification information set. Then, it determines the number of initial predicted classification information sets containing this at least one initial predicted classification information set, which is designated as the first predicted classification number. Finally, it determines the ratio of the first predicted classification number to the total number of initial predicted classification information sets included in the initial predicted classification information set, which is then used as the predicted classification accuracy.
[0030] Sub-step 7: In response to the prediction accuracy being greater than or equal to a preset classification accuracy threshold, the initial grouped image feature extraction model corresponding to the number of executions is determined as a successfully trained grouped image feature extraction model. The preset classification accuracy threshold can be a pre-set threshold used to determine whether the initial grouped image feature extraction model has been successfully trained. For example, the preset classification accuracy threshold could be 95%.
[0031] Third, in response to the prediction accuracy being less than the preset classification accuracy threshold, the first group of case image sample set is reselected to obtain at least one group of case image samples. Based on the intra-group classification loss value, the parameters of the initial group image feature extraction model and the initial intra-group classification prediction network corresponding to the number of executions are adjusted to obtain an adjusted group image feature extraction model and an adjusted intra-group classification prediction network, which serve as the initial group image feature extraction model and the initial intra-group classification prediction network, to be executed again in accordance with the above determination steps. In practice, the executing entity can utilize the BGD (Batch Gradient Descent) algorithm to adjust the parameters of the initial group image feature extraction model and the initial intra-group classification prediction network corresponding to the number of executions, obtaining an adjusted group image feature extraction model and an adjusted intra-group classification prediction network, which serve as the initial group image feature extraction model and the initial intra-group classification prediction network, to be executed again in accordance with the above determination steps.
[0032] In some optional implementations of certain embodiments, the case-level fusion feature extraction model and the case-level classification model trained above can be obtained through the following training steps: The first step involves obtaining a case-level training sample set, a case-level validation sample set, and an initial case-level classification model for the initial case-level fusion feature extraction model. The case-level training sample set includes a multi-type case image set and a corresponding multi-classification label information set. The case-level training samples in the case-level training sample set can be samples used to train the initial case-level fusion feature extraction model. The case-level validation samples in the case-level validation sample set can be samples used to validate the output accuracy of the model trained on the case-level training sample set. The multi-type case images in the multi-type case image set can be images used for model prediction by inputting the grouped image feature extraction model set and the initial case-level fusion feature extraction model. The multi-classification label information in the corresponding multi-classification label information set can be the labeled, real disease classification information of the multi-type case images.
[0033] The second step involves inputting the aforementioned multi-type case image set into the intra-group feature encoding network set included in the trained grouped image feature extraction model set, thereby obtaining a multi-type sample feature vector set. The multi-type sample feature vectors in this set can represent the visual feature information of the multi-type case images.
[0034] The third step involves performing a linear transformation on the aforementioned set of feature vectors for multiple sample types to obtain the transformed set of feature vectors for multiple sample types. In practice, the execution entity can input the set of feature vectors for multiple sample types into a linear layer for linear mapping to obtain the transformed set of feature vectors for multiple sample types. For example, the set of feature vectors for multiple sample types with different output dimensions can be mapped to a set of feature vectors with a dimension of 512.
[0035] The fourth step involves performing the following joint training steps based on the transformed set of feature vectors from multiple sample types.
[0036] Sub-step 1 involves inputting at least one of the transformed multi-type sample feature vector sets into the initial case-level fusion feature extraction model to obtain the case-level fusion feature vector.
[0037] Sub-step 2 involves inputting the aforementioned case-level fusion feature vector into the initial case-level classification model to obtain a multi-task prediction classification result set. The multi-task prediction classification results in this set can be the predicted category information of the disease type from the case-level training sample set.
[0038] Sub-step 3: Based on the multi-task prediction classification result set and the multi-class label information set, determine the main task loss value and the auxiliary task loss value. The main task loss value can be the degree of difference between the multi-task prediction classification result set and the multi-class label information set corresponding to the transformed multi-type sample feature vector set. The auxiliary task loss value can characterize the degree of difference between the predicted classification result of the auxiliary task corresponding to the multi-task prediction classification result set and the classification label information of the auxiliary task corresponding to the multi-class label information set. The auxiliary task can be a more detailed sub-classification task under the major classification category corresponding to the disease. For example, the major classification category can be, but is not limited to, at least one of the following: profile type classification, overbite classification, overjet classification, maxillary crowding classification, mandibular crowding classification, left canine relationship classification, right canine relationship classification, and left molar relationship classification. The auxiliary task can be a skeletal category classification task for skeletal type I, skeletal type II, and skeletal type III.
[0039] Sub-step 4 involves applying regularization constraints to the type grouping weight set corresponding to the multi-task prediction classification result set to obtain the regularization loss value. The type grouping weights in this set serve to prevent the initial case-level fusion feature extraction model from over-relying on the transformed multi-type sample feature vector groups of one category, and to balance the feature distribution of the transformed multi-type sample feature vector groups of different categories. The regularization loss value characterizes the constraint penalty determined after applying regularization constraints to the type grouping weight set. In practice, the executing entity can calculate the sum of squares of the type grouping weight set and then take the average to obtain the regularization loss value.
[0040] Sub-step 5 involves performing a weighted summation of the main task loss value, the auxiliary task loss value, and the regularization term loss value to obtain the total loss value. The weights in this weighted summation can be manually set. For example, the weights for the main task loss value, the auxiliary task loss value, and the regularization term loss value could be 1.0, 0.5, and 0.005, respectively.
[0041] Sub-step 6, in response to determining that the number of joint training executions of the above joint training steps has reached a second preset training count threshold, identifies the initial case-level fusion feature extraction model and the initial case-level classification model corresponding to the above number of joint training executions as the trained case-level fusion feature extraction model and the trained case-level classification model. The second preset training count threshold can be considered a pre-set maximum number of times the above joint training steps can be executed. For example, the preset training count threshold could be 300.
[0042] Sub-step 7: In response to the determination that the number of joint training executions has not reached the second preset training count threshold, the transformed multi-type sample feature vectors corresponding to the multi-type validation case image sets included in the case-level validation sample set are sequentially input into the initial case-level fusion feature extraction model and the initial case-level classification model corresponding to the number of joint training executions, to obtain the validation prediction classification result set. The validation prediction classification results in the validation prediction classification result set can be the prediction classification results of the transformed multi-type sample feature vectors under the current number of joint training executions.
[0043] Sub-step 8: Based on the aforementioned validation prediction classification result set and the multi-task validation classification label set included in the aforementioned initial case-level validation sample set, determine the validation accuracy. The validation accuracy can be the ratio of the number of validation prediction classification results that are identical to their corresponding multi-task validation classification labels to the total number of classifications. The total number of classifications can be the number of validation prediction classification results included in the aforementioned validation prediction classification result set. In practice, the executing entity can first determine at least one validation prediction classification result in the aforementioned validation prediction classification result set that is identical to its corresponding multi-task validation classification label. Then, it can determine the number of validation prediction classification results included in the at least one validation prediction classification result set, as the first classification prediction number. Finally, it can determine the ratio of the first classification prediction number to the total number of classifications, as the validation accuracy.
[0044] Sub-step 9, in response to determining that the above verification accuracy is greater than or equal to a preset accuracy threshold, identifies the initial case-level fusion feature extraction model and the initial case-level classification model corresponding to the above joint training execution counts as the trained case-level fusion feature extraction model and the trained case-level classification model. The preset accuracy threshold can represent a pre-set critical value used to determine whether the initial case-level fusion feature extraction model and the initial case-level classification model have been trained successfully. For example, the preset accuracy threshold could be 95%.
[0045] Fifth, in response to the determination that the above verification accuracy is less than the above preset accuracy threshold, at least one transformed multi-type sample feature vector is reselected from the above transformed multi-type sample feature vector set. Based on the above total loss value, the parameters of the initial case-level fusion feature extraction model and the initial case-level classification model corresponding to the above joint training execution count are adjusted to obtain the adjusted case-level fusion feature extraction model and the adjusted case-level classification model, which serve as the initial case-level fusion feature extraction model and the initial case-level classification model, for re-execution of the above joint training step. In practice, the above execution entity can utilize the BGD (Batch Gradient Descent) algorithm to adjust the parameters of the initial case-level fusion feature extraction model and the initial case-level classification model corresponding to the above joint training execution count based on the above total loss value, obtaining the adjusted case-level fusion feature extraction model and the adjusted case-level classification model, which serve as the initial case-level fusion feature extraction model and the initial case-level classification model.
[0046] Step 102: Group the images from the multi-source image set of medical cases to obtain the multi-source image set of medical cases.
[0047] In some embodiments, the executing entity may group the aforementioned multi-source medical case images to obtain a multi-source medical case image set, wherein there is a one-to-one correspondence between the multi-source medical case image set and the grouped image feature extraction model set. The multi-source medical case image sets within the multi-source medical case image set may be image sets representing the actual condition of the diseased area, obtained by grouping according to different components of the diseased area (e.g., facial group, occlusal group, occlusal surface group, etc. of the oral cavity). The multi-source medical case image set may include: a first multi-source medical case image set, a second multi-source medical case image set, a third multi-source medical case image set, and a fourth multi-source medical case image set. The first, second, third, and fourth multi-source medical case image sets may be images displaying the current state of the case from different locations. For example, the first multi-source medical case image set may include, but is not limited to, at least one of the following: a frontal view of the oral face under facial grouping, a frontal smiling view of the oral face, and a 90-degree view of the oral face from the right side. The aforementioned second medical case multi-source image group may include, but is not limited to, at least one of the following: oral occlusal images displayed from the front, left side, and right side under the occlusal group. The aforementioned third medical case multi-source image group may include, but is not limited to, at least one of the following: oral occlusal surface images displayed from the maxillary occlusal surface and oral occlusal surface images displayed from the mandibular occlusal surface under the occlusal surface group. The aforementioned fourth medical case multi-source image group may include, but is not limited to, at least one of the following: oral radiographic images displayed from the lateral cephalometric view and oral radiographic images displayed from the curved tomographic view under the xray group. The intra-group feature encoding network included in the trained grouped image feature extraction model corresponding to the aforementioned first, second, and third medical case multi-source image groups may be a ConvNeXt-Tiny model. The intra-group feature encoding network included in the trained grouped image feature extraction model corresponding to the aforementioned fourth medical case multi-source image group may be a DenseNet121 (Densely Connected Convolutional Networks121) model. The number 121 mentioned above can be the total number of network layers included in DenseNet.
[0048] Step 103: Using the trained set of grouped image feature extraction models, perform grouped feature extraction and fusion on the multi-source image set of medical cases to obtain a set of grouped image feature vectors.
[0049] In some embodiments, the execution entity may utilize the trained set of grouped image feature extraction models to perform grouped feature extraction and fusion on the multi-source image set of medical cases, thereby obtaining a set of grouped image feature vectors. The grouped image feature vectors in the set of grouped image feature vectors can represent the visual feature information of different groups of multi-source image sets of medical cases.
[0050] In some optional implementations of certain embodiments, the set of grouped image feature extraction models that have been trained includes: a set of intra-group feature encoding networks that have been trained and a set of intra-group feature aggregation layers that have been trained.
[0051] The above-mentioned grouped image feature extraction model set, trained as described above, is used to extract and fuse grouped features from the above-mentioned multi-source image set of medical cases to obtain a grouped image feature vector set. This process may include the following steps: The first step is to perform the following feature extraction steps for each medical case multi-source image group in the above medical case multi-source image group set: Sub-step 1: From the set of grouped image feature extraction models trained above, determine the grouped image feature extraction model corresponding to the multi-source image group of the above medical cases, and use it as the target grouped image feature extraction model.
[0052] Sub-step 2 involves inputting the aforementioned multi-source image set of medical cases into the trained intra-group feature encoding network included in the target group image feature extraction model to obtain an initial image feature vector set. The initial image feature vectors in this initial image feature vector set can represent the visual feature information of images describing the patient's disease at different locations corresponding to the aforementioned multi-source image set of medical cases.
[0053] Sub-step 3 involves inputting the initial image feature vector group into the trained intra-group feature aggregation layer of the target grouped image feature extraction model to obtain grouped image feature vectors. These grouped image feature vectors can be feature vectors obtained by nonlinearly fitting the dependency and complementarity relationships between the initial image feature vector group after feature alignment (e.g., feature concatenation, weighted summation, element-level operations, etc.).
[0054] The second step is to determine the obtained multiple grouped image feature vectors as a grouped image feature vector set.
[0055] Optionally, before using the grouped image feature extraction model set trained above to perform grouped feature extraction and fusion on the above-mentioned multi-source image set of medical cases to obtain a grouped image feature vector set, the above method may further include the following steps: The first step is to perform the following standardized processing steps on the above-mentioned medical case multi-source imaging group.
[0056] Sub-step 1 involves scaling the aforementioned multi-source medical case images to obtain a scaled multi-source medical case image set. The scaled multi-source images in this set can be images of different sizes scaled to a uniform preset size. This uniform preset size can be a pre-defined numerical value. For example, this value could be 224×224 pixels.
[0057] Sub-step 2 involves performing color mode conversion processing on the scaled medical case multi-source image group to obtain a color-converted medical case multi-source image group. The color-converted medical case multi-source images in the color-converted medical case multi-source image group can be images located in a unified color mode. This unified color mode can be RGB (Red, Green, Blue) color mode.
[0058] Sub-step 3 involves standardizing the color-converted multi-source medical case images to obtain a standardized multi-source medical case image set. In practice, the executing entity can use Z-score standardization to standardize the color-converted multi-source medical case images to obtain a pre-processed multi-source medical case image set.
[0059] In addressing the technical problems mentioned in the background section, the application scenario—specifically, high-precision correction of lesions with extremely small size and low density contrast (e.g., pulmonary micronodules, cardiovascular correction)—often presents the following challenges: During the acquisition of multi-source medical images, various factors such as equipment differences, lighting conditions, exposure parameters, and operator skill levels can affect the images. This leads to uneven brightness distribution, overall dark or bright images with noise, insufficient local contrast, and high-texture images being overwhelmed by noise. Consequently, the resolution and image quality of these multi-source images are low, resulting in lower quality extracted feature vectors and lower accuracy of image classification information sets. This necessitates repeated adjustments to the correction equipment parameters, increasing the equipment's failure rate. Considering the following requirements for this application scenario—adaptability to high precision, high resolution, high-precision correction processing, non-destructive reconstruction of multi-source medical images, and the complexity and subtlety of the correction area—we have decided to adopt the following solution: Optionally, before inputting the aforementioned multi-source image group of medical cases into the trained intra-group feature encoding network included in the aforementioned target group image feature extraction model to obtain the initial image feature vector group, the following steps may also be included: The first step is to perform the following reconstruction steps for each medical case's multi-source images in the above medical case multi-source imaging group: Sub-step 1 involves extracting a grayscale histogram from the aforementioned multi-source images of the medical case. This grayscale histogram represents the pixel distribution information of the multi-source images of the medical case at different grayscale levels. In practice, the executing entity can first perform grayscale processing on the multi-source images of the medical case to obtain grayscale images of the medical case. Then, iterate through each pixel of the multi-source images of the medical case, count the number of pixels appearing at each grayscale level (0-255), and construct the grayscale histogram by combining each grayscale level and its corresponding pixel count. The difference between the horizontal coordinates of adjacent grayscale values in the grayscale histogram can be one grayscale value.
[0060] Sub-step 2: Based on the aforementioned grayscale histogram, determine the first percentile grayscale value and the second percentile grayscale value. The first percentile grayscale value can be the image grayscale value corresponding to the preset first percentile achieved by summing the number of pixels corresponding to each grayscale level included in the grayscale histogram. The preset first percentile can be a pre-set lower percentile value representing a specified proportion of the total number of pixels in the grayscale histogram. For example, the preset first percentile can be 5%. The second percentile grayscale value can be the image grayscale value corresponding to the preset second percentile achieved by summing the number of pixels corresponding to each grayscale level included in the grayscale histogram. The preset second percentile can be a pre-set upper percentile value representing a specified proportion of the total number of pixels in the grayscale histogram. For example, the preset second percentile can be 95%. In practice, the aforementioned executing entity can accumulate and sum the ratios of the number of pixels included in the aforementioned grayscale histogram to the total number of pixels, arranged in ascending order of grayscale level, to obtain the cumulative pixel percentage. The total number of pixels can be the number of grayscale values included in the aforementioned medical case grayscale image. Then, the grayscale value of the aforementioned grayscale histogram corresponding to the first time the cumulative pixel percentage reaches a preset first percentile (e.g., 5%) is determined as the first percentile grayscale value. Finally, the grayscale value of the aforementioned grayscale histogram corresponding to the first time the cumulative pixel percentage reaches a preset second percentile (e.g., 95%) is determined as the second percentile grayscale value.
[0061] Sub-step 4 involves normalizing the aforementioned grayscale histogram to obtain normalized grayscale distribution information. This normalized grayscale distribution information can be the probability distribution of each grayscale level in the grayscale image of the medical case. In practice, the executing entity can divide the number of pixels at each grayscale level in the grayscale histogram by the total number of pixels in the multi-source image of the medical case to obtain the proportion corresponding to each grayscale level, which serves as the normalized grayscale distribution information.
[0062] Sub-step 5: Based on the normalized grayscale distribution information described above, determine the skewness of the normalized grayscale distribution information. The skewness can be a statistic describing the direction and degree of asymmetry in the probability distribution. In practice, the executing entity first uses the formula for calculating the third central moment to determine the third central moment of the normalized grayscale distribution information. Finally, the ratio of the third central moment to the cube of the standard deviation of the normalized grayscale distribution information is determined as the skewness.
[0063] Sub-step 6: Based on the aforementioned skewness, perform multi-scale decomposition processing on the multi-source images of the aforementioned medical cases to obtain a set of low-frequency residual images and high-frequency detail images. The low-frequency residual images can be the lowest scale image retained after multi-scale decomposition, reflecting the overall brightness and slow change trend of the image. The high-frequency detail images in the high-frequency detail image set can be images containing high-frequency information such as image edges, textures, and noise from the multi-source images of the aforementioned medical cases. In practice, the executing entity can first perform Gaussian low-pass filtering on the multi-source images of the aforementioned medical cases to obtain a filtered image. Then, interlaced sampling is performed on the filtered image to obtain a Gaussian image at the next scale. Next, the absolute value of the aforementioned skewness is determined. When the absolute value of the aforementioned skewness is less than or equal to a first preset skewness threshold, the number of decomposition layers is determined to be 2; when the absolute value of the aforementioned skewness is greater than the first skewness threshold and less than or equal to a second skewness threshold, the number of decomposition layers is determined to be 3; when the absolute value of the aforementioned skewness is greater than the second skewness threshold, the number of decomposition layers is determined to be 4. The first and second preset skewness thresholds can be pre-set thresholds used to determine the number of multi-size decomposition layers. For example, the first and second preset skewness thresholds can be 0.4 and 0.8 respectively, with the first preset skewness threshold being less than the second preset skewness threshold. Next, the Gaussian image at the next scale is used as the current Gaussian image, and the Gaussian low-pass filtering and interlaced sampling are repeatedly performed until the Gaussian image corresponding to the decomposition layer is obtained, forming a Gaussian pyramid from the Gaussian images at each scale. Then, the Gaussian images at adjacent scales in the Gaussian pyramid are interpolated. Specifically, the coarser Gaussian image is upsampled to the size of the finer Gaussian image and then interpolated pixel-by-pixel with the finer Gaussian image to obtain high-frequency detail images at each scale. Finally, the coarsest Gaussian image in the Gaussian pyramid is identified as the low-frequency residual image. The high-frequency detail images at each scale form a high-frequency detail image set. The filtered image can be a smoothed image that retains low-frequency information.
[0064] Sub-step 7: Based on the aforementioned first percentile gray value and second percentile gray value, perform contrast stretching processing on the aforementioned low-frequency residual image to obtain a stretched low-frequency residual image. In practice, the executing entity can first extract multiple pixel gray values located between the aforementioned first percentile gray value and the aforementioned second percentile gray value from the pixel gray value set included in the aforementioned low-frequency residual image, as a low-frequency residual pixel gray value set. Then, for each low-frequency residual pixel gray value, determine the difference between the low-frequency residual pixel gray value and the aforementioned first percentile gray value. Divide the aforementioned difference by the difference between the aforementioned second percentile gray value and the aforementioned first percentile gray value to obtain a ratio value. Then multiply the aforementioned ratio value by the interval length of the preset gray level interval, and add the lower bound value of the aforementioned preset gray level interval to obtain the mapped gray value. Finally, determine the multiple mapped gray values obtained as the pixel gray value set included in the stretched low-frequency residual image. The aforementioned preset grayscale range can be a pre-defined target grayscale range (e.g., 5 to 250) used to support the grayscale values after stretching.
[0065] Sub-step 8 involves performing soft-threshold denoising on each high-frequency detail image in the aforementioned high-frequency detail image set to obtain a denoised high-frequency detail image set. The denoised high-frequency detail images in this set can be high-frequency images obtained by setting low-amplitude high-frequency components (e.g., noise) to zero or suppressing them, while retaining and appropriately shrinking effective high-frequency components (e.g., edges, textures, etc.). In practice, the executing entity can, in the first step, perform the following processing steps for each high-frequency detail image in the aforementioned high-frequency detail image set: First, determine each pixel value included in the high-frequency detail image as a denoising coefficient. The denoising coefficient can be the pixel value at each pixel location in the high-frequency detail image. Then, in response to determining that the denoising coefficient is less than the aforementioned preset denoising threshold, set it to zero. The aforementioned preset denoising threshold is a pre-set critical value for whether or not it is noise. In response to determining that the denoising coefficient is greater than or equal to the aforementioned preset denoising threshold, subtract the aforementioned preset denoising threshold from the denoising coefficient, retaining the positive or negative sign of the original denoising coefficient, to obtain the shrunken denoising coefficient. The second step is to use the obtained multiple denoising coefficients after shrinkage to determine the enhanced high-frequency detail image, including the individual pixel values.
[0066] Sub-step 9 involves performing multi-size reconstruction processing on the above-mentioned stretched low-frequency residual image and the above-mentioned denoised high-frequency detail image set to obtain reconstructed multi-source images of medical cases, which serve as multi-source images of medical cases. Based on these multi-source images, an image classification information set is determined, and based on this image classification information set, adjustment parameters for the correction device targeting the above-mentioned multi-source image set of medical cases are determined to control the correction device to perform correction. In practice, the executing entity can first use the stretched low-frequency residual image as the current layer image. Then, the current layer image is upsampled to the same size as the denoised high-frequency detail image adjacent to the current layer image in the above-mentioned denoised high-frequency detail image set. The upsampled current layer image is then pixel-by-pixel added to the adjacent denoised high-frequency detail image to obtain the next layer image. This next layer image is then used as the current layer image, and upsampling and pixel-by-pixel addition are performed again until all enhanced high-frequency detail images in the above-mentioned denoised high-frequency detail image set have been traversed, resulting in reconstructed multi-source images of medical cases, which serve as multi-source images of medical cases. Then, based on the multi-source images of the medical case, an image classification information set is determined, and based on the image classification information set, adjustment parameters for the correction device for the multi-source image set of the medical case are determined to control the correction device to perform correction. It should be noted that the specific implementation method of this step can refer to the implementation methods included in steps 103-107.
[0067] The aforementioned technical content and related content, as an inventive point of this disclosure, solve the technical problem of "resulting in low quality of extracted feature vectors and low accuracy of image classification information sets, requiring repeated adjustments to the parameters of the correction equipment, thus increasing the damage rate of the correction equipment." Factors contributing to the low resolution and image quality of multi-source images of medical cases, leading to low quality of extracted feature vectors and low accuracy of image classification information sets, necessitating repeated adjustments to the parameters of the correction equipment and increasing the damage rate of the correction equipment, are often as follows: During the acquisition process of multi-source images of medical cases, various factors such as equipment differences, lighting conditions, exposure parameters, and operator skill levels can affect the images. This results in uneven brightness distribution, overall dark or bright noise in the images, insufficient local contrast, and high texture being submerged by noise, leading to low resolution and image quality in the multi-source images of medical cases, which in turn leads to low quality of extracted feature vectors and low accuracy of image classification information sets, requiring repeated adjustments to the parameters of the correction equipment and increasing the damage rate of the correction equipment. If the above factors are addressed, the quality of the extracted feature vectors can be improved, the accuracy of the image classification information set can be increased, multiple adjustments can be avoided, and the damage rate of the correction equipment can be reduced. To achieve this effect, this disclosure cites the fact that when the grayscale distribution of a multi-source image is too dark or too bright, the skewness of its normalized grayscale histogram will be positive or negative accordingly. The larger the absolute value of the skewness, the more uneven the brightness distribution of the image, and the more layer enhancement is needed. The upper and lower thresholds required for contrast stretching are determined by the first and second percentile grayscale values of the image's own grayscale histogram. The first percentile removes extreme dark noise, and the second percentile removes extreme bright noise. Together, they define the actual boundary of the effective grayscale range of the image. Since skewness measures the overall brightness bias of an image, and the first and second percentiles can adaptively define the effective grayscale range, a multi-scale decomposition strategy driven by skewness, a contrast stretching strategy for low-frequency residual images driven by percentile thresholds, and soft thresholding noise reduction for high-frequency detail images driven by normalized grayscale distribution information, work together to achieve adaptive enhancement of images with different acquisition qualities. Finally, the enhanced low-frequency residual images and enhanced high-frequency detail images are reconstructed at multiple scales to obtain enhanced multi-source images of medical cases with balanced brightness and controlled noise, reducing the low quality problem caused by acquisition equipment limitations in multi-source images of medical cases. Finally, by performing group feature extraction and case-level fusion on high-quality multi-source image sets of medical cases, a highly accurate and comprehensive image classification information set is obtained, thereby reducing the number of parameter modifications required for correction equipment, lowering the damage rate of correction equipment, and improving the safety of correction equipment.
[0068] Step 104: In response to the determination that the target image set is missing in the multi-source image group set of medical cases, the group image feature vector set corresponding to the target image set is masked to obtain the masked feature vector set.
[0069] In some embodiments, the executing entity may, in response to determining that a target image set is missing from the multi-source image group of medical cases, perform masking processing on the grouped image feature vector set corresponding to the target image set to obtain a masked feature vector set. The target image set is one of the following: a multi-source image group of medical cases, or an image set within a multi-source image group of medical cases. The target image in the target image set may be a missing image from the multi-source image group of medical cases. The masked feature vector in the masked feature vector set may be a feature vector where the feature value corresponding to the missing target image in the grouped image feature vector set is set to 0, and the feature value corresponding to the existing multi-source image set of medical cases is set to 1. When the target image set can be a multi-source image group of medical cases, it represents the case where the images in the corresponding group after grouping are empty. When the target image set can be an image set within a multi-source image group of medical cases, it represents the case where there is a missing image in the group after grouping.
[0070] In some optional implementations of certain embodiments, the above response to determining that a target image set is missing in the multi-source image set of the medical case, and performing masking processing on the grouped image feature vector set corresponding to the target image set to obtain a masked feature vector set, may include the following steps: The first step is to perform the following masking steps for each medical case multi-source imaging group in the above medical case multi-source imaging set: Sub-step 1: In response to determining that the missing target image set is a missing multi-source image group of medical cases, the feature vector of the grouped image corresponding to the target image set is set to zero, resulting in a first mask feature vector. Here, the missing multi-source image group of medical cases can be the complete missing multi-source image group of medical cases within the multi-source image group set. The first mask feature vector can be a feature vector in which the feature vector of the grouped image corresponding to the missing multi-source image group of medical cases is set to zero.
[0071] Sub-step 2, in response to determining that the missing image in the target image set is at least one missing multi-source image from a multi-source medical case image group, performs average pooling on the grouped image feature vectors corresponding to the multi-source medical case image group to obtain pooled grouped image feature vectors. Here, the missing multi-source medical case image in the multi-source medical case image group can be at least one missing multi-source medical case image within the multi-source medical case image group. In practice, the executing entity can average the initial image feature vector group corresponding to the at least one non-missing multi-source medical case image in the multi-source medical case image group to obtain the pooled grouped image feature vector group.
[0072] The first sub-step involves masking the pooled grouped image feature vectors and the zero vector set corresponding to the target image set to obtain a second masked feature vector. This second masked feature vector can be a feature vector where the feature values at the locations of the grouped image feature vectors corresponding to the non-missing multi-source image sets of the medical case multi-source image group are set to 1, and the feature values at the locations of the grouped image feature vectors corresponding to the missing multi-source image sets of the medical case multi-source image group are set to 0. In practice, the execution entity can combine and concatenate the pooled grouped image feature vector group and the zero vector set corresponding to the target image set, and then input this concatenation into the trained intra-group feature aggregation layer to obtain the second masked feature vector.
[0073] The second step involves determining the aforementioned first mask feature vector set and / or the aforementioned second mask feature vector set as the aforementioned mask feature vector set. Specifically, when all medical case multi-source image groups included in the aforementioned medical case multi-source image set are entirely missing, the aforementioned mask feature vector set can be the first mask feature vector set. When all medical case multi-source image groups included in the aforementioned medical case multi-source image set have one or more missing medical case multi-source images, the aforementioned mask feature vector set can be the second mask feature vector set. When all medical case multi-source image groups included in the aforementioned medical case multi-source image set have either an entire missing medical case multi-source image set or at least one missing medical case multi-source image in a medical case multi-source image set, the aforementioned mask feature vector set can be composed of the first mask feature vector set and the second mask feature vector set.
[0074] In addressing the technical problems mentioned in the background section, when employing technical solutions to solve these problems, the following technical issues often arise in the application scenarios: scenarios where medical image acquisition is incomplete, image loss rates are high, and timely correction is required (e.g., emergency medical image acquisition scenarios with low patient cooperation or incomplete examination equipment in primary healthcare institutions). These issues are often accompanied by incomplete images due to factors such as equipment malfunction, incomplete examination items, insufficient patient cooperation, substandard image quality, or abnormal data transmission. Existing technologies typically directly discard incomplete images for image classification, resulting in a waste of effective case data. Alternatively, they may simply replace missing images with zero vectors, making it difficult for case-level fusion models to distinguish between "true feature values close to zero" and "image loss." Or, they may use a fixed mean to fill in missing images, making it difficult to adaptively complete them based on the information carried by the non-missing images in the same case. Consequently, the grouped image feature vectors and case-level fusion feature vectors easily deviate from the normal feature distribution of complete cases, leading to low accuracy of the image classification information set. This causes deviations in the parameter adjustment of the correction equipment, requiring multiple adjustments and increasing the damage rate of the correction equipment. To address the following requirements for this application scenario: adapting to the loss of the entire image set, adapting to the loss of a single or multiple images within the set, adapting to the explicit representation of the missing state, adapting to information completion based on the remaining valid images, and adapting to highly robust joint discrimination in incomplete cases, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the above response to determining that a target image set is missing in the multi-source image set of the medical case, and performing masking processing on the grouped image feature vector set corresponding to the target image set to obtain a masked feature vector set, may include the following steps: The first step is to perform the following merging steps for each of the above-mentioned medical case multi-source imaging sets: Sub-step 1, in response to determining that the above-mentioned target image set is missing as a multi-source image group missing in the above-mentioned medical cases, performs the following additive processing steps: The first sub-step involves obtaining a zero vector with the same dimension as the feature vector of the grouped images in the aforementioned multi-source image group of medical cases, as the initial zero vector. This initial zero vector can represent the all-zero occupancy information of the missing image group in the feature space. The aforementioned missing multi-source image group of medical cases can refer to the complete absence of the entire multi-source image group of medical cases within the set of multi-source image groups of medical cases.
[0075] The second sub-step involves retrieving the corresponding type encoding vector from the modality encoding vector table based on the image source type corresponding to the aforementioned multi-source image group of medical cases. The image source type can be a component of the diseased area corresponding to the multi-source image group of medical cases (e.g., facial image). Each type encoding vector in the modality encoding vector table is a learnable parameter vector representing the feature information of multi-source images of medical cases, learned by the grouped image feature extraction model through backpropagation during model training. The type encoding vector can represent the common feature pattern information learned by the corresponding image source type during training. In practice, the executing entity can first use the image source type as a retrieval index to find the type encoding vector corresponding to the image source type in the modality encoding vector table. During model training, the executing entity can use the type encoding vector as model parameters in forward computation and backpropagate the gradient corresponding to each vector element in the type encoding vector based on the total loss value. Finally, the type encoding vector is updated based on the gradient, enabling the grouped image feature extraction model to learn the type semantics and missing semantics of the multi-source image group of medical cases corresponding to the type encoding vector.
[0076] The third sub-step involves element-wise addition of the initial zero vector and the type-coded vector to obtain the first mask feature vector. This first mask feature vector can represent alternative feature information for the missing image group. The initial zero vector can be a vector with the same shape as the type-coded vector.
[0077] Sub-step 2, in response to determining that the above-mentioned target image set is missing as the medical case multi-source image group includes the medical case multi-source image missing, performs the following input steps: The first sub-step involves performing key mapping processing on the initial image feature vector set corresponding to the multi-source image set of the medical cases without missing images, resulting in a key vector set. Here, each key vector in the key vector set can be represented as a key vector that needs to be matched with the initial image feature vector. In practice, the executing entity can perform matrix multiplication on each initial image feature vector in the initial image feature vector set corresponding to the multi-source image set of the medical cases without missing images, along with trained key mapping weight parameters, to obtain a key vector. Here, the trained key mapping weight parameters can be represented as linear mapping information transforming the initial image feature vector into a key vector.
[0078] The second sub-step involves performing value mapping processing on the initial image feature vector set corresponding to the aforementioned multi-source image set of medical cases without missing images, resulting in a value vector set. The value vectors in this set can be represented as feature information from the initial image feature vectors used to generate the weighted output. In practice, the executing entity can perform matrix multiplication on each initial image feature vector in the initial image feature vector set corresponding to the aforementioned multi-source image set of medical cases with the trained value mapping weight parameters to obtain a value vector. The trained value mapping weight parameters can be represented as linear mapping information transforming the initial image feature vector into a value vector.
[0079] The third sub-step involves searching for the corresponding image-level coding vector in the modality coding vector table based on the intra-group image perspective type corresponding to the missing multi-source images of the medical case. The intra-group image perspective type can be the type of different perspectives (e.g., frontal view) of the components of the diseased area corresponding to the missing multi-source images of the medical case. The image-level coding vector can represent the set of shooting perspective information for the missing multi-source images of the medical case. In practice, the executing entity can use the type identifier set of the missing multi-source images of the medical case as an index to search for the image-level coding vector matching the type identifier in the modality coding vector table.
[0080] The fourth sub-step involves performing query mapping processing on the aforementioned image-level encoded vector to obtain a query vector. This query vector can represent query feature information about the image viewpoint type within a group. In practice, the executing entity can perform matrix multiplication on the aforementioned image-level encoded vector and the trained query mapping weight parameters to obtain the query vector. The trained query mapping weight parameters can be learnable parameters obtained through backpropagation during the model training phase.
[0081] The fifth sub-step involves determining the dot product between the query vector and each key vector in the key vector set to obtain an initial similarity set. The initial similarity in this initial similarity set can be characterized as the initial correlation information between multi-source images of medical cases with missing features and images without missing features.
[0082] The sixth sub-step involves normalizing the initial similarity set to obtain an attention weight vector. This attention weight vector represents the contribution ratio of each non-missing image to the missing multi-source images of the medical case. In practice, the executing entity can input the initial similarity set into a normalization exponential function (Softmax function) for normalization to obtain the attention weight vector.
[0083] The seventh sub-step involves performing a weighted summation of the value vector set based on the aforementioned attention weight vector to obtain the missing image mask embedding vector. This missing image mask embedding vector can be represented as the alternative feature information of the missing multi-source images of the medical case after attention aggregation. In practice, the executing entity can first use a broadcasting mechanism to broadcast and copy the attention weight values included in the aforementioned attention weight vector, then multiply them element-wise with the corresponding value vectors in the aforementioned value vector set to obtain a weighted value vector. Then, the element-wise summation of each weighted value vector yields the missing image mask embedding vector.
[0084] The eighth sub-step involves inputting the missing image mask embedding vector and the initial image feature vector set corresponding to the non-missing multi-source medical case image set into the trained intra-group feature aggregation layer to determine the second mask feature vector. This second mask feature vector can be represented as the group-level feature information of image groups with missing images after imputation.
[0085] The second step is to determine the mask feature vector set by combining the obtained first mask feature vectors and / or the obtained second mask feature vectors.
[0086] The above-mentioned technical content, combined with steps 105 to 107 and related content, serves as an inventive point of this disclosure, solving the technical problem of "low accuracy of image classification information set, causing deviations in the parameter adjustment of correction equipment, requiring multiple adjustments, and increasing the damage rate of correction equipment." The factors contributing to low accuracy of image classification information set, causing deviations in the parameter adjustment of correction equipment, requiring multiple adjustments, and increasing the damage rate of correction equipment are often as follows: factors such as equipment malfunction, incomplete examination items, insufficient patient cooperation, substandard image quality, or abnormal data transmission can affect the acquisition of incomplete images from cases. Existing technologies typically directly discard cases with incomplete images for image classification, resulting in a waste of effective case data; or they may simply replace missing images with zero vectors, making it difficult for case-level fusion models to distinguish between "true feature values close to zero" and "image missing" states; or they may use a fixed mean to fill in missing images, making it difficult to adaptively complete them based on the information carried by the non-missing images in the same case. Incomplete images in acquired cases can be caused by factors such as equipment malfunction, incomplete examination items, insufficient patient cooperation, substandard image quality, or abnormal data transmission. Current technologies typically discard cases with incomplete images for classification, wasting valuable case data. Alternatively, they may simply replace missing images with zero vectors, making it difficult for case-level fusion models to distinguish between "true feature values close to zero" and "missing images." Or, they may use a fixed mean to fill in missing images, failing to adaptively complete them based on information carried by non-missing images within the same case. Addressing these issues would improve the accuracy of the image classification information set, reduce parameter adjustments to the correction equipment, avoid repeated adjustments, and decrease the equipment damage rate. To achieve this effect, this disclosure introduces a learnable type encoding vector and an attention-based feature reconstruction mechanism: When the entire group is missing, the zero vector is added element-wise to the learnable type encoding vector to obtain the first mask feature vector, which maintains the positional integrity of the missing group in the fusion process and carries the type information of the image group; when some images within a group are missing, the type encoding vector of the missing image is used as the query vector, and the features of the non-missing images are used as the key vector and value vector, respectively. Attention weights are obtained through dot product similarity and normalization processing. The value vectors are then weighted and summed to reconstruct the missing image mask embedding vector. This is then fused with the features of the non-missing images using an in-group feature aggregation layer to obtain the second mask feature vector. When the similarity between the type of the missing image and the features of the non-missing images is high, the attention weight is large, and the features of the non-missing images occupy a high proportion in the reconstruction of the missing image mask embedding vector; conversely, when the similarity is low, the attention weight is small, and its proportion is low.The aforementioned attention weights are determined by normalizing the dot product of the query vector and the key vector. The type encoding vector and image-level encoding vector in the modality encoding vector table are learnable parameter vectors obtained through backpropagation of errors during model training. Since the type encoding vector can carry the type information of the missing image at the missing location, and the attention mechanism can adaptively reconstruct the feature embedding of the missing image using information from the non-missing images, the feature vector obtained through masking can maintain consistency with the feature semantics under normal conditions. Therefore, after determining the first mask feature vector and / or the second mask feature vector as the mask feature vector, the case-level fusion feature extraction model can stably complete feature fusion in image-missing scenarios. Finally, the aforementioned mask feature vector set is used for subsequent case-level fusion and classification, thereby improving the accuracy of classification results based on multi-source images of medical cases in image-missing scenarios. This reduces the number of parameter modifications to the correction device, lowers the damage rate of the correction device, and improves the safety of the correction device.
[0087] Step 105: Using the trained case-level fusion feature extraction model, perform case-level fusion on the target group image feature vector set to obtain the case-level fusion feature vector.
[0088] In some embodiments, the execution entity can utilize the trained case-level fusion feature extraction model to perform case-level fusion on the target grouped image feature vector set to obtain a case-level fusion feature vector. The target grouped image feature vector set includes one of the following: the mask feature vector set and the grouped image feature vector set corresponding to the non-missing multi-source medical case image set, or the grouped image feature vector set. The case-level fusion feature vector can be characterized as a feature vector that uniformly embeds the target grouped image feature vectors located in different embedding spaces into a unified embedding space. The target grouped image feature vector set can be the mask feature vector set and the grouped image feature vector set corresponding to the non-missing multi-source medical case image set, or it can be a grouped image feature vector set, or it can be a set composed of the mask feature vector set, the grouped image feature vector set corresponding to the non-missing multi-source medical case image set, and the grouped image feature vector set.
[0089] In some optional implementations of certain embodiments, the above-trained case-level fusion feature extraction model is used to perform case-level fusion on the target group image feature vector set to obtain a case-level fused feature vector, which may include the following steps: The first step involves linearly mapping the target group image feature vector set to obtain a mapped group image feature vector set. The mapped group image feature vectors in this set can be feature vectors that uniformly map target group image feature vectors with different output dimensions to a unified spatial dimension. For example, the target group image feature vectors with different output dimensions could be 768-dimensional feature vectors output by the ConvNeXt-Tiny model, or 1024-dimensional feature vectors output by the DenseNet121 model. The mapped group image feature vectors can also be 512-dimensional feature vectors. In practice, the execution entity can input the target group image feature vector set into a linear layer for linear mapping to obtain the mapped group image feature vector set.
[0090] The second step involves concatenating the feature vectors of each mapped group image in the aforementioned mapped group image feature vector set along the channel dimension to obtain a concatenated feature vector. This concatenated feature vector can be a 2048-dimensional feature vector. In practice, the executing entity can concatenate the feature vectors of each mapped group image included in the mapped feature vector set along the channel dimension to obtain the concatenated feature vector.
[0091] The third step is to input the above-mentioned concatenated feature vector into the trained case-level fusion feature extraction model to obtain the above-mentioned case-level fusion feature vector.
[0092] In addressing the technical problems mentioned in the background section, the following technical issues often arise in the application scenarios: high-precision medical correction scenarios involving multimodal image joint interpretation and cross-modal correlation analysis of lesions (e.g., multi-sequence joint localization of brain lesions, joint assessment of multiple site injuries). These scenarios often involve complex nonlinear interactions between multiple image groups in medical cases. Existing case-level fusion methods typically involve directly concatenating, simply adding, or weighting the feature vectors of each image group. Essentially, these methods only linearly combine the features of each image group, failing to explicitly characterize the second-order interaction relationships reflected by the dimension-wise product between different image groups. Furthermore, different image groups and combinations of image groups contribute differently to the case classification task. Simple weighted fusion includes interaction information and noise with low relevance to the current case classification task, diluting the discriminative information with high relevance. This results in low discriminative power of the case-level fused feature vectors, leading to low accuracy of the image classification information set. Consequently, the parameters of the correction device need to be adjusted repeatedly, increasing the device's damage rate. To address the following requirements for this application scenario: adaptability to modeling nonlinear interactions between multiple imaging groups, adaptability to differences in importance between different modalities, adaptability to redundancy information and noise suppression, adaptability to high-precision case-level joint discrimination, and adaptability to structurally complex and subtle correction sites, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the above-trained case-level fusion feature extraction model is used to perform case-level fusion on the target group image feature vector set to obtain a case-level fused feature vector, which may include the following steps: The first step involves performing a linear mapping process on the aforementioned target group image feature vector set to obtain a mapped group image feature vector set. The mapped group image feature vectors in this set can be represented as information about the visual features of the multi-source image set of the corresponding medical case within a unified embedding space. In practice, the execution entity can first determine the input dimension of each target group image feature vector in the aforementioned target group image feature vector set. Then, each target group image feature vector is input into the mapping corresponding to its input dimension to obtain the mapped group image feature vector corresponding to the target group image feature vector. Finally, the mapped group image feature vectors are combined according to the order of the target group image feature vector set to obtain the mapped group image feature vector set. The linear mapping layer can be a fully connected layer used to perform a linear transformation on the input vectors. The input dimension can be the vector dimension of the aforementioned target group image feature vectors.
[0093] The second step involves performing an outer product operation on the set of mapped group image feature vector pairs corresponding to the aforementioned mapped group image feature vector set, resulting in a set of inter-group interaction matrices. Here, each mapped group image feature vector pair in the aforementioned set can represent the similarity relationship information between two mapped group image feature vectors. The inter-group interaction matrix in the aforementioned set can be represented as a matrix that characterizes the interaction relationship between the features of two image groups, obtained after the outer product operation of two mapped group image feature vectors. In practice, the executing entity can perform an outer product operation on each mapped group image feature vector pair in the aforementioned set of mapped group image feature vector pairs to obtain the inter-group interaction matrix.
[0094] The third step involves performing channel concatenation on each mapping group image feature vector pair in the aforementioned set of mapping group image feature vector pairs to generate a modal pair concatenation vector, resulting in a modal pair concatenation vector set. This modal pair concatenation vector can be characterized as the joint representation information of the two mapping group image feature vectors in the mapping group image feature vector pair after channel concatenation. In practice, the executing entity can, for each mapping group image feature vector pair in the aforementioned set of mapping group image feature vector pairs, concatenate the two mapping group image feature vectors along the channel dimension to obtain the modal pair concatenation vector.
[0095] The fourth step involves performing a gated linear transformation on each modal pair stitching vector in the aforementioned modal pair stitching vector set to obtain a set of initial modal pair gating values. These initial modal pair gating values can be characterized as the initial importance information of the mapped grouped image feature vector pairs in case-level fusion. In practice, the executing entity can use trained weight parameters and bias parameters to perform a weighted summation of each element in the aforementioned modal pair stitching vectors to obtain the initial modal pair gating values. The initial modal pair gating values constitute the set of initial modal pair gating values. The aforementioned weight parameters and bias parameters can be learnable parameters obtained through backpropagation during the model training phase.
[0096] The fifth step involves activating the aforementioned modality pair gating initial value set to obtain a modality pair gating value set. The modality pair gating values in this set can be represented as adaptive fusion weight information of the activated mapping grouped image feature vector pairs. In practice, the executing entity can first input the aforementioned modality pair gating initial value set into an activation function (e.g., the Sigmoid function) for activation, mapping each of the initial modality pair gating values to a value between 0 and 1, thus obtaining the modality pair gating value set corresponding to the initial modality pair gating values.
[0097] Step 6: Based on the aforementioned modality pair gating value set, perform a weighted summation process on the aforementioned inter-group interaction matrix set to obtain the aggregated interaction matrix. This aggregated interaction matrix can be characterized as a matrix containing the interaction information of each mapped group image feature vector pair after gating and weighted fusion. In practice, the executing entity can first multiply each modality pair gating value in the aforementioned modality pair gating value set with the corresponding inter-group interaction matrix in the aforementioned inter-group interaction matrix set to obtain the weighted interaction matrix corresponding to that modality pair. Then, the weighted interaction matrices are summed element-wise to obtain the aforementioned aggregated interaction matrix.
[0098] Step 7: Perform L2 normalization on the above aggregated interaction matrix to obtain a normalized interaction matrix. This normalized interaction matrix can be represented as the standardized second-order interaction information of the aggregated interaction matrix after scale normalization. In practice, the executing entity can first determine the arithmetic square root of the sum of squares of all elements in the aggregated interaction matrix to obtain the L2 norm of the matrix. Then, divide each element of the aggregated interaction matrix by the L2 norm to obtain the normalized interaction matrix.
[0099] Step 8: Vectorize the normalized interaction matrix to obtain the interaction feature vector. This interaction feature vector can be represented as the global second-order interaction feature information after the normalized interaction matrix has been vectorized and expanded. In practice, the executing entity can arrange the elements of the normalized interaction matrix row-wise to obtain the interaction feature vector.
[0100] Step 9: Input the above-mentioned interactive feature vector into the trained case-level fusion feature extraction model to obtain the above-mentioned case-level fusion feature vector.
[0101] The above-described technical content, combined with steps 106 and 107 and their related content, serves as an inventive point of this disclosure, solving the technical problem of "low discriminative ability of case-level fusion feature vectors, leading to low accuracy of image classification information sets, requiring repeated adjustments to the parameters of the correction device, and increasing the damage rate of the correction device." The factors leading to low discriminative ability of case-level fusion feature vectors, resulting in low accuracy of image classification information sets, requiring repeated adjustments to the parameters of the correction device, and increasing the damage rate of the correction device are often as follows: Medical case multi-source image groups are not independent of each other, but rather have complex nonlinear interaction relationships. Existing case-level fusion methods typically only directly splice, simply add, or equally weighted average the feature vectors of each image group. Essentially, they can only linearly combine the features of each image group, making it difficult to explicitly characterize the second-order interaction relationship reflected by the dimension-wise product between different image groups. Meanwhile, different image groups and combinations of image groups contribute differently to the case classification task. Simple equal-weight fusion will include interactive information and noise with low relevance to the current case classification task, thus diluting the discriminative information with high relevance to the current case classification task. This results in low discriminative power of the case-level fused feature vector, leading to low accuracy of the image classification information set. Consequently, the parameters of the correction device need to be adjusted repeatedly, increasing the damage rate of the correction device. If the above factors are addressed, the discriminative power of the case-level fused feature vector can be improved, thereby increasing the accuracy of the image classification information set, avoiding the need for repeated adjustments of the correction device parameters, and reducing the damage rate of the correction device. To achieve this effect, this disclosure introduces an inter-group interaction matrix and a gated weighted aggregation mechanism: the mapping group image feature vectors linearly mapped to a unified feature space are subjected to pairwise outer product processing to explicitly construct the inter-group interaction matrix to capture the dimensional interaction relationship between any two group image features. A gated linear transformation and activation processing are performed on the spliced vector of each modality pair to obtain the modality pair gating value, which characterizes the importance of each modality pair in the discrimination process. The interaction matrices between each group are weighted and summed using modality pair gating values to obtain an aggregated interaction matrix. This aggregated interaction matrix is then normalized using L2 to eliminate scale differences. When the interaction relationship between a group pair is highly relevant to the discrimination task, the gating value corresponding to that group pair is larger, resulting in a higher weight for the interaction matrix in the aggregated interaction matrix. Conversely, when the relevance is low, the gating value is smaller, and the corresponding weight for the interaction matrix is lower. These gating values are determined by performing a gating linear transformation and activation function processing on the modality pair concatenation vector. The parameters of this gating linear transformation are learned through error backpropagation during the model training phase.Since the outer product operation can explicitly capture the interaction relationship between any two grouped image features, and the gating mechanism can adaptively adjust the importance of the interaction matrix between each group, the aggregated interaction matrix can comprehensively represent the cross-group association information of cases and highlight the key interaction information relevant to the discrimination task. Therefore, the interaction feature vector obtained by normalizing the aggregated interaction matrix to L2 and then vectorizing it can provide more comprehensive and stable discrimination information for the case-level fusion feature extraction model. Finally, the above interaction feature vector is input into the trained case-level fusion feature extraction model to obtain the case-level fusion feature vector. Classification is performed based on the case-level fusion feature vector to obtain the image classification information set, and the correction device is controlled to adjust and correct parameters. This can improve the accuracy of multi-branch multi-task joint discrimination, reduce the number of repeated modifications to the correction device parameters, reduce the damage rate of the correction device, and improve the safety of the correction device.
[0102] Step 106: Input the case-level fusion feature vector into the trained case-level classification model to obtain the image classification information set for the multi-source image set of medical cases.
[0103] In some embodiments, the executing entity can input the case-level fusion feature vector into the trained case-level classification model to obtain an image classification information set for a multi-source image set of medical cases. This image classification information set can be characterized as a collection of comprehensive classification results covering multiple dimensions. The image classification information in the image classification information set can be the disease type of the diseased portion corresponding to the multi-source image set of the medical cases. For example, the image classification information set may include: degree of overbite, degree of overjet, and left molar relationship.
[0104] Step 107: Based on the image classification information set, determine the adjustment parameters of the correction device for the multi-source image set of medical cases, so as to control the correction device to perform correction.
[0105] In some embodiments, the executing entity can determine adjustment parameters of the correction device for the multi-source image set of the medical case based on the image classification information set, so as to control the correction device to perform correction. The correction device can be an adjustable medical device for applying corrective action to the patient's area to be corrected. For example, the correction device can be an orthodontic appliance. The adjustment parameters can be a set of parameters for controlling the adjustment state of the correction device. The adjustment parameters can include, but are not limited to, at least one of the following: target position, correction angle, force applied to the area to be corrected, and duration of application to the area to be corrected. In practice, the executing entity can first use a multi-objective optimization algorithm to obtain the adjustment parameters of the correction device based on the image classification information set. The multi-objective optimization algorithm can be a GA (Genetic Algorithm). Then, using FEA (Finite Element Analysis), the adjustment parameters are converted into correction control signals to control the correction device to perform correction. For example, the control signal can control the drive device of the correction appliance to perform correction according to the adjustment parameters. The above embodiments of this disclosure have the following beneficial effects: The correction method of the correction device based on grouped case-level fusion classification of some embodiments of this disclosure can effectively distinguish the feature information of grouped image feature vectors under different categories by extracting grouped features from multi-source image sets of medical cases. The case-level fusion feature vector obtained by case fusion can effectively fuse shared feature vectors of different categories, improve classification accuracy, improve the precision of parameter adjustment of the correction device, and reduce the wear and tear of the correction device. Specifically, the reason why the classification results based on multi-source images of medical cases are inaccurate and the image stitching requires a large amount of computing power, resulting in a waste of computing resources and the need for multiple adjustments to the correction device is that: directly stitching images from the multi-source image set of medical cases does not take into account the case information of different groups included in the images from different sources. Instead, the stitched medical case images are directly input into the model for classification. The model does not fully learn the distinguishing features between different groups and is prone to confusion. The extracted feature information is relatively simple, resulting in low accuracy of classification results based on multi-source images of medical cases and a large amount of computing power required for image stitching, resulting in a waste of computing resources. This, in turn, leads to low precision in adjusting the parameters of the correction device, requiring repeated adjustments to the correction device, increasing the damage and wear rate of the correction device. Based on this, some embodiments of the correction device correction method based on grouped case-level fusion classification in this disclosure firstly, in response to determining that there is a correction abnormality in the correction device, the multi-source image set of medical cases, the trained grouped image feature extraction model set, the trained case-level fusion feature extraction model, and the trained case-level classification model are obtained.Therefore, the multi-source image set of medical cases is used as input to the subsequently trained grouped image feature extraction model set. The trained case-level fusion feature extraction model and the trained case-level classification model are used to perform case-level feature fusion and classification on the multi-source image set of medical cases. Next, the above-mentioned multi-source image set of medical cases is grouped to obtain multi-source image sets of medical cases, where there is a one-to-one correspondence between the multi-source image sets of medical cases and the grouped image feature extraction model set. Thus, image grouping facilitates the effective learning of visual information from different categories of multi-source image sets of medical cases, improving the quality of the grouped image feature vector set obtained by grouped feature extraction and fusion. Then, using the trained grouped image feature extraction model set, the multi-source image sets of medical cases are subjected to grouped feature extraction and fusion to obtain a grouped image feature vector set. This allows for the extraction of feature vectors within different categories of groups, effectively distinguishing feature information from different categories, reducing the probability of confusion between feature information from different categories, effectively adapting to the discrimination space of different categories, and preserving the differences between different categories. Next, in response to the determination that the target image set is missing in the aforementioned multi-source image set of medical cases, the feature vector set of the grouped images corresponding to the target image set is masked to obtain a masked feature vector set. The target image set is one of the following: the multi-source image set of medical cases, or an image set within the multi-source image set of medical cases. Thus, masking can handle missing images when there are gaps in the multi-source image set of medical cases, ensuring the stability and comprehensiveness of the masked feature vector set, thereby improving the accuracy of the case-level fusion feature extraction model output and reducing underfitting. Then, using the trained case-level fusion feature extraction model, case-level fusion is performed on the target grouped image feature vector set to obtain a case-level fused feature vector. The target grouped image feature vector set includes one of the following: the masked feature vector set and the grouped image feature vector set corresponding to the non-missing multi-source image set of medical cases, or the aforementioned grouped image feature vector set. Therefore, case-level fusion can fuse feature information from different categories, i.e., different components of the same part, improving the comprehensiveness of the case-level fused feature vector. Secondly, the aforementioned case-level fusion feature vectors are input into the trained case-level classification model to obtain an image classification information set for the multi-source image set of medical cases. Therefore, due to the comprehensiveness of the case-level fusion feature vectors, the case-level classification model can capture more feature information, and the inclusion of multiple classification heads in the case-level classification model can improve the accuracy of the image classification information set for multi-classification tasks. Finally, based on the aforementioned image classification information set, the adjustment parameters of the correction device for the aforementioned multi-source image set of medical cases are determined to control the correction device's correction process.Therefore, determining adjustment parameters and controlling the correction device based on the image classification information set can improve the accuracy of the adjustment parameters, reduce repeated adjustments caused by classification bias, reduce wear and tear on the correction device, and improve the stability and safety of the device. Thus, this correction device correction method based on grouped case-level fusion classification can effectively distinguish the feature information of grouped image feature vectors under different categories by extracting grouped features from multi-source medical case image sets. The case-level fusion feature vector obtained by case fusion can effectively fuse shared feature vectors from different categories, improving classification accuracy, enhancing the precision of parameter adjustment for the correction device, and reducing wear and tear on the correction device.
[0106] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an orthodontic device based on grouped case-level fusion classification. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the corrective device based on grouped case-level fusion classification can be specifically applied to various electronic devices.
[0107] like Figure 2As shown, an orthodontic device 200 based on grouped case-level fusion classification includes: an acquisition unit 201, a grouping unit 202, an extraction and fusion unit 203, a masking processing unit 204, a fusion unit 205, an input unit 206, and a determination unit 207. The acquisition unit 201 is configured to: in response to determining that an orthodontic device has an orthodontic abnormality, acquire a multi-source image set of medical cases, a trained set of grouped image feature extraction models, a trained case-level fusion feature extraction model, and a trained case-level classification model. The grouping unit 202 is configured to: group the aforementioned multi-source image set of medical cases to obtain a multi-source image set of medical cases, wherein there is a one-to-one correspondence between the multi-source image set of medical cases and the grouped image feature extraction model set. The extraction and fusion unit 203 is configured to: use the trained set of grouped image feature extraction models to perform grouped feature extraction and fusion on the multi-source image set of medical cases to obtain a set of grouped image feature vectors. The masking unit 204 is configured to: in response to determining that a target image set is missing in the aforementioned multi-source image set of medical cases, perform masking processing on the group image feature vector set corresponding to the target image set to obtain a masked feature vector set, wherein the target image set is one of the following: a multi-source image set of medical cases, or an image set within a multi-source image set of medical cases. The fusion unit 205 is configured to: utilize the aforementioned trained case-level fusion feature extraction model to perform case-level fusion on the target group image feature vector set to obtain a case-level fusion feature vector, wherein the target group image feature vector set includes one of the following: the aforementioned masked feature vector set and the group image feature vector set corresponding to the non-missing multi-source image set of medical cases, or the aforementioned group image feature vector set. The input unit 206 is configured to: input the aforementioned case-level fusion feature vector into the aforementioned trained case-level classification model to obtain an image classification information set for the multi-source image set of medical cases. The determining unit 207 is configured to: determine the adjustment parameters of the correction device for the multi-source image set of the medical case based on the above image classification information set, so as to control the correction device to perform correction.
[0108] It is understandable that the units described in the orthodontic device 200 based on grouped case-level fusion classification are similar to those in the reference device. Figure 1 The steps described in the method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the orthodontic device 200 and its constituent units based on grouped case-level fusion classification, and will not be repeated here.
[0109] The following is for reference. Figure 3 It shows a method suitable for implementing this disclosure. Figure 1 A schematic diagram of the structure of an electronic device 300 in some corresponding embodiments. Figure 3The electronic device shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of this disclosure. Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory 302 or a program loaded from a storage device 308 into a random access memory 303. The random access memory 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, the read-only memory 302, and the random access memory 303 are interconnected via a bus 304. An input / output interface 305 is also connected to the bus 304.
[0110] Typically, the following devices can be connected to the input / output interface 305: input devices 306 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 308 including, for example, magnetic tape, hard disk, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.
[0111] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a read-only memory 302. When the computer program is executed by the processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.
[0112] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0113] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.
[0114] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: in response to determining that an orthodontic device has an orthodontic abnormality, acquire a multi-source image set of medical cases, a trained set of grouped image feature extraction models, a trained case-level fusion feature extraction model, and a trained case-level classification model; group the aforementioned multi-source image set of medical cases to obtain a multi-source image group set of medical cases, wherein the aforementioned multi-source image group set of medical cases and the aforementioned set of grouped image feature extraction models have a one-to-one correspondence; use the aforementioned trained set of grouped image feature extraction models to perform grouped feature extraction and fusion on the aforementioned multi-source image group set of medical cases to obtain a set of grouped image feature vectors; in response to determining that a target image set is missing from the aforementioned multi-source image group set of medical cases, perform grouped image feature extraction and fusion on the grouped image set corresponding to the target image set. The feature vector set is masked to obtain a masked feature vector set. The target image set is one of the following: a multi-source image group of medical cases, or an image set within the multi-source image group of medical cases. Using the trained case-level fusion feature extraction model, the target group image feature vector set is fused at the case level to obtain a case-level fusion feature vector. The target group image feature vector set includes one of the following: the masked feature vector set and the group image feature vector set corresponding to the non-missing multi-source image group of medical cases, or the group image feature vector set. The case-level fusion feature vector is input into the trained case-level classification model to obtain an image classification information set for the multi-source image group of medical cases. Based on the image classification information set, the adjustment parameters of the correction device for the multi-source image group of medical cases are determined to control the correction device.
[0115] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0117] The units described in some embodiments of this disclosure can be implemented in software or in hardware. The described units can also be located in a processor.
[0118] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.
[0119] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.
Claims
1. A method for orthodontic device correction based on grouped case-level fusion classification, characterized in that, include: In response to the determination that the correction device has a correction abnormality, the system acquires a multi-source image set of medical cases, a set of trained grouped image feature extraction models, a set of trained case-level fusion feature extraction models, and a set of trained case-level classification models. The medical case multi-source image set is grouped to obtain a medical case multi-source image group set, wherein there is a one-to-one correspondence between the medical case multi-source image group set and the grouped image feature extraction model set; Using the trained set of grouped image feature extraction models, the multi-source image set of the medical case is subjected to grouped feature extraction and fusion to obtain a set of grouped image feature vectors; In response to determining that a target image set is missing in the multi-source image group of medical cases, the group image feature vector set corresponding to the target image set is masked to obtain a masked feature vector set, wherein the target image set is one of the following: multi-source image group of medical cases, or image set in multi-source image group of medical cases; Using the trained case-level fusion feature extraction model, the target group image feature vector set is fused at the case level to obtain a case-level fusion feature vector. The target group image feature vector set includes one of the following: the mask feature vector set and the group image feature vector set corresponding to the non-missing medical case multi-source image group set, or the group image feature vector set. The case-level fusion feature vector is input into the trained case-level classification model to obtain an image classification information set for the multi-source image set of medical cases. Based on the image classification information set, the adjustment parameters of the correction device for the multi-source image set of the medical case are determined in order to control the correction device to perform correction.
2. The method according to claim 1, characterized in that, Before using the trained grouped image feature extraction model set to perform grouped feature extraction and fusion on the multi-source image set of medical cases to obtain a grouped image feature vector set, the method further includes: For the medical case multi-source image set in the aforementioned medical case multi-source image set, the following standardized processing steps are performed: The multi-source image group of the medical case is scaled to obtain the scaled multi-source image group of the medical case. The scaled medical case multi-source image group is subjected to color mode conversion processing to obtain a color-converted medical case multi-source image group. The color-converted medical case multi-source image group is standardized to obtain a standardized medical case multi-source image group, which is used as a medical case multi-source image group set.
3. The method according to claim 1, characterized in that, The trained set of grouped image feature extraction models includes: a trained set of intra-group feature encoding networks and a trained set of intra-group feature aggregation layers; and The step of using the trained set of grouped image feature extraction models to extract and fuse grouped features from the multi-source image set of medical cases to obtain a set of grouped image feature vectors includes: for each multi-source image set of medical cases, performing the following feature extraction steps: From the set of grouped image feature extraction models that have been trained, determine the grouped image feature extraction model corresponding to the multi-source image group of the medical case, and use it as the target grouped image feature extraction model; The multi-source image group of the medical case is input into the trained intra-group feature encoding network included in the target group image feature extraction model to obtain an initial image feature vector group; The initial image feature vector group is input into the trained intra-group feature aggregation layer of the target grouped image feature extraction model to obtain the grouped image feature vector; The obtained multiple grouped image feature vectors are determined as a grouped image feature vector set.
4. The method according to claim 1, characterized in that, In response to determining that a target image set is missing in the multi-source image set of the medical case, the feature vector set of the grouped images corresponding to the target image set is masked to obtain a masked feature vector set, including: For each medical case multi-source image group in the aforementioned medical case multi-source image group set, perform the following masking step: In response to determining that the missing target image set is a missing multi-source image group of the medical case, the grouped image feature vector corresponding to the target image set is set to a zero vector to obtain a first mask feature vector; In response to determining that the missing target image set is at least one missing medical case multi-source image in a medical case multi-source image group, average pooling is performed on the grouped image feature vector corresponding to the medical case multi-source image group to obtain the pooled grouped image feature vector. The pooled grouped image feature vectors and the zero vector set corresponding to the target image set are masked to obtain the second mask feature vector; The obtained first mask feature vector set and / or the obtained second mask feature vector set are determined as the mask feature vector set.
5. The method according to claim 1, characterized in that, The trained case-level fusion feature extraction model is used to perform case-level fusion on the target group image feature vector set to obtain a case-level fused feature vector, including: The target group image feature vector set is linearly mapped to obtain the mapped group image feature vector set. Channel splicing is performed on each feature vector of the mapped grouped image in the set of mapped grouped image feature vectors to obtain spliced feature vectors; The concatenated feature vector is input into the trained case-level fusion feature extraction model to obtain the case-level fusion feature vector.
6. The method according to claim 1, characterized in that, Each of the trained grouped image feature extraction models in the set of trained grouped image feature extraction models is obtained through the following steps: Obtain a first group of case image sample set and a second group of case image sample set for the initial group image feature extraction model, wherein the first group of case image sample set includes: the first group of case image set and the first classification label information corresponding to the first group of case image set, and the second group of case image sample set includes: the second group of case image set and the second classification label information corresponding to the second group of case image set; Based on the image sample set of the first group of cases, the following determination steps were performed: Input at least one first group of case images included in the first group of case image sample set into the initial group image feature extraction model to obtain a group image sample feature vector set; The grouped image sample feature vector set is input into the initial intra-group classification prediction network to obtain image prediction classification information; Determine the within-group classification loss value for the image prediction classification information and the classification label information; In response to determining that the number of times the determination step is executed reaches a first preset training number threshold, the initial grouped image feature extraction model corresponding to the number of executions is determined as the grouped image feature extraction model that has been trained. In response to determining that the number of executions has not reached the preset first training number threshold, the second group of case images is input into the initial group image feature extraction model corresponding to the number of executions to obtain an initial prediction classification information set; Determine the prediction classification accuracy of the initial prediction classification information set and the second classification label information; In response to the prediction accuracy being greater than or equal to a preset classification accuracy threshold, the initial grouped image feature extraction model corresponding to the number of executions is determined as the trained grouped image feature extraction model. In response to the prediction accuracy being less than the preset classification accuracy threshold, the first group of case image sample set is reselected to obtain at least one group of case image sample. Based on the intra-group classification loss value, the parameters of the initial group image feature extraction model and the initial intra-group classification prediction network corresponding to the number of executions are adjusted to obtain the adjusted group image feature extraction model and the adjusted intra-group classification prediction network, which are then used as the initial group image feature extraction model and the initial intra-group classification prediction network to execute the determination step again.
7. The method according to claim 1, characterized in that, The trained case-level fusion feature extraction model and the case-level classification model were obtained through the following training steps: Obtain a case-level training sample set, a case-level validation sample set, and an initial case-level classification model for the initial case-level fusion feature extraction model. The case-level training sample set includes: a multi-type case image set and a multi-classification label information set corresponding to the multi-type case image set. The multi-type case image set is input into the intra-group feature encoding network set included in the trained group image feature extraction model set to obtain a multi-type sample feature vector set; Perform a feature linear transformation on the set of feature vectors of the multi-type samples to obtain the transformed set of feature vectors of the multi-type samples. Based on the case-level training sample set, perform the following joint training steps: The transformed multi-type sample feature vector set corresponding to at least one multi-type case image in the case-level training sample set is input into the initial case-level fusion feature extraction model to obtain the case-level fusion feature vector. The case-level fusion feature vector is input into the initial case-level classification model to obtain a multi-task prediction classification result set; Based on the multi-task prediction classification result set and the multi-class label information set corresponding to the at least one multi-type case image, determine the main task loss value and the auxiliary task loss value; The type grouping weight set corresponding to the multi-task prediction classification result set is subjected to regularization constraint processing to obtain the regularization term loss value; The total loss value is obtained by weighted summing of the main task loss value, the auxiliary task loss value, and the regularization term loss value. In response to determining that the number of joint training executions of the joint training step has reached a second preset training number threshold, the initial case-level fusion feature extraction model and the initial case-level classification model corresponding to the number of joint training executions are determined as the trained case-level fusion feature extraction model and the trained case-level classification model. In response to determining that the number of joint training executions has not reached the second preset training number threshold, the transformed multi-type sample feature vectors corresponding to the multi-type verification case image sets included in the case-level verification sample set are sequentially input into the initial case-level fusion feature extraction model and the initial case-level classification model corresponding to the number of joint training executions, to obtain the verification prediction classification result set; The validation accuracy is determined based on the validation prediction classification result set and the multi-task validation classification label set included in the initial case-level validation sample set; In response to determining that the verification accuracy is greater than or equal to a preset accuracy threshold, the initial case-level fusion feature extraction model and the initial case-level classification model corresponding to the number of joint training executions are determined as the trained case-level fusion feature extraction model and the trained case-level classification model. In response to determining that the verification accuracy is less than the preset accuracy threshold, at least one multi-type case image is reselected from the case-level training sample set, and the parameters of the initial case-level fusion feature extraction model and the initial case-level classification model corresponding to the number of joint training executions are adjusted according to the total loss value to obtain the adjusted case-level fusion feature extraction model and the adjusted case-level classification model, which are used as the initial case-level fusion feature extraction model and the initial case-level classification model, so as to execute the joint training step again.
8. A correction device and method apparatus based on grouped case-level fusion classification, characterized in that, include: The acquisition unit is configured to acquire, in response to determining that there is an abnormality in the correction device, a multi-source image set of medical cases, a set of trained grouped image feature extraction models, a set of trained case-level fusion feature extraction models, and a set of trained case-level classification models. The grouping unit is configured to group the medical case multi-source image set to obtain a medical case multi-source image group set, wherein there is a one-to-one correspondence between the medical case multi-source image group set and the grouped image feature extraction model set; The extraction and fusion unit is configured to use the trained set of grouped image feature extraction models to perform grouped feature extraction and fusion on the multi-source image set of the medical case to obtain a set of grouped image feature vectors. The masking unit is configured to, in response to determining that there is a missing target image set in the multi-source image group of the medical case, perform masking processing on the grouped image feature vector set corresponding to the target image set to obtain a masked feature vector set, wherein the target image set is one of the following: the multi-source image group of the medical case, or the image set in the multi-source image group of the medical case. The fusion unit is configured to use the trained case-level fusion feature extraction model to perform case-level fusion on the target group image feature vector set to obtain a case-level fusion feature vector, wherein the target group image feature vector set includes one of the following: the mask feature vector set and the group image feature vector set corresponding to the non-missing medical case multi-source image group set, or the group image feature vector set. The input unit is configured to input the case-level fused feature vector into the trained case-level classification model to obtain an image classification information set for a multi-source image set of medical cases. The determining unit is configured to determine adjustment parameters of the correction device for the multi-source image set of the medical case based on the image classification information set, so as to control the correction device to perform correction.
9. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.
10. A computer-readable medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.
