Image soft tissue detection analysis method and device based on deep learning, equipment, medium and product

Abstract

This application discloses a method, apparatus, device, medium, and product for image soft tissue detection and analysis based on deep learning, relating to the fields of deep learning vision and intelligent analysis technology. The method includes: acquiring facial image data to be detected; classifying the facial image data to be detected based on a first image classification neural network; calling a corresponding second keypoint analysis neural network based on the classification results to perform coarse position prediction of preset keypoints in the facial image data to be detected, obtaining coarse position prediction results for the keypoints; inputting the facial image data to be detected and the coarse position prediction results for the keypoints into a third keypoint analysis neural network to obtain precise position prediction results for the keypoints; and performing soft tissue detection and analysis based on the precise position prediction results for the keypoints to obtain facial soft tissue analysis results. This application can improve the accuracy and efficiency of image soft tissue detection and analysis.

Description

Technical Field

[0001] This application relates to the field of deep learning vision and intelligent analysis technology, and in particular to a method, apparatus, device, medium and product for image soft tissue detection and analysis based on deep learning. Background Technology

[0002] In medical image analysis, especially in the field of facial soft tissue assessment, frontal and lateral photographs (i.e., frontal and lateral views) are frequently used to assist in diagnosis, treatment planning, and efficacy evaluation. Traditional methods mainly rely on doctors or technicians manually annotating key anatomical landmarks, which is not only time-consuming and labor-intensive but also easily influenced by subjective experience, resulting in poor repeatability and consistency. In recent years, with the development of computer vision and deep learning technologies, facial landmark detection methods based on Convolutional Neural Networks (CNNs) or Transformer architectures have made significant progress in general scenarios.

[0003] However, soft tissue analysis in medical settings places higher demands on accuracy, robustness, and anatomical plausibility, making it difficult to directly apply general-purpose models. Furthermore, frontal and lateral images differ significantly in imaging angle, visible structures, and feature distribution; using a single model can easily lead to localization errors or missed detections. Therefore, there is an urgent need for a soft tissue analysis method that can combine image type priors with staged, refined localization to balance analytical efficiency and clinical accuracy, meeting the practical needs of professional fields such as orthodontics and craniofacial surgery. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, device, medium, and product for image soft tissue detection and analysis based on deep learning, which can improve the accuracy and efficiency of image soft tissue detection and analysis.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a deep learning-based method for detecting and analyzing soft tissue in images, including: Acquire facial image data to be detected; The first image classification neural network classifies the facial image data to be detected to obtain classification results; the classification results include frontal and side views; the first image classification neural network is trained with the encoder in the autoencoder network as the skeleton and based on a multilayer perceptron, according to the training dataset, with the goal of minimizing the cross-entropy loss function; the training dataset includes facial image data with known classification results and precise key point locations. Based on the classification results, the corresponding second key point analysis neural network is invoked, and based on the second key point analysis neural network, the rough position prediction of the preset key points is performed on the facial image data to be detected, and the rough position prediction result of the key points is obtained. The second key point analysis neural network is obtained by using the encoder in the autoencoder network as the skeleton and based on the multilayer perceptron, and is trained with the goal of minimizing the loss function according to the training dataset. The facial image data to be detected and the rough position prediction results of the key points are input into the third key point analysis neural network to obtain the precise position prediction results of the key points. The third key point analysis neural network is obtained by using the encoder and decoder in the autoencoder network as the skeleton, performing key point position drift iterative processing, and comparing and regressing training based on the training dataset. Based on the predicted precise location of the key points, soft tissue detection and analysis are performed to obtain facial soft tissue analysis results; the soft tissue detection and analysis includes point, line distance, and angle analysis.

[0006] In one embodiment, the expression corresponding to the cross-entropy loss function is: ; in, The cross-entropy loss function; This represents the probability that the image in the classification results is a frontal view. and All are true classifications of the images. When the image is a frontal view... =1, otherwise The value is 0 when the image is a side view. =1, otherwise =0; This represents the probability that the image in the classification results is a side view.

[0007] In one embodiment, the expression corresponding to the loss function is: ; in, The loss function; The weight of L1 loss in the loss function; For the first The weight of the loss at each point relative to the total loss at all points; The maximum threshold range for L1 loss; For serial numbers; The first in the rough location prediction results of key points The coordinates of the points; For the first The actual coordinates of each point; For L1 regularization; The weight of L2 loss in the loss function; This is for L2 regularization.

[0008] In one embodiment, key point position drift iteration processing is performed, specifically including: ; ; The final output of the iterative process is expressed as follows: ; in, After the first iteration of the neural network, which analyzes the third key point, the... The coordinates to which the point drifts; Analyze the neural network for the third key point; Analyze the network parameters of the neural network for the third key point; The second key point is analyzed by the neural network prediction of the first key point. The coordinates of the points; The facial image data to be detected; For the analysis of the neural network after the third key point After the nth iteration, the th The coordinates to which the point drifts; For the analysis of the neural network after the third key point After the nth iteration, the th The coordinates to which the point drifts; The coordinates of all points output after the neural network iteration is completed for the third key point analysis.

[0009] In one embodiment, the expression for the loss function corresponding to the comparison and regression training based on the training dataset in the third key point analysis neural network is: ; ; ; in, For the first Loss function after wheel drift; This is the weighted sum of the loss functions after multiple rounds of drifting; For the first The weight of the loss at each point relative to the total loss at all points; The weight of L1 loss in the loss function; The maximum threshold range for L1 loss; The third key point is analyzed by the neural network prediction of the first key point. The coordinates of the points; For the first The actual coordinates of each point; for The weight of the loss in the loss function; For L1 regularization; For L2 regularization; This represents the total number of iterations. For the first The weight of the loss in each iteration in the loss function; As a weighting factor; For serial numbers.

[0010] In one implementation, within the second keypoint analysis neural network, the output of the multilayer perceptron is a one-dimensional vector: ; in, The number of key points in a frontal photograph; The number of key points in a side view photograph; Analyze the neural network for the second key point; In order to target the Zhang image, second key point analysis, one-dimensional vector output by neural network; Analyze the network parameters of the neural network for the second key point; For the first Zhang Image; Dimensions The real vector space; Split a one-dimensional vector into , 4-dimensional vectors, respectively corresponding to Direction coordinate vector , Direction coordinate vector Linear transformation matrix ; Based on template coordinate parameters The formula for generating a rough prediction of key point locations is as follows: ; in, The coordinates predicted by the neural network are analyzed for the second key point; It is a 2×2 identity matrix.

[0011] Secondly, this application provides a deep learning-based image soft tissue detection and analysis device, comprising: The data acquisition module is used to acquire facial image data to be detected; The classification processing module is used to classify the facial image data to be detected based on the first image classification neural network to obtain the classification results; the classification results include frontal and side views; the first image classification neural network is trained with the encoder in the autoencoder network as the skeleton and based on the multilayer perceptron, according to the training dataset, with the goal of minimizing the cross-entropy loss function; the training dataset includes facial image data with known classification results and precise key point locations. The coarse location prediction module is used to call the corresponding second key point analysis neural network based on the classification results, and to perform coarse location prediction of preset key points on the facial image data to be detected based on the second key point analysis neural network, so as to obtain the coarse location prediction result of the key points; the second key point analysis neural network is trained with the encoder in the autoencoder network as the skeleton and based on the multilayer perceptron, according to the training dataset, with the goal of minimizing the loss function. The precise location prediction module is used to input the facial image data to be detected and the coarse location prediction results of the key points into the third key point analysis neural network to obtain the precise location prediction results of the key points. The third key point analysis neural network is obtained by using the encoder and decoder in the autoencoder network as the skeleton, performing key point location drift iterative processing, and performing comparison and regression training based on the training dataset. The detection and analysis module is used to perform soft tissue detection and analysis based on the predicted precise location of the key points to obtain facial soft tissue analysis results; the soft tissue detection and analysis includes point, line distance, and angle analysis.

[0012] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the deep learning-based image soft tissue detection and analysis method described above.

[0013] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned deep learning-based image soft tissue detection and analysis method.

[0014] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned deep learning-based image soft tissue detection and analysis method.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method, apparatus, device, medium, and product for image soft tissue detection and analysis based on deep learning. By determining the classification results, and based on these results, a second keypoint analysis neural network for the corresponding category is invoked to perform preliminary localization and bounding to obtain the approximate location of soft tissue feature points. For each approximate location of a soft tissue feature point, an operation to optimize the feature point location is performed, i.e., based on a third keypoint analysis neural network, to obtain the precise location of each soft tissue feature point, which is the keypoint precise location prediction result. Image soft tissue detection is achieved through three steps: classification, keypoint approximate location prediction, and keypoint precise location prediction, thereby enabling soft tissue detection and analysis to obtain facial soft tissue analysis results and improving the accuracy and efficiency of image soft tissue detection and analysis. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a deep learning-based image soft tissue detection and analysis method. Figure 2 This is a flowchart illustrating the practical application of a deep learning-based image soft tissue detection and analysis method. Figure 3 This is an example of the user interface for positioning, measuring, and analyzing soft tissue in frontal radiographs. Figure 4 This is an example diagram of the user interface for positioning, measuring, and analyzing soft tissue in lateral radiographs. Figure 5 This is a structural diagram of a deep learning-based image soft tissue detection and analysis device. Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] In one exemplary embodiment, such as Figure 1 As shown, a deep learning-based image soft tissue detection and analysis method is provided, including: Step 100: Obtain the facial image data to be detected.

[0021] Step 200: Classify the facial image data to be detected based on the first image classification neural network to obtain classification results. The classification results include frontal and side views; the first image classification neural network is trained with the encoder in the autoencoder network as the skeleton and based on a multilayer perceptron, using the training dataset and aiming to minimize the cross-entropy loss function; the training dataset includes facial image data with known classification results and precise key point locations.

[0022] The expression for the cross-entropy loss function is: .

[0023] in, The cross-entropy loss function; This represents the probability that the image in the classification results is a frontal view. and All are true classifications of the images. When the image is a frontal view... =1, otherwise The value is 0 when the image is a side view. =1, otherwise =0; This represents the probability that the image in the classification results is a side view.

[0024] Step 300: Based on the classification results, the corresponding second keypoint analysis neural network is invoked. Then, based on this network, a rough prediction of the preset keypoint positions is performed on the facial image data to be detected, yielding the rough keypoint position prediction results. The second keypoint analysis neural network uses the encoder in an autoencoder network as its skeleton and is based on a multilayer perceptron, trained using a training dataset with the goal of minimizing the loss function.

[0025] The expression for the loss function is: .

[0026] in, The loss function; The weight of L1 loss in the loss function; For the first The weight of the loss at each point relative to the total loss at all points; The maximum threshold range for L1 loss; For serial numbers; The first in the rough location prediction results of key points The coordinates of the points; For the first The actual coordinates of each point; For L1 regularization; The weight of L2 loss in the loss function; This is for L2 regularization.

[0027] Step 400: Input the facial image data to be detected and the rough position prediction results of key points into the third key point analysis neural network to obtain the precise position prediction results of key points. The third key point analysis neural network is obtained by using the encoder and decoder in the autoencoder network as the skeleton, performing iterative processing of key point position drift, and performing comparison and regression training based on the training dataset.

[0028] Perform keypoint position drift iteration processing, specifically including: ; .

[0029] The final output of the iterative process is expressed as follows: .

[0030] in, After the first iteration of the neural network, which analyzes the third key point, the... The coordinates to which the point drifts; Analyze the neural network for the third key point; Analyze the network parameters of the neural network for the third key point; The second key point is analyzed by the neural network prediction of the first key point. The coordinates of the points; The facial image data to be detected; For the analysis of the neural network after the third key point After the nth iteration, the th The coordinates to which the point drifts; For the analysis of the neural network after the third key point After the nth iteration, the th The coordinates to which the point drifts; The coordinates of all points output after the neural network iteration is completed for the third key point analysis.

[0031] The third key point in analyzing neural networks is that the expression for the loss function corresponding to comparison and regression training based on the training dataset is: .

[0032] ; .

[0033] in, For the first Loss function after wheel drift; This is the weighted sum of the loss functions after multiple rounds of drifting; For the first The weight of the loss at each point relative to the total loss at all points; The weight of L1 loss in the loss function; The maximum threshold range for L1 loss; The third key point is analyzed by the neural network prediction of the first key point. The coordinates of the points; For the first The actual coordinates of each point; for The weight of the loss in the loss function; For L1 regularization; For L2 regularization; This represents the total number of iterations. For the first The weight of the loss in each iteration in the loss function; As a weighting factor; For serial numbers.

[0034] Step 500: Perform soft tissue detection and analysis based on the predicted precise locations of key points to obtain facial soft tissue analysis results. Soft tissue detection and analysis includes point, line spacing, and angle analysis.

[0035] As an optional implementation, within the second keypoint analysis neural network, the output of the multilayer perceptron is a one-dimensional vector: .

[0036] in, The number of key points in a frontal photograph; The number of key points in a side view photograph; Analyze the neural network for the second key point; In order to target the Zhang image, second key point analysis, one-dimensional vector output by neural network; Analyze the network parameters of the neural network for the second key point; For the first Zhang Image; Dimensions The real vector space.

[0037] Split a one-dimensional vector into , 4-dimensional vectors, respectively corresponding to Direction coordinate vector , Direction coordinate vector Linear transformation matrix .

[0038] Based on template coordinate parameters The formula for generating a rough prediction of key point locations is as follows: .

[0039] in, The coordinates predicted by the neural network are analyzed for the second key point; It is a 2×2 identity matrix.

[0040] Human face and facial landmark detection is a common topic in computer vision. Products and technologies with good results and wide application include RetinaFace and OpenFace.

[0041] Its core is a multi-task learning framework capable of simultaneously performing three tasks: face detection, 5-point landmark localization, and pixel-level face segmentation. It employs a Feature Pyramid Network (FPN) to effectively detect faces of varying sizes through multi-scale feature fusion. Based on improvements to the RetinaNet model, it is specifically optimized for face detection by designing dense regression branches to refine face bounding boxes.

[0042] OPENFACE is an open-source facial behavior analysis toolkit developed by Tadas Baltrusaitis et al. It was the first open-source tool capable of simultaneously performing facial landmark detection, head pose estimation, facial action unit recognition, and eye expression estimation. OPENFACE implemented state-of-the-art facial behavior analysis algorithms at the time, providing a complete framework for building interactive applications based on facial behavior analysis. The core functionality of this toolkit includes facial landmark detection, which accurately locates facial key points, providing a foundation for subsequent analysis tasks. Due to its open-source nature and comprehensive functionality, OPENFACE has been widely adopted in academia and industry.

[0043] However, the main shortcomings of the aforementioned products and technologies are: While commonly used facial landmark detection algorithms such as OPENFACE and RetinaFace perform well in general image processing, these algorithms are primarily designed for common scenarios such as everyday photos or video surveillance, and do not fully consider the specific needs of medical image analysis. Medical soft tissue analysis requires extremely high localization accuracy and anatomical plausibility. The landmarks to be identified are often located in delicate anatomical structures, such as the nasolabial folds and mentolabial folds, and general-purpose algorithms often lack sufficient accuracy at these medical landmarks, failing to meet the millimeter-level accuracy requirements of clinical diagnosis.

[0044] Currently, the analysis process for frontal and lateral photographs is not effectively integrated. Most systems treat them as completely independent tasks, lacking a unified analysis framework and data flow mechanism. This results in the inability to fully utilize the spatial geometric relationships and anatomical correspondences between frontal and lateral photographs. For example, there is a three-dimensional spatial correspondence between the highest point of the cheekbone in the frontal photograph and the protruding point of the cheekbone in the lateral photograph. This relationship has not been effectively modeled and utilized in current technology, making the analysis results lack integrity and coordination, and failing to provide doctors with a comprehensive three-dimensional soft tissue assessment.

[0045] Current algorithms generally perform poorly in recognizing side views because the anatomical structure distribution in side views differs fundamentally from that in frontal views. Many landmarks that are clearly visible in frontal views are either obscured or have completely different shapes in side views. For example, recognizing key anatomical landmarks such as the tip of the nose and the labial protuberance in side views is much more difficult than recognizing points such as the inner canthus of the eye and the corner of the mouth in frontal views. Current algorithms typically use a single model architecture to process all types of images and have not been specifically optimized for the unique anatomical feature distribution and spatial relationships of key points in side views.

[0046] The flowchart of the operation of the deep learning-based image soft tissue detection and analysis method in practical applications is as follows: Figure 2 As shown, this application performs classification operations on frontal and side-view images to determine the image category; based on the image category, it calls the corresponding category model to perform preliminary point localization and bounding operations to obtain the approximate location of soft tissue feature points; for each soft tissue feature point's approximate location, it performs feature point location optimization operations to obtain the precise location of each soft tissue feature point. Therefore, this application can improve the accuracy and efficiency of frontal and side-view image analysis. That is, it helps improve the resolution accuracy of facial soft tissue images, thereby improving the accuracy and efficiency of facial aesthetic analysis.

[0047] The training process corresponding to this application includes: Construct a frontal and side view image dataset, i.e., a training dataset. This dataset includes several frontal and side view images, as well as category information for each image and manually annotated keypoint locations (precise keypoint locations).

[0048] For a dataset of frontal and side-view images, the images in the dataset are sequentially input into an autoencoder network, which includes an encoder and a decoder. The autoencoder network can be based on a convolutional neural network model, a self-attention neural network model, etc., and its computation can be expressed as: .

[0049] in, This is the output of the autoencoder network; This refers to the decoder portion of an autoencoder network. This refers to the encoder portion of an autoencoder network. The input to the autoencoder network is the image from the dataset; where, and The dimensions are equal, both being x×y×c.

[0050] The purpose of comparing the output of the autoencoder with the input image is to enable the autoencoder to learn an efficient, low-dimensional representation of the data in an unsupervised or self-supervised manner, while preserving as much key information as possible from the original data.

[0051] For the frontal and side profile image dataset, the dataset is input into a first image classification neural network, which can be based on a convolutional neural network model, a self-attention neural network model, etc. The image classification neural network uses the encoder in the autoencoder network as the skeleton and inputs the encoder's output into a multilayer perceptron. The first image classification network outputs a binary classification result, which is compared with the true category. The cross-entropy loss function is used to calculate the correct classification of the image.

[0052] .

[0053] For the frontal and side profile dataset, the dataset is input into a second keypoint analysis neural network, which can be based on a convolutional neural network model, a self-attention neural network model, etc.; the encoder in the autoencoder network is used as the skeleton, and the output of the encoder is input into a multilayer perceptron; the second keypoint analysis neural network is denoted as model2, and the output of the multilayer perceptron is a one-dimensional vector: .

[0054] The output one-dimensional vector of the neural network for the second key point analysis is split into two groups: (M+N), (M+N), and a 4-dimensional vector, corresponding to the x-axis coordinate vectors, respectively. y-direction coordinate vector Linear transformation matrix Based on learnable template coordinate parameters Generate the approximate locations of key points for predicted front and side views (size (M+N)×2), calculated using the following formula: .

[0055] Based on the category of the front and side profile photos, the key points of the corresponding category are compared with manually labeled key points, and a regression task is performed for training. The loss function in the training is: .

[0056] It is a weighted sum of L1 mean absolute error and L2 mean square error, thereby achieving the purpose of roughly predicting the location of key points in two types of images.

[0057] For a frontal and side profile dataset, the dataset, along with the coarse keypoint positions calculated by the second keypoint analysis neural network, is input into a third keypoint analysis neural network. This third network can be based on a convolutional neural network model, a self-attention neural network model, etc. Using the encoder and decoder in an autoencoder network as the backbone, the output of each layer in the decoder of the second keypoint analysis neural network is extracted, scaled to the same size, and stacked channel by channel to form an encoded feature map. Based on the coarse keypoint positions, features corresponding to those positions are extracted from the encoded feature map to form a coarse keypoint position image feature sequence. This sequence is then fused with the coordinate information of the coarse keypoint positions and input into a keypoint position drift network. This keypoint position drift network can be an autoencoder network. The output of the keypoint position drift network is added to the original coarse keypoint positions to obtain the drifted keypoint positions. The third keypoint analysis neural network is denoted as model3. The keypoint position drift can be described as an iterative process, expressed as: .

[0058] In each iteration, the input is the output of the previous iteration. The final output can be represented as: .

[0059] The location is compared with the manually labeled key points, and a regression task is performed for training. The loss function during training is: .

[0060] .

[0061] This achieves the goal of accurately predicting the location of key points in two types of images.

[0062] The image to be detected (facial image data) is input into a first image classification neural network for classification to obtain the category of the image. For each image, based on its classification result, a second keypoint analysis neural network corresponding to that category is invoked to obtain the approximate location of all keypoints in the image (predicted approximate keypoint location). For each image, based on its classification result, a third keypoint analysis neural network is invoked to obtain the precise location of all keypoints in the image (predicted precise keypoint location). After obtaining the precise locations, point-to-point, line-segment, and angle measurements are performed on the frontal image. Through point, line, and angle analysis, soft tissue analysis is performed on the frontal image, including but not limited to the three facial proportions (large and small), five eyes, golden ratio, and aspect ratio.

[0063] After obtaining the precise location, the lateral photograph is precisely pinpointed to obtain points, lines, angles, and line spacing. Preset parameters include: nasofrontal angle (nasofrontal angle, nasofacial angle, nasal tip angle, nasolabial angle); labiomental angle (upper lip concave angle, upper lip mentitis angle, upper lip inclination angle, lower lip mentitis angle, lower lip inclination angle, upper and lower lip angles, mentolabial sulcus angle, mentoneck angle); facial shape (overall protrusion angle, facial protrusion angle, upper triangle, lower triangle, G-Trg-Prn, Prn-Trg-Gn', facial angle Peck, facial angle Stoner, Z angle, T angle); and facial proportions (mid-face height / morphological face height, mid-face height / lower face height, upper lip height / lower face height, Sn-Gn' / C-Gn'). Soft tissue analysis of the lateral photograph is then performed based on these points, lines, angles, and line spacing.

[0064] A comprehensive facial soft tissue analysis report will be obtained after obtaining the above-mentioned frontal and side soft tissue analyses.

[0065] The benefits of this application are: To address the adaptability issues in medical scenarios, a network architecture was specifically designed for the needs of medical soft tissue analysis, overcoming the shortcomings of general models in terms of accuracy, robustness, and anatomical rationality.

[0066] Optimize the differentiated processing of front and side views: Adopt a strategy of classification first and then analysis, and call dedicated models for the characteristics of images from different perspectives to avoid positioning errors and missed detections caused by a single model processing multiple perspective images.

[0067] By employing a three-stage processing flow of initial location determination, range definition, and precise optimization, positioning accuracy is improved while ensuring efficiency.

[0068] In one specific embodiment, the dataset construction includes: collecting 1000 clinical facial frontal and lateral images (500 frontal and 500 lateral images), and having three senior orthodontists jointly annotate the soft tissue key points in each image: 32 key points are annotated for the frontal images (such as the inner canthus, outer canthus, nasal ala, corner of the mouth, and midpoint of the chin), and 28 key points are annotated for the lateral images (such as the nasal root point, nasal tip point, upper lip protrusion point, lower lip protrusion point, and prechin point). Each image is also labeled with its category label (frontal or lateral). In this specific embodiment, the key points to be included in the frontal and lateral images are shown in Table 1.

[0069] Table 1. Key data points to be included in frontal and side views.

[0070] In this specific embodiment, the autoencoder pre-training includes: constructing a U-Net structured autoencoder network, where the encoder part uses ResNet-34 as the backbone network. All 1000 images are input into the autoencoder, and by minimizing the L2 loss between the input and reconstructed images, the encoder learns discriminative low-dimensional feature representations. For an image with dimensions X and Y and three color channels, the autoencoder outputs an image with dimensions X and Y and three color channels. .

[0071] The formula for calculating the L2 loss is: .

[0072] in, This represents the value of the c-th channel in the pixel at row x and column y of the image output by the autoencoder; This represents the value of the c-th channel in the pixel at row x and column y of the input image.

[0073] After training is complete, the encoder parameters in the autoencoder are fixed for subsequent tasks.

[0074] In this specific embodiment, the training of the first image classification neural network includes: using a pre-trained encoder as the feature extraction backbone, its output feature map is input into a two-layer MLP (Multilayer Perceptron) after global average pooling, and the output is a frontal / lateral binary classification probability, represented by a two-dimensional vector: ;in p f , p l Let represent the probability that the image is frontal and the probability that the image is sideways, respectively. The cross-entropy loss function is used, and the loss function formula is: When the image is correctly classified as a frontal view, y f =1,y l =0; when the image is correctly classified as a side view, y f =0,y l =1. Randomly select 900 images for training and 100 images for validation.

[0075] In this specific embodiment, the second key point analysis neural network training (coarse localization) includes: using a pre-trained encoder as the backbone, connecting it to a fully connected neural network, outputting a vector with a dimension of (32+28)×2 + 4 = 124. The computation of this neural network can be represented as ;img i For the i-th image, θ m2 The weights of the neural network model are analyzed for the second key point. This vector is decomposed into: : Indicates offset in the x-direction; : Indicates offset in the y-direction; : Represents the affine transformation matrix (linear transformation matrix). The second key point analysis shows that the neural network contains a learnable template coordinate system, represented as... The formula for roughly calculating the location of key points is as follows: .

[0076] Coordinate correction is performed to generate approximate key point locations. Among these, This is a 60×2 matrix representing the predicted locations of 60 coarse keypoints. During training, the loss function is calculated by taking only 32 or 28 points corresponding to the image category (front / side view) and comparing them with the ground truth annotations.

[0077] In calculating the loss function, this specific embodiment adopts the loss function of this application that combines long-distance keypoint drift and local optimization: .

[0078] E i This indicates whether the i-th point exists in the current image (front view or side view). , These represent L1 and L2 regularization, respectively.

[0079] After introducing Tikhonov regularization L2, the optimization objective of the second keypoint analysis neural network is: .

[0080] N is the total number of images used in the training process.

[0081] This mechanism enables global deformation (through a linear transformation matrix) and local deformation (through coordinate offset) of the template, adaptively adjusting the standard template to the approximate position of the current image, providing an initial estimate for subsequent precise localization.

[0082] In this specific embodiment, the training (fine localization) of the third keypoint analysis neural network includes: constructing a feature extractor based on U-Net decoder enhancement; extracting feature maps from each layer of the U-Net decoder of the second keypoint analysis neural network; aggregating features from different regions using the self-attention mechanism of a deformable encoder-decoder to form sparse encoded features; and calculating the coordinate correction amount for each coarse keypoint position in the coefficient encoded feature map using the self-attention mechanism.

[0083] Deformable encoder-decoder networks are based on encoder-decoder architectures with deformable attention mechanisms and have the following key characteristics: Image feature extraction: First, multi-scale feature maps are extracted using a convolutional neural network (such as ResNet) backbone.

[0084] Deformable attention mechanism: Unlike standard attention mechanism encoders that focus on all pixels, it samples only a small number (e.g., 4) of reference points around each query; and the positions of these sampling points are not fixed grids, but offsets dynamically predicted by the network, thus focusing on the target-related region.

[0085] Treating keypoint drift as an object query: The drift process from the coarse position of a keypoint to its precise position is regarded as a learnable query vector Q, the number of Q equals the number of keypoints to be predicted, and the feature sequence output by the encoder-decoder of the deformable attention mechanism is regarded as key values ​​K and V, and the output is the number of keypoint drift values ​​equal to the number of Q.

[0086] Backbone network feature extraction: Assuming ResNet is used as the backbone network, it outputs four feature pyramids of different sizes, denoted as L1 to L4.

[0087] Position enhancement fusion: concatenates L1~L4 visual features with the position embedding of key point coordinates to provide spatial position information for the encoder-decoder network of deformable attention mechanism.

[0088] Query initialization: for each coarse point (x) i ,y i Initialize a keypoint query qi∈Rd, whose position is encoded as (x i ,y i Centered on ).

[0089] Constrained Deformable Sampling: In standard Deformable DETR, each query predicts a sampling offset across the entire feature map. This application improves upon this by restricting the sampling range—allowing offset prediction only within a local region centered at (xi,yi) with a radius of r (e.g., 20 pixels). Implementation: The predicted offset Δpi is constrained to the range [−r,r] by applying tanh activation and scaling.

[0090] Deformable attention feature aggregation: for each query q i Within its limited neighborhood, N (e.g., 4) points are sampled, and features are obtained through bilinear interpolation; after weighted aggregation, q is updated. i .

[0091] Position drift optimization: q for each query i Coordinate correction values ​​were obtained through MLP regression. Final precise positioning point: .

[0092] Iterative convergence: The input image is The approximate location obtained through the second key point analysis neural network model is: The iterative process can be written as: .

[0093] The output offset is added to the original coarse position to obtain the optimized position; this process can be repeated multiple times until the position converges to a stable state, obtaining the final precise position. This process can be represented as: .

[0094] In this specific embodiment, the end-to-end training of the third key point analysis neural network training (fine localization) stage, its loss function can be expressed as: .

[0095] .

[0096] Where T is the number of iterations for convergence, and FineLoss (the loss function for training the neural network in the third keypoint analysis) is the weighted sum of the loss functions for each iteration.

[0097] Therefore, after introducing Tikhonov regularization L2, the optimization objective of the third keypoint analysis neural network is: .

[0098] N is the total number of images used in the training process.

[0099] Input a face photo to be tested: First, preprocess the photo by normalizing the pixel values ​​to between 0 and 1, and convert it into a two-dimensional image containing only grayscale information.

[0100] Scale the image to 224 A 22-pixel image is input into a first image classification neural network, which outputs a side profile view. Then, coarse localization is performed: a second keypoint analysis neural network corresponding to the side profile view is called, outputting the approximate positions of 28 keypoints, for example: the tip of the nose is located at (215, 180), and the anterior chin is located at (220, 310) (in pixels). Then, fine localization is performed: the original image and the approximate points are input into a third keypoint analysis neural network to obtain optimized, precise positions, such as the tip of the nose corrected to (216.3, 181.7), and the anterior chin corrected to (219.8, 309.4).

[0101] Soft tissue analysis and report generation includes: After outputting precise key point positioning, the system automatically performs measurements based on preset measurement items. Preset measurement items for frontal photos include: the three main facial sections, the three secondary facial sections, the five facial features, the golden ratio, and the length-to-width ratio. Preset measurement items for side photos include: nasofrontal angle (nasofrontal angle, nasofacial angle, nasal tip angle, nasolabial angle), labiomental angle (upper lip concave angle, upper lip-mental protrusion angle, upper lip inclination angle, lower lip-mental protrusion angle, lower lip inclination angle, upper and lower lip angles, mentolabial sulcus angle, and chin-neck angle), facial shape (overall protrusion angle, facial protrusion angle, upper triangle, lower triangle, G-Trg-Prn, Prn-Trg-Gn', facial angle Peck, facial angle Stoner, Z angle, T angle), and facial proportions (midface height / morphological face height, midface height / lower face height, upper lip height / lower face height, Sn-Gn' / C-Gn). The user interface allows for fine-tuning and adding of points, lines, and angles using feature point adjustment and addition, as well as measurement tools, to achieve measurements that ultimately meet clinical requirements and generate a final report. The system integrates these measurement results with clinical standards (such as the normal range of Z-angle being 70°–75°) to output a structured analysis report. Figure 3 This is an example of the user interface for positioning, measuring, and analyzing soft tissue in frontal radiographs. Figure 4 This is an example of the user interface for positioning, measuring, and analyzing soft tissue in lateral radiographs.

[0102] In one exemplary embodiment, such as Figure 5 As shown, a deep learning-based image soft tissue detection and analysis device is provided, comprising: The data acquisition module is used to acquire facial image data to be detected.

[0103] The classification processing module is used to classify the facial image data to be detected based on the first image classification neural network to obtain classification results; the classification results include frontal and side views; the first image classification neural network is trained with the encoder in the autoencoder network as the skeleton and based on the multilayer perceptron, according to the training dataset, with the goal of minimizing the cross-entropy loss function; the training dataset includes facial image data with known classification results and precise key point locations.

[0104] The coarse location prediction module is used to call the corresponding second key point analysis neural network based on the classification results, and to perform coarse location prediction of preset key points on the facial image data to be detected based on the second key point analysis neural network, so as to obtain the coarse location prediction result of the key points; the second key point analysis neural network is trained with the encoder in the autoencoder network as the skeleton and based on the multilayer perceptron, according to the training dataset, with the goal of minimizing the loss function.

[0105] The precise location prediction module is used to input the facial image data to be detected and the coarse location prediction results of the key points into the third key point analysis neural network to obtain the precise location prediction results of the key points. The third key point analysis neural network is obtained by using the encoder and decoder in the autoencoder network as the skeleton, performing key point location drift iteration processing, and performing comparison and regression training based on the training dataset.

[0106] The detection and analysis module is used to perform soft tissue detection and analysis based on the predicted precise location of the key points to obtain facial soft tissue analysis results; the soft tissue detection and analysis includes point, line distance, and angle analysis.

[0107] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores image soft tissue detection and analysis data based on deep learning. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a deep learning-based image soft tissue detection and analysis method.

[0108] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0109] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0110] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0111] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0113] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0114] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logic devices, etc., and are not limited to these.

[0115] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting and analyzing soft tissue in images based on deep learning, characterized in that, include: Acquire facial image data to be detected; The first image classification neural network classifies the facial image data to be detected to obtain classification results; the classification results include frontal and side views; the first image classification neural network is trained with the encoder in the autoencoder network as the skeleton and based on a multilayer perceptron, according to the training dataset, with the goal of minimizing the cross-entropy loss function; the training dataset includes facial image data with known classification results and precise key point locations. Based on the classification results, the corresponding second key point analysis neural network is invoked, and based on the second key point analysis neural network, the rough position prediction of the preset key points is performed on the facial image data to be detected, so as to obtain the rough position prediction result of the key points. The second key point is that the neural network is based on the encoder in the autoencoder network as the skeleton and is trained on a multilayer perceptron with the goal of minimizing the loss function based on the training dataset. The facial image data to be detected and the rough position prediction results of the key points are input into the third key point analysis neural network to obtain the precise position prediction results of the key points. The third keypoint analysis neural network is obtained by using the encoder and decoder in the autoencoder network as the skeleton, performing keypoint position drift iteration processing, and comparing and regressing training based on the training dataset; Based on the predicted precise location of the key points, soft tissue detection and analysis are performed to obtain facial soft tissue analysis results; The soft tissue detection and analysis includes point, line spacing, and angle analysis.

2. The image soft tissue detection and analysis method based on deep learning according to claim 1, characterized in that, The expression corresponding to the cross-entropy loss function is: ； in, The cross-entropy loss function; This represents the probability that the image in the classification results is a frontal view. and All are true classifications of the images. When the image is a frontal view... =1, otherwise The value is 0 when the image is a side view. =1, otherwise =0; This represents the probability that the image in the classification results is a side view.

3. The image soft tissue detection and analysis method based on deep learning according to claim 1, characterized in that, The expression corresponding to the loss function is: ； in, The loss function; The weight of L1 loss in the loss function; For the first The weight of the loss at each point relative to the total loss at all points; The maximum threshold range for L1 loss; For serial numbers; The first in the rough location prediction results of key points The coordinates of the points; For the first The actual coordinates of each point; For L1 regularization; The weight of L2 loss in the loss function; This is for L2 regularization.

4. The image soft tissue detection and analysis method based on deep learning according to claim 1, characterized in that, Perform keypoint position drift iteration processing, specifically including: ； ； The final output of the iterative process is expressed as follows: ； in, After the first iteration of the neural network, which analyzes the third key point, the... The coordinates to which the point drifts; Analyze the neural network for the third key point; Analyze the network parameters of the neural network for the third key point; The second key point is analyzed by the neural network prediction of the first key point. The coordinates of the points; The facial image data to be detected; For the analysis of the neural network after the third key point After the nth iteration, the th The coordinates to which the point drifts; For the analysis of the neural network after the third key point After the nth iteration, the th The coordinates to which the point drifts; The coordinates of all points output after the neural network iteration is completed for the third key point analysis.

5. The image soft tissue detection and analysis method based on deep learning according to claim 1, characterized in that, The third key point in analyzing neural networks is that the expression for the loss function corresponding to comparison and regression training based on the training dataset is: ； ； ； in, For the first Loss function after wheel drift; This is the weighted sum of the loss functions after multiple rounds of drifting; For the first The weight of the loss at each point relative to the total loss at all points; The weight of L1 loss in the loss function; The maximum threshold range for L1 loss; The third key point is analyzed by the neural network prediction of the first key point. The coordinates of the points; For the first The actual coordinates of each point; for The weight of the loss in the loss function; For L1 regularization; For L2 regularization; This represents the total number of iterations. For the first The weight of the loss in each iteration in the loss function; As a weighting factor; For serial numbers.

6. The image soft tissue detection and analysis method based on deep learning according to claim 1, characterized in that, Within the second key point analysis neural network, the output of the multilayer perceptron is a one-dimensional vector: ； in, The number of key points in a frontal photograph; The number of key points in a side view photograph; Analyze the neural network for the second key point; In order to target the Zhang image, second key point analysis, one-dimensional vector output by neural network; Analyze the network parameters of the neural network for the second key point; For the first Zhang Image; For dimension The real vector space; Decompose a one-dimensional vector into , 4-dimensional vectors, respectively corresponding to Direction coordinate vector , Direction coordinate vector Linear transformation matrix ; Based on template coordinate parameters The formula for generating a rough prediction of key point locations is as follows: ； in, The coordinates predicted by the neural network are analyzed for the second key point; It is a 2×2 identity matrix.

7. A deep learning-based image soft tissue detection and analysis device, characterized in that, include: The data acquisition module is used to acquire facial image data to be detected; The classification processing module is used to classify the facial image data to be detected based on the first image classification neural network to obtain the classification results; the classification results include frontal and side views; the first image classification neural network is trained with the encoder in the autoencoder network as the skeleton and based on the multilayer perceptron, according to the training dataset, with the goal of minimizing the cross-entropy loss function; the training dataset includes facial image data with known classification results and precise key point locations. The coarse location prediction module is used to call the corresponding second key point analysis neural network based on the classification results, and to perform coarse location prediction of preset key points on the facial image data to be detected based on the second key point analysis neural network, so as to obtain the coarse location prediction result of key points. The second key point is that the neural network is based on the encoder in the autoencoder network as the skeleton and is trained on a multilayer perceptron with the goal of minimizing the loss function based on the training dataset. The precise location prediction module is used to input the facial image data to be detected and the coarse location prediction results of the key points into the third key point analysis neural network to obtain the precise location prediction results of the key points. The third keypoint analysis neural network is obtained by using the encoder and decoder in the autoencoder network as the skeleton, performing keypoint position drift iteration processing, and comparing and regressing training based on the training dataset; The detection and analysis module is used to perform soft tissue detection and analysis based on the predicted precise location of the key points to obtain facial soft tissue analysis results. The soft tissue detection and analysis includes point, line spacing, and angle analysis.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the deep learning-based image soft tissue detection and analysis method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the deep learning-based image soft tissue detection and analysis method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the deep learning-based image soft tissue detection and analysis method as described in any one of claims 1-6.

Images