An orthodontic diagnosis report automatic generation method based on image text alignment

Abstract

The application discloses an orthodontic diagnosis report automatic generation method based on image and text alignment. First, the orthodontic lateral film is pretreated, and the image features are extracted by using Gaussian convolution. At the same time, the state space model is used to model and extract features of the orthodontic report. Then, the image and text features are aligned through the cross attention mechanism, so that the image feature extraction module can more accurately identify the semantic information related to the image, thereby closely associating the image features with the corresponding language description. Finally, the extracted orthodontic knowledge features are input into the text decoder to generate high-quality orthodontic diagnosis reports. The application significantly improves the accuracy and efficiency of the diagnosis report generation, and solves the problems of low quality and low efficiency in the traditional manual diagnosis process.

Description

Technical Field

[0001] The present application belongs to the field of medical image processing technology, and in particular relates to a method for automatically generating an oral orthodontic diagnosis report based on image-text alignment. Background Art

[0002] Orthodontics is a key branch of dentistry, primarily involving the study and correction of teeth, jaws, and facial structures. The core purpose of orthodontic diagnosis is to provide accurate diagnostic reports by analyzing the patient's oral images and clinical data to guide subsequent treatment plans. These reports typically require a detailed analysis of images such as orthodontic lateral radiographs by experienced professionals, combined with extensive clinical medical knowledge. The traditional manual diagnostic process is time-consuming and labor-intensive, and is susceptible to human influence, limiting the accuracy and consistency of diagnostic results.

[0003] Furthermore, orthodontic diagnosis requires a comprehensive consideration of multiple factors, including clinical examination and medical history, to develop a personalized treatment plan. This process requires extensive clinical experience and a deep medical knowledge base. As patients' expectations for treatment effectiveness and comfort increase, the demand for more accurate and personalized orthodontic diagnostic reports is also increasing.

[0004] In recent years, the rapid development of artificial intelligence (AI) and deep learning technologies has brought new possibilities for automated orthodontic diagnostic report generation. Convolutional Neural Networks (CNNs) within deep learning have demonstrated powerful capabilities in image feature extraction and analysis, making medical image analysis possible through image recognition. Simultaneously, advances in natural language processing (NLP) technology for text generation and medical knowledge extraction have made it possible to extract useful information from large volumes of medical literature and reports and generate structured text content.

[0005] While some research has attempted to apply deep learning to orthodontic image analysis and report generation, these techniques primarily focus on overall image classification and simple text generation, lacking the deep integration of image and text information. Existing methods have limitations in image feature extraction and medical knowledge alignment, making it difficult to achieve highly accurate and personalized diagnostic reports. Furthermore, the alignment and integration of image and text features remains a challenge, with existing solutions focusing on this area. Summary of the Invention

[0006] The purpose of this application is to provide a method for automatically generating orthodontic diagnostic reports based on image-text alignment, aiming to overcome the shortcomings of the existing technology. On the basis of improving the existing technology, by aligning and fusing multimodal features, the efficient and accurate generation of diagnostic reports is further achieved. This not only solves the time-consuming and labor-intensive problems in the traditional manual diagnosis process, but also effectively improves the generation quality of orthodontic diagnostic reports, solving the problems of low report quality, lack of personalization and accuracy in the existing technology.

[0007] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0008] A method for automatically generating an orthodontic diagnosis report based on image-text alignment, comprising:

[0009] Preprocess the orthodontic lateral radiograph image to obtain a multi-scale Gaussian pyramid , input the image feature extraction module to obtain image features ;

[0010] Input the orthodontic report into the medical knowledge extraction module to obtain the orthodontic knowledge set ;

[0011] Orthodontic knowledge collection Image characteristics of orthodontic lateral radiographs Input the medical knowledge alignment module together, perform cross-attention operations, obtain features from orthodontic knowledge to image and image to orthodontic knowledge, calculate the loss value, and use backpropagation to update the network parameters of the image feature extraction module to obtain a trained image feature extraction module;

[0012] The user's oral orthodontic lateral radiograph image is input into the trained image feature extraction module to obtain image features, and the image features are input into the text decoder to generate a diagnosis report.

[0013] Furthermore, the orthodontic lateral radiograph image is preprocessed to obtain a multi-scale Gaussian pyramid ,include:

[0014] According to the prior distribution of the features of the orthodontic lateral radiograph image, random initial coordinates of the feature points of the orthodontic lateral radiograph image that obey the normal distribution are obtained;

[0015] Gaussian convolution is used on the orthodontic lateral radiograph to obtain a Gaussian blurred image. Regions of different sizes are cropped from the image with the initial coordinates as the center to obtain a set of multi-scale Gaussian pyramids. ;

[0016] The input image feature extraction module obtains image features ,include:

[0017] Gaussian pyramid set The images of each layer are downsampled to the same size;

[0018] The Gaussian pyramid set after downsampling Input to the convolutional encoder to get the features ;

[0019] The features Input to the convolutional embedding layer and reduce the dimension to obtain features ;

[0020] The features Input to the graph attention module to obtain features ;

[0021] The features Input into the multi-layer perceptron MLP for regression to obtain the relative coordinate offset, update the coordinates of the image feature points, crop areas of different sizes in the image with the updated coordinates as the center, obtain a new set of multi-scale Gaussian pyramids, and then enter the image feature extraction module for iteration until the preset number of iterations is reached to obtain the image features. .

[0022] Furthermore, the Gaussian pyramid set after downsampling Input to the convolutional encoder to get the features ,include:

[0023] The downsampled Gaussian pyramid set Input into the convolutional layer in the convolutional encoder to obtain the feature map ;

[0024] The feature map Perform maximum pooling and then input it into the first convolution block in the convolution encoder to obtain the feature map ;

[0025] The feature map is sequentially passed through the convolution blocks of each layer of the convolution encoder. Process and generate subsequent feature maps to , the feature map dimension changes with the processing of each convolution block, and finally we get ;

[0026] The feature map Perform average pooling to obtain the feature vector .

[0027] Furthermore, the feature Input into the convolutional embedding layer for embedding operation and dimensionality reduction to obtain features ,include:

[0028] In Features Perform an expansion operation on the second dimension to obtain a vector ;

[0029] The vector Input to the convolutional embedding layer, use a one-dimensional convolution layer to Perform convolution operation to obtain the feature vector after dimensionality reduction .

[0030] Furthermore, the feature Input to the graph attention module to obtain features ,include:

[0031] The features Input three linear layers Q, K, V to get the vector 、 、 ;

[0032] The vector With the transposed Multiply to get the attention matrix ;

[0033] The attention matrix The input is sent to the batch normalization layer for normalization, and then the sigmoid function is used for activation and normalization to obtain the attention heat map. ;

[0034] Attention heat map and vector Perform point multiplication to obtain the attention features between each feature point ;

[0035] Attention Features Input the linear layer R to get the attention feature vector, and then add the attention feature vector and feature Perform residual connection operation to obtain features .

[0036] Furthermore, the orthodontic report is input into the medical knowledge extraction module to obtain the orthodontic knowledge set ,include:

[0037] Input the orthodontic report R into the embedding layer to obtain the word embedding vector ;

[0038] Embedding words into vectors Input to the linear mapping layer to get vector ;

[0039] The vector Input the convolution layer to obtain a higher-level vocabulary feature vector ;

[0040] The feature vector The vectors in the state space module are input one by one. For each input vector , select the state space module to output the corresponding vector , all output vectors Splice and get the feature vector ;

[0041] The feature vector With vector Multiply element by element to get the eigenvector With vector Hadamard ;

[0042] Hadamard Input to the linear mapping layer, map it to a low dimension, and obtain the orthodontic knowledge set .

[0043] Furthermore, the vector Input the selected state space module and get the feature vector ,include:

[0044] The vector Input to the linear layer of the selected state-space module , get vector B, and transform vector Input to the linear layer of the state-space module , get vector C;

[0045] The vector Input to the linear layer of the state-space model , and get the vector ;

[0046] The vector and select the predefined learnable parameter vector in the state space module Add and input to SoftPlus activation function to get the feature ;

[0047] The predefined learnable parameters Discretization to obtain parameters , the calculation formula is as follows, where is the identity matrix:

[0048]

[0049] Discretize vector B to get , the calculation formula is as follows, where is the identity matrix:

[0050]

[0051] Combine the vector C with the predefined hidden state matrix in the selected state space module Perform matrix multiplication to obtain the eigenvector , the hidden state matrix Used to store the hidden state of all input vectors and update the hidden state matrix , the calculation formula is:

[0052] .

[0053] Furthermore, the oral orthodontic knowledge set Image characteristics of orthodontic lateral radiographs The medical knowledge alignment module is input together, and cross-attention operations are performed to obtain features from orthodontic knowledge to image and from image to orthodontic knowledge. The loss value is calculated, and the network parameters of the image feature extraction module are updated using backpropagation to obtain a trained image feature extraction module, which includes:

[0054] Orthodontic knowledge collection Input the average pooling layer to obtain the global features of orthodontic knowledge ;

[0055] The image features Input the average pooling layer to obtain the global image features of the orthodontic lateral radiograph ;

[0056] Global Features Stitch together to get features ;

[0057] The features Input linear layer Get features ,Will Input linear layer separately and linear layers Get features and ;

[0058] Will 、 、 As input, we calculate cross-attention and get features. , the calculation formula is as follows, where express Dimensions, express The transpose of :

[0059] ;

[0060] The features Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent orthodontic knowledge to images ;

[0061] High-dimensional features Input linear layer, map it to low dimension, and get features ;

[0062] The global image features Stitch together to get features ;

[0063] The features Input to the linear layer Get features ,Will Input linear layer separately and linear layers Get features and ;

[0064] Will 、 、 As input, we calculate cross-attention and get features. , the calculation formula is as follows, where express Dimensions, express The transpose of :

[0065] ;

[0066] The features Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent image to orthodontic knowledge ;

[0067] High-dimensional features Input to the linear layer, map it to a low dimension, and get the features ;

[0068] in accordance with and Calculate the loss value and use back propagation to update the network parameters of the image feature extraction module to obtain the trained image feature extraction module.

[0069] The present application proposes a method for automatically generating orthodontic diagnostic reports based on image-text alignment. On the one hand, by aligning and fusing the multimodal features of images and texts, the model can better extract and understand the information in orthodontic lateral films, thereby significantly improving the accuracy of the diagnostic report. On the other hand, by introducing the state-space model to model and extract features of text sequences, the expressive power of text information is enhanced. In addition, the use of Gaussian convolution to extract features from images improves the effectiveness of the model in processing image detail information. At the same time, the cross-attention mechanism is used to align text and images, so that the model can more effectively combine multimodal information and enhance the accuracy and consistency of report generation. The present application effectively solves the problems of low quality, low efficiency and reliance on human factors in the traditional manual diagnostic report generation process, and provides an efficient, accurate and consistent solution for orthodontic diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a flow chart of the method for automatically generating orthodontic diagnosis reports based on image-text alignment for this application.

[0071] Figure 2 This is a schematic diagram of the training network model structure for this application.

[0072] Figure 3 Schematic diagram of the convolutional encoder module structure in an embodiment of the present application.

[0073] Figure 4 This is a structural diagram of the medical knowledge extraction module of the embodiment of this application.

[0074] Figure 5 This is a structural diagram of the medical knowledge alignment module of an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0076] One embodiment of the present application, such as Figure 1 As shown, a method for automatically generating an orthodontic diagnosis report based on image-text alignment is provided, comprising:

[0077] Step S1: Preprocess the orthodontic lateral radiograph to obtain a multi-scale Gaussian pyramid , input the image feature extraction module to obtain image features .

[0078] In order to distinguish the existing technology from the problem that the image feature extraction module only extracts isolated visual features, this application introduces a medical knowledge extraction module and a medical knowledge alignment module to participate in the training of the image feature extraction module, so that the trained image feature extraction module can extract features that contain both visual features and semantic information from the input orthodontic lateral radiographs. The network structure during the entire training process can be found in Figure 2 ,After the training is completed, the features extracted by the ,image feature extraction module are input into the text decoder to ,generate a diagnosis report.

[0079] In a specific embodiment, the orthodontic lateral radiograph image is preprocessed to obtain a multi-scale Gaussian pyramid ,include:

[0080] Step 1.1: According to the prior distribution of the features of the orthodontic lateral radiograph image, random initial coordinates of the feature points of the orthodontic lateral radiograph image that obey the normal distribution are obtained.

[0081] An orthodontic lateral radiograph is an image taken from the side of the head using X-rays. It can show the patient's facial bones and tooth arrangement in detail. This application generates an orthodontic diagnostic report based on the orthodontic lateral radiograph.

[0082] This step calculates the prior distribution of the features of the orthodontic lateral radiograph image based on the annotations in the known data set to calculate the random initial coordinates of the feature points of the orthodontic lateral radiograph image that obey the normal distribution. The prior distribution and how to calculate the initial coordinates are relatively mature technologies in this field and will not be elaborated here.

[0083] Step 1.2: Use Gaussian convolution to obtain a Gaussian blurred image on the orthodontic lateral radiograph. Cut out regions of different sizes from the image with the initial coordinates as the center to obtain a set of multi-scale Gaussian pyramids. .

[0084] This step includes indivual A collection of multi-scale Gaussian pyramids .

[0085] For each Gaussian pyramid , and the image dimensions of each layer are ,in is the height of the first layer image, is the width of the first layer image, The corresponding A Gaussian pyramid, The corresponding layer images, is the number of predefined feature points.

[0086] Then, the Gaussian pyramid set Input into the image feature extraction module to obtain image features In a specific embodiment, the image feature extraction module performs the following operations:

[0087] Step 1.3, assemble the Gaussian pyramids The images of each layer are downsampled to the same size.

[0088] For example, after downsampling, The dimension becomes , and are the length and width of the second layer of the Gaussian pyramid. The size of the second layer is chosen as the baseline for downsampling to maintain image information while effectively reducing computational complexity. This layer size provides a good balance between the computational requirements of subsequent processing and the preservation of key image information, ensuring that the image's structural features are well preserved while reducing data volume, thus providing an efficient and reliable foundation for subsequent processing steps.

[0089] Step 1.4: Set the downsampled Gaussian pyramid Input to the convolutional encoder to get the features .

[0090] After the convolutional encoder, its dimension is , The dimension of a single feature vector after convolution encoding.

[0091] Step 1.5: Features Input into the convolutional embedding layer for embedding operation and dimensionality reduction to obtain features .

[0092] feature The dimension is , is the dimension of a single feature vector after the embedding operation.

[0093] Step 1.6: Features Input to the graph attention module to obtain features .

[0094] feature The dimension is .

[0095] Step 1.7: The features Input into the multi-layer perceptron MLP to regress and get the relative coordinate offset, update the coordinates of the image feature points, cut out areas of different sizes in the image with the updated coordinates as the center, get a new set of multi-scale Gaussian pyramids, and then enter the image feature extraction module for iteration until the preset number of iterations is reached to get the image features. .

[0096] For example, the first time through the multi-layer perceptron MLP, the relative coordinate offset is obtained , and according to and Update the coordinates of image feature points , with coordinates Cut out regions of different sizes in the image as the center, and perform multiple iterations to update the final feature point coordinates, that is, to obtain the image features. , whose dimensions are , is the number of feature points, is the feature point coordinate dimension.

[0097] In a specific embodiment, Figure 3 As shown, the Gaussian pyramid set after downsampling Input to the convolutional encoder to get the features ,include:

[0098] Step 2.1, collect the downsampled Gaussian pyramids Input into the convolutional layer in the convolutional encoder to obtain the feature map , whose dimension is ,in Indicates the number of channels.

[0099] Step 2.2: Feature map Perform maximum pooling and then input it into the first convolution block in the convolution encoder to obtain the feature map , whose dimension is .

[0100] Step 2.3, pass through each convolution block of the convolution encoder in turn, and perform the feature map Process and generate subsequent feature maps to , the feature map dimension changes with the processing of each convolution block, and finally we get , feature map The dimension size is .

[0101] Step 2.4: Feature map Perform average pooling to obtain the feature vector , whose dimension is , Indicates that the vector length is equal to the number of channels .

[0102] In a specific embodiment, the feature Input into the convolutional embedding layer for embedding operation and dimensionality reduction to obtain features ,include:

[0103] Step 3.1, in the feature Perform an expansion operation on the second dimension to obtain a vector , the dimension is ,in Indicates that the expansion operation is The medium length is The eigenvectors of Second splicing.

[0104] Step 3.2, vector Input to the convolutional embedding layer, use a one-dimensional convolution layer to Perform convolution operation to obtain the feature vector after dimensionality reduction , the dimension is , is the length of the vector.

[0105] In a specific embodiment, the feature Input to the graph attention module to obtain features ,include:

[0106] Step 4.1, the features Input three linear layers Q, K, V to get the vector 、 、 , whose dimensions are , where the vector length is the output dimension of the linear layer Q, K, V.

[0107] Step 4.2, vector With the transposed Multiply to get the attention matrix , whose dimensions are .

[0108] Step 4.3: Attention matrix The input is sent to the batch normalization layer for normalization, and then the sigmoid function is used for activation and normalization to obtain the attention heat map. , whose dimensions are .

[0109] Step 4.4: Attention heat map and vector Perform point multiplication to obtain the attention features between each feature point , whose dimensions are .

[0110] Step 4.5: Attention features Input linear layer, the dimension is Attention feature vector, and then the attention feature vector and feature Perform residual connection operation to obtain features , whose dimensions are .

[0111] In a specific embodiment, the Input into the multi-layer perceptron MLP for regression to obtain the relative coordinate offset , and based on and Update the coordinates of image feature points , with coordinates As the center, cut out areas of different sizes in the image and perform multiple iterations to get the final feature point coordinates ,include:

[0112] Step 5.1: Transform the feature vector Input to the linear layer 𝑃1 for full connection operation, activated and dropped out by the Relu function, and the feature vector is obtained , whose dimensions are , where the vector length is the number of output nodes of the linear layer 𝑃1.

[0113] Step 5.2: The feature vector Input to the linear layer 𝑃2 for full connection operation, activated and dropped out by the Relu function, and the feature vector is obtained. , whose dimensions are , where the vector length is the number of output nodes of the linear layer 𝑃2.

[0114] Step 5.3: Transform the feature vector Input to the linear layer 𝑃3 for full connection operation, and regress to get the relative offset , whose dimensions are .

[0115] Step 5.4: Update according to the following formula , and Construct a new multi-scale Gaussian pyramid set for the center , as the input of step 1.3, and iterate the result after 10 iterations. is the final position of each feature point, that is, the final feature point coordinates :

[0116]

[0117] Where t represents the current iteration number, ranging from 1 to 10.

[0118] Step S2: Input the orthodontic report into the medical knowledge extraction module to obtain the orthodontic knowledge set .

[0119] In this embodiment, the orthodontic report (R) is input into the medical knowledge extraction module to obtain the orthodontic knowledge set ,like Figure 4 Shown, including:

[0120] Step 6.1: Input the orthodontic report into the word embedding layer to obtain the word embedding vector , whose dimensions are ,in represents the number of word embedding vectors, is the word embedding vector dimension.

[0121] Step 6.2: embed the word into vector Input to the linear mapping layer to get vector , whose dimensions are .

[0122] Step 6.3, vector Input the convolution layer to obtain a higher-level vocabulary feature vector , the dimension is .

[0123] Step 6.4: Transform the feature vector in The state space module is selected by inputting vectors one by one. For each input vector , select the state space module to output the corresponding vector , all output vectors Splice and get the feature vector , the dimension is ,vector and The dimension is , correspond No. vectors and their corresponding state-space module outputs.

[0124] Step 6.5: Transform the feature vector With vector Multiply element by element to get the eigenvector With vector Hadamard , the dimension is .

[0125] Step 6.6, Hadamard product Input to the linear mapping layer, map it to a low dimension, and obtain the orthodontic knowledge set , the dimension is ,in Indicates the amount of knowledge.

[0126] In a specific embodiment, the feature vector Input the selected state space module and get the feature vector ,include:

[0127] Step 7.1, vector Input to the linear layer of the selected state-space module , we get vector B, whose dimension is , the vector Input to the linear layer of the state-space module , we get vector C, whose dimension is .

[0128] Step 7.2, vector Input to the linear layer of the state-space model , and get the vector , whose dimensions are .

[0129] Step 7.3, vector and select the predefined learnable parameter vector in the state space module Add and input to SoftPlus activation function to get the feature , whose dimensions are , The dimension is .

[0130] Step 7.4: Set the predefined learnable parameters Discretization to obtain parameters , whose dimensions are , the calculation formula is as follows, where is the identity matrix:

[0131]

[0132] Step 7.5: Discretize vector B to obtain , whose dimensions are , the calculation formula is as follows, where is the identity matrix:

[0133]

[0134] Step 7.6: Compare the vector C with the predefined hidden state matrix in the selected state space module Perform matrix multiplication to obtain the eigenvector , whose dimensions are , the hidden state matrix Used to store the hidden state of all input vectors, the dimension is .

[0135] Step 7.7, Update the hidden state matrix , the calculation formula is:

[0136] .

[0137] Step S3: Orthodontic knowledge set Image characteristics of orthodontic lateral radiographs The medical knowledge alignment module is input together, cross-attention operation is performed, the features of orthodontic knowledge to image and image to orthodontic knowledge are obtained, the loss value is calculated, and the network parameters of the image feature extraction module are updated using back propagation to obtain the trained image feature extraction module.

[0138] This application describes the collection of oral orthodontic knowledge Image characteristics of orthodontic lateral radiographs Input the medical knowledge alignment module together, perform cross-attention operation, obtain the features of orthodontic knowledge to image and image to orthodontic knowledge, calculate the loss value, use back propagation to update the network parameters of the image feature extraction module, and obtain the trained image feature extraction module, such as Figure 5 Shown, including:

[0139] Step 8.1: Orthodontic knowledge set Input the average pooling layer to obtain the global features of orthodontic knowledge , the dimension is ,in Represents the number of orthodontic knowledge sets.

[0140] Step 8.2: Image features Input the average pooling layer to obtain the global image features of the orthodontic lateral radiograph , the dimension is ,in represents the number of orthodontic lateral radiographs input to the alignment module, and .

[0141] Step 8.3, Global features Stitch together to get features , the dimension is .

[0142] Step 8.4, the features Input linear layer Get features ,Will Input linear layer separately and linear layers Get features and .

[0143] Step 8.5, 、 、 As input, we calculate cross-attention and get features. , the calculation formula is as follows, where express Dimensions, express The transpose of :

[0144] .

[0145] Step 8.6, the features Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent orthodontic knowledge to images , the dimension is .

[0146] Step 8.7: High-dimensional features Input linear layer, map it to low dimension, and get features , the dimension is .

[0147] Step 8.8, Global image features Stitch together to get features , the dimension is .

[0148] Step 8.9, Input to the linear layer Get features ,Will Input linear layer separately and linear layers Get features and .

[0149] Step 8.10, 、 、 As input, we calculate cross-attention and get features. , the calculation formula is as follows, where express Dimensions, express The transpose of :

[0150] .

[0151] Step 8.11, the features Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent image to orthodontic knowledge , the dimension is .

[0152] Step 8.12, Input to the linear layer, map it to a low dimension, and get the features , the dimension is .

[0153] Step 8.13, based on and Calculate loss value , the calculation formula is as follows, Indicates that it is located [i,j] The element at The image and The similarity of the texts, Indicates that it is located [i,j] The element at The text and Similarity of images:

[0154]

[0155] Step 8.14: Use back propagation to update the network parameters of the image feature extraction module to obtain a trained image feature extraction module.

[0156] This embodiment uses the back propagation algorithm to optimize the network parameters of the image feature extraction module, and multiple iterations make the loss value Minimize the number of features and obtain a trained image feature extraction module. The trained image feature extraction module not only has good image feature extraction capabilities but also can identify semantic information related to the image. This property helps the image feature extraction module learn more discriminative features, thereby more accurately associating image features with corresponding language descriptions, rather than simply extracting isolated visual features.

[0157] Step S4: input the user's oral orthodontic lateral radiograph image into a trained image feature extraction module to obtain image features, and input the image features into a text decoder to generate a diagnosis report.

[0158] Specifically, the trained image feature extraction module can extract features that contain both visual features and semantic information from the input orthodontic lateral radiographs. These features are called orthodontic knowledge features. Orthodontic knowledge features can be used for subsequent operations in the orthodontic diagnosis report generation task. For example, orthodontic knowledge features can first be fused with other relevant features, or directly input into a text decoder to generate a detailed orthodontic diagnosis report. This approach combines visual and semantic information to make the generated diagnosis report more accurate and clinically relevant.

[0159] In the existing technology, text decoders basically use Transformer or large language models, which are relatively mature technologies and this application does not impose any restrictions on them.

[0160] This application uses Gaussian convolution to help extract features at different levels of detail in orthodontic lateral radiographs, and introduces a state-space model for modeling and feature extraction of text sequences, thus addressing the limitations of traditional convolutional networks in processing text. The use of a cross-attention mechanism for alignment of text and images enables the model to more effectively combine multimodal information, improving the accuracy and consistency of report generation. In addition, the use of advanced deep learning technology enhances the model's ability to process complex medical images and generate structured reports, greatly reducing model parameters and improving computational efficiency. Through the combined application of these technologies, this application significantly improves the quality and efficiency of orthodontic diagnostic report generation, and addresses the problems of low quality, low efficiency, and reliance on human factors in the traditional manual diagnostic report generation process.

[0161] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for automatically generating an orthodontic diagnosis report based on image-text alignment, characterized in that: The method for automatically generating an orthodontic diagnosis report based on image-text alignment comprises: Preprocess the orthodontic lateral radiograph image to obtain a multi-scale Gaussian pyramid , input the image feature extraction module to obtain image features ; Input the orthodontic report into the medical knowledge extraction module to obtain the orthodontic knowledge set ; Orthodontic knowledge collection Image characteristics of orthodontic lateral radiographs Input the medical knowledge alignment module together, perform cross-attention operations, obtain features from orthodontic knowledge to image and image to orthodontic knowledge, calculate the loss value, and use backpropagation to update the network parameters of the image feature extraction module to obtain a trained image feature extraction module; Input the user's orthodontic lateral radiograph image into the trained image feature extraction module to obtain image features, and then input the image features into the text decoder to generate a diagnosis report; Wherein, the orthodontic lateral radiograph image is preprocessed to obtain a multi-scale Gaussian pyramid ,include: According to the prior distribution of the features of the orthodontic lateral radiograph image, random initial coordinates of the feature points of the orthodontic lateral radiograph image that obey the normal distribution are obtained; Gaussian convolution is used on the orthodontic lateral radiograph to obtain a Gaussian blurred image. Regions of different sizes are cropped from the image with the initial coordinates as the center to obtain a set of multi-scale Gaussian pyramids. ; The input image feature extraction module obtains image features ,include: Gaussian pyramid set The images of each layer are downsampled to the same size; The downsampled Gaussian pyramid set Input to the convolutional encoder to get the features ; The features Input to the convolutional embedding layer and reduce the dimension to obtain features ; The features Input to the graph attention module to obtain features ; The features Input into the multi-layer perceptron MLP for regression to obtain the relative coordinate offset, update the coordinates of the image feature points, crop areas of different sizes in the image with the updated coordinates as the center, obtain a new set of multi-scale Gaussian pyramids, and then enter the image feature extraction module for iteration until the preset number of iterations is reached to obtain the image features. .

2. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 1, wherein: The Gaussian pyramid set after downsampling Input to the convolutional encoder to get the features ,include: The downsampled Gaussian pyramid set Input into the convolutional layer in the convolutional encoder to obtain the feature map ; The feature map Perform maximum pooling and then input it into the first convolution block in the convolution encoder to obtain the feature map ; The feature map is sequentially passed through the convolution blocks of each layer of the convolution encoder. Process and generate subsequent feature maps to , the feature map dimension changes with the processing of each convolution block, and finally we get ; The feature map Perform average pooling to obtain the feature vector .

3. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 1, wherein: The characteristics Input into the convolutional embedding layer for embedding operation and dimensionality reduction to obtain features ,include: In Features Perform an expansion operation on the second dimension to obtain a vector ; The vector Input to the convolutional embedding layer, use a one-dimensional convolution layer to Perform convolution operation to obtain the feature vector after dimensionality reduction .

4. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 1, wherein: The characteristics Input to the graph attention module to obtain features ,include: The features Input three linear layers Q, K, V to get the vector 、 、 ; The vector With the transposed Multiply to get the attention matrix ; The attention matrix The input is sent to the batch normalization layer for normalization, and then the sigmoid function is used for activation and normalization to obtain the attention heat map. ; Attention heat map and vector Perform point multiplication to obtain the attention features between each feature point ; Attention Features Input the linear layer R to get the attention feature vector, and then add the attention feature vector and feature Perform residual connection operation to obtain features .

5. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 1, wherein: The orthodontic report is input into the medical knowledge extraction module to obtain the orthodontic knowledge set ,include: Input the orthodontic report R into the embedding layer to obtain the word embedding vector ; Embedding words into vectors Input to the linear mapping layer to get vector ; The vector Input the convolution layer to obtain a higher-level vocabulary feature vector ; The feature vector The vectors in the state space module are input one by one. For each input vector , select the state space module to output the corresponding vector , all output vectors Splice and get the feature vector ; The feature vector With vector Multiply element by element to get the eigenvector With vector Hadamard ; Hadamard Input to the linear mapping layer, map it to a low dimension, and obtain the orthodontic knowledge set .

6. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 5, characterized in that: The vector Input the selected state space module and get the feature vector ,include: The vector Input to the linear layer of the selected state-space module , get vector B, and transform vector Input to the linear layer of the state-space module , get vector C; The vector Input to the linear layer of the state-space model , and get the vector ; The vector and select the predefined learnable parameter vector in the state space module Add and input to SoftPlus activation function to get the feature ; The predefined learnable parameters Discretization to obtain parameters , the calculation formula is as follows, where is the identity matrix: ； Discretize vector B to get , the calculation formula is as follows, where is the identity matrix: ； Combine the vector C with the predefined hidden state matrix in the selected state space module Perform matrix multiplication to obtain the eigenvector , the hidden state matrix Used to store the hidden state of all input vectors and update the hidden state matrix , the calculation formula is: 。 7. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 1, wherein: The oral orthodontic knowledge set Image characteristics of orthodontic lateral radiographs The medical knowledge alignment module is input together, and cross-attention operations are performed to obtain features from orthodontic knowledge to image and from image to orthodontic knowledge. The loss value is calculated, and the network parameters of the image feature extraction module are updated using backpropagation to obtain a trained image feature extraction module, which includes: Orthodontic knowledge collection Input the average pooling layer to obtain the global features of orthodontic knowledge ; The image features Input the average pooling layer to obtain the global image features of the orthodontic lateral radiograph ; Global Features Stitch together to get features ; The features Input linear layer Get features ,Will Input linear layer separately and linear layers Get features and ; Will 、 、 As input, we calculate cross-attention and get features. , the calculation formula is as follows, where express Dimensions, express The transpose of : ； The features Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent orthodontic knowledge to images ; High-dimensional features Input linear layer, map it to low dimension, and get features ; The global image features Stitch together to get features ; The features Input to the linear layer Get features ,Will Input linear layer separately and linear layers Get features and ; Will 、 、 As input, we calculate cross-attention and get features. , the calculation formula is as follows, where express Dimensions, express The transpose of : ； The features Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent image to orthodontic knowledge ; High-dimensional features Input to the linear layer, map it to a low dimension, and get the features ; in accordance with and Calculate the loss value and use back propagation to update the network parameters of the image feature extraction module to obtain the trained image feature extraction module.

Images