Method for automatically generating orthodontic diagnosis report based on image text alignment

Abstract

The invention discloses a method for automatically generating an orthodontic diagnosis report based on image and text alignment, which comprises the following steps of: firstly, preprocessing an orthodontic lateral film, and extracting image features by using Gaussian convolution; meanwhile, modeling and feature extraction are carried out on an orthodontic report by utilizing a state space model. Then, the image and text features are aligned through a cross attention mechanism, so that an image feature extraction module can recognize semantic information related to the image more accurately, and the image features are closely associated with corresponding language descriptions; finally, the extracted orthodontic knowledge features are input into a text decoder, and a high-quality orthodontic diagnosis report is generated. According to the method, the accuracy and efficiency of diagnosis report generation are remarkably improved, and the problems of low quality and low efficiency in the traditional manual diagnosis process are solved.

Description

Technical Field

[0001] The present application belongs to the field of medical image processing technology, and in particular relates to a method for automatically generating an oral orthodontic diagnosis report based on image-text alignment. Background Art

[0002] Orthodontic diagnosis is an important branch of stomatology, mainly involving the study and correction of teeth, jaws and facial structures. The core purpose of orthodontic diagnosis is to provide accurate diagnostic reports through the analysis of patients' oral images and clinical data to guide subsequent treatment plans. This kind of diagnostic report usually requires experienced professional doctors to conduct detailed analysis of images such as orthodontic lateral radiographs, and write it in combination with extensive clinical medical knowledge. The traditional manual diagnosis process is time-consuming and labor-intensive, and is easily affected by human factors, resulting in limited accuracy and consistency of diagnostic results.

[0003] In addition, orthodontic diagnosis also needs to be combined with the patient's clinical examination and medical history, taking into account a variety of factors and formulating a personalized treatment plan. This process requires doctors to have rich clinical experience and a deep medical knowledge base. As patients' requirements for treatment effects and comfort increase, the accuracy and personalization of orthodontic diagnostic reports are also increasing.

[0004] In recent years, the rapid development of artificial intelligence (AI) and deep learning technology has brought new possibilities for the automated generation of orthodontic diagnostic reports. Convolutional Neural Networks (CNNs) in deep learning have demonstrated powerful capabilities in image feature extraction and analysis, making it possible to perform medical image analysis through image recognition technology. At the same time, the progress of natural language processing (NLP) technology in text generation and medical knowledge extraction has made it possible to extract useful information from a large amount of medical literature and reports and generate structured text content.

[0005] Currently, although some studies have attempted to apply deep learning to the analysis and report generation of orthodontic images, these technologies are mostly focused on the overall classification of images and simple text generation, lacking the deep fusion of image and text information. Existing methods have certain limitations in image feature extraction and medical knowledge alignment, making it difficult to achieve highly accurate and personalized diagnostic report generation. In addition, the alignment and fusion between image features and text features remains a challenge, and existing solutions have little research in this area. Summary of the invention

[0006] The purpose of this application is to provide an automatic generation method of orthodontic diagnostic reports based on image-text alignment, aiming to overcome the defects in the prior art. On the basis of improving the prior art, by aligning and fusing multimodal features, the efficient and accurate generation of diagnostic reports is further achieved, which not only solves the time-consuming and labor-intensive problems in the traditional manual diagnosis process, but also effectively improves the generation quality of orthodontic diagnostic reports, and solves the problems of low report quality, lack of personalization and accuracy in the prior art.

[0007] In order to achieve the above purpose, the technical solution of this application is as follows: A method for automatically generating an orthodontic diagnosis report based on image and text alignment, comprising: Preprocess the orthodontic lateral radiograph image to obtain a multi-scale Gaussian pyramid , input the image feature extraction module to obtain image features ; Input the orthodontic report into the medical knowledge extraction module to obtain the orthodontic knowledge set ; Orthodontic knowledge collection Image characteristics of lateral orthodontic radiographs Input the medical knowledge alignment module together, perform cross-attention operation, obtain the features of orthodontic knowledge to image and image to orthodontic knowledge, calculate the loss value, use back propagation to update the network parameters of the image feature extraction module, and obtain the trained image feature extraction module; The user's oral orthodontic lateral radiograph image is input into a trained image feature extraction module to obtain image features, and the image features are input into a text decoder to generate a diagnosis report.

[0008] Furthermore, the orthodontic lateral radiograph image is preprocessed to obtain a multi-scale Gaussian pyramid ,include: According to the prior distribution of the features of the orthodontic lateral radiograph image, the random initial coordinates of the feature points of the orthodontic lateral radiograph image that obey the normal distribution are obtained; Gaussian convolution is used on the orthodontic lateral radiograph to obtain a Gaussian blurred image. Regions of different sizes are cut out from the image with the initial coordinates as the center to obtain a set of multi-scale Gaussian pyramids. ; The input image feature extraction module obtains image features ,include: The Gaussian pyramid set The images of each layer are downsampled to the same size; The downsampled Gaussian pyramid set Input to the convolution encoder to get the features ; The characteristics Input to the convolution embedding layer, reduce the dimension to get features ; The characteristics Input to the graph attention module to obtain features ; The characteristics Input into the multi-layer perceptron MLP for regression, obtain the relative coordinate offset, update the coordinates of the image feature points, cut areas of different sizes in the image with the updated coordinates as the center, obtain a new set of multi-scale Gaussian pyramids, and then enter the image feature extraction module for iteration until the preset number of iterations is reached to obtain the image feature. .

[0009] Furthermore, the Gaussian pyramid set after downsampling Input to the convolution encoder to get the features ,include: The downsampled Gaussian pyramid set Input into the convolutional layer in the convolutional encoder to obtain the feature map ; The feature map Perform maximum pooling and then input it into the first convolution block in the convolution encoder to obtain the feature map ; Through each layer of convolutional encoder, the feature map Processing to generate subsequent feature maps to , the feature map dimension changes with the processing of each layer of convolutional blocks, and finally we get ; The feature map Perform average pooling to obtain the feature vector .

[0010] Furthermore, the feature Input to the convolution embedding layer for embedding operation and dimension reduction to obtain features ,include: In Features Expand the second dimension to get the vector ; Vector Input to the convolutional embedding layer, use a one-dimensional convolutional layer to Perform convolution operation to obtain the feature vector after dimensionality reduction .

[0011] Furthermore, the feature Input to the graph attention module to obtain features ,include: The characteristics Input three linear layers Q, K, V to get the vector , , ; Vector With the transposed Multiply them together to get the attention matrix ; The attention matrix The input is sent to the batch normalization layer for normalization, and then the sigmoid function is used for activation and normalization to obtain the attention heat map. ; Attention heat map and vector Perform point multiplication to obtain the attention features between feature points ; Attention Features Input the linear layer R to get the attention feature vector, and then add the attention feature vector and feature Perform residual connection operation to obtain features .

[0012] Furthermore, the orthodontic report is input into the medical knowledge extraction module to obtain the orthodontic knowledge set ,include: Input the orthodontic report R into the embedding layer to obtain the word embedding vector ; Embedding words into vectors Input to the linear mapping layer to get the vector ; Vector Input the convolutional layer to obtain a higher-level vocabulary feature vector ; The feature vector The vectors in the state-space module are input one by one. For each input vector , select the state space module to output the corresponding vector , all output vectors Concatenate and get the feature vector ; The feature vector With vector Multiply element by element to get the eigenvector With vector Hadamard ; Hadamard Input to the linear mapping layer, map it to a low dimension, and obtain the orthodontic knowledge set .

[0013] Furthermore, the vector Input select state space module and get feature vector ,include: Vector Input to the linear layer of the selected state-space module , get vector B, and transform the vector Input to the linear layer of the state-space module , get vector C; Vector Input to the linear layer of the state-space model , we get the vector ; Vector and select the predefined learnable parameter vector in the state space module Add and input to the SoftPlus activation function to get the feature ; The predefined learnable parameters Discretization to get parameters , the calculation formula is as follows, where is the identity matrix: Discretize vector B to get , the calculation formula is as follows, where is the identity matrix: Combine the vector C with the predefined hidden state matrix in the selected state space module Perform matrix multiplication to obtain the eigenvector , the hidden state matrix Used to store the hidden states of all input vectors and update the hidden state matrix , the calculation formula is: .

[0014] Furthermore, the oral orthodontic knowledge collection Image characteristics of lateral orthodontic radiographs Input the medical knowledge alignment module together, perform cross-attention operation, obtain the features of orthodontic knowledge to image and image to orthodontic knowledge, calculate the loss value, use back propagation to update the network parameters of the image feature extraction module, and obtain the trained image feature extraction module, including: Orthodontic knowledge collection Input the average pooling layer to obtain the global features of orthodontic knowledge ; The image features Input the average pooling layer to obtain the global image features of the orthodontic lateral radiograph ; Global Features Stitch together to get the features ; The characteristics Input Linear Layer Get Features ,Will Input linear layer separately and the linear layer Get Features and ; Will , , As input, we calculate cross-attention and get the feature , the calculation formula is as follows, where express The dimension of express The transpose of : ; The characteristics Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent orthodontic knowledge to images ; High-dimensional features Input the linear layer, map it to a low dimension, and get the features ; The global image features Stitch together to get the features ; The characteristics Input to the linear layer Get Features ,Will Input linear layer separately and the linear layer Get Features and ; Will , , As input, we calculate cross-attention and get the feature , the calculation formula is as follows, where express The dimension of express The transpose of : ; The characteristics Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent image-to-orthodontic knowledge ; High-dimensional features Input to the linear layer, map it to a low dimension, and get the features ; in accordance with and Calculate the loss value and use back propagation to update the network parameters of the image feature extraction module to obtain the trained image feature extraction module.

[0015] The present application proposes a method for automatically generating an orthodontic diagnostic report based on image-text alignment. On the one hand, by aligning and fusing the multimodal features of images and texts, the model can better extract and understand the information in the orthodontic lateral radiographs, thereby significantly improving the accuracy of the diagnostic report. On the other hand, by introducing the state-space model to model and extract features of text sequences, the expressive power of text information is enhanced. In addition, the use of Gaussian convolution to extract features from images improves the effect of the model in processing image detail information. At the same time, the cross-attention mechanism is used to align text and images, so that the model can more effectively combine multimodal information and enhance the accuracy and consistency of report generation. The present application effectively solves the problems of low quality, low efficiency and reliance on human factors in the traditional manual diagnostic report generation process, and provides an efficient, accurate and consistent solution for orthodontic diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of the method for automatically generating orthodontic diagnosis reports based on image-text alignment for this application.

[0017] Figure 2 Schematic diagram of the training network model structure for this application.

[0018] Figure 3 Schematic diagram of the structure of the convolutional encoder module in an embodiment of the present application.

[0019] Figure 4 This is a structural diagram of the medical knowledge extraction module of the embodiment of the present application.

[0020] Figure 5 This is a structural diagram of the medical knowledge alignment module of an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0022] An embodiment of the present application, such as Figure 1 As shown, a method for automatically generating an orthodontic diagnosis report based on image-text alignment is provided, comprising: Step S1: preprocess the orthodontic lateral radiograph image to obtain a multi-scale Gaussian pyramid , input the image feature extraction module to obtain image features .

[0023] In order to distinguish the problem that the image feature extraction module in the prior art simply extracts isolated visual features, this application introduces a medical knowledge extraction module and a medical knowledge alignment module to participate in the training of the image feature extraction module, so that the trained image feature extraction module can extract features that contain both visual features and semantic information from the input orthodontic lateral radiographs. The network structure during the entire training can be found in Figure 2 ,After the training is completed, the features extracted by the ,image feature extraction module are input into the text decoder to generate a ,diagnosis report.

[0024] In a specific embodiment, the orthodontic lateral radiograph image is preprocessed to obtain a multi-scale Gaussian pyramid. ,include: Step 1.1, according to the prior distribution of the features of the orthodontic lateral radiograph image, obtain the random initial coordinates of the feature points of the orthodontic lateral radiograph image that obey the normal distribution.

[0025] An orthodontic lateral radiograph image is an image taken from the side of the head using X-rays, which can show the patient's facial bones and tooth arrangement in detail. This application generates an orthodontic diagnosis report based on the orthodontic lateral radiograph image.

[0026] This step calculates the prior distribution of the features of the orthodontic lateral radiograph image based on the annotations in the known data set to calculate the random initial coordinates of the feature points of the orthodontic lateral radiograph image that obey the normal distribution. The prior distribution and how to calculate the initial coordinates are relatively mature technologies in this field and will not be described here.

[0027] Step 1.2: Use Gaussian convolution to obtain a Gaussian blurred image on the orthodontic lateral radiograph. Cut regions of different sizes in the image with the initial coordinates as the center to obtain a set of multi-scale Gaussian pyramids. .

[0028] This step includes indivual A collection of multi-scale Gaussian pyramids .

[0029] For each Gaussian pyramid , and the image dimensions of each layer are ,in is the height of the first layer image, is the width of the first layer image, The corresponding A Gaussian pyramid, The corresponding layer image, is the number of predefined feature points.

[0030] Then, the Gaussian pyramid set Input into the image feature extraction module to obtain image features In a specific embodiment, the image feature extraction module performs the following operations: Step 1.3: Assemble the Gaussian pyramids The images of each layer are downsampled to the same size.

[0031] For example, after downsampling, The dimension becomes , and is the length and width of the second layer image of the Gaussian pyramid. The size of the second layer is selected as the basis for downsampling in order to effectively reduce the computational complexity while maintaining the image information. The size of this layer can well balance the computational requirements in subsequent processing and the retention of key information of the image, ensuring that the structural features of the image are well preserved while reducing the amount of data, thus providing an efficient and reliable basis for subsequent processing steps.

[0032] Step 1.4: Set the downsampled Gaussian pyramid Input to the convolution encoder to get the features .

[0033] After the convolutional encoder, its dimension is , The dimension of a single feature vector after convolution encoding.

[0034] Step 1.5: Features Input to the convolution embedding layer for embedding operation and dimension reduction to obtain features .

[0035] feature The dimension is , is the dimension of a single feature vector after the embedding operation.

[0036] Step 1.6: Features Input to the graph attention module to obtain features .

[0037] feature The dimension is .

[0038] Step 1.7: Features Input it into the multi-layer perceptron MLP for regression to obtain the relative coordinate offset, update the coordinates of the image feature points, cut areas of different sizes in the image with the updated coordinates as the center, obtain a new set of multi-scale Gaussian pyramids, and then enter the image feature extraction module for iteration until the preset number of iterations is reached to obtain the image feature. .

[0039] For example, the first time through the multi-layer perceptron MLP, the relative coordinate offset is obtained , and according to and Update the coordinates of image feature points , with coordinates Cut out regions of different sizes in the image as the center, and perform multiple iterations to update the final feature point coordinates, that is, to obtain the image features. , whose dimensions are , is the number of feature points, is the feature point coordinate dimension.

[0040] In a specific embodiment, Figure 3 As shown, the Gaussian pyramid set after downsampling Input to the convolution encoder to get the features ,include: Step 2.1: Set the downsampled Gaussian pyramid Input into the convolutional layer in the convolutional encoder to obtain the feature map , whose dimension size is ,in Indicates the number of channels.

[0041] Step 2.2: Feature map Perform maximum pooling and then input it into the first convolution block in the convolution encoder to obtain the feature map , whose dimension size is .

[0042] Step 2.3: Pass the convolution blocks of each layer of the convolution encoder in turn to the feature map Processing to generate subsequent feature maps to , the feature map dimension changes with the processing of each layer of convolutional blocks, and finally we get , feature map The dimension size is .

[0043] Step 2.4: Feature map Perform average pooling to obtain the feature vector , whose dimension size is , Indicates that the vector length is equal to the number of channels .

[0044] In a specific embodiment, the feature Input to the convolution embedding layer for embedding operation and dimension reduction to obtain features ,include: Step 3.1: In the feature Expand the second dimension to get the vector , the dimension is ,in Indicates that the expansion operation is The medium length is The feature vector of Second splicing.

[0045] Step 3.2: Vector Input to the convolutional embedding layer, use a one-dimensional convolutional layer to Perform convolution operation to obtain the feature vector after dimensionality reduction , the dimension is , is the length of the vector.

[0046] In a specific embodiment, the feature Input to the graph attention module to obtain features ,include: Step 4.1: Features Input three linear layers Q, K, V to get the vector , , , whose dimensions are , where the vector length is the output dimension of the linear layers Q, K, and V.

[0047] Step 4.2: Vector With the transposed Multiply them together to get the attention matrix , whose dimensions are .

[0048] Step 4.3: Attention matrix The input is sent to the batch normalization layer for normalization, and then the sigmoid function is used for activation and normalization to obtain the attention heat map. , whose dimensions are .

[0049] Step 4.4: Attention heat map and vector Perform point multiplication to obtain the attention features between feature points , whose dimensions are .

[0050] Step 4.5: Attention features Input linear layer, the dimension is The attention feature vector is then combined with the feature Perform residual connection operation to obtain features , whose dimensions are .

[0051] In a specific embodiment, the Input into the multi-layer perceptron MLP for regression to obtain the relative coordinate offset , and based on and Update the coordinates of image feature points , with coordinates As the center, cut out areas of different sizes in the image, and perform multiple iterations to update the final feature point coordinates ,include: Step 5.1: Transform the feature vector Input to the linear layer 𝑃1 for full connection operation, activated and dropped out through the Relu function, and obtain the feature vector , whose dimensions are , where the vector length is the number of output nodes of the linear layer 𝑃1.

[0052] Step 5.2: Transform the feature vector Input to the linear layer 𝑃2 for full connection operation, activated and dropped out through the Relu function, and obtain the feature vector , whose dimensions are , where the vector length is the number of output nodes of the linear layer 𝑃2.

[0053] Step 5.3: Transform the feature vector Input to the linear layer 𝑃3 for full connection operation, and regress to get the relative offset , whose dimensions are .

[0054] Step 5.4: Update according to the following formula , and Construct a new set of multi-scale Gaussian pyramids for the center , as the input of step 1.3 for iterative operation, and the value after 10 iterations is is the final position of each feature point, that is, the final feature point coordinates : Where t represents the current iteration number, ranging from 1 to 10.

[0055] Step S2: Input the orthodontic report into the medical knowledge extraction module to obtain the orthodontic knowledge set .

[0056] In this embodiment, the orthodontic report (R) is input into the medical knowledge extraction module to obtain the orthodontic knowledge set. ,like Figure 4 As shown, including: Step 6.1: Input the orthodontic report into the word embedding layer to obtain the word embedding vector , whose dimensions are ,in represents the number of word embedding vectors, is the word embedding vector dimension.

[0057] Step 6.2: embed the word into vector Input to the linear mapping layer to get the vector , whose dimensions are .

[0058] Step 6.3: Vector Input the convolutional layer to obtain a higher-level vocabulary feature vector , the dimension is .

[0059] Step 6.4: Transform the feature vector In vectors are input one by one to select the state space module. For each input vector , select the state space module to output the corresponding vector , all output vectors Concatenate and get the feature vector , the dimension is ,vector and The dimension is , correspond No. vectors and their corresponding state-space module outputs.

[0060] Step 6.5: Transform the feature vector With vector Multiply element by element to get the eigenvector With vector Hadamard , the dimension is .

[0061] Step 6.6: Hadamard product Input to the linear mapping layer, map it to a low dimension, and obtain the orthodontic knowledge set , the dimension is ,in Indicates the amount of knowledge.

[0062] In a specific embodiment, the feature vector Input select state space module and get feature vector ,include: Step 7.1: Convert the vector Input to the linear layer of the selected state-space module , we get vector B, whose dimension is , the vector Input to the linear layer of the state-space module , we get a vector C whose dimension is .

[0063] Step 7.2: Vector Input to the linear layer of the state-space model , we get the vector , whose dimensions are .

[0064] Step 7.3: Vector and select the predefined learnable parameter vector in the state space module Add and input to the SoftPlus activation function to get the feature , whose dimensions are , The dimension is .

[0065] Step 7.4: Set the predefined learnable parameters Discretization to get parameters , whose dimensions are , the calculation formula is as follows, where is the identity matrix: Step 7.5: Discretize vector B to obtain , whose dimensions are , the calculation formula is as follows, where is the identity matrix: Step 7.6: Compare the vector C with the hidden state matrix predefined in the selected state space module Perform matrix multiplication to obtain the eigenvector , whose dimensions are , the hidden state matrix Used to store the hidden state of all input vectors, the dimension is .

[0066] Step 7.7: Update the hidden state matrix , the calculation formula is: .

[0067] Step S3: Orthodontic knowledge set Image characteristics of lateral orthodontic radiographs The two modules are input into the medical knowledge alignment module together, and the cross-attention operation is performed to obtain the features of orthodontic knowledge to image and image to orthodontic knowledge. The loss value is calculated, and the network parameters of the image feature extraction module are updated using back propagation to obtain the trained image feature extraction module.

[0068] This application describes the collection of oral orthodontic knowledge Image characteristics of lateral orthodontic radiographs The medical knowledge alignment module is input together, the cross-attention operation is performed, the features of orthodontic knowledge to image and image to orthodontic knowledge are obtained, the loss value is calculated, and the network parameters of the image feature extraction module are updated using back propagation to obtain the trained image feature extraction module, such as Figure 5 As shown, including: Step 8.1: Orthodontic knowledge collection Input the average pooling layer to obtain the global features of orthodontic knowledge , the dimension is ,in Represents the number of orthodontic knowledge sets.

[0069] Step 8.2: Image features Input the average pooling layer to obtain the global image features of the orthodontic lateral radiograph , the dimension is ,in represents the number of orthodontic lateral radiographs input to the alignment module, and .

[0070] Step 8.3: Global Features Stitch together to get the features , the dimension is .

[0071] Step 8.4: Add features Input Linear Layer Get Features ,Will Input linear layer separately and the linear layer Get Features and .

[0072] Step 8.5: , , As input, we calculate cross-attention and get the feature , the calculation formula is as follows, where express The dimension of express The transpose of : .

[0073] Step 8.6: Features Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent orthodontic knowledge to images , the dimension is .

[0074] Step 8.7: High-dimensional features Input the linear layer, map it to a low dimension, and get the features , the dimension is .

[0075] Step 8.8: Global image features Stitch together to get the features , the dimension is .

[0076] Step 8.9: Input to the linear layer Get Features ,Will Input linear layer separately and the linear layer Get Features and .

[0077] Step 8.10: , , As input, we calculate cross-attention and get the feature , the calculation formula is as follows, where express The dimension of express The transpose of : .

[0078] Step 8.11: Add features Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent image-to-orthodontic knowledge , the dimension is .

[0079] Step 8.12: Input to the linear layer, map it to a low dimension, and get the features , the dimension is .

[0080] Step 8.13, based on and Calculate the loss value , the calculation formula is as follows, Indicates that it is located [i,j] The element at The image and The similarity of the texts, Indicates that it is located [i,j] The element at The text and Similarity of images: Step 8.14: Use back propagation to update the network parameters of the image feature extraction module to obtain a trained image feature extraction module.

[0081] This embodiment uses the back propagation algorithm to optimize the network parameters of the image feature extraction module, and multiple iterations make the loss value Minimize and obtain a trained image feature extraction module. The trained image feature extraction module not only has good image feature extraction capabilities, but also can recognize semantic information related to the image. This feature helps the image feature extraction module learn more discriminative features, so as to more accurately associate image features with corresponding language descriptions, rather than simply extracting isolated visual features.

[0082] Step S4: input the user's oral orthodontic lateral radiograph image into a trained image feature extraction module to obtain image features, and input the image features into a text decoder to generate a diagnosis report.

[0083] Specifically, the trained image feature extraction module can extract features that contain both visual features and semantic information from the input orthodontic lateral radiographs. Such features are called orthodontic knowledge features. Orthodontic knowledge features can be used for subsequent operations in the orthodontic diagnosis report generation task. For example, orthodontic knowledge features can be first fused with other related features, or directly input into a text decoder to generate a detailed orthodontic diagnosis report. This method combines visual and semantic information to make the generated diagnosis report more accurate and clinically relevant.

[0084] In the prior art, text decoders basically use Transformer or large language models, which are relatively mature technologies and this application does not impose any restrictions on this.

[0085] This application uses Gaussian convolution to help extract features of orthodontic lateral radiographs at different levels of detail, and introduces a state-space model for modeling and feature extraction of text sequences, thereby addressing the limitations of traditional convolutional networks when processing text. The use of a cross-attention mechanism for alignment of text and images enables the model to more effectively combine multimodal information and improve the accuracy and consistency of report generation. In addition, advanced deep learning techniques are used to enhance the model's ability to process complex medical images and generate structured reports, greatly reducing model parameters and improving computational efficiency. Through the combined application of these technologies, this application significantly improves the quality and efficiency of orthodontic diagnostic report generation, and addresses the problems of low quality, low efficiency, and reliance on human factors in the traditional manual diagnostic report generation process.

[0086] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for automatically generating an orthodontic diagnosis report based on image-text alignment, characterized in that: The method for automatically generating an orthodontic diagnosis report based on image-text alignment comprises: Preprocess the orthodontic lateral radiograph image to obtain a multi-scale Gaussian pyramid , input the image feature extraction module to obtain image features ; Input the orthodontic report into the medical knowledge extraction module to obtain the orthodontic knowledge set ; Orthodontic knowledge collection Image characteristics of lateral orthodontic radiographs Input the medical knowledge alignment module together, perform cross-attention operation, obtain the features from orthodontic knowledge to image and from image to orthodontic knowledge, calculate the loss value, use back propagation to update the network parameters of the image feature extraction module, and obtain the trained image feature extraction module; The user's oral orthodontic lateral radiograph image is input into a trained image feature extraction module to obtain image features, and the image features are input into a text decoder to generate a diagnosis report.

2. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 1, characterized in that: The orthodontic lateral radiograph image is preprocessed to obtain a multi-scale Gaussian pyramid ,include: According to the prior distribution of the features of the orthodontic lateral radiograph image, the random initial coordinates of the feature points of the orthodontic lateral radiograph image that obey the normal distribution are obtained; Gaussian convolution is used on the orthodontic lateral radiograph to obtain a Gaussian blurred image. Regions of different sizes are cut from the image with the initial coordinates as the center to obtain a set of multi-scale Gaussian pyramids. ; The input image feature extraction module obtains image features ,include: The Gaussian pyramid set The images of each layer are downsampled to the same size; The downsampled Gaussian pyramid set Input to the convolution encoder to get the features ; The characteristics Input to the convolution embedding layer, reduce the dimension to get features ; The characteristics Input to the graph attention module to obtain features ; The characteristics Input into the multi-layer perceptron MLP for regression, obtain the relative coordinate offset, update the coordinates of the image feature points, cut areas of different sizes in the image with the updated coordinates as the center, obtain a new set of multi-scale Gaussian pyramids, and then enter the image feature extraction module for iteration until the preset number of iterations is reached to obtain the image feature. .

3. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 2, characterized in that: The downsampled Gaussian pyramid set Input to the convolution encoder to get the features ,include: The downsampled Gaussian pyramid set Input into the convolutional layer in the convolutional encoder to obtain the feature map ; The feature map Perform maximum pooling and then input it into the first convolution block in the convolution encoder to obtain the feature map ; Through each layer of convolutional encoder, the feature map Processing to generate subsequent feature maps to , the feature map dimension changes with the processing of each layer of convolutional blocks, and finally we get ; The feature map Perform average pooling to obtain the feature vector .

4. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 2, characterized in that: The characteristics Input to the convolution embedding layer for embedding operation and dimension reduction to obtain features ,include: In Features Expand the second dimension to get the vector ; Vector Input to the convolutional embedding layer, use a one-dimensional convolutional layer to Perform convolution operation to obtain the feature vector after dimensionality reduction .

5. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 2, characterized in that: The characteristics Input to the graph attention module to obtain features ,include: The characteristics Input three linear layers Q, K, V and get the vector , , ; Vector With the transposed Multiply them together to get the attention matrix ; The attention matrix The input is sent to the batch normalization layer for normalization, and then the sigmoid function is used for activation and normalization to obtain the attention heat map. ; Attention heat map and vector Perform point multiplication to obtain the attention features between feature points ; Attention Features Input the linear layer R to get the attention feature vector, and then add the attention feature vector and feature Perform residual connection operation to obtain features .

6. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 1, characterized in that: The orthodontic report is input into the medical knowledge extraction module to obtain the orthodontic knowledge set ,include: Input the orthodontic report R into the embedding layer to obtain the word embedding vector ; Embedding words into vectors Input to the linear mapping layer to get the vector ; Vector Input the convolutional layer to obtain a higher-level vocabulary feature vector ; The feature vector The vectors in the state-space module are input one by one. For each input vector , select the state space module to output the corresponding vector , all output vectors Concatenate and get the feature vector ; The feature vector With vector Multiply element by element to get the eigenvector With vector Hadamard ; Hadamard Input to the linear mapping layer, map it to a low dimension, and obtain the orthodontic knowledge set .

7. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 6, characterized in that: The vector Input select state space module and get feature vector ,include: Vector Input to the linear layer of the selected state-space module , get vector B, and transform the vector Input to the linear layer of the state-space module , get vector C; Vector Input to the linear layer of the state-space model , we get the vector ; Vector and select the predefined learnable parameter vector in the state space module Add and input to the SoftPlus activation function to get the feature ; The predefined learnable parameters Discretization to get parameters , the calculation formula is as follows, where is the identity matrix: Discretize vector B to get , the calculation formula is as follows, where is the identity matrix: Combine the vector C with the predefined hidden state matrix in the selected state space module Perform matrix multiplication to obtain the eigenvector , the hidden state matrix Used to store the hidden states of all input vectors and update the hidden state matrix , the calculation formula is: 。 8. The method for automatically generating an orthodontic diagnosis report based on image-text alignment according to claim 1, characterized in that: The oral orthodontic knowledge Image characteristics of lateral orthodontic radiographs Input the medical knowledge alignment module together, perform cross-attention operation, obtain the features of orthodontic knowledge to image and image to orthodontic knowledge, calculate the loss value, use back propagation to update the network parameters of the image feature extraction module, and obtain the trained image feature extraction module, including: Orthodontic knowledge collection Input the average pooling layer to obtain the global features of orthodontic knowledge ; The image features Input the average pooling layer to obtain the global image features of the orthodontic lateral radiograph ; Global Features Stitch together to get the features ; The characteristics Input Linear Layer Get Features ,Will Input linear layer separately and the linear layer Get Features and ; Will , , As input, we calculate cross-attention and get the feature , the calculation formula is as follows, where express The dimension of express The transpose of : ； The characteristics Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent orthodontic knowledge to images ; High-dimensional features Input the linear layer, map it to a low dimension, and get the features ; The global image features Stitch together to get the features ; The characteristics Input to the linear layer Get Features ,Will Input linear layer separately and the linear layer Get Features and ; Will , , As input, we calculate cross-attention and get the feature , the calculation formula is as follows, where express The dimension of express The transpose of : ； The characteristics Input the multi-layer perceptron MLP to obtain high-dimensional features used to represent image-to-orthodontic knowledge ; High-dimensional features Input to the linear layer, map it to a low dimension, and get the features ; in accordance with and Calculate the loss value and use back propagation to update the network parameters of the image feature extraction module to obtain the trained image feature extraction module.

Images