Method, device and equipment for training report generation model and storage medium

Abstract

The application relates to the computer field, in particular to the artificial intelligence field, and provides a report generation model training method and device, equipment and a storage medium. The method comprises the following steps: iteratively training a report generation model by using multiple sample image pictures; an initial classification feature and multiple initial image features of each sample image picture are obtained; a target classification feature is obtained by performing fusion coding processing on the initial classification feature and the initial image features; the target classification feature is classified and recognized to obtain a prediction label used for determining a disease category to which the picture belongs; and the model parameters are adjusted based on the prediction label and a reference label. The model parameters of the report generation model are iteratively updated based on the prediction label and the reference label obtained from the target classification feature, the cognition of the model to the disease category to which the sample image picture belongs is strengthened, and then the attention of the model is redirected to an abnormal area of the sample image picture, and the attention of the model to the abnormal area is strengthened.

Description

Technical Field

[0001] This application relates to the field of computer science, and more particularly to the field of artificial intelligence, and provides a training method, apparatus, device, and storage medium for a report generation model. Background Technology

[0002] Image caption generation aims to generate a natural language description for a given image. It is a multimodal task involving both computer vision and natural language processing. In recent years, this technology has begun to be applied in the medical field, with the aim of automatically generating diagnostic reports based on medical images such as CT scans and chest X-rays. Highly accurate medical image reports can not only assist doctors in diagnosis and reduce their workload, but also provide diagnostic results to help patients with self-diagnosis.

[0003] Most commonly used report generation models are based on multimodal encoder-decoder structures, with R2GEN being a classic example. For instance... Figure 1 As shown, a medical image is input into R2GEN. The visual feature extraction module of R2GEN extracts the corresponding visual features from the medical image. Then, a visual encoder-text decoder based on multimodal Transformer is used to convert the visual features into text features. Finally, a word generator based on multilayer perceptron converts the text features into words to form the target medical image report.

[0004] While existing report generation models can produce fluent reports, they are hampered by an imbalanced distribution of positive and negative samples in the existing training datasets and the small proportion of abnormal regions in a complete medical image. These models often focus only on normal areas in medical images, neglecting abnormal regions that represent lesions. Consequently, the medical image reports generated by these models mostly describe normal areas, lacking descriptions of abnormal regions. Therefore, the quality of medical image reports generated based on existing models varies greatly, and some low-quality reports can affect doctors' accurate judgment and treatment of patients. Summary of the Invention

[0005] This application provides a training method, apparatus, device, and storage medium for a report generation model to address the problem of low recognition rate of abnormal regions in report generation models.

[0006] In a first aspect, embodiments of this application provide a method for training a report generation model, including:

[0007] The system employs an iterative approach, sequentially reading image samples from the training dataset to train the report generation model until the fully trained model is output. Each iteration includes:

[0008] Feature extraction is performed on a sample image to obtain initial classification features for determining the disease category to which the sample image belongs, and multiple initial image features for characterizing the visual information of the sample image.

[0009] The initial classification features are fused and encoded with multiple initial image features to obtain the corresponding target classification features;

[0010] The target classification features are classified and identified to obtain a predicted label representing the disease category to which the sample image belongs. Based on the predicted label and the reference label of the sample image, the model parameters of the report generation model are updated.

[0011] Secondly, embodiments of this application also provide a training apparatus for a report generation model, comprising:

[0012] The model training unit is used to iteratively read each sample image from the training dataset, train the report generation model, and output the trained report generation model until the model is fully trained. Each iteration includes:

[0013] The feature extraction unit is used to extract features from a read sample image to obtain initial classification features for determining the disease category to which the sample image belongs, and multiple initial image features for characterizing the visual information of the sample image.

[0014] An encoding unit is used to perform fusion encoding processing on the initial classification features and multiple initial image features to obtain the corresponding target classification features;

[0015] A classification unit is used to classify and identify the target classification features to obtain a predicted label that characterizes the disease category to which the sample image belongs;

[0016] The parameter adjustment unit is used to update the model parameters of the report generation model based on the predicted label and the reference label of the sample image.

[0017] Optionally, the classification unit is used for:

[0018] Feature extraction is performed on the target classification features to obtain the predicted probability that a sample image belongs to each disease category;

[0019] The disease category corresponding to the predicted probability exceeding a set threshold is determined as the predicted label of the sample image.

[0020] Optionally, the parameter adjustment unit is used for:

[0021] Based on the predicted label and the reference label of the sample image, the classification balance loss value for this iteration is determined;

[0022] Based on the obtained classification balance loss value, update the model parameters of the report generation model.

[0023] Thirdly, embodiments of this application also provide a computer device, including a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of any of the above-described training methods for a report generation model.

[0024] Fourthly, embodiments of this application also provide a computer-readable storage medium including program code, which, when the program product is run on a computer device, is used to cause the computer device to perform the steps of any of the above-described training methods for the report generation model.

[0025] Fifthly, embodiments of this application also provide a computer program product, including computer instructions, which are executed by a processor to perform the steps of any of the above-described training methods for a report generation model.

[0026] The beneficial effects of this application are as follows:

[0027] This application proposes a training method, apparatus, device, and storage medium for a report generation model. The method includes: sequentially reading sample image images from a training dataset in an iterative manner to train the report generation model until the trained model is output; wherein each iteration includes: extracting features from a single sample image to obtain initial classification features for determining the disease category of the sample image, and multiple initial image features for characterizing the visual information of the sample image; fusing and encoding the initial classification features and multiple initial image features to obtain corresponding target classification features; classifying and recognizing the target classification features to obtain a predicted label characterizing the disease category of the sample image; and updating the model parameters of the report generation model based on the predicted label and a reference label of the sample image.

[0028] This application embodiment utilizes target classification features for classification and recognition. Based on the predicted labels obtained from classification and recognition and the reference labels of the corresponding sample image images, the model parameters of the report generation model are iteratively updated to enhance the model's understanding of the disease category to which the sample image images belong. This allows the model's attention to be redirected to abnormal regions of the sample image images, increasing the model's focus on abnormal regions and reducing its focus on normal regions. This solves the problem of low abnormal region recognition rate caused by the imbalance of positive and negative sample distribution in the existing training dataset and the small proportion of abnormal regions in a complete medical image.

[0029] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0030] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0031] Figure 1 This is a schematic diagram of the R2GEN model architecture;

[0032] Figure 2 This is an optional schematic diagram of an application scenario in the embodiments of this application;

[0033] Figure 3A This is a schematic diagram of the overall architecture of the report generation model provided in the embodiments of this application;

[0034] Figure 3B A flowchart illustrating a training report generation model provided in an embodiment of this application;

[0035] Figure 3C A schematic diagram of the structure of a multi-modal Transformer encoder based on a multilayer perceptron provided in this application embodiment;

[0036] Figure 3D A schematic diagram of the structure of a multimodal Transformer encoder based on an attention mechanism provided in this application embodiment;

[0037] Figure 4 A schematic diagram of the training report generation subnetwork provided in the embodiments of this application;

[0038] Figure 5A A schematic diagram of the process for generating a medical imaging report of a chest X-ray provided in an embodiment of this application;

[0039] Figure 5B A logical diagram illustrating the generation of a medical imaging report for a chest X-ray provided in an embodiment of this application;

[0040] Figure 6 A schematic diagram of the structure of a training device for a report generation model provided in an embodiment of this application;

[0041] Figure 7 This is a schematic diagram of the composition structure of a computer device provided in the embodiments of this application;

[0042] Figure 8 This is a schematic diagram of the structure of a computing device in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0044] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0045] 1. Artificial Intelligence (AI):

[0046] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0047] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0048] With the research and advancement of artificial intelligence technology, AI is being studied and applied in various fields, such as smart homes, intelligent customer service, virtual assistants, smart speakers, intelligent marketing, autonomous driving, robotics, and smart healthcare. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0049] 2. Machine Learning:

[0050] Machine learning is a multidisciplinary field that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0051] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are found in all areas of artificial intelligence, including deep learning, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0052] 3. Computer vision is a comprehensive discipline that integrates computer science, signal processing, physics, applied mathematics, statistics, neurophysiology and other disciplines. It is also a challenging and important research direction in the scientific field.

[0053] This discipline uses various imaging systems to replace visual organs as input methods, with computers handling processing and interpretation instead of the brain. This allows computers to observe and understand the world visually, just like humans. Subfields of computer vision include face detection, face comparison, facial feature detection, blink detection, liveness detection, and fatigue detection.

[0054] 4. Natural Language Processing (NLP) is an important field in computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language.

[0055] Natural Language Processing (NLP) is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it is closely related to linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0056] The design concept of the embodiments of this application is briefly introduced below:

[0057] Image caption generation aims to generate a natural language description for a given image. It is a multimodal task involving both computer vision and natural language processing. In recent years, this technology has begun to be applied in the medical field, with the aim of automatically generating diagnostic reports based on medical images such as CT scans and chest X-rays. Highly accurate medical image reports can not only assist doctors in diagnosis and reduce their workload, but also provide diagnostic results to help patients with self-diagnosis.

[0058] Most commonly used report generation models are based on multimodal encoder-decoder structures, with R2GEN being a classic example. For instance... Figure 1 As shown, a medical image is input into R2GEN. The visual feature extraction module of R2GEN extracts the corresponding visual features from the medical image. Then, a visual encoder-text decoder based on multimodal Transformer is used to convert the visual features into text features. Finally, a word generator based on multilayer perceptron converts the text features into words to form the target medical image report.

[0059] While existing report generation models can produce fluent reports, they suffer from several drawbacks. These include an imbalanced distribution of positive and negative samples in the training dataset and the small percentage of abnormal regions within a complete medical image (e.g., in a chest X-ray, pixels representing lung nodules typically account for only about 5% of the total image pixels). This leads these models to learn from a large number of normal medical image reports, attempting to manipulate the model's loss function. While models trained in this way may appear to have good learning outcomes, they often focus solely on normal areas, neglecting abnormal regions that represent lesions. Consequently, medical image reports generated by existing models mostly describe normal areas, lacking descriptions of abnormal regions, resulting in inconsistent report quality. Low-quality medical image reports, especially those generated from images of infected patients, can significantly impair doctors' accurate diagnosis and treatment.

[0060] In view of this, embodiments of this application propose a training method, apparatus, device, and storage medium for a report generation model. The method includes: sequentially reading sample image images from a training dataset in an iterative manner to train the report generation model until the trained model is output; wherein each iteration includes: extracting features from a read sample image image to obtain initial classification features for determining the disease category to which the sample image image belongs, and multiple initial image features for characterizing the visual information of the sample image image; fusing and encoding the initial classification features and multiple initial image features to obtain corresponding target classification features; classifying and recognizing the target classification features to obtain a predicted label characterizing the disease category to which the sample image image belongs; and updating the model parameters of the report generation model based on the predicted label and the reference label of the sample image image.

[0061] This application embodiment utilizes target classification features for classification and recognition. Based on the predicted labels obtained from classification and recognition and the reference labels of the corresponding sample image images, the model parameters of the report generation model are iteratively updated to enhance the model's understanding of the disease category to which the sample image images belong. This allows the model's attention to be redirected to abnormal regions of the sample image images, increasing the model's focus on abnormal regions and reducing its focus on normal regions. This solves the problem of low abnormal region recognition rate caused by the imbalance of positive and negative sample distribution in the existing training dataset and the small proportion of abnormal regions in a complete medical image.

[0062] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0063] Figure 2 One application scenario is shown, which includes two physical terminal devices 210 and a server 230. Each physical terminal device 210 establishes a communication connection with the server 230 through a wired network or a wireless network.

[0064] In this embodiment, the physical terminal device 210 is a computer device used by a user. Computer devices include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft.

[0065] The server 230 in this application embodiment can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. This application does not impose any restrictions on these services.

[0066] A report generation model is deployed on server 230. Medical images sent by physical terminal device 210 are input into this model. Through neural network processing within the model, a predicted label and a medical image report are obtained for each image. The predicted label represents the disease category to which the image belongs. Then, physical terminal device 210 receives the predicted label and medical image report returned by server 230 and displays this content on display 220 to assist doctors in diagnosing conditions, reducing their workload, or to help patients with self-diagnosis.

[0067] The report generation model is trained iteratively using a training dataset until a preset iteration stopping condition is met, at which point the trained report generation model is output. Each iteration includes: extracting features from a sample image to obtain initial classification features for determining the disease category of the sample image, and multiple initial image features for characterizing the visual information of the sample image; fusing and encoding the initial classification features and multiple initial image features to obtain the corresponding target classification features; classifying and recognizing the target classification features to obtain a predicted label for characterizing the disease category of the sample image; and updating the model parameters of the report generation model based on the predicted label and the reference label of the sample image.

[0068] like Figure 3A As shown, the report generation model mainly consists of two sub-networks: a classification and recognition sub-network and a report generation sub-network. The classification and recognition sub-network includes a visual feature extractor, a multimodal Transformer encoder, and a multi-label classifier. In contrast, the report generation sub-network has a simpler model architecture, mainly composed of a multimodal Transformer decoder.

[0069] Therefore, when training the report generation model, the model parameters of the two sub-networks can be updated together based on the predicted labels and corresponding reference labels of multiple sample images. However, to further improve the model's performance and enhance its sensitivity to abnormal medical image reports, the model parameters of the classification and recognition sub-network can be updated first based on the predicted labels and corresponding reference labels of multiple sample images, and then the trained classification and recognition sub-network can be used to assist in training the report generation sub-network.

[0070] Next, see Figure 3B The flowchart shown is for reference only. We will first introduce the specific implementation process of the first model training method.

[0071] S301: The report generation model reads a sample image from the training dataset.

[0072] To obtain sufficient and rich training data, medical images of different organs were used as sample images. Medical images consisting entirely of normal areas were considered positive samples, which helped the report generation model learn the features of normal areas. Medical images containing abnormal areas were considered negative samples, which helped the model learn the features of abnormal areas.

[0073] In practice, currently available training datasets typically only contain two types of information: medical image images and medical image reports. However, medical image reports contain rich semantic information, such as disease names, lesion locations, and disease severity. Therefore, this application embodiment uses natural language processing technology to extract disease information from the reports and uses it as reference labels for image classification. Examples of such reference labels include: pneumonia, fracture, pneumothorax, pulmonary nodules, mediastinal enlargement, cardiopulmonary enlargement, lung injury, lung shadow, pulmonary edema, pleural effusion, and no findings. Therefore, in this application embodiment, each sample image includes: a medical image, a corresponding medical image report, and a reference label representing the disease category to which the image belongs.

[0074] S302: The report generation model extracts features from a sample image to obtain initial classification features for determining the disease category to which the sample image belongs, as well as multiple initial image features for characterizing the visual information of the sample image.

[0075] As shown in Equation 1, convolutional neural networks such as DenseNet, ResNet, and VGGNet are used as visual feature extractors to extract features from a sample image. This yields initial classification features used to determine the disease category of the sample image, as well as multiple initial image features used to characterize the visual information of the sample image. In Equation 1, X refers to the input sample image, f(X) represents feature extraction from the sample image, CLS refers to the initial classification features, and v... n This refers to the nth initial image feature.

[0076] {CLS,v1,v2,……,v n}=f(X) Formula 1;

[0077] S303: The report generation model fuses and encodes the initial classification features with multiple initial image features to obtain the corresponding target classification features.

[0078] To enable the model to fully understand the image features, as shown in Equation 2, the multimodal Transformer encoder first encodes multiple initial image features to obtain the target image features corresponding to each initial image feature; then, it fuses the initial classification features with each target image feature to obtain the corresponding target classification features. In Equation 2, v... n This refers to the nth initial image feature, Encoder({v1,v2,……,v...). n} indicates that the initial image features are encoded, v i 3 This refers to the target image feature output by the i-th layer in the 3rd layer.

[0079]

[0080] A multimodal Transformer encoder consists of multiple Transformer encoders, such as... Figure 3C As shown, each Transformer encoder is built upon a multilayer perceptron. Multiple feature extractions are performed on multiple initial image features using multiple multilayer perceptrons to obtain the corresponding target image features.

[0081] The human brain possesses a selective ability: when receiving a large amount of input information, it chooses a small portion of useful information for focused processing, while ignoring other input information, ensuring that it can function systematically. In cognitive neuroscience, this selective ability is called "attention," an indispensable and complex cognitive function.

[0082] Similarly, when the computing power of neural networks is limited, the attention mechanism explores the relationship between input and output, allowing the neural network to shift from "global attention" to "local attention," enabling the neural network to allocate computing resources to more important tasks and effectively solving the problem of information overload in neural networks.

[0083] Therefore, as Figure 3D As shown, each Transformer encoder can also be built based on an attention mechanism, and each encoder includes structures such as attention units and normalization layers. Specifically, multiple initial image features are first input into the attention unit, and then the attention features extracted by the attention unit are input into the feedforward neural network for feature extraction. The above process is repeated multiple times to obtain the corresponding target image features.

[0084] Each attention unit contains three attention weight matrices: a query weight matrix, a key weight matrix, and a value weight matrix. An initial image feature matrix, composed of multiple initial image features, is subjected to matrix operations with the three attention weight matrices of that unit to obtain the corresponding query matrix, key matrix, and value matrix. Then, a matrix operation is performed between the query matrix and the transpose of the key matrix, and the result is mapped to the (0,1) interval. Finally, a matrix operation is performed between the mapped feature matrix and the value matrix to obtain the image association feature matrix.

[0085] Each image association feature reflects the correlation between the corresponding initial image feature and other initial image features. When encoding one of the initial image features X, the neural network will also pay attention to other initial image features that are highly correlated with the initial image feature X, even if the initial image feature is far away from the initial image feature X in the time series, thus breaking the limitations of the neural network.

[0086] After obtaining the features of each target image, the multimodal Transformer encoder determines the classification contribution of each target image feature in the classification and recognition subnetwork based on the initial classification features and the features of each target image; then, based on each target image feature and the corresponding classification contribution, it obtains the target classification features.

[0087] The process of determining the classification contribution of each target image feature is as follows:

[0088] As shown in Formula 3, the multimodal Transformer encoder multiplies the initial classification features with each target image feature to determine the classification contribution features of each target image feature in the classification recognition subnetwork; then, it maps each classification contribution feature to the same probability space, and determines each probability distribution value in the probability space as the classification contribution of each target image feature.

[0089] In Formula 3, CLS refers to the initial classification feature. This refers to the target image feature output by the i-th element in the 3rd layer, processed using softmax representation mapping, α i This refers to the classification contribution of the i-th target image feature.

[0090]

[0091] The process of obtaining the target classification features is as follows:

[0092] As shown in Equation 4, the multimodal Transformer encoder multiplies each target image feature by its corresponding classification contribution to obtain multiple candidate classification features for the sample image. These candidate features are then summed to obtain the target classification feature. Therefore, the target classification feature is not a feature specific to a single pixel region, but rather an image feature representing the visual information of the entire image, which can be used to classify and recognize the input sample image.

[0093] α in Formula 4 i This refers to the classification contribution of the i-th target image feature. This refers to the target image feature of the i-th output in the 3rd layer, and refers to the target classification feature.

[0094]

[0095] S304: The report generation model classifies and identifies the target classification features, obtains a predicted label to represent the disease category to which the sample image belongs, and updates its own model parameters based on the predicted label and the reference label of the sample image.

[0096] The fused target classification features are input into a multi-label classifier. The multi-label classifier extracts features from the target classification features to obtain the predicted probability of the sample image belonging to each disease category. Disease categories whose predicted probabilities exceed a set threshold are determined as the predicted labels of the sample image.

[0097] For example, inputting a chest X-ray image yields the corresponding target classification features, and classification is performed based on these features. The recognition results are shown in Table 1. Assuming a threshold value of 0.67 is set, the predicted label for this chest X-ray image would be pneumonia.

[0098] Table 1

[0099] 0.7 0.01 0.1 0.03 …… 0.1 0.01

[0100] After obtaining the predicted label of the sample image, the classification balance loss value for this iteration is determined based on the predicted label and the reference label of the sample image. Then, the model parameters of the report generation model are updated based on the obtained classification balance loss value.

[0101] In this embodiment, the classification balance loss function can be selected from Focal Loss, Asymmtrical Loss, etc. Equation 5 shows the calculation formula for Focal Loss, which is used to calculate the classification balance loss. In Equation 5, y refers to the reference label of the positive or negative sample, P refers to the predicted label, β refers to the penalty factor, and Loss... cls This refers to the classification balance loss value. The closer the predicted label is to the reference label, the smaller the classification balance loss value of the report generation model; conversely, the closer the predicted label is to the reference label, the larger the classification balance loss value. Under this model training mechanism, the report generation model will also fit appropriately for disease categories with a smaller proportion, thus training a model with better performance.

[0102] Loss cls =-[y(1-P) β logP+(1-P)P β Formula 5;

[0103] S305: Determine whether the model has been trained. If yes, output a training completion report and generate the model; otherwise, return to step 301.

[0104] The model training is complete when at least one of the following conditions is met, and the adjusted report generation model for the current round is output; otherwise, return to step 301 and begin the next round of iterative training:

[0105] (1) The classification balance loss value is less than or equal to the set loss value;

[0106] (2) The current iteration round has reached the set iteration round;

[0107] (3) All sample images in the training dataset have been read.

[0108] In the second model training method, the classification and recognition sub-network is first trained using steps 301-305, and then an auxiliary training report is generated based on the trained classification and recognition sub-network to create a new sub-network. Since the specific implementation process of the above steps has already been described above, it will not be repeated here. Next, please refer to... Figure 4 The flowchart shown illustrates the specific training process of the report generation subnetwork.

[0109] S401: The classification and recognition subnetwork reads a sample image from the training dataset.

[0110] The two networks can use the same training dataset or different training datasets; this application does not impose too many restrictions on this.

[0111] S402: The classification and recognition sub-network extracts features from a sample image to obtain initial classification features for determining the disease category to which the sample image belongs, as well as multiple initial image features for characterizing the visual information of the sample image.

[0112] S403: The classification and recognition subnetwork encodes multiple initial image features to obtain corresponding target image features, and sends each target image feature as a feature to be decoded to the report generation subnetwork.

[0113] S404: The report generation subnetwork decodes the features of each image to be decoded, obtains the corresponding words, and updates the model parameters of the report generation task based on the features of each image to be decoded and the corresponding words.

[0114] As shown in Equation 6, a multimodal Transformer decoder is used to decode the features of each target image to obtain the corresponding words. In Equation 6, z... i This refers to the i-th image feature to be decoded, w i This refers to the i-th word obtained through decoding.

[0115] {w1,w2,……,w i} = Decoder({z1,z2,……,z) i}) Formula 6;

[0116] The structure of the multimodal Transformer encoder and decoder is the same. If the multimodal Transformer encoder is based on a multilayer perceptron, then the multimodal Transformer decoder is also based on a multilayer perceptron; if the multimodal Transformer encoder is based on an attention mechanism, then the multimodal Transformer decoder is also based on an attention mechanism. Therefore, the decoding process is similar to the encoding process. Since the encoding process has been described in detail above, the specific decoding process will not be repeated here.

[0117] As shown in Equation 7, based on the features of each image to be decoded and the corresponding words, the decoding loss value for this iteration is determined. Then, based on the obtained decoding loss value, the model parameters of the report generation sub-network are updated. Wherein, z in Equation 7... i This refers to the i-th image feature to be decoded, w i This refers to the i-th decoded word, l refers to the total length of the text, and ρ θθ refers to the word prediction value, and Loss(θ) refers to the decoding loss value.

[0118]

[0119] However, in order to further optimize the network performance of the report generation subnetwork, Formula 3 can be used to obtain the classification contribution of each target image feature, arrange each target image feature in descending order of classification contribution, and determine the target image features with the top m classification contributions as the image features to be decoded and perform decoding processing on them.

[0120] In addition, Formula 3 can be used to obtain the classification contribution of each target image feature. The target image features are then arranged from high to low according to their classification contribution to obtain the top m target image features with the highest classification contribution. As shown in Formula 8, the top m target image features with the highest classification contribution and the corresponding initial image features are determined as the image features to be decoded, and then decoded.

[0121]

[0122] S405: Determine whether the report generation subnetwork has been trained. If yes, output the trained report generation subnetwork; otherwise, return to step 401.

[0123] The report generation subnetwork is considered complete when at least one of the following conditions is met, and the adjusted report generation subnetwork for the current round is output; otherwise, return to step 401 and begin the next round of iterative training:

[0124] (1) The classification balance loss value is less than or equal to the set loss value;

[0125] (2) The current iteration round has reached the set iteration round;

[0126] (3) All sample images in the training dataset have been read.

[0127] For both sub-networks mentioned above, the AdamW optimizer was used to optimize the model parameters of the sub-networks, with the initial learning rate set to 1×10⁻⁶. -4 Then, slow start and cosine function are used to adjust the learning rate.

[0128] Since the report generation subnetwork is trained on top of the already trained classification subnetwork, the number of iterations for the report generation subnetwork can be fewer than that for the classification subnetwork. For example, the classification subnetwork can have 200 iterations, while the report generation subnetwork can have 100 iterations.

[0129] The report generation model can be applied in medical systems to automatically generate corresponding medical imaging reports based on medical images acquired by machines, such as ultrasound, MRI, colonoscopy, endoscopy, CT, and magnetic resonance imaging (MRI). This model can also be used in online mini-programs to generate online medical imaging reports, making it convenient for patients to view their reports and assisting them in self-diagnosis.

[0130] See Figure 5A The flowchart shown is consistent with Figure 5B The logic diagram shown illustrates the process of inputting a chest X-ray into the report generation model to generate a medical imaging report, as follows:

[0131] S501: Extract features from chest X-rays to obtain initial classification features and multiple initial image features;

[0132] S502: Encode multiple initial image features to obtain the target image features corresponding to each initial image feature;

[0133] S503: The initial classification features are fused with the features of each target image to obtain a target classification feature, and the target classification feature is classified and identified to obtain a predictive label for representing the disease category to which the chest X-ray belongs.

[0134] S504: The top m class contribution target image features and their corresponding initial image features are determined as the image features to be decoded;

[0135] S505: Decode the features of each image to be decoded and generate a corresponding medical image report;

[0136] S506: Output predicted labels for chest X-rays and corresponding medical imaging reports.

[0137] Based on the same inventive concept as the above-described method embodiments, this application also provides a training apparatus for a report generation model. For example... Figure 6 As shown, the training device 600 for the report generation model may include:

[0138] The model training unit 601 is used to iteratively read each sample image from the training dataset, train the report generation model, and output the trained report generation model until it is completed. Each iteration includes:

[0139] The feature extraction unit 602 is used to extract features from a read sample image to obtain initial classification features for determining the disease category to which the sample image belongs, and multiple initial image features for characterizing the visual information of the sample image.

[0140] The encoding unit 603 is used to perform fusion encoding processing on the initial classification features and multiple initial image features to obtain the corresponding target classification features;

[0141] Classification unit 604 is used to classify and identify target classification features to obtain a predicted label that represents the disease category to which a sample image belongs;

[0142] The parameter adjustment unit 605 is used to update the model parameters of the report generation model based on the predicted label and the reference label of a sample image.

[0143] Optionally, encoding unit 603 is used for:

[0144] Encode multiple initial image features to obtain the target image features corresponding to each initial image feature;

[0145] The initial classification features are fused with the features of each target image to obtain the corresponding target classification features.

[0146] Optionally, encoding unit 603 is used for:

[0147] Based on the initial classification features and the features of each target image, determine the classification contribution of each target image feature in the classification and recognition subnetwork;

[0148] Based on the features of each target image and its corresponding classification contribution, target classification features are obtained.

[0149] Optionally, encoding unit 603 is used for:

[0150] The initial classification features are multiplied by the features of each target image to determine the classification contribution features of each target image feature in the classification and recognition subnetwork.

[0151] Each classification contribution feature is mapped to the same probability space, and each probability distribution value in the probability space is determined as the classification contribution of each target image feature.

[0152] Optionally, encoding unit 603 is used for:

[0153] Each target image feature is multiplied by its corresponding classification contribution to obtain multiple candidate classification features for a sample image.

[0154] The target classification feature is obtained by summing up multiple candidate classification features.

[0155] Optionally, classification unit 604 is used for:

[0156] Feature extraction is performed on the target classification features to obtain the predicted probability of a sample image belonging to each disease category;

[0157] The disease category corresponding to the predicted probability exceeding a set threshold is determined as the predicted label of a sample image.

[0158] Optionally, the parameter adjustment unit 605 is used for:

[0159] Based on the predicted label and the reference label of a sample image, the classification balance loss value for this iteration is determined.

[0160] Based on the obtained classification balance loss value, update the model parameters of the report generation model.

[0161] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0162] Having introduced the training method and apparatus for the report generation model according to exemplary embodiments of this application, the computer device according to another exemplary embodiment of this application will now be described.

[0163] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0164] Based on the same inventive concept as the above-described method embodiments, this application also provides a computer device, see reference. Figure 7 As shown, the computer device 700 may include at least a processor 701 and a memory 702. The memory 702 stores program code, which, when executed by the processor 701, causes the processor 701 to perform the steps of any of the above-described training methods for the report generation model.

[0165] In some possible implementations, the computing device according to this application may include at least one processor and at least one memory. The memory stores program code that, when executed by the processor, causes the processor to perform the steps in the training method for the report generation model according to various exemplary embodiments of this application described above. For example, the processor may perform actions such as... Figure 3B The steps are shown in the figure.

[0166] The following reference Figure 8 To describe a computing device 800 according to this embodiment of the present application. Figure 8The computing device 800 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0167] like Figure 8 As shown, the computing device 800 is presented in the form of a general-purpose computing device. The components of the computing device 800 may include, but are not limited to: at least one processing unit 801, at least one storage unit 802, and a bus 803 connecting different system components (including storage unit 802 and processing unit 801).

[0168] Bus 803 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or a local bus using any of the various bus structures.

[0169] Storage unit 802 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 8021 and / or cache memory unit 8022, and may further include read-only memory (ROM) 8023.

[0170] Storage unit 802 may also include a program / utility 8025 having a set (at least one) program module 8024, such program module 8024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0171] The computing device 800 can also communicate with one or more external devices 804 (e.g., keyboard, pointing device, etc.), one or more devices that enable a user to interact with the computing device 800, and / or any device that enables the computing device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 805. Furthermore, the computing device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 806. As shown, network adapter 806 communicates with other modules for the computing device 800 via bus 803. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the computing device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0172] Based on the same inventive concept as the above-described method embodiments, various aspects of the training method for the report generation model provided in this application can also be implemented as a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to execute the steps in the training method for the report generation model according to the various exemplary embodiments of this application described above. For example, the computer device can execute, for example, Figure 3B The steps are shown in the figure.

[0173] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0174] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0175] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A training method for a report generation model, characterized in that, include: The system employs an iterative approach, sequentially reading image samples from the training dataset to train the report generation model until the fully trained model is output. Each iteration includes: Feature extraction is performed on a sample image to obtain initial classification features for determining the disease category to which the sample image belongs, and multiple initial image features for characterizing the visual information of the sample image. The multiple initial image features are encoded to obtain multiple target image features; each target image feature represents the association relationship between an initial image feature and multiple other initial image features. The initial classification features are multiplied by the multiple target image features respectively to determine the classification contribution features of the multiple target image features in the classification and recognition subnetwork. The classification contribution of the multiple target image features in the classification and recognition subnetwork is determined by mapping the multiple classification contribution features. Based on the multiple target image features and their corresponding classification contributions, target classification features are obtained; The target classification features are classified and identified to obtain a predicted label representing the disease category to which the sample image belongs. Based on the predicted label and the reference label of the sample image, the model parameters of the report generation model are updated. The report generation model is used to generate a medical image report, which is used to interpret the disease information implied in the classification results of the medical image.

2. The method as described in claim 1, characterized in that, The process of obtaining target classification features based on the multiple target image features and their corresponding classification contributions includes: Each of the target image features is multiplied by its corresponding classification contribution to obtain multiple candidate classification features for the sample image. The target classification feature is obtained by summing up multiple candidate classification features.

3. The method according to any one of claims 1 to 2, characterized in that, The process of classifying and identifying the target classification features to obtain a predicted label characterizing the disease category to which the sample image belongs includes: Feature extraction is performed on the target classification features to obtain the predicted probability that a sample image belongs to each disease category; The disease category corresponding to the predicted probability exceeding a set threshold is determined as the predicted label of the sample image.

4. The method as described in claim 3, characterized in that, The process of updating the model parameters of the report generation model based on the predicted label and the reference label of the sample image includes: Based on the predicted label and the reference label of the sample image, the classification balance loss value for this iteration is determined; Based on the obtained classification balance loss value, update the model parameters of the report generation model.

5. A training device for a report generation model, characterized in that, include: The model training unit is used to iteratively read each sample image from the training dataset, train the report generation model, and output the trained report generation model until the model is fully trained. Each iteration includes: The feature extraction unit is used to extract features from a read sample image to obtain initial classification features for determining the disease category to which the sample image belongs, and multiple initial image features for characterizing the visual information of the sample image. An encoding unit is used to encode the plurality of initial image features to obtain a plurality of target image features; each target image feature represents the association relationship between an initial image feature and a plurality of other initial image features. The initial classification features are multiplied by the multiple target image features respectively to determine the classification contribution features of the multiple target image features in the classification and recognition subnetwork. The classification contribution of the multiple target image features in the classification and recognition subnetwork is determined by mapping the multiple classification contribution features. Based on the multiple target image features and their corresponding classification contributions, target classification features are obtained; A classification unit is used to classify and identify the target classification features to obtain a predicted label that characterizes the disease category to which the sample image belongs; The parameter adjustment unit is used to update the model parameters of the report generation model based on the predicted label and the reference label of the sample image; the report generation model is used to generate a medical image report, and the medical image report is used to interpret the disease information implied in the classification results of the medical image.

6. The apparatus as claimed in claim 5, characterized in that, The encoding unit is used for: Each of the target image features is multiplied by its corresponding classification contribution to obtain multiple candidate classification features for the sample image. The target classification feature is obtained by summing up multiple candidate classification features.

7. A computer device, characterized in that, It includes a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, It includes program code that, when run on a computer device, causes the computer device to perform the steps of the method according to any one of claims 1 to 4.

9. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 4.

Images