A method of generating a medical report and related apparatus

Abstract

The embodiment of the application discloses a method for generating a medical report and a related device, at least relates to computer vision technology and machine learning technology in artificial intelligence, and can be applied to the fields of medical cloud, artificial intelligence and the like. The method comprises the following steps: acquiring a first medical image of a target object, the first medical image being related to a first disease of the target object; performing visual feature extraction processing on each first lesion area in the first medical image based on a preset feature extraction model to obtain a first visual feature of each first lesion area; acquiring a first word embedding feature vector; and processing the first visual feature and the first word embedding feature vector based on a report generation model to generate a medical report related to the first disease. Through the above method, more visual features related to the lesion area are considered, richer feature information is provided for model training, and the quality of the medical report is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method and related apparatus for generating medical reports. Background Technology

[0002] Medical imaging, also known as medical images, refers to non-invasive images of the internal tissues of the human body or a part of the body, helping doctors understand a patient's health condition. Medical images come with corresponding medical reports, which contain the analysis results of the medical images. For example, a medical report may include information determined based on the medical images, such as the location of the disease, the extent of the lesion, and the organs affected.

[0003] In existing solutions, graph convolution methods are used to learn the relationships between diseases, enabling information to be transferred from common diseases to rare diseases, thereby generating medical reports for rare diseases. However, during the training process of the medical report generation model, the training dataset needs to include not only disease annotations for common diseases but also those for rare diseases. This means that in scenarios where rare disease annotations are unavailable, the graph convolution method cannot generate corresponding medical reports for those rare diseases. Furthermore, when using graph convolution to extract features from medical images, only global features are extracted, failing to consider more details related to the lesion area. This results in poor quality medical reports that cannot provide doctors with a satisfactory understanding of the relevant disease conditions. Summary of the Invention

[0004] This application provides a method and related apparatus for generating medical reports. It is applicable not only to scenarios where corresponding medical reports cannot be generated due to the lack of disease labeling information for the primary disease, but also takes into account more visual features related to the lesion area, providing richer feature information for model training, thereby improving the quality of medical reports.

[0005] In a first aspect, embodiments of this application provide a method for generating a medical report. The method includes acquiring a first medical image of a target object, the first medical image being related to a first disease of the target object; performing visual feature extraction processing on each first lesion region in the first medical image based on a preset feature extraction model to obtain a first visual feature of each first lesion region; acquiring a first word embedding feature vector, the first word embedding feature vector being used to indicate the association between the first disease and a second disease; inputting the first visual feature and the first word embedding feature vector into a report generation model to obtain a medical report related to the first disease. The report generation model is a machine learning model obtained by iteratively training a preset initial model with the goal of generating a medical report related to the first disease, using the second visual features of each second lesion region in a sample medical image related to the second disease and the word embedding feature vector between the first and second diseases as training data.

[0006] Secondly, embodiments of this application provide a medical report generation apparatus. This medical report generation apparatus includes an acquisition unit and a processing unit. The acquisition unit is used to acquire a first medical image of a target object, the first medical image being related to a first disease of the target object; the processing unit is used to perform visual feature extraction processing on each first lesion region in the first medical image based on a preset feature extraction model to obtain a first visual feature of each first lesion region; the acquisition unit is used to acquire a first word embedding feature vector, the first word embedding feature vector being used to indicate the association between the first disease and a second disease; the processing unit is used to input the first visual feature and the first word embedding feature vector into a report generation model to obtain a medical report related to the first disease. The report generation model is a machine learning model obtained by iteratively training a preset initial model with the goal of generating a medical report related to the first disease, using the second visual features of each second lesion region in a sample medical image related to the second disease and the word embedding feature vector between the first and second diseases as training data.

[0007] In some examples, the processing unit is also used to display the bounding box of each first lesion region in the medical report after inputting the first visual features and the first word embedding feature vector into the report generation model to obtain a medical report related to the first disease.

[0008] In other examples, the processing unit is also used to determine the region information of the corresponding first lesion region based on the first visual features before displaying the bounding box of each first lesion region in the medical report; determine the location and size of the first lesion region based on the region information of the first lesion region; and generate the bounding box of the first lesion region based on the location and size of the first lesion region.

[0009] In other examples, the processing unit is also used to, after inputting the first visual features and the first word embedding feature vector into the report generation model to obtain a medical report related to the first disease, label the category of the first lesion region in the medical report. The category of the first lesion region is used to indicate the disease type to which the corresponding first lesion region belongs.

[0010] In other examples, the processing unit is used to: concatenate each first visual feature with the first word embedding feature vector to obtain each concatenated visual feature; calculate the predicted category probability value of each first lesion region; determine the target visual feature based on each concatenated visual feature and the predicted category probability value of the corresponding first lesion region; and input the target visual feature into the report generation model to generate a medical report related to the first disease.

[0011] In other examples, the acquisition unit is used to: acquire disease information of a first disease and disease information of a second disease. The processing unit is used to determine a first word embedding feature vector based on the disease information of the first disease and the disease information of the second disease.

[0012] In other examples, the acquisition unit is also used to acquire sample medical images related to a second disease before inputting the first visual features and the first word embedding feature vector into the report generation model to obtain a medical report related to the first disease. The processing unit is also used to: perform visual feature extraction processing on each second lesion region in the sample medical images based on a preset feature extraction model to obtain a second visual feature for each second lesion region; determine the semantic features of each second lesion region based on a preset projection model and each second visual feature; determine the word embedding feature vector between the first and second diseases based on the disease information of the first and second diseases; determine a target loss value based on the semantic features of the second lesion region, the word embedding feature vector, and the second visual features; and adjust the model parameters of the preset initial model based on the target loss value to obtain the report generation model.

[0013] In other examples, the processing unit is used to: calculate a first loss value based on the semantic features and word embedding feature vectors of the second lesion region; calculate a second loss value based on the semantic features and second visual features of the second lesion region; calculate a third loss value based on the probability value of each word in the medical prediction report, wherein the medical prediction report is a report generated based on the second visual features and word embedding feature vectors; and determine a target loss value based on the first loss value, the second loss value, and the third loss value.

[0014] In other examples, the processing unit is used to: calculate a first similarity between the word embedding feature vector and the semantic features of the second lesion region; calculate a second similarity between every two word embedding feature vectors in the word embedding feature vector; and calculate the difference between the first similarity and the second similarity to obtain a first loss value. In other examples, the processing unit is used to: determine the predicted label of the corresponding second lesion region based on the semantic features and second visual features of the second lesion region; calculate the difference between the predicted label and the preset labeled label of the second lesion region to obtain the label loss value; determine the predicted bounding box of the corresponding second lesion region based on the second visual features; calculate the difference between the predicted bounding box and the preset labeled bounding box of the second lesion region to obtain the bounding box loss value; and determine the second loss value based on the label loss value and the bounding box loss value.

[0015] In other examples, the processing unit is used to: determine the predicted probability value of the semantic features based on the similarity between the semantic features of the second lesion region and the word embedding feature vector; perform category prediction processing on the second visual features based on a preset classification branch model to obtain the predicted probability value of the category of the corresponding second lesion region; and determine the predicted label of the corresponding second lesion region based on the predicted probability value of the semantic features and the predicted probability value of the category of the corresponding second lesion region. In other examples, the processing unit is used to: generate a medical prediction report related to a second disease based on the second visual features and word embedding feature vectors; calculate the probability value of each word appearing in the medical prediction report; and determine a third loss value based on the probability value of each word appearing.

[0016] A third aspect of this application provides a medical report generation apparatus, including: a memory, an input / output (I / O) interface, and a processor. The memory stores program instructions. The processor executes the program instructions in the memory to perform the method for generating a medical report corresponding to the embodiment of the first aspect described above.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method corresponding to the embodiments of the first aspect described above.

[0018] The fifth aspect of this application provides a computer program product containing instructions that, when run on a computer or processor, causes the computer or processor to execute the method described above for performing the implementation method of the first aspect.

[0019] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0020] In this embodiment, since the first medical image of the target object is related to the first disease, after acquiring the first medical image of the target object, visual feature extraction processing can be performed on each first lesion region in the first medical image based on a preset feature extraction model to obtain the first visual feature of each first lesion region. Then, the first word embedding feature vector is obtained, which is used to indicate the association between the first disease and the second disease. Moreover, since the report generation model is a machine learning model trained by iteratively training a preset initial model with the visual features of each second lesion region in the sample medical image related to the second disease and the word embedding feature vector between the first disease and the second disease as training data, the first visual feature and the first word embedding feature vector are processed based on the report generation model to generate a medical report related to the first disease. In the above way, the disease annotation information of the first disease does not need to be considered during the training process. Instead, the visual features of the lesion regions in the medical image related to the second disease and the word embedding feature vector between the first disease and the second disease are comprehensively considered to train the corresponding report generation model. In this way, the report generation model processes the visual features of lesion regions in medical images related to the first disease, as well as word embedding feature vectors, to generate medical reports related to the first disease. This approach is not only applicable to scenarios where disease annotation information for the first disease is unavailable, preventing the generation of corresponding medical reports, but also considers more visual features related to lesion regions, providing richer feature information for model training and thus improving the quality of medical reports. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram of an implementation environment provided by an embodiment of this application is shown;

[0023] Figure 2 This illustration shows a framework flowchart provided by an embodiment of this application;

[0024] Figure 3 A flowchart of a method for generating a medical report according to an embodiment of this application is shown;

[0025] Figure 4 The diagram illustrates the training flowchart of the report generation model provided in an embodiment of this application;

[0026] Figure 5 This illustration shows a schematic diagram of a medical report provided in an embodiment of this application;

[0027] Figure 6 Another schematic diagram of a medical report provided in an embodiment of this application is shown;

[0028] Figure 7 Another flowchart of the method for generating medical reports provided in this application embodiment is shown;

[0029] Figure 8 A schematic diagram of the structure of the medical report generation device provided in the embodiments of this application is shown;

[0030] Figure 9 A schematic diagram of the hardware structure of the medical report generation device provided in the embodiments of this application is shown. Detailed Implementation

[0031] This application provides a method and related apparatus for generating medical reports. It is applicable not only to scenarios where corresponding medical reports cannot be generated due to the lack of disease labeling information for the primary disease, but also takes into account more visual features related to the lesion area, providing richer feature information for model training, thereby improving the quality of medical reports.

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] The method for generating medical reports provided in this application is based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making functions.

[0035] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0036] In the embodiments of this application, the main artificial intelligence technologies involved include the aforementioned computer vision (CV) and machine learning (ML) technologies. For example, it may involve generating medical reports in computer vision technology; it may also involve deep learning in machine learning, including neural networks such as convolutional neural networks (CNNs).

[0037] The method for generating medical reports provided in this application can be applied to medical report generation devices with data processing capabilities, such as terminal devices or servers, etc., without specific limitations in this application. Terminal devices may include, but are not limited to, smartphones, desktop computers, laptops, tablets, smart speakers, in-vehicle devices, smartwatches, etc. Servers may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing cloud computing services, etc., without specific limitations in this application. Furthermore, the mentioned terminal devices and servers can be directly or indirectly connected via wired or wireless communication, without specific limitations in this application.

[0038] The aforementioned medical report generation device possesses the processing capabilities to implement computer vision technology. Computer vision technology is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for target recognition, trajectory tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technology typically includes image processing, medical report generation, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0039] Furthermore, this medical report generation device also possesses machine learning capabilities. Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0040] The method for generating medical reports provided in this application uses an artificial intelligence model, which mainly involves the application of technologies such as neural networks, machine learning, and computer vision to generate disease-related medical reports through neural networks.

[0041] For example, the method for generating medical reports provided in this application embodiment can also be implemented based on cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize data computation, storage, processing, and sharing.

[0042] Cloud technology is a collective term for network technology, information technology, integration technology, management platform technology, and application technology applied to the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will all require robust system support, which can only be achieved through cloud computing.

[0043] Cloud computing refers to the delivery and usage model of IT infrastructure, meaning obtaining necessary resources through a network in an on-demand and easily scalable manner. In a broader sense, cloud computing also refers to the delivery and usage model of services, meaning obtaining necessary services through a network in an on-demand and easily scalable manner. These services can be IT and software related, internet-related, or other services. Cloud computing is a product of the convergence and development of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. Driven by the development of the internet, real-time data streams, the diversification of connected devices, and the demands of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Unlike previous parallel and distributed computing, the emergence of cloud computing will fundamentally revolutionize the entire internet model and enterprise management model.

[0044] For example, the method for generating medical reports provided in this application can also be applied to scenarios such as medical cloud. Medical cloud refers to the creation of a healthcare service cloud platform using cloud computing, mobile technology, multimedia, 4G, big data, and the Internet of Things, combined with medical technology, thereby achieving the sharing of medical resources and expanding the scope of medical care. Because of the application and integration of cloud computing technology, medical cloud improves the efficiency of medical institutions and facilitates access to medical care for residents. For example, hospital appointment registration, electronic medical records, and medical insurance are all products of the integration of cloud computing and the medical field. Medical cloud also has advantages such as data security, information sharing, dynamic expansion, and a global layout.

[0045] Figure 1 A schematic diagram of an implementation environment provided by an embodiment of this application is shown.

[0046] like Figure 1 As shown, the implementation environment may include a model training device and a model usage device. The model training device may be a computer device such as a computer or server, used to train the report generation model. The described report generation model can be understood as a model used to generate medical reports. In this embodiment, the report generation model is a machine learning model that automatically generates corresponding medical reports based on medical images. The model training device can use machine learning to train the report generation model to achieve good automated medical report generation performance.

[0047] Furthermore, the trained report generation model can be deployed on the device where the model is used. This device can be a terminal device or a server, etc., and is not specifically limited in this embodiment. The described terminal devices may include, but are not limited to, smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart voice interaction devices, smart home appliances, in-vehicle terminals, aircraft, medical devices, multimedia playback devices, etc. When a medical report needs to be generated, the device using the model can automatically generate a disease-related medical report through this report generation model.

[0048] It should be understood that, in practical applications, it can also be applied to other than... Figure 1 Other use cases besides the implementation environment shown are not specifically limited in the embodiments of this application.

[0049] The method for generating medical reports according to embodiments of this application will now be described in detail with reference to the accompanying drawings. This method for generating medical reports can be applied to the framework flowchart shown in Figure 2. Figure 2 As shown, after acquiring medical images of the target object's disease at different stages (e.g., ... After N≥1 and N is an integer, each medical image can be extracted first using a preset feature extraction model (such as a feature extractor). The global features are then extracted. Next, a region proposal network (RPN) model is used to generate bounding boxes for lesion regions in each medical image, and a region of interest (ROI) pooling model is used to extract the visual features of each lesion region. Visual characteristics of all lesion areas. The following operations ① to ③ can be performed. Operation ①: Using the bounding box regression branch model Visual features of the lesion area The process involves detecting the location and size of the lesion region and generating corresponding bounding boxes. Operation ②: Using a classification branch model. Visual features of the lesion area The process involves processing to predict the category of each lesion region. Operation ③: Semantic prediction branch. The model will incorporate visual features Mapping to a semantic space to obtain the semantic features of the lesion region. Furthermore, by learning the relationship between common and rare diseases, corresponding word embedding feature vectors are obtained. Finally, visual features Its corresponding word embedding feature vector Merging to obtain target visual features The data is input into the report generation model to generate a corresponding medical report for the disease. It should be noted that the specific details of the report generation model described herein can be found in subsequent sections. Figure 4 We will understand the content described in the text, but will not elaborate further here.

[0050] Figure 3 A flowchart illustrating a method for generating medical reports according to an embodiment of this application is shown. Figure 3 As shown, the method for generating a medical report may include the following steps:

[0051] 301. Obtain the first medical image of the target object, which is related to the target object's first disease.

[0052] In this example, the first disease described can be understood as a rare disease or other uncommon disease, etc., and is not specifically limited in this application. Additionally, the described target object can be understood as a patient, diseased person, animal, or plant, etc., and is not specifically limited in this embodiment.

[0053] The first medical image refers to a medical image related to the first disease acquired for the target object. The described medical image may be an X-ray image, a computed tomography (CT) image, a positron emission tomography (PET) image, a magnetic resonance imaging (MRI) image, a medical ultrasound image, a medical microscope image, etc., and this application embodiment does not specify a particular type. Furthermore, this application embodiment does not specifically limit the human body part targeted by the medical image; for example, it may include, but is not limited to, the abdomen, brain, limbs, internal organs, bones, or blood vessels. For example, the medical image may be a medical image of an animal such as a cat or dog, or a plant such as a flower or tree, and this application embodiment does not specify a particular type.

[0054] 302. Based on the preset feature extraction model, perform visual feature extraction processing on each first lesion region in the first medical image to obtain the first visual features of each first lesion region.

[0055] In this example, the preset feature extraction model can be a region-of-interest pooling model or other types of feature extraction models, etc., which are not specifically limited in this embodiment. After obtaining the first medical image of the target object, the position and size of each first lesion region in the first medical image can be determined by a region proposal network model, etc., thereby obtaining the bounding box of each first lesion region. Then, for each first lesion region in the first medical image, visual feature extraction processing can be performed by the preset feature extraction model to obtain the first visual features of the respective first lesion region. Exemplarily, the described first visual features may include, but are not limited to, image features such as texture features, shape features, and color features corresponding to the first lesion region. The mentioned color features describe the surface properties of the image corresponding to the first lesion region. Texture features describe the properties of the scene corresponding to the image corresponding to the first lesion region. Shape features may include the contour features of the first lesion region, etc., which are not specifically limited in this application.

[0056] 303. Obtain the first word embedding feature vector, which is used to indicate the association between the first disease and the second disease.

[0057] In this example, since no labeling information for the first disease was collected, the correlation between the first and second diseases can be explored. Specifically, the relevant information about the first disease can be mined by embedding the feature vector of the first word, thereby guiding the generation of the corresponding medical report. It should be noted that the described second disease can be understood as a common illness, such as diabetes, sprains, or meniscus injuries; this application does not impose a specific limitation. Furthermore, there is no overlap between the categories of the second and first diseases.

[0058] For example, for a first disease, disease information related to the first disease can be obtained, such as the name and symptoms of the first disease. Similarly, for a second disease, disease information related to the second disease can also be obtained, such as the name and symptoms of the second disease. In this way, after obtaining the disease information of the first disease and the disease information of the second disease, the embedding feature vector of the first word can be determined based on the disease information of the first disease and the disease information of the second disease.

[0059] 304. Input the first visual feature and the first word embedding feature vector into the report generation model to obtain a medical report related to the first disease. The report generation model is a machine learning model that is trained by iteratively training a preset initial model with the visual features of each second lesion region in the sample medical images related to the second disease and the word embedding feature vector between the first disease and the second disease as training data.

[0060] In this example, the described report generation model is a machine learning model trained iteratively on a pre-set initial model, using the visual features of each second lesion region in sample medical images related to the second disease, and the word embedding feature vectors between the first and second diseases as training data, with the goal of generating medical reports related to the first disease. For details, please refer to the subsequent sections. Figure 4 The process of model training shown will be understood, but will not be elaborated upon here.

[0061] Thus, after obtaining the first word embedding feature vector, it can be combined with the previously obtained first visual features as input to the report generation model. The model then processes the first visual features and the first word embedding feature vector to generate a medical report related to the first disease. Illustratively, generating a medical report related to the first disease based on the report generation model can be achieved as follows: Each first visual feature is concatenated with the first word embedding feature vector to obtain a concatenated visual feature. Then, the predicted category probability value for each first lesion region is calculated, and the target visual feature is determined based on each concatenated visual feature and the corresponding predicted category probability value of the first lesion region. For example, the predicted category probability value of each first lesion region can be considered as a weight, and a weighted average of the concatenated visual features of all first lesion regions can be performed to obtain the target visual feature. Finally, the target visual feature is input into the report generation model to generate a medical report related to the first disease.

[0062] The training process for the aforementioned report generation model can be referred to the following: Figure 4 The content of the illustrated embodiment is understood. Figure 4 As shown, the model training process for the report generation model includes at least the following steps:

[0063] 401. Obtain medical images of samples related to the second disease.

[0064] In this example, the described second disease can be understood as a common disease, and its details can be found in the preceding content, which will not be repeated here. Additionally, the sample medical images can be medical images related to the second disease collected from the target subject. The described medical images can be understood in the context of the medical images shown in step 301 above, and will not be repeated here.

[0065] 402. Based on the preset feature extraction model, visual feature extraction processing is performed on each second lesion region in the sample medical image to obtain the second visual features of each second lesion region.

[0066] In this example, directly using the global features of the sample medical image as input to the report generation model would directly affect the quality of the generated medical report. Therefore, after acquiring the sample medical image, a preset feature extraction model can be used to extract visual features from each secondary lesion region in the sample medical image. This extracts the secondary visual features of each secondary lesion region, which are then used as input to the model training process to provide more representative feature information to the report generation model.

[0067] For example, after acquiring the sample medical image, the location and size of each second lesion region in the sample medical image can be determined using a model such as the RPN, and then the bounding box of each second lesion region can be calculated. Then, for each second lesion region in the sample medical image, visual feature extraction processing can be performed using the aforementioned preset feature extraction model to obtain the second visual features of each second lesion region. It should be noted that the described second visual features can also be understood with reference to the content of the first visual features described in step 302 above, and will not be repeated here.

[0068] 403. Based on the preset projection model and each second visual feature, determine the semantic features of each second lesion region.

[0069] In this example, since medical report generation is an image-to-text task, and both visual and semantic features are rich in valuable information that can be used to improve the quality of medical report generation, a semantic space can be created to enable the finally trained report generation model to be transferred from the second disease to the first disease for generating medical reports for the first disease. Thus, after extracting the second visual features corresponding to each second lesion region, each second visual feature can be projected into the semantic space based on a pre-defined projection model, thereby determining the semantic features of each second lesion region. For example, the second visual features can be projected into the semantic space using a linear projection model, such as... ,in, For semantic features, As a second visual feature, For projection model, For this projection model The parameters.

[0070] 404. Determine the word embedding feature vector between the first disease and the second disease based on the disease information of the first disease and the disease information of the second disease.

[0071] In this example, since no labels or other information for the first disease were collected, the correlation between the first and second diseases can be explored through word embedding feature vectors to uncover relevant information about the first disease and guide the generation of a corresponding medical report. For example, for the first disease, related disease information can be obtained, such as its name and symptoms. Similarly, for the second disease, related disease information can be obtained, such as its name and symptoms. Thus, after obtaining the disease information for both the first and second diseases, the word embedding feature vector can be determined based on this information. For example, the disease information for both diseases can be used as input to a pre-trained word vector model (Biobert). This pre-trained model processes the disease information of both diseases to obtain the word embedding feature vector between them. , where C is the union of the categories of the first disease and the categories of the second disease.

[0072] 405. Determine the target loss value based on the semantic features of the second lesion region, word embedding feature vectors, and second visual features.

[0073] In this example, after obtaining the semantic features and second visual features of each second lesion region, as well as the corresponding word embedding feature vectors, the target loss value can be determined based on the semantic features, word embedding feature vectors, and second visual features of the second lesion region. This target loss value can be used to update and adjust the model parameters of the preset initial model to complete the iterative training of the first generative model, thereby obtaining a trained first generative model. For example, determining the target loss value based on the semantic features, word embedding feature vectors, and second visual features of the second lesion region can be implemented as follows: calculate the first loss value based on the semantic features and word embedding feature vectors of the second lesion region; calculate the second loss value based on the semantic features and second visual features of the second lesion region; calculate the third loss value based on the probability value of each word in the medical prediction report, where the medical prediction report is generated based on the second visual features and word embedding feature vectors. Finally, the target loss value is determined based on the first loss value, the second loss value, and the third loss value. For example, the first loss value, the second loss value, and the third loss value can be summed to calculate the target loss value.

[0074] In some examples, the calculation of the first loss value mentioned above can also be achieved by calculating the first similarity between the word embedding feature vector and the semantic features of the second lesion region, such as: ,in, Indicates the first similarity. Semantic features representing the second lesion region This represents the word embedding feature vector. Similarly, the second similarity between every two word embedding feature vectors is calculated, such as... ,in, For the current word embedding feature vector, Embed a feature vector for another word. This is the second similarity score. Thus, after calculating the first and second similarities, the first loss value can be obtained by calculating the difference between the first and second similarities. It should be noted that the description It can be understood as a cosine similarity function, but this is not a limitation here.

[0075] In other examples, the calculation process for the second loss value can also be understood as follows: first, based on the semantic features and second visual features of the second lesion region, determine the predicted label for the corresponding second lesion region. For example, the predicted probability value of the semantic features can be determined based on the similarity between the semantic features of the second lesion region and the word embedding feature vector, i.e. ,in, The similarity between the semantic features of the second lesion region and the word embedding feature vector is calculated. Then, the second visual features are processed for category prediction according to a pre-defined classification branch model to obtain the predicted probability value of the corresponding category of the second lesion region. ,in, These are the predicted probability values ​​for the categories of each secondary lesion region. This represents the number of categories for the second disease. Finally, the predicted label for the corresponding second lesion region is determined based on the predicted probability values ​​of the semantic features and the predicted probability values ​​of the corresponding second lesion region categories. For example, the predicted probability values ​​of the semantic features and the predicted probability values ​​of the corresponding second lesion region categories can be averaged, and the resulting average can indicate the predicted label for the corresponding second lesion region, such as... Thus, after obtaining the predicted label, the label loss value can be obtained by calculating the difference between the predicted label and the preset label of the second lesion region. ,in, Pre-defined labels for the second lesion area.

[0076] Similarly, the predicted bounding box of the corresponding second lesion region can be determined based on the second visual features, and then the difference between the predicted bounding box and the preset labeled bounding box of the second lesion region can be calculated to obtain the bounding box loss value. ,in, Indicates the predicted bounding box. This represents the predefined bounding box indicating the region of the second lesion. More specifically, This indicates the position and size of the pre-defined bounding box for the second lesion area.

[0077] Thus, the second loss value is determined based on the label loss value and the bounding box loss value. For example, the second loss value can be obtained by summing the label loss value and the bounding box loss value, i.e.: ,in, This represents the second loss value.

[0078] In other examples, the aforementioned third loss value can also be determined as follows: A medical prediction report related to the second disease is generated based on the second visual features and word embedding feature vectors. Then, the probability value of each word appearing in the medical prediction report is calculated, and this probability value is used as the third loss value. ,in, This refers to the (T-1)th word in the medical prediction report.

[0079] It should be noted that, in addition to using the methods mentioned above to calculate the corresponding first loss value, second loss value and third loss value, other methods may be used in practical applications to calculate them, and no specific limitations are made in the embodiments of this application.

[0080] 406. Adjust the model parameters of the preset initial model based on the target loss value to obtain the report generation model.

[0081] In this example, after calculating the target loss value, the model parameters of the preset initial model can be updated and adjusted using this target loss value, thereby training the report generation model. For example, to ensure that the model can accurately detect the first disease not seen during model training using the limited weakly labeled information in the support set S, the model parameters can be adjusted using the medical images and weakly labeled information of the first disease in the support set S to obtain the report generation model. It should be noted that the described weakly labeled information may include the category and bounding box of the lesion region of the first disease in the support set S. For example, given the support set S for the first disease, the images of the lesion regions of each category in each medical image in the support set S can be determined, and then the images of the lesion regions of each category are input into the global visual feature extraction model to obtain the corresponding visual features. ,in This represents the number of lesion regions where the labels are located. At this point, new weights are calculated:

[0082] ,in The category weight for the first disease.

[0083] The category classification weight of the first disease is calculated. Then, the category classification weights in the model can be adjusted by fine-tuning to eliminate feature bias in the feature space, thereby obtaining the report generation model.

[0084] Thus, after obtaining the first word embedding feature vector and the first visual feature, the first word embedding feature vector and the first visual feature can be used as input to the report generation model. Then, the report generation model processes the first visual feature and the first word embedding feature vector to generate a medical report related to the first disease.

[0085] It should be noted that the described report generation model may include, but is not limited to, the Transformer structure, but no specific limitation is made in this application embodiment.

[0086] In other examples, the region information of the corresponding first lesion area can be determined based on the first visual features. Then, the location and size of the first lesion area are determined based on the region information, and a bounding box of the first lesion area is generated based on the location and size of the first lesion area. Thus, after obtaining the medical report through the above step 304, the bounding box of each first lesion area can be displayed in the medical report. It should be noted that the described region information may include, but is not limited to, the coordinates of the upper left corner of the lesion area, length information, width information, etc., and is not specifically limited in this embodiment.

[0087] In other examples, the category of the corresponding first lesion region can be determined based on the first visual features. Then, after obtaining the medical report through step 304 above, the category of the first lesion region can also be identified in the medical report. It should be noted that the category of the first lesion region is used to indicate the disease type to which the corresponding first lesion region belongs.

[0088] For example, Figure 5 A schematic diagram of a medical report provided in an embodiment of this application is shown. Figure 5As shown, when a medical report for an uncommon primary disease needs to be generated, the first medical image of the primary disease and a small number of annotations are obtained, and then processed by the report generation model provided in this application embodiment to generate a medical report related to the primary disease. The content displayed in the medical report shows that each primary medical image displays and marks the bounding box of the corresponding primary lesion region, as well as the category of the corresponding primary lesion region.

[0089] Figure 6 This illustration shows another schematic diagram of a medical report provided in an embodiment of this application. (See attached diagram.) Figure 6 As shown, when it is necessary to generate medical reports for common second diseases, the trained report generation model can also be used to generate medical reports related to the second disease. The medical reports also display and mark the bounding boxes of the corresponding second lesion areas, as well as the categories of the corresponding second lesion areas.

[0090] Figure 7 Another flowchart illustrating the method for generating medical reports provided in this application is shown. Figure 7 As shown, the method for generating the report includes at least the following steps: First, acquiring sample medical images related to the second disease, and performing visual feature extraction processing on each second lesion region in the sample medical images according to a preset feature extraction model to obtain the second visual features of each second lesion region. Then, based on a preset projection model and each second visual feature, determining the semantic features of each second lesion region; and determining the word embedding feature vector between the first and second diseases based on the disease information of the first and second diseases. Finally, determining the target loss value based on the semantic features of the second lesion region, the word embedding feature vector, and the second visual features, and then adjusting the model parameters of the preset initial model based on the target loss value to obtain the report generation model.

[0091] Then, after training the report generation model, the system acquires a first medical image of the target object, which is related to the target object's first disease. Based on a pre-defined feature extraction model, visual features are extracted from each first lesion region in the first medical image to obtain the first visual features of each first lesion region. A first word embedding feature vector is also obtained, indicating the association between the first and second diseases. Each first visual feature is then concatenated with the first word embedding feature vector to obtain a concatenated visual feature. The predicted category probability value for each first lesion region is calculated. Based on each concatenated visual feature and the corresponding predicted category probability value for the first lesion region, the target visual features are determined. These target visual features are then input into the report generation model to generate a medical report related to the first disease.

[0092] In this way, after receiving the medical report, the medical report will display the bounding box of each primary lesion region, as well as the category of the primary lesion region.

[0093] It should be noted that the above Figure 7 The content shown can be referred to in the foregoing. Figure 4 The content in the text will be understood in detail here, and will not be elaborated upon further.

[0094] In this embodiment, since the first medical image of the target object is related to the first disease, after acquiring the first medical image of the target object, visual feature extraction processing can be performed on each first lesion region in the first medical image based on a preset feature extraction model to obtain the first visual feature of each first lesion region. Then, the first word embedding feature vector is obtained, which is used to indicate the association between the first disease and the second disease. Moreover, since the report generation model is a machine learning model trained by iteratively training a preset initial model with the visual features of each second lesion region in the sample medical image related to the second disease and the word embedding feature vector between the first disease and the second disease as training data, the first visual feature and the first word embedding feature vector are processed based on the report generation model to generate a medical report related to the first disease. In the above way, the disease annotation information of the first disease does not need to be considered during the training process. Instead, the visual features of the lesion regions in the medical image related to the second disease and the word embedding feature vector between the first disease and the second disease are comprehensively considered to train the corresponding report generation model. In this way, the report generation model processes the visual features of lesion regions in medical images related to the first disease, as well as word embedding feature vectors, to generate a medical report related to the first disease. This approach is not only applicable to scenarios where disease annotation information for the first disease cannot be determined, but also considers more visual features related to the lesion region, providing richer feature information for model training and thus improving the quality of medical reports.

[0095] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. It is understood that to achieve the above functions, corresponding hardware structures and / or software modules are included to execute each function. Those skilled in the art should readily recognize that, based on the modules and algorithm steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] This application embodiment can divide the device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0097] The medical report generation device in the embodiments of this application will be described in detail below. Figure 8 This is a schematic diagram of one embodiment of the medical report generation device provided in this application. Figure 8 As shown, the medical report generation device may include an acquisition unit 801 and a processing unit 802.

[0098] The system includes an acquisition unit 801 for acquiring a first medical image of the target object, which is related to a first disease of the target object. A processing unit 802 is used to perform visual feature extraction processing on each first lesion region in the first medical image based on a preset feature extraction model, obtaining a first visual feature for each first lesion region. The acquisition unit 801 is also used to acquire a first word embedding feature vector, which indicates the association between the first disease and a second disease. The processing unit 802 inputs the first visual feature and the first word embedding feature vector into a report generation model to obtain a medical report related to the first disease. The report generation model is trained with the goal of generating a medical report related to the first disease, using the second visual features of each second lesion region in a sample medical image related to the second disease and the word embedding feature vector between the first and second diseases as training data to iteratively train a preset initial model to obtain a machine learning model.

[0099] In some examples, the processing unit 802 is also used to display the bounding box of each first lesion region in the medical report after inputting the first visual features and the first word embedding feature vector into the report generation model to obtain a medical report related to the first disease.

[0100] In other examples, the processing unit 802 is also configured to determine the region information of the corresponding first lesion region based on the first visual features before displaying the bounding box of each first lesion region in the medical report; determine the location and size of the first lesion region based on the region information of the first lesion region; and generate the bounding box of the first lesion region based on the location and size of the first lesion region.

[0101] In other examples, the processing unit 802 is also used to, after inputting the first visual feature and the first word embedding feature vector into the report generation model to obtain a medical report related to the first disease, label the category of the first lesion region in the medical report, the category of the first lesion region being used to indicate the disease type to which the corresponding first lesion region belongs.

[0102] In other examples, the processing unit 802 is used to: concatenate each first visual feature with the first word embedding feature vector to obtain each concatenated visual feature; calculate the predicted category probability value of each first lesion region; determine the target visual feature based on each concatenated visual feature and the predicted category probability value of the corresponding first lesion region; and input the target visual feature into the report generation model to generate a medical report related to the first disease.

[0103] In other examples, the acquisition unit 801 is used to acquire disease information of a first disease and disease information of a second disease. The processing unit 802 is used to determine a first word embedding feature vector based on the disease information of the first disease and the disease information of the second disease.

[0104] In other examples, the acquisition unit 801 is further configured to acquire sample medical images related to a second disease before inputting the first visual features and the first word embedding feature vector into the report generation model to obtain a medical report related to the first disease. The processing unit 802 is further configured to: perform visual feature extraction processing on each second lesion region in the sample medical images based on a preset feature extraction model to obtain a second visual feature for each second lesion region; determine the semantic features of each second lesion region based on a preset projection model and each second visual feature; determine the word embedding feature vector between the first and second diseases based on the disease information of the first and second diseases; determine a target loss value based on the semantic features of the second lesion region, the word embedding feature vector, and the second visual features; and adjust the model parameters of the preset initial model based on the target loss value to obtain a report generation model.

[0105] In other examples, the processing unit 802 is used to: calculate a first loss value based on the semantic features and word embedding feature vectors of the second lesion region; calculate a second loss value based on the semantic features and second visual features of the second lesion region; calculate a third loss value based on the probability value of each word in the medical prediction report, wherein the medical prediction report is a report generated based on the second visual features and word embedding feature vectors; and determine a target loss value based on the first loss value, the second loss value, and the third loss value.

[0106] In other examples, the processing unit 802 is used to: calculate a first similarity between the word embedding feature vector and the semantic features of the second lesion region; calculate a second similarity between every two word embedding feature vectors in the word embedding feature vector; and calculate the difference between the first similarity and the second similarity to obtain a first loss value. In other examples, the processing unit 802 is used to: determine the predicted label of the corresponding second lesion region based on the semantic features and second visual features of the second lesion region; calculate the difference between the predicted label and the preset labeled label of the second lesion region to obtain the label loss value; determine the predicted bounding box of the corresponding second lesion region based on the second visual features; calculate the difference between the predicted bounding box and the preset labeled bounding box of the second lesion region to obtain the bounding box loss value; and determine the second loss value based on the label loss value and the bounding box loss value.

[0107] In other examples, the processing unit 802 is used to: determine the predicted probability value of the semantic features based on the similarity between the semantic features of the second lesion region and the word embedding feature vector; perform category prediction processing on the second visual features based on a preset classification branch model to obtain the predicted probability value of the category of the corresponding second lesion region; and determine the predicted label of the corresponding second lesion region based on the predicted probability value of the semantic features and the predicted probability value of the category of the corresponding second lesion region. In other examples, the processing unit 802 is used to: generate a medical prediction report related to a second disease based on the second visual features and the word embedding feature vector; calculate the probability value of each word appearing in the medical prediction report; and determine a third loss value based on the probability value of each word appearing.

[0108] The above describes the medical report generation device in the embodiments of this application from the perspective of modular functional entities. The following describes the medical report generation device in the embodiments of this application from the perspective of hardware processing. Figure 9 This is a schematic diagram of the structure of a medical report generation device provided in an embodiment of this application. The medical report generation device can vary considerably due to differences in configuration or performance. The medical report generation device may include at least one processor 901, a communication line 907, a memory 903, and at least one communication interface 904.

[0109] The processor 901 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (server IC), or one or more integrated circuits used to control the execution of the program of the present application.

[0110] Communication line 907 may include a path for transmitting information between the aforementioned components.

[0111] Communication interface 904 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0112] The memory 903 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions. The memory can exist independently and be connected to the processor via communication line 907. The memory can also be integrated with the processor.

[0113] The memory 903 stores computer execution instructions for implementing the scheme of this application, and its execution is controlled by the processor 901. The processor 901 executes the computer execution instructions stored in the memory 903, thereby implementing the method provided in the above embodiments of this application.

[0114] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0115] In a specific implementation, as one example, the medical report generation device may include multiple processors, such as... Figure 9 Processors 901 and 902 are described in the text. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0116] In a specific implementation, as one embodiment, the medical report generation device may further include an output device 905 and an input device 906. The output device 905 communicates with the processor 901 and can display information in various ways. The input device 906 communicates with the processor 901 and can receive input from the target object in various ways. For example, the input device 906 may be a mouse, a touch screen device, or a sensing device, etc.

[0117] The aforementioned medical report generation device can be a general-purpose device or a dedicated device. In specific implementations, the medical report generation device can be a server, terminal equipment, or other similar devices. Figure 9A device with a similar structure. The embodiments of this application do not limit the type of medical report generation device.

[0118] It should be noted that Figure 9 The processor 901 can invoke computer execution instructions stored in the memory 903 to cause the medical report generation device to perform actions such as... Figure 3 Or the method in the method embodiment corresponding to 7.

[0119] Specifically, Figure 8 The function / implementation process of the processing unit 802 in the middle can be achieved through Figure 9 The processor 901 in the memory calls computer execution instructions stored in the memory 903 to achieve this. Figure 8 The function / implementation process of the acquisition unit 801 can be achieved through... Figure 9 It is implemented using the 904 communication interface.

[0120] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods for identifying sensitive events described in the above method embodiments.

[0121] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods for generating medical reports described in the above method embodiments.

[0122] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0127] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, they can be implemented in whole or in part in the form of a computer program product.

[0129] A computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, they generate, in whole or in part, the processes or functions according to embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.

[0130] It is understood that in the specific implementation of this application, user information, user personal data and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0131] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating medical reports, characterized in that, include: Acquire a first medical image of a target object, wherein the first medical image is related to a first disease of the target object; Based on a preset feature extraction model, visual feature extraction processing is performed on each first lesion region in the first medical image to obtain the first visual feature of each first lesion region. Obtain the first word embedding feature vector, which includes: obtaining the disease information of the first disease and the disease information of the second disease; The first word embedding feature vector is determined based on the disease information of the first disease and the disease information of the second disease. The first word embedding feature vector is used to indicate the association between the first disease and the second disease. The first visual feature and the first word embedding feature vector are input into the report generation model to obtain a medical report related to the first disease. The report generation model is a machine learning model that is trained iteratively on a preset initial model with the second visual feature of each second lesion region in the sample medical image related to the second disease and the word embedding feature vector between the first disease and the second disease as training data, with the goal of generating a medical report related to the first disease.

2. The method according to claim 1, characterized in that, After inputting the first visual feature and the first word embedding feature vector into the report generation model to obtain a medical report related to the first disease, the method further includes: The medical report displays the bounding box of each of the first lesion regions.

3. The method according to claim 2, characterized in that, Before displaying the bounding box of each of the first lesion regions in the medical report, the method further includes: Based on the first visual feature, determine the corresponding regional information of the first lesion area; The location and size of the first lesion region are determined based on the regional information of the first lesion region. The bounding box of the first lesion region is generated based on the location and size of the first lesion region.

4. The method according to any one of claims 1 to 3, characterized in that, After inputting the first visual feature and the first word embedding feature vector into the report generation model to obtain a medical report related to the first disease, the method further includes: The category of the first lesion area is marked in the medical report, and the category of the first lesion area is used to indicate the disease type to which the corresponding first lesion area belongs.

5. The method according to any one of claims 1 to 3, characterized in that, The first visual feature and the first word embedding feature vector are input into the report generation model to obtain a medical report related to the first disease, including: Each of the first visual features is concatenated with the first word embedding feature vector to obtain each concatenated visual feature; Calculate the predicted category probability value for each of the first lesion regions; Based on each stitched visual feature and the corresponding predicted category probability value of the first lesion region, the target visual feature is determined. The target visual features are input into the report generation model to generate a medical report related to the first disease.

6. The method according to any one of claims 1 to 3, characterized in that, Before inputting the first visual feature and the first word embedding feature vector into the report generation model to obtain a medical report related to the first disease, the method further includes: Acquire medical images of samples related to the second disease; Based on the preset feature extraction model, visual feature extraction processing is performed on each second lesion region in the sample medical image to obtain the second visual feature of each second lesion region. Based on the preset projection model and each of the second visual features, the semantic features of each of the second lesion regions are determined; Based on the disease information of the first disease and the disease information of the second disease, determine the word embedding feature vector between the first disease and the second disease; The target loss value is determined based on the semantic features of the second lesion region, the word embedding feature vector, and the second visual features. The model parameters of the preset initial model are adjusted based on the target loss value to obtain the report generation model.

7. The method according to claim 6, characterized in that, The step of determining the target loss value based on the semantic features of the second lesion region, the word embedding feature vector, and the second visual features includes: Based on the semantic features of the second lesion region and the word embedding feature vector, a first loss value is calculated; The second loss value is calculated based on the semantic features of the second lesion region and the second visual features; A third loss value is calculated based on the probability value of each word in the medical prediction report, wherein the medical prediction report is a report generated based on the second visual feature and the word embedding feature vector; The target loss value is determined based on the first loss value, the second loss value, and the third loss value.

8. The method according to claim 7, characterized in that, The calculation of the first loss value based on the semantic features of the second lesion region and the word embedding feature vector includes: Calculate the first similarity between the word embedding feature vector and the semantic features of the second lesion region; Calculate the second similarity between every two word embedding feature vectors in the word embedding feature vector; Calculate the difference between the first similarity and the second similarity to obtain the first loss value.

9. The method according to claim 7, characterized in that, The calculation of the second loss value based on the semantic features of the second lesion region and the second visual features includes: Based on the semantic features and the second visual features of the second lesion region, a predictive label for the corresponding second lesion region is determined; Calculate the difference between the predicted label and the preset label of the second lesion region to obtain the label loss value; Based on the second visual feature, determine the predicted bounding box of the corresponding second lesion region; Calculate the difference between the predicted bounding box and the preset labeled bounding box of the second lesion region to obtain the bounding box loss value; The second loss value is determined based on the label loss value and the bounding box loss value.

10. The method according to claim 9, characterized in that, The step of determining the predicted label for the corresponding second lesion region based on the semantic features and the second visual features of the second lesion region includes: Based on the similarity between the semantic features of the second lesion region and the word embedding feature vector, the predicted probability value of the semantic features is determined; The second visual feature is subjected to category prediction processing based on a preset classification branch model to obtain the predicted probability value of the category of the corresponding second lesion region. Based on the predicted probability value of the semantic features and the predicted probability value of the corresponding second lesion region category, the predicted label of the corresponding second lesion region is determined.

11. The method according to claim 7, characterized in that, The calculation of the third loss value based on the probability value of each word in the medical prediction report includes: A medical prediction report related to the second disease is generated based on the second visual features and the word embedding feature vector. Calculate the probability value of each word appearing in the medical prediction report; The third loss value is determined based on the probability value of each word appearing.

12. A medical report generation device, characterized in that, include: An acquisition unit is used to acquire a first medical image of a target object, wherein the first medical image is related to a first disease of the target object; The processing unit is used to perform visual feature extraction processing on each first lesion region in the first medical image based on a preset feature extraction model to obtain the first visual feature of each first lesion region. The acquisition unit is used to acquire the first word embedding feature vector, including: acquiring the disease information of the first disease and the disease information of the second disease; The first word embedding feature vector is determined based on the disease information of the first disease and the disease information of the second disease. The first word embedding feature vector is used to indicate the association between the first disease and the second disease. The processing unit is used to input the first visual feature and the first word embedding feature vector into the report generation model to obtain a medical report related to the first disease. The report generation model is a machine learning model obtained by iteratively training a preset initial model with the second visual feature of each second lesion region in the sample medical image related to the second disease and the word embedding feature vector between the first disease and the second disease as training data, with the goal of generating a medical report related to the first disease.

13. A medical report generation device, characterized in that, The medical report generation device includes: an input / output (I / O) interface, a processor, and a memory, wherein the memory stores program instructions; The processor is configured to execute program instructions stored in the memory to perform the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer or processor, cause the computer or processor to perform the method as described in any one of claims 1 to 11.

Images