A cross-modal text generation method, device, electronic device and storage medium

Abstract

The present invention discloses a cross-modal text generation method, device, electronic device and storage medium, the method comprising: loading an image-text migration model, the image-text migration model comprising an image encoder and a text encoder; constructing an image classification model based on the image encoder, fine-tuning the image classification model, and updating the image classification model; acquiring medical image data, classifying the medical image data based on the image classification model, and obtaining a classification result; the classification result comprises classification labels of multiple dimensions and prompt data corresponding to the classification labels; encoding the prompt data corresponding to the classification labels based on the text encoder to form a feature space; searching in the feature space to obtain a prompt data group matching the medical image features of the medical image data; and generating text data corresponding to the medical image data based on the prompt data group. Cross-modal generation from image to text is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image analysis, and in particular to a cross-modal text generation method, device, electronic device and storage medium. Background Art

[0002] With the rapid development and widespread application of deep learning and other related technologies, remarkable achievements have been made in computer vision, natural language processing, and auxiliary medical diagnosis, and great potential has been shown. The continuous advancement and widespread use of medical equipment have led to the emergence of many forms of medical testing, including multi-modal data such as images, sound waves, and reports. However, with the sharp increase in the number of patient test results, multi-modal data is helpful in achieving more accurate disease detection, but it also adds more workload to clinicians.

[0003] At present, compared with single-modal data, multimodal data is difficult to obtain. Summary of the invention

[0004] The present invention provides a cross-modal text generation method, device, electronic device and storage medium to achieve cross-modal generation from image to text.

[0005] According to one aspect of the present invention, a cross-modal text generation method is provided, the method comprising:

[0006] Loading an image-to-text migration model, wherein the image-to-text migration model includes an image encoder and a text encoder;

[0007] Building an image classification model based on the image encoder, and fine-tuning the image classification model to update the image classification model; the image classification model includes the image encoder and the classifier;

[0008] Acquire medical image data, and classify the medical image data based on the image classification model to obtain a classification result; the classification result includes classification labels of multiple dimensions and prompt data corresponding to the classification labels;

[0009] Based on the text encoder, the prompt data corresponding to the classification label is encoded to form a feature space; searching in the feature space to obtain a prompt data group matching the medical image features of the medical image data;

[0010] The text data corresponding to the medical image data is generated based on the prompt data group.

[0011] According to another aspect of the present invention, a cross-modal text generation device is provided, comprising:

[0012] An image-text migration model loading module, used to load an image-text migration model, wherein the image-text migration model includes an image encoder and a text encoder;

[0013] An image classification model fine-tuning module, used to construct an image classification model based on the image encoder, and fine-tune the image classification model to update the image classification model; the image classification model includes the image encoder and the classifier;

[0014] An image classification module, used to obtain medical image data, classify the medical image data based on the image classification model, and obtain a classification result; the classification result includes classification labels of multiple dimensions and prompt data corresponding to the classification labels;

[0015] A prompt data group retrieval module is used to encode the prompt data corresponding to the classification label based on the text encoder to form a feature space; search in the feature space to obtain a prompt data group matching the medical image feature of the medical image data;

[0016] A text data generating module is used to generate text data corresponding to the medical image data based on the prompt data group.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the cross-modal text generation method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the cross-modal text generation method described in any embodiment of the present invention when executed.

[0022] The technical solution of the embodiment of the present invention is to load an image-text migration model, which includes an image encoder and a text encoder; construct an image classification model based on the image encoder, and fine-tune the image classification model to update the image classification model; obtain medical image data, classify the medical image data based on the image classification model, and obtain classification results; the classification results include classification labels of multiple dimensions and prompt data corresponding to the classification labels; encode the prompt data corresponding to the classification labels based on the text encoder to form a feature space; search in the feature space to obtain a prompt data group that matches the medical image features of the medical image data; generate text data corresponding to the medical image data based on the prompt data group. Perform cross-modal migration on medical image data based on the image-text migration model to achieve cross-modal generation from image to text.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flow chart of a cross-modal text generation method provided in Embodiment 1 of the present invention;

[0026] Figure 2 is a flow chart of a training method for an image-text migration model provided in Embodiment 2 of the present invention;

[0027] Figure 3 is a structural schematic diagram of an image-text migration model provided by Embodiment 2 of the present invention;

[0028] Figure 4 It is a structural schematic diagram of a cross-modal text generation device provided in Embodiment 3 of the present invention;

[0029] Figure 5 It is a structural schematic diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1 This is a flow chart of a cross-modal text generation method provided in the first embodiment of the present invention. This embodiment is applicable to the case of cross-modal generation of medical diagnosis reports. The method can be executed by a cross-modal text generation device. The cross-modal text generation device can be implemented in the form of hardware and / or software. The cross-modal text generation device can be configured in electronic devices such as computers and servers. Figure 1 As shown, the method includes:

[0034] S110 . Load an image-to-text migration model, where the image-to-text migration model includes an image encoder and a text encoder.

[0035] Among them, the image encoder is used to extract image features of image data. Specifically, the image encoder can be SwinTransformer pre-trained based on ImageNet or RestNet-18; the text encoder is used to extract text features of text data. The text encoder can be BioClinicalBERT, which is a variant based on the BERT natural language understanding model and is used to process medical text data.

[0036] In some embodiments, optionally, to unify the input dimension, an additional linear projection layer may be added at the end of the encoder.

[0037] S120, constructing an image classification model based on the image encoder, and fine-tuning the image classification model to update the image classification model; the image classification model includes the image encoder and the classifier.

[0038] In this embodiment, based on the loaded image encoder, a classifier can be added to build an image classification model. The classifier takes the output of the image encoder as input and outputs a classification result. The classifier includes but is not limited to a fully connected neural network, a support vector machine SVM, etc. Further, after the image classification model is built, the image classification model is fine-tuned based on a medical image dataset. It is understandable that the classifier is trained on the medical image dataset while keeping most of the parameters of the image encoder fixed so that the image classification model can learn the unique features of medical images.

[0039] S130, obtaining medical image data, and classifying the medical image data based on the image classification model to obtain a classification result; the classification result includes classification labels of multiple dimensions and prompt data corresponding to the classification labels.

[0040] Among them, the medical image data includes but is not limited to X-ray images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, ultrasound images, etc. In this embodiment, the medical image data is classified and processed based on the image classification model to obtain a classification result. Among them, the classification result includes classification labels of multiple dimensions and prompt data corresponding to the classification labels. For example, taking a chest X-ray image as an example, if the classification label is pleural effusion, the prompt data includes right small bilateral pleural effusion, left small bilateral pleural effusion, left small subpulmonary pleural effusion, right small subpulmonary pleural effusion, etc.

[0041] On the basis of the above embodiment, optionally, the medical image data is classified based on the image classification model to obtain a classification result, including: encoding the medical image based on the image encoder to obtain medical image features; inputting the medical image features into the classifier to obtain classification labels of multiple dimensions; and determining prompt data corresponding to the classification label based on the classification label.

[0042] In this embodiment, the medical image can be encoded based on the image encoder to obtain medical image features; the medical image features are input into the classifier to obtain classification labels of multiple dimensions; further, the prompt data corresponding to the classification label can be determined based on a preset mapping table of classification labels to prompt data; wherein the classification label in the preset mapping table corresponds to at least one prompt data. Optionally, the prompt data can also be dynamically generated using natural language processing (NLP) technology. For example, a template filling method can be used to generate prompt data corresponding to the classification label based on the classification label and a preset template.

[0043] S140, encoding the prompt data corresponding to the classification label based on the text encoder to form a feature space; searching in the feature space to obtain a prompt data group matching the medical image features of the medical image data.

[0044] The feature space includes prompt data text features of the prompt data corresponding to the classification label, and the prompt data text features are text features extracted by encoding the prompt data by the text encoder. In this embodiment, the prompt data corresponding to the classification label is encoded based on the text encoder to obtain the prompt data text features; the feature space is constructed based on the prompt data text features, and a search is performed in the feature space to obtain a prompt data group that matches the medical image features of the medical image data.

[0045] Based on the above embodiment, optionally, the image-text migration model also includes an image-text mapping module; the retrieval in the feature space to obtain a prompt data group that matches the medical image features of the medical image data includes: converting the medical image features into medical text features based on the image-text mapping module; in the feature space, comparing the medical text features with the prompt data text features to determine the prompt data group that matches the medical image features of the medical image data.

[0046] The medical text feature is a text feature converted from the medical image feature. In this embodiment, the medical image feature is converted into the medical text feature based on the image-text mapping module; in the feature space, the similarity between the medical text feature and each prompt data text feature is determined based on the similarity measurement; according to the similarity, the prompt data is sorted to determine the candidate prompt data, and the prompt data group is determined in the candidate prompt data. The prompt data group includes a plurality of prompt data that match the image features of the medical image data.

[0047] S150. Generate text data corresponding to the medical image data based on the prompt data group.

[0048] In this embodiment, text data corresponding to the medical image data may be generated based on a hard-coded template and a prompt data group; wherein the hard-coded template includes a fixed sentence structure and placeholders for subsequent insertion of prompt data and related information.

[0049] The technical solution of this embodiment is to load an image-text migration model, which includes an image encoder and a text encoder; construct an image classification model based on the image encoder, and fine-tune the image classification model to update the image classification model; obtain medical image data, classify the medical image data based on the image classification model, and obtain classification results; the classification results include classification labels of multiple dimensions and prompt data corresponding to the classification labels; encode the prompt data corresponding to the classification labels based on the text encoder to form a feature space; search in the feature space to obtain a prompt data group that matches the medical image features of the medical image data; generate text data corresponding to the medical image data based on the prompt data group. Perform cross-modal migration of medical image data based on the image-text migration model to achieve cross-modal generation from image to text.

[0050] Embodiment 2

[0051] Figure 2 It is a flow chart of a training method for an image-text migration model provided in the second embodiment of the present invention. This embodiment introduces the training method for the image-text migration model in the above embodiments, wherein the explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0052] like Figure 2 As shown, the method includes:

[0053] S210: Establish an image-text migration model, wherein the image-text migration model includes an image encoder, a text encoder, and two mapping modules.

[0054] Among them, the mapping module is composed of a linear layer and a TransformerEncoder superimposed.

[0055] S220: Obtain an image-text dataset.

[0056] The image-text dataset is composed of image-text pairs, each of which includes image data, text data and at least one label data, and the label data corresponds to the image data and the text data respectively; illustratively, imaging reports can be randomly selected from public medical datasets such as MIMIC-CXR, CheXpert-5x200, and RSNA Pneumonia as image-text datasets.

[0057] S230. In a first training phase, the image encoder and the text encoder are trained based on the image-text dataset to obtain a target text encoder and a target image encoder.

[0058] In this embodiment, the training in the first training phase is used for intramodal contrast, and the image encoder and the text encoder are trained based on the image-text dataset to obtain a target text encoder and a target image encoder. The target text encoder and the target image encoder are the image encoder and the text encoder obtained by the training in the first training phase.

[0059] On the basis of the above embodiment, optionally, the image encoder and the text encoder are trained based on the image-text pair to obtain a target text encoder and a target image encoder, including: encoding the image data based on the image encoder to obtain image features, and constructing an image similarity matrix based on the image features; encoding the text data based on the text encoder to obtain text features, and constructing a text similarity matrix based on the text features; constructing a label similarity matrix based on the label data; determining a first contrast loss function based on the image similarity matrix and the label similarity matrix; determining a second contrast loss function based on the text similarity matrix and the label similarity matrix; determining a target loss function based on the first contrast loss function and the second contrast loss function; adjusting the parameters of the image encoder and the text encoder respectively based on the target loss function; iteratively executing the above steps to obtain a target text encoder and a target image encoder when the first training end condition is met.

[0060] Figure 3 is a structural diagram of an image-text migration model provided by the second embodiment of the present invention, such as Figure 3 As shown, the training process of the first training stage is as follows:

[0061] 1) Encoding the image data based on an image encoder to obtain image features, determining the image features based on a similarity measurement method to determine the similarity between the image data, and constructing an image similarity matrix based on the similarity between the image data;

[0062] 2) encoding the text data based on a text encoder to obtain text features, determining the similarity between the text data based on the text features based on a similarity measurement method, and constructing a text similarity matrix based on the similarity between the text data;

[0063] 3) Determine the label features based on the similarity measurement method to determine the similarity between the label data, and construct a label similarity matrix based on the similarity between the label data;

[0064] Among them, the similarity matrix determines the sample similarity within the modality, that is, the image similarity matrix determines the similarity of image data, the text similarity matrix determines the similarity of text data, and the label similarity matrix determines the similarity of label data. It can be understood that this similarity is used to ensure that the features of images (or texts) with the same label are as close as possible in the feature space, while the features of images (or texts) with different labels will be far away. Similarity measurement methods include but are not limited to cosine similarity, Euclidean distance, etc., which are not limited here.

[0065] It should be noted that steps 1) to 3) can be executed in parallel.

[0066] 4) determining a first contrast loss function based on the image similarity matrix and the label similarity matrix;

[0067] 5) determining a second contrast loss function based on the text similarity matrix and the label similarity matrix;

[0068] 6) determining a target loss function based on the first contrast loss function and the second contrast loss function;

[0069] Among them, the first contrast loss function is the loss function between the image similarity matrix and the label similarity matrix, the second contrast loss function is the loss function between the text similarity matrix and the label similarity matrix, and the target loss function is the total loss function of the first training stage. The formula of the total loss function is:

[0070] Loss = loss Contrasitve1 +loss Contrasitve2

[0071] Among them, Loss is the total loss function, loss Contrasitve1 is the first contrast loss, loss Contrasitve2 is the second contrast loss. The contrast loss function is Contrasitve Loss.

[0072] 7) adjusting the parameters of the image encoder and the text encoder respectively based on the target loss function;

[0073] 8) Iteratively execute the above steps to obtain the target text encoder and the target image encoder when the first training end condition is met.

[0074] The first training end condition refers to the condition for ending the first training stage. Specifically, the first training end condition may be that the number of iterations of the first training stage reaches a first preset number of iterations, or that the loss value of the total loss function remains unchanged.

[0075] S240. In a second training phase, the two mapping modules are trained based on the image-text dataset, the target text encoder, and the target image encoder to obtain an image-text mapping module and a text-image mapping module.

[0076] In this embodiment, the training in the second training phase is used for cross-modal feature transfer, and two mapping modules are trained based on the image-text dataset, the target text encoder, and the target image encoder, respectively, to obtain an image-text mapping module and a text-image mapping module. Among them, the image-text mapping module is used to convert image features into text features, and the text-image mapping module is used to convert text features into image features.

[0077] On the basis of the above embodiment, optionally, the two mapping modules are trained respectively based on the image-text data set, the target text encoder and the target image encoder to obtain an image-text mapping module and a text-image mapping module, including: encoding the image data based on the target image encoder to obtain a first image feature; converting the first image feature into a first text feature based on the first mapping module; encoding the text data based on the target text encoder to obtain a second text feature; converting the two text features into a second image feature based on the second mapping module; determining a text loss function based on the first text feature and the second text feature, and adjusting the parameters of the first mapping module based on the text loss function; determining an image loss function based on the first image feature and the second image feature, and adjusting the parameters of the second mapping module based on the image loss function; iteratively performing the above steps to obtain the image-text mapping module and the text-image mapping module when the second training end condition is met.

[0078] like Figure 3 As shown, the training process of the second training stage is as follows:

[0079] 1) encoding the image data based on the target image encoder to obtain a first image feature;

[0080] 2) converting the first image feature into a first text feature based on the first mapping module;

[0081] Encoding the text data based on the target text encoder to obtain a second text feature;

[0082] 3) converting the second text feature into a second image feature based on a second mapping module;

[0083] Among them, the first image feature is the image feature encoded by the target image encoder obtained by the first stage training, the first text feature is the text feature converted from the first image feature, the second text feature is the text feature encoded by the target text encoder obtained by the first stage training, and the second image feature is the image feature converted from the second text feature.

[0084] 4) determining a text loss function based on the first text feature and the second text feature, and adjusting parameters of the first mapping module based on the text loss function;

[0085] 5) determining an image loss function based on the first image feature and the second image feature, and adjusting parameters of the second mapping module based on the image loss function;

[0086] The text loss function is the loss function between the first text feature and the second text feature, and the image loss function is the loss function between the first image feature and the second image feature. The text loss function and the image loss function can be a mean square error loss function, a mean absolute error loss, a root mean square error loss, etc., which are not limited here.

[0087] 6) Iteratively execute the above steps to obtain an image-text mapping module and a text-image mapping module when the second training end condition is met.

[0088] Among them, the second training end condition refers to the condition for ending the training of the second training stage. Specifically, the second training end condition can be that the number of iterations of the second training stage reaches a second preset number of iterations, or the loss values ​​of the text loss function and the image loss function remain unchanged.

[0089] It can be understood that the first mapping module is trained to obtain an image-text mapping module, and the second mapping module is trained to obtain a text-image mapping module.

[0090] The technical solution of this embodiment uses multimodal data to perform two-stage training on the text-image migration model, performs feature extraction training in the first training stage, and performs cross-modal feature migration training in the second training stage, so as to improve the accuracy of cross-modal migration.

[0091] Embodiment 3

[0092] Figure 4 is a schematic diagram of the structure of a cross-modal text generation device provided by Embodiment 3 of the present invention. Figure 4 As shown, the device comprises:

[0093] An image-to-text migration model loading module 310, used to load an image-to-text migration model, wherein the image-to-text migration model includes an image encoder and a text encoder;

[0094] An image classification model fine-tuning module 320, configured to construct an image classification model based on the image encoder, and fine-tune the image classification model to update the image classification model; the image classification model includes the image encoder and the classifier;

[0095] An image classification module 330 is used to obtain medical image data, and classify the medical image data based on the image classification model to obtain a classification result; the classification result includes classification labels of multiple dimensions and prompt data corresponding to the classification labels;

[0096] A prompt data group retrieval module 340 is used to encode the prompt data corresponding to the classification label based on the text encoder to form a feature space; and to search in the feature space to obtain a prompt data group matching the medical image feature of the medical image data;

[0097] The text data generating module 350 is configured to generate text data corresponding to the medical image data based on the prompt data group.

[0098] The technical solution of this embodiment is to load an image-text migration model, which includes an image encoder and a text encoder; construct an image classification model based on the image encoder, and fine-tune the image classification model to update the image classification model; obtain medical image data, classify the medical image data based on the image classification model, and obtain classification results; the classification results include classification labels of multiple dimensions and prompt data corresponding to the classification labels; encode the prompt data corresponding to the classification labels based on the text encoder to form a feature space; search in the feature space to obtain a prompt data group that matches the medical image features of the medical image data; generate text data corresponding to the medical image data based on the prompt data group. Perform cross-modal migration of medical image data based on the image-text migration model to achieve cross-modal generation from image to text.

[0099] On the basis of the above embodiment, optionally, the device also includes a training module of an image-text migration model, which is used to establish an image-text migration model, wherein the image-text migration model includes an image encoder, a text encoder and two mapping modules; an image-text data set is obtained; in a first training stage, the image encoder and the text encoder are trained based on the image-text data set to obtain a target text encoder and a target image encoder; in a second training stage, the two mapping modules are respectively trained based on the image-text data set, the target text encoder and the target image encoder to obtain an image-text mapping module and a text-image mapping module.

[0100] Based on the above embodiment, optionally, the mapping module is composed of a linear layer and a TransformerEncoder superimposed.

[0101] Based on the above embodiment, optionally, the image-text data set is composed of image-text pairs, each of which includes image data, text data and at least one label data, and the label data corresponds to the image data and the text data respectively; the training module of the image-text migration model includes an encoder training unit, which is used to encode the image data based on the image encoder to obtain image features, and construct an image similarity matrix based on the image features; encode the text data based on the text encoder to obtain text features, and construct a text similarity matrix based on the text features; construct a label similarity matrix based on the label data; determine a first contrast loss function based on the image similarity matrix and the label similarity matrix; determine a second contrast loss function based on the text similarity matrix and the label similarity matrix; determine a target loss function based on the first contrast loss function and the second contrast loss function; adjust the parameters of the image encoder and the text encoder respectively based on the target loss function; iterate the above steps to obtain a target text encoder and a target image encoder when the first training end condition is met.

[0102] Based on the above embodiment, optionally, the training module of the image-text migration model also includes a mapping module training unit, which is used to encode the image data based on the target image encoder to obtain a first image feature; convert the first image feature into a first text feature based on the first mapping module; encode the text data based on the target text encoder to obtain a second text feature; convert the second text feature into a second image feature based on a second mapping module; determine a text loss function based on the first text feature and the second text feature, and adjust the parameters of the first mapping module based on the text loss function; determine an image loss function based on the first image feature and the second image feature, and adjust the parameters of the second mapping module based on the image loss function; iteratively perform the above steps to obtain the image-text mapping module and the text-image mapping module when the second training end condition is met.

[0103] Based on the above embodiment, optionally, the image classification module 330 is specifically used to encode the medical image based on the image encoder to obtain medical image features; input the medical image features into the classifier to obtain classification labels of multiple dimensions; and determine the prompt data corresponding to the classification label based on the classification label.

[0104] Based on the above embodiment, optionally, the prompt data group retrieval module 340 is specifically used to convert the medical image features into medical text features based on the image-text mapping module; in the feature space, the medical text features and the prompt data text features are compared to determine the prompt data group that matches the medical image features of the medical image data.

[0105] The cross-modal text generation device provided in the embodiment of the present invention can execute the cross-modal text generation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0106] Embodiment 4

[0107] Figure 5 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0108] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0109] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0110] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the cross-modal text generation method.

[0111] In some embodiments, the cross-modal text generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the cross-modal text generation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the cross-modal text generation method in any other appropriate manner (e.g., by means of firmware).

[0112] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The computer programs for implementing the cross-modal text generation method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0114] Embodiment 5

[0115] Embodiment 5 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a processor to execute a cross-modal text generation method, the method comprising:

[0116] Loading an image-to-text migration model, wherein the image-to-text migration model includes an image encoder and a text encoder;

[0117] Building an image classification model based on the image encoder, and fine-tuning the image classification model to update the image classification model; the image classification model includes the image encoder and the classifier;

[0118] Acquire medical image data, and classify the medical image data based on the image classification model to obtain a classification result; the classification result includes classification labels of multiple dimensions and prompt data corresponding to the classification labels;

[0119] Based on the text encoder, the prompt data corresponding to the classification label is encoded to form a feature space; searching in the feature space to obtain a prompt data group matching the medical image features of the medical image data;

[0120] The text data corresponding to the medical image data is generated based on the prompt data group.

[0121] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0124] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-modal text generation method, characterized in that: The method comprises: Loading an image-to-text migration model, wherein the image-to-text migration model includes an image encoder and a text encoder; Building an image classification model based on the image encoder, and fine-tuning the image classification model to update the image classification model; the image classification model includes the image encoder and the classifier; Acquire medical image data, and classify the medical image data based on the image classification model to obtain a classification result; the classification result includes classification labels of multiple dimensions and prompt data corresponding to the classification labels; Based on the text encoder, the prompt data corresponding to the classification label is encoded to form a feature space; searching in the feature space to obtain a prompt data group matching the medical image features of the medical image data; The text data corresponding to the medical image data is generated based on the prompt data group.

2. The method according to claim 1, characterized in that The training method of the image-text migration model includes: Establishing an image-text migration model, wherein the image-text migration model includes an image encoder, a text encoder and two mapping modules; Get the image-text dataset; In a first training stage, the image encoder and the text encoder are trained based on the image-text dataset to obtain a target text encoder and a target image encoder; In the second training stage, the two mapping modules are trained respectively based on the image-text dataset, the target text encoder and the target image encoder to obtain an image-text mapping module and a text-image mapping module.

3. The method according to claim 2, characterized in that The mapping module is composed of a linear layer and a TransformerEncoder stacked together.

4. The method according to claim 2, characterized in that: The image-text data set is composed of image-text pairs, each of which includes image data, text data and at least one label data, and the label data corresponds to the image data and the text data respectively; The step of training the image encoder and the text encoder based on the image-text pair to obtain a target text encoder and a target image encoder includes: Performing encoding processing on the image data based on the image encoder to obtain image features, and constructing an image similarity matrix based on the image features; Encoding the text data based on the text encoder to obtain text features, and constructing a text similarity matrix based on the text features; Constructing a label similarity matrix based on the label data; Determine a first contrast loss function based on the image similarity matrix and the label similarity matrix; Determine a second contrast loss function based on the text similarity matrix and the label similarity matrix; Determine a target loss function based on the first contrast loss function and the second contrast loss function; Adjusting parameters of the image encoder and the text encoder respectively based on the target loss function; The above steps are iteratively performed to obtain a target text encoder and a target image encoder when the first training end condition is met.

5. The method according to claim 4, characterized in that The two mapping modules are trained based on the image-text dataset, the target text encoder and the target image encoder to obtain an image-text mapping module and a text-image mapping module, including: Encoding the image data based on the target image encoder to obtain a first image feature; converting the first image feature into a first text feature based on a first mapping module; Based on the target text encoder, the text data is encoded to obtain a second text feature; based on a second mapping module, the second text feature is converted into a second image feature; Determine a text loss function based on the first text feature and the second text feature, and adjust parameters of the first mapping module based on the text loss function; determining an image loss function based on the first image feature and the second image feature, and adjusting parameters of the second mapping module based on the image loss function; The above steps are iteratively performed to obtain the image-text mapping module and the text-image mapping module when the second training end condition is met.

6. The method according to claim 1, characterized in that The classifying the medical image data based on the image classification model to obtain a classification result includes: Performing encoding processing on the medical image based on the image encoder to obtain medical image features; Inputting the medical image features into the classifier to obtain classification labels of multiple dimensions; Prompt data corresponding to the classification label is determined based on the classification label.

7. The method according to claim 6, characterized in that The image-text migration model further includes an image-text mapping module; the feature space includes prompt data text features of the prompt data corresponding to the classification label; The step of searching in the feature space to obtain a prompt data group matching the medical image features of the medical image data includes: Converting the medical image features into medical text features based on the image-text mapping module; In the feature space, the medical text features and the prompt data text features are compared to determine a prompt data group that matches the medical image features of the medical image data.

8. A cross-modal text generation device, characterized in that: include: An image-text migration model loading module, used to load an image-text migration model, wherein the image-text migration model includes an image encoder and a text encoder; An image classification model fine-tuning module, used to construct an image classification model based on the image encoder, and fine-tune the image classification model to update the image classification model; the image classification model includes the image encoder and the classifier; An image classification module, used to obtain medical image data, classify the medical image data based on the image classification model, and obtain a classification result; the classification result includes classification labels of multiple dimensions and prompt data corresponding to the classification labels; A prompt data group retrieval module is used to encode the prompt data corresponding to the classification label based on the text encoder to form a feature space; Searching in the feature space to obtain a prompt data group matching the medical image features of the medical image data; A text data generating module is used to generate text data corresponding to the medical image data based on the prompt data group.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the cross-modal text generation method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the cross-modal text generation method described in any one of claims 1-7 when executed.

Images