Image information identification method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of image information recognition, is applied to medical image information recognition and textualized description scenes, and relates to an image information recognition method, device and equipment and a storage medium thereof.The method comprises the steps that an original medical image needing information recognition is acquired; performing medical image visual feature extraction on the preprocessed medical image; mapping the medical image visual features to a target language embedding space through a preset linear projection layer, and obtaining language embedding vector representation; generating target medical information corresponding to the original medical image according to the language embedded vector representation code; and outputting target medical information through the multi-modal medical information generation model to complete image information identification of the original medical image. The image information identification method is applied to medical image information identification, the medical image information can be identified in an artificial intelligence mode, the medical image analysis time of medical workers is shortened, the method is more accurate, and the misjudgment risk of the medical workers is reduced.

Description

Technical Field

[0001] The present application relates to the field of image information recognition technology, and is applied to medical image information recognition and textual description scenarios, and in particular to an image information recognition method, device, equipment and storage medium thereof. Background Art

[0002] Medical imaging technology is a method of observing and analyzing the internal structure and function of the human body by using various imaging technologies to assist doctors in diagnosing and making treatment decisions for diseases and lesions.

[0003] The industry products in the field of medical imaging technology are not mature enough. In order to improve the accuracy and efficiency of medical imaging diagnosis, automated and intelligent diagnostic methods have become a hot topic of research. In the past few decades, computer-aided diagnosis systems (CAD) have been widely studied and applied. These systems can automatically detect and identify abnormal areas in medical images through image processing and analysis technology, providing doctors with valuable reference information. At present, when analyzing medical images, there are few medical images to be analyzed, which leads to the problem of inaccurate analysis results. Moreover, the analysis results often require medical experts and assistants to discuss together to finally generate a medical description text, which consumes a lot of analysis time and is not intelligent enough. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an image information recognition method, device, equipment and storage medium thereof to solve the problem that when analyzing medical images, there are few medical images to be analyzed, resulting in inaccurate analysis results, and the analysis results often require discussion between medical experts and assistants to finally generate a medical description text, which consumes a lot of analysis time and is not intelligent enough.

[0005] In order to solve the above technical problems, the present application provides an image information recognition method, which adopts the following technical solutions:

[0006] A method for image information recognition comprises the following steps:

[0007] Obtaining original medical images that require information identification;

[0008] Preprocessing the original medical image to obtain a preprocessed medical image;

[0009] Inputting the preprocessed medical image into a pre-trained visual feature extraction model to extract visual features of the medical image;

[0010] Mapping the visual features of the medical image to the target language embedding space through a preset linear projection layer to obtain a language embedding vector representation corresponding to the original medical image;

[0011] Inputting the language embedding vector representation into a pre-trained multimodal medical information generation model, and encoding and generating target medical information corresponding to the original medical image based on pre-deployed medical information knowledge;

[0012] The target medical information is outputted through the multimodal medical information generation model to complete the image information recognition of the original medical image.

[0013] Furthermore, the step of preprocessing the original medical image to obtain the preprocessed medical image specifically includes:

[0014] According to a preset sampling size and sampling interval, the original medical image is segmented and sampled to obtain a medical image set after segmentation and sampling;

[0015] Performing magnification processing on all medical images in the medical image set, magnifying all medical images to the original size of the original medical images, and completing preprocessing of the original medical images;

[0016] The enlarged images of all the medical images are used as the preprocessed medical images.

[0017] Furthermore, the visual feature extraction model includes a visual feature extraction model based on a ResNet-50 network structure. Before executing the step of inputting the preprocessed medical image into the pretrained visual feature extraction model to extract the visual features of the medical image, the method further includes:

[0018] Obtaining the center point position information corresponding to each of the medical images in the medical image set in the original medical image;

[0019] According to the center position information, all medical images in the medical image set are numbered with distinguishing numbers in a sequence from left to right and from top to bottom, wherein the distinguishing numbers are numbered from 1 to N, and N is a positive integer greater than 1;

[0020] The step of inputting the preprocessed medical image into a pre-trained visual feature extraction model to extract visual features of the medical image specifically includes:

[0021] Inputting the preprocessed medical images into the visual feature extraction model based on the ResNet-50 network structure in sequence according to the distinguishing digital numbers;

[0022] The ResNet-50 network structure is used to extract multi-scale feature maps of the medical images corresponding to all distinguishing digital numbers, and a multi-scale feature map corresponding to each distinguishing digital number is obtained;

[0023] The multi-scale feature maps corresponding to the same distinguishing digital number are uniformly scaled to the sampling size, and the feature maps are fused to obtain a fused feature map corresponding to each distinguishing digital number;

[0024] According to the sampling interval, the fused feature maps corresponding to all distinguishing digital numbers are stacked and stitched to obtain a final feature map with the same scale as the original medical image;

[0025] The final feature map is input into a preset convolution layer for convolution processing to obtain an image feature vector corresponding to the visual feature of the medical image.

[0026] Furthermore, before executing the step of mapping the visual features of the medical image to the target language embedding space via a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image, the method further includes:

[0027] Acquire a batch of medical images annotated with language description texts to generate a training atlas, wherein the language description texts include medical lesion representation texts;

[0028] Preprocessing all medical images in the training atlas to obtain preprocessed medical images;

[0029] Inputting the preprocessed medical image into a visual feature extraction model to be trained, performing learning training on the visual feature extraction model to obtain a pretrained visual feature extraction model, and extracting image feature vectors of all medical images in the training atlas;

[0030] Inputting the language description texts corresponding to all the medical images into the language feature extraction component to be trained, performing learning and training on the language feature extraction component, obtaining the trained language feature extraction component, and extracting the language feature vectors corresponding to all the language description texts respectively;

[0031] Deploy the paired image feature vectors and language feature vectors as prior knowledge to the linear projection layer according to the annotation relationship;

[0032] The step of mapping the medical image visual features to the target language embedding space via a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image specifically includes:

[0033] Mapping the image feature vector corresponding to the visual feature of the medical image to the target language embedding space through the linear projection layer to convert the image feature vector into the corresponding language feature vector;

[0034] The image feature vector and the language feature vector are fused by using a multimodal fusion technique, and a fusion result is obtained as a language embedding vector representation corresponding to the original medical image.

[0035] Furthermore, before executing the step of inputting the language embedding vector representation into a pre-trained multimodal medical information generation model and encoding and generating target medical information corresponding to the original medical image according to pre-deployed medical information knowledge, the method further includes:

[0036] Deploy all medical images in the training atlas and the language description texts corresponding to all medical images as medical information knowledge into the multimodal medical information generation model to be trained;

[0037] According to the annotation relationship, the image feature vectors and language feature vectors of all the medical images in the training atlas are fused using multimodal fusion technology to obtain language embedding vector representations corresponding to all the medical images in the training atlas;

[0038] Inputting the language embedding vector representations corresponding to all medical images in the training atlas into the multimodal medical information generation model to be trained, performing learning and training on the multimodal medical information generation model, and obtaining a pre-trained multimodal medical information generation model;

[0039] The step of inputting the language embedding vector representation into a pre-trained multimodal medical information generation model, and encoding and generating target medical information corresponding to the original medical image according to the pre-deployed medical information knowledge, specifically includes:

[0040] Inputting the language embedding vector representation corresponding to the original medical image into a pre-trained multimodal medical information generation model;

[0041] According to the language embedding vector representation corresponding to the original medical image, all language description texts covered by the original medical image are screened out from the pre-deployed medical information knowledge;

[0042] All the language description texts are encoded to generate a medical information encoding result corresponding to the original medical image.

[0043] Furthermore, the multimodal medical information generation model includes a medical image generation component, a language description text generation component, a first comparison component, a second comparison component, a learning and training adjustment component, and a loss verification component. The language embedding vector representations corresponding to all medical images in the training atlas are input into the multimodal medical information generation model to be trained, the multimodal medical information generation model is trained, and the step of obtaining a pre-trained multimodal medical information generation model specifically includes:

[0044] Step 601, encoding the language embedding vector representations corresponding to all medical images in the training atlas to obtain encoding processing results;

[0045] Step 602, inputting the encoding processing result into the medical image generation component for decoding processing, generating medical images corresponding to all language embedding vector representations as training generated images;

[0046] Step 603, inputting the encoding processing result into the language description text generation component for decoding processing, generating language description texts corresponding to all language embedding vector representations respectively as training generated texts;

[0047] Step 604, comparing the training generated image with all medical images in the training atlas through the first comparison component to obtain a first comparison result;

[0048] Step 605, comparing the training generated text with the language description texts corresponding to all medical images in the training atlas through the second comparison component to obtain a second comparison result;

[0049] Step 606: input the first comparison result and the second comparison result into the loss verification component to obtain a learning training loss value;

[0050] Step 607, if the loss value exceeds the preset loss threshold, the multimodal medical information generation model is adjusted by the learning and training adjustment component, and steps 601 to 606 are repeatedly performed until the loss value does not exceed the preset loss threshold, and the learning and training of the multimodal medical information generation model is completed;

[0051] Step 608: If the loss value does not exceed the preset loss threshold, the multimodal medical information generation model that has been learned and trained is obtained as the pre-trained multimodal medical information generation model.

[0052] Furthermore, the step of outputting the target medical information through the multimodal medical information generation model to complete the step of recognizing the image information of the original medical image specifically includes:

[0053] Using the language description text generation component to decode the medical information encoding result corresponding to the original medical image to obtain the target medical information;

[0054] The target medical information is used as the image information recognition result of the original medical image.

[0055] In order to solve the above technical problems, the embodiment of the present application also provides an image information recognition device, which adopts the following technical solution:

[0056] An image information recognition device, comprising:

[0057] The original medical image acquisition module is used to acquire the original medical image that needs to be identified;

[0058] A medical image preprocessing module, used to preprocess the original medical image to obtain a preprocessed medical image;

[0059] A visual feature extraction module, used to input the preprocessed medical image into a pre-trained visual feature extraction model to extract visual features of the medical image;

[0060] A linear mapping module, used to map the visual features of the medical image to the target language embedding space through a preset linear projection layer, and obtain a language embedding vector representation corresponding to the original medical image;

[0061] A multimodal encoding processing module, used for inputting the language embedding vector representation into a pre-trained multimodal medical information generation model, and encoding and generating target medical information corresponding to the original medical image according to pre-deployed medical information knowledge;

[0062] The target medical information output module is used to output the target medical information through the multimodal medical information generation model to complete the image information recognition of the original medical image.

[0063] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0064] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned image information recognition method when executing the computer-readable instructions.

[0065] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0066] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the image information recognition method as described above.

[0067] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0068] The image information recognition method described in the embodiment of the present application is to obtain the original medical image that needs to be recognized; pre-process the original medical image to obtain the pre-processed medical image; input the pre-processed medical image into a pre-trained visual feature extraction model to extract the visual features of the medical image; map the visual features of the medical image to the target language embedding space through a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image; input the language embedding vector representation into a pre-trained multimodal medical information generation model, and encode and generate the target medical information corresponding to the original medical image based on the pre-deployed medical information knowledge; output the target medical information through the multimodal medical information generation model to complete the image information recognition of the original medical image. By using the image information recognition method in medical image information recognition, it is possible to use artificial intelligence to recognize medical image information, reduce the medical image analysis time of medical workers, and be more accurate, reducing the risk of misjudgment of medical workers. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0071] Figure 2 is a flow chart of an embodiment of the image information recognition method according to the present application;

[0072] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;

[0073] Figure 4 It is a flowchart of a specific embodiment of the linear projection layer construction method in the image information recognition method described in this application;

[0074] Figure 5 It is a flowchart of a specific embodiment of learning and training the multimodal medical information generation model in the image information recognition method described in the present application;

[0075] Figure 6 yes Figure 5 A flowchart of a specific embodiment of step 503 is shown;

[0076] Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown;

[0077] Figure 8 is a structural schematic diagram of an embodiment of an image information recognition device according to the present application;

[0078] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0079] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0080] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0081] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0082] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0083] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0084] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0085] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0086] It should be noted that the image information recognition method provided in the embodiment of the present application is generally executed by a terminal device, and accordingly, the image information recognition device is generally arranged in the terminal device.

[0087] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0088] Continue to refer Figure 2 , shows a flow chart of an embodiment of an image information recognition method according to the present application. The image information recognition method comprises the following steps:

[0089] Step 201: Acquire the original medical image that needs to be identified.

[0090] In this embodiment, the original medical image includes a two-dimensional image of the human body generated by using X-ray radiation and a medical detector, such as an X-ray medical image, a three-dimensional image with a higher resolution scanned by a CT scanner, such as a CT medical scan image, a three-dimensional image with a higher contrast created by magnetic resonance imaging technology, an image of the human body generated by ultrasonic imaging technology, and a medical image of human organs generated by nuclear medical imaging technology. The original medical image refers to the latest medical image that has just been generated without adding medical information description. By using the image information recognition method in medical image information recognition, medical image information can be recognized by artificial intelligence, which reduces the medical image analysis time of medical workers, is more accurate, and reduces the risk of misjudgment by medical workers.

[0091] Step 202: pre-process the original medical image to obtain a pre-processed medical image.

[0092] In this embodiment, the step of preprocessing the original medical image to obtain the preprocessed medical image specifically includes: segmenting and sampling the original medical image according to a preset sampling size and sampling interval to obtain a medical image set after the segmentation and sampling; enlarging all medical images in the medical image set to enlarge all medical images to the original size of the original medical image to complete the preprocessing of the original medical image; and using the enlarged images of all medical images as the preprocessed medical image.

[0093] The original medical image is segmented and sampled by using a preset sampling size and sampling interval to obtain a medical image set after segmentation and sampling. This achieves the goal of expanding the medical image set through sampling when performing medical information recognition on the original medical image, thereby avoiding the problem of low persuasiveness of the result when only a single image is used for medical information recognition. Afterwards, all the medical images are enlarged to the original size of the original medical image, thereby ensuring the consistency of the size of the medical images used for subsequent visual feature extraction.

[0094] Step 203: input the preprocessed medical image into a pre-trained visual feature extraction model to extract the visual features of the medical image.

[0095] In this embodiment, the visual feature extraction model includes a visual feature extraction model based on the ResNet-50 network structure, and the visual feature extraction model based on the ResNet-50 network structure is, for example, a medical image visual feature extraction model based on MedCL IP. Among them, the feature extraction structure of the MedCL IP medical image visual feature extraction model adopts the ResNet-50 residual network structure, and the specific image processing and feature extraction can be combined with the Matlab image processing toolbox, which provides a wealth of functions and methods for image processing, including but not limited to image filtering, image segmentation, feature extraction, etc. In this embodiment, only the visual feature extraction structure of the MedCL IP medical image visual feature extraction model is used to obtain the visual features of the medical image.

[0096] Specifically, before executing the step of inputting the preprocessed medical image into the pre-trained visual feature extraction model to extract the visual features of the medical image, the method also includes: obtaining the corresponding center point position information of all medical images in the medical image set in the original medical image; according to the center position information, all medical images in the medical image set are numbered with distinguishing numbers in order from left to right and from top to bottom, wherein the distinguishing numbers are numbered from 1 to N, and N is a positive integer greater than 1.

[0097] By combining the center point position information with distinguishing digital numbers for all medical images in the medical image set, it is convenient to subsequently perform layered comprehensive splicing processing on the fused feature maps corresponding to all distinguishing digital numbers to obtain a final feature map with the same scale as the original medical image.

[0098] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes:

[0099] Step 301, inputting the preprocessed medical images into the visual feature extraction model based on the ResNet-50 network structure in sequence according to the distinguishing digital numbers;

[0100] Step 302, using the ResNet-50 network structure to extract multi-scale feature maps of the medical images corresponding to all distinguishing digital numbers, to obtain a multi-scale feature map corresponding to each distinguishing digital number;

[0101] Specifically, a pyramid sampling method can be used to extract multi-scale feature maps of medical images corresponding to all distinguishing digital numbers, and obtain a multi-scale feature map corresponding to each distinguishing digital number, thereby ensuring that the extracted medical image feature information is richer.

[0102] Step 303, uniformly scaling the multi-scale feature maps corresponding to the same distinguishing digital number to the sampling size, and performing feature map fusion to obtain a fused feature map corresponding to each distinguishing digital number;

[0103] By uniformly scaling the multi-scale feature maps corresponding to the same distinguishing digital number to the sampling size and performing feature map fusion, the fused feature map corresponding to each distinguishing digital number finally obtained contains richer image features, ensuring that subsequent image information recognition results are more accurate.

[0104] Step 304, performing a stacking and comprehensive splicing process on the fused feature maps corresponding to all distinguishing digital numbers according to the sampling interval distance, to obtain a final feature map having a scale consistent with the original medical image;

[0105] In essence, the fused feature maps corresponding to all distinguishing digital numbers are subjected to stacked comprehensive splicing processing, that is, the fused feature maps corresponding to all distinguishing digital numbers are subjected to secondary fusion processing to obtain the fused feature map corresponding to the original medical image. In the processing of the original medical image, sampling processing, multi-scale feature map extraction, and two feature map fusions are successively performed, so that the final feature map obtained with the same scale as the original medical image has richer and more complex image features, thereby improving the accuracy of subsequent image information recognition results.

[0106] Step 305: input the final feature map into a preset convolutional layer for convolution processing to obtain an image feature vector corresponding to the visual feature of the medical image.

[0107] Specifically, the convolution layer can be implemented using a conventional convolutional neural network, the purpose of which is to obtain an image feature vector in the form of a numerical matrix by convolution processing the image feature map.

[0108] Step 204 , mapping the medical image visual features to the target language embedding space via a preset linear projection layer, and obtaining a language embedding vector representation corresponding to the original medical image.

[0109] In this embodiment, before executing the step of mapping the visual features of the medical image to the target language embedding space via a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image, the method also includes a linear projection layer construction method.

[0110] Continue to refer Figure 4 , Figure 4 It is a flowchart of a specific embodiment of the linear projection layer construction method in the image information recognition method described in this application, including:

[0111] Step 401, obtaining a batch of medical images annotated with language description texts to generate a training atlas, wherein the language description texts include medical lesion representation texts;

[0112] By using a batch of medical images with annotated language description text to generate a training atlas, the learning and training of the visual feature extraction model and the multimodal medical information generation model have a large amount of data support and basis, ensuring that the trained model has practical persuasiveness when processing new medical images, and also ensuring that the construction of the linear projection layer and the feature conversion are based on the training atlas, ensuring that the feature conversion has a strong scientific basis. Specifically, the medical lesion representation text includes the volume, size, and disease information of the lesion tissue.

[0113] Step 402, preprocessing all medical images in the training atlas to obtain preprocessed medical images;

[0114] Specifically, the preprocessing is consistent with the preprocessing in step 202, ensuring that the method used in the subsequent preprocessing of the original medical image is consistent with the method used in the preprocessing of all medical images in the training atlas.

[0115] Step 403, inputting the preprocessed medical image into the visual feature extraction model to be trained, performing learning training on the visual feature extraction model, acquiring the pretrained visual feature extraction model, and extracting image feature vectors of all medical images in the training atlas;

[0116] Specifically, the visual feature extraction model may be the visual feature extraction model based on the ResNet-50 network structure, and a pre-trained visual feature extraction model may be obtained through a learning and training method.

[0117] Step 404, inputting the language description texts corresponding to all the medical images into the language feature extraction component to be trained, performing learning training on the language feature extraction component, obtaining the trained language feature extraction component, and extracting the language feature vectors corresponding to all the language description texts;

[0118] Specifically, the language feature extraction component includes a BERT-based medical information feature extraction component, which mainly adopts natural language understanding technology and combines the language description texts corresponding to all the medical images to learn the language feature vectors, and obtains the trained language feature extraction component, so as to facilitate the subsequent inference of the corresponding language description text when obtaining the language feature vectors.

[0119] Step 405 : Deploy the paired image feature vectors and language feature vectors as prior knowledge to the linear projection layer according to the annotation relationship.

[0120] By deploying paired image feature vectors and language feature vectors as prior knowledge to the linear projection layer, when the image feature vector of the original medical image is subsequently obtained and mapped through the linear projection layer, the language feature vector corresponding to the image feature vector is obtained, which plays a role in feature conversion, that is, converting image features into corresponding text features, so as to facilitate the subsequent generation of medical description information based on the text features.

[0121] In this embodiment, the step of mapping the visual features of the medical image to the target language embedding space via a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image specifically includes: mapping the image feature vector corresponding to the visual features of the medical image to the target language embedding space via the linear projection layer to convert the image feature vector into a corresponding language feature vector; and fusing the image feature vector and the language feature vector using a multimodal fusion technique to obtain a fusion result as the language embedding vector representation corresponding to the original medical image.

[0122] Specifically, the multimodal fusion technology includes splicing the image feature vector and the language feature vector, or first calculating the eigenvalues ​​of the image feature vector and the language feature vector respectively, and then splicing the two eigenvalues. Its main purpose is to integrate the image feature vector and the language feature vector. The specific integration method is determined by the actual application scenario and no feature limitation is made here.

[0123] Step 205: input the language embedding vector representation into a pre-trained multimodal medical information generation model, and encode and generate target medical information corresponding to the original medical image based on pre-deployed medical information knowledge.

[0124] Continue to refer Figure 5 , Figure 5 is a flowchart of a specific embodiment of learning and training the multimodal medical information generation model in the image information recognition method described in the present application, including:

[0125] Step 501, deploying all medical images in the training atlas and the language description texts corresponding to all medical images as medical information knowledge into the multimodal medical information generation model to be trained;

[0126] In this embodiment, the multimodal medical information generation model includes a medical image generation component, a language description text generation component, a first comparison component, a second comparison component, a learning training adjustment component and a loss verification component. Specifically, the medical image generation component can use an image feature decoding component, the language description text generation component can use a text feature decoding component, the first comparison component can use an image similarity comparison component, and the second comparison component can use a text similarity comparison component. The multimodal medical information generation model, for example: a multimodal medical information generation model based on MLLM, includes at least two modal parts: image processing and text processing.

[0127] Step 502, according to the annotation relationship, the image feature vectors and language feature vectors of all the medical images in the training atlas are fused using a multimodal fusion technology to obtain language embedding vector representations corresponding to all the medical images in the training atlas;

[0128] Specifically, after obtaining the language embedding vector representations corresponding to all the medical images in the training atlas, since the language embedding vector representations are generated by corresponding pairs of image feature vectors and language feature vectors through a multimodal fusion method, each language embedding vector representation contains both the image feature vector and the language feature vector corresponding to the image feature vector. Therefore, a decoding method can be used subsequently to obtain the decoded output image and descriptive text information.

[0129] Step 503: input the language embedding vector representations corresponding to all the medical images in the training atlas into the multimodal medical information generation model to be trained, perform learning training on the multimodal medical information generation model, and obtain a pre-trained multimodal medical information generation model.

[0130] In this embodiment, since the language description text generation component is the component actually used to process the latest medical images, it can be understood that the training goal is to train the language description text generation component in the multimodal medical information generation model. As for the medical image generation component, the first comparison component, the second comparison component, the learning training adjustment component and the loss verification component, they are only components for auxiliary training and comparison verification adjustment to ensure the accuracy of the multimodal medical information generation model. They are only used in the training process, and these components are not involved in the subsequent actual generation of medical knowledge information for the latest medical images.

[0131] Continue to refer Figure 6 , Figure 6 yes Figure 5 The flowchart of a specific embodiment of step 503 shown includes:

[0132] Step 601, encoding the language embedding vector representations corresponding to all medical images in the training atlas to obtain encoding processing results;

[0133] Specifically, the language embedding vector representations corresponding to all medical images in the training atlas are encoded to obtain the encoding results. An encoder that performs comprehensive visual and text encoding can be used to obtain the encoding results. At this time, the encoding results can be decoded by the medical image generation component to obtain the corresponding medical image, and can also be decoded by the language description text generation component to obtain the corresponding language description text; however, the language embedding vector representation can also be encoded by a visual encoder and a text encoder respectively. After that, the medical image generation component is followed by the visual encoder to decode the corresponding medical image, and the language description text generation component is followed by the text encoder to decode the corresponding language description text. Specifically, whether to use the encoder that performs comprehensive visual and text encoding, or to use a visual encoder and a text encoder respectively, is determined by the actual R&D personnel according to needs and is not limited here.

[0134] Step 602, inputting the encoding processing result into the medical image generation component for decoding processing, generating medical images corresponding to all language embedding vector representations as training generated images;

[0135] Step 603, inputting the encoding processing result into the language description text generation component for decoding processing, generating language description texts corresponding to all language embedding vector representations respectively as training generated texts;

[0136] Step 604, comparing the training generated image with all medical images in the training atlas through the first comparison component to obtain a first comparison result;

[0137] Specifically, the first comparison component can use an image similarity comparison component to compare the training generated image with all medical images in the training atlas. In fact, if the similarity between the training generated image and all medical images in the training atlas exceeds a preset image similarity threshold, the training of the medical image generation component is completed.

[0138] Step 605, comparing the training generated text with the language description texts corresponding to all medical images in the training atlas through the second comparison component to obtain a second comparison result;

[0139] Specifically, the second comparison component can use a text similarity comparison component to compare the training generated text with the language description texts corresponding to all medical images in the training atlas. In essence, if the training generated text and the language description texts corresponding to all medical images in the training atlas exceed a preset text similarity threshold, the training of the language description text generation component is completed. The text similarity comparison component has a built-in text similarity comparison algorithm, such as a cosine similarity algorithm, a Jaccard similarity algorithm, a text similarity algorithm based on Word2Vec and word vectors, etc.

[0140] Through the first comparison component and the second comparison component, the output items of the medical image generation component and the text similarity comparison component are compared and verified with all medical images in the training atlas and all medical images deployed in the multimodal medical information generation model to be trained, thereby ensuring the learning and training results of the multimodal medical information generation model and ensuring the accuracy of the model output results in subsequent actual use.

[0141] Step 606: input the first comparison result and the second comparison result into the loss verification component to obtain a learning training loss value;

[0142] Step 607, if the loss value exceeds the preset loss threshold, the multimodal medical information generation model is adjusted by the learning and training adjustment component, and steps 601 to 606 are repeatedly performed until the loss value does not exceed the preset loss threshold, and the learning and training of the multimodal medical information generation model is completed;

[0143] Step 608: If the loss value does not exceed the preset loss threshold, the multimodal medical information generation model that has been learned and trained is obtained as the pre-trained multimodal medical information generation model.

[0144] Continue to refer Figure 7 , Figure 7 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes:

[0145] Step 701, inputting the language embedding vector representation corresponding to the original medical image into a pre-trained multimodal medical information generation model;

[0146] Step 702, based on the language embedding vector representation corresponding to the original medical image, filter out all language description texts covered by the original medical image from the pre-deployed medical information knowledge;

[0147] Step 703: Encode all the language description texts to generate a medical information encoding result corresponding to the original medical image.

[0148] Specifically, the image information recognition task is converted into a language text information encoding task. Compared with image information recognition, the use of language text information encoding makes the model more interpretable and improves the persuasiveness of the output results.

[0149] Step 206: Output the target medical information through the multimodal medical information generation model to complete the image information recognition of the original medical image.

[0150] In this embodiment, the step of outputting the target medical information through the multimodal medical information generation model to complete the image information recognition of the original medical image specifically includes: using the language description text generation component to decode the medical information encoding result corresponding to the original medical image to obtain the target medical information; and using the target medical information as the image information recognition result of the original medical image.

[0151] Specifically, in the actual processing of the original medical image, only the visual feature extraction model is used to extract the image visual features, and then the linear projection layer is applied to perform feature conversion to convert the image feature vector into a language feature vector. Then, the multimodal fusion technology is used to fuse the image feature vector and the language feature vector to generate a language embedding vector representation. Finally, the trained language description text generation component is used to decode the language embedding vector representation to obtain the target medical information.

[0152] This application obtains the original medical image that needs to be identified; pre-processes the original medical image to obtain the pre-processed medical image; inputs the pre-processed medical image into a pre-trained visual feature extraction model to extract the visual features of the medical image; maps the visual features of the medical image to the target language embedding space through a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image; inputs the language embedding vector representation into a pre-trained multimodal medical information generation model, and encodes and generates the target medical information corresponding to the original medical image based on the pre-deployed medical information knowledge; outputs the target medical information through the multimodal medical information generation model to complete the image information recognition of the original medical image. By using the image information recognition method in medical image information recognition, it is possible to use artificial intelligence to identify medical image information, reduce the medical image analysis time of medical workers, and be more accurate, reducing the risk of misjudgment of medical workers.

[0153] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0154] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0155] In the embodiment of the present application, the original medical image that needs to be identified is obtained; the original medical image is preprocessed to obtain the preprocessed medical image; the preprocessed medical image is input into the pre-trained visual feature extraction model to extract the visual features of the medical image; the medical image visual features are mapped to the target language embedding space through a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image; the language embedding vector representation is input into the pre-trained multimodal medical information generation model, and according to the pre-deployed medical information knowledge, the target medical information corresponding to the original medical image is encoded and generated; the target medical information is output through the multimodal medical information generation model to complete the image information recognition of the original medical image. By using the image information recognition method in medical image information recognition, it is possible to use artificial intelligence to identify medical image information, reduce the medical image analysis time of medical workers, and be more accurate, reducing the risk of misjudgment of medical workers.

[0156] Further references Figure 8 , as a response to the above Figure 2 The present application provides an embodiment of an image information recognition device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0157] like Figure 8 As shown, the image information recognition device 800 described in this embodiment includes: an original medical image acquisition module 801, a medical image preprocessing module 802, a visual feature extraction module 803, a linear mapping module 804, a multimodal encoding processing module 805 and a target medical information output module 806.

[0158] in:

[0159] The original medical image acquisition module 801 is used to acquire the original medical image that needs to be identified;

[0160] The medical image preprocessing module 802 is used to preprocess the original medical image to obtain a preprocessed medical image;

[0161] A visual feature extraction module 803 is used to input the preprocessed medical image into a pre-trained visual feature extraction model to extract visual features of the medical image;

[0162] A linear mapping module 804 is used to map the medical image visual features to a target language embedding space via a preset linear projection layer to obtain a language embedding vector representation corresponding to the original medical image;

[0163] The multimodal encoding processing module 805 is used to input the language embedding vector representation into a pre-trained multimodal medical information generation model, and encode and generate target medical information corresponding to the original medical image according to the pre-deployed medical information knowledge;

[0164] The target medical information output module 806 is used to output the target medical information through the multimodal medical information generation model to complete the image information recognition of the original medical image.

[0165] The image information recognition device proposed in this application is applied to medical image information recognition and textual description scenarios, by obtaining the original medical image that needs to be recognized; preprocessing the original medical image to obtain the preprocessed medical image; inputting the preprocessed medical image into the pre-trained visual feature extraction model to extract the medical image visual features; mapping the medical image visual features to the target language embedding space through a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image; inputting the language embedding vector representation into the pre-trained multimodal medical information generation model, and encoding and generating the target medical information corresponding to the original medical image according to the pre-deployed medical information knowledge; outputting the target medical information through the multimodal medical information generation model to complete the image information recognition of the original medical image. By using the image information recognition method in medical image information recognition, it is possible to use artificial intelligence to recognize medical image information, reduce the medical image analysis time of medical workers, and be more accurate, reducing the risk of misjudgment of medical workers.

[0166] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0167] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0168] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0169] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 9 having components 9a-9c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0170] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0171] The memory 9a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 9a can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 9a can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 9a can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 9a is generally used to store the operating system and various application software installed on the computer device 9, such as a computer-readable instruction of an image information recognition method, etc. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or are to be output.

[0172] The processor 9b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 9b is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to run the computer-readable instructions stored in the memory 9a or process data, such as computer-readable instructions for running the image information recognition method.

[0173] The network interface 9c may include a wireless network interface or a wired network interface. The network interface 9c is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0174] The computer device proposed in this embodiment belongs to the field of image information recognition technology and is applied to medical image information recognition and textual description scenarios. This application obtains the original medical image that needs to be recognized; pre-processes the original medical image to obtain the pre-processed medical image; inputs the pre-processed medical image into a pre-trained visual feature extraction model to extract the visual features of the medical image; maps the visual features of the medical image to the target language embedding space through a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image; inputs the language embedding vector representation into a pre-trained multimodal medical information generation model, and encodes and generates the target medical information corresponding to the original medical image based on the pre-deployed medical information knowledge; outputs the target medical information through the multimodal medical information generation model to complete the image information recognition of the original medical image. By using the image information recognition method in medical image information recognition, it is possible to use artificial intelligence to recognize medical image information, reduce the medical image analysis time of medical workers, and be more accurate, reducing the risk of misjudgment of medical workers.

[0175] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the image information recognition method as described above.

[0176] The computer-readable storage medium proposed in this embodiment is applied to medical image information recognition and textual description scenarios. This application obtains the original medical image that needs to be recognized; pre-processes the original medical image to obtain the pre-processed medical image; inputs the pre-processed medical image into a pre-trained visual feature extraction model to extract the visual features of the medical image; maps the visual features of the medical image to the target language embedding space through a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image; inputs the language embedding vector representation into a pre-trained multimodal medical information generation model, and encodes and generates the target medical information corresponding to the original medical image based on the pre-deployed medical information knowledge; outputs the target medical information through the multimodal medical information generation model to complete the image information recognition of the original medical image. By using the image information recognition method in medical image information recognition, it is possible to use artificial intelligence to recognize medical image information, reduce the medical image analysis time of medical workers, and be more accurate, reducing the risk of misjudgment of medical workers.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0178] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for image information recognition, characterized in that: The steps include: Obtaining original medical images that require information identification; Preprocessing the original medical image to obtain a preprocessed medical image; Inputting the preprocessed medical image into a pre-trained visual feature extraction model to extract visual features of the medical image; Mapping the visual features of the medical image to the target language embedding space through a preset linear projection layer to obtain a language embedding vector representation corresponding to the original medical image; Inputting the language embedding vector representation into a pre-trained multimodal medical information generation model, and encoding and generating target medical information corresponding to the original medical image based on pre-deployed medical information knowledge; The target medical information is outputted through the multimodal medical information generation model to complete the image information recognition of the original medical image.

2. The image information recognition method according to claim 1, characterized in that: The step of preprocessing the original medical image to obtain the preprocessed medical image specifically includes: According to a preset sampling size and sampling interval, the original medical image is segmented and sampled to obtain a medical image set after segmentation and sampling; Performing magnification processing on all medical images in the medical image set, magnifying all medical images to the original size of the original medical images, and completing preprocessing of the original medical images; The enlarged images of all the medical images are used as the preprocessed medical images.

3. The image information recognition method according to claim 2, characterized in that: The visual feature extraction model includes a visual feature extraction model based on a ResNet-50 network structure. Before executing the step of inputting the preprocessed medical image into the pretrained visual feature extraction model to extract the visual features of the medical image, the method further includes: Obtaining the center point position information corresponding to each of the medical images in the medical image set in the original medical image; According to the center position information, all medical images in the medical image set are numbered with distinguishing numbers in a sequence from left to right and from top to bottom, wherein the distinguishing numbers are numbered from 1 to N, and N is a positive integer greater than 1; The step of inputting the preprocessed medical image into a pre-trained visual feature extraction model to extract visual features of the medical image specifically includes: Inputting the preprocessed medical images into the visual feature extraction model based on the ResNet-50 network structure in sequence according to the distinguishing digital numbers; The ResNet-50 network structure is used to extract multi-scale feature maps of the medical images corresponding to all distinguishing digital numbers, and a multi-scale feature map corresponding to each distinguishing digital number is obtained; The multi-scale feature maps corresponding to the same distinguishing digital number are uniformly scaled to the sampling size, and the feature maps are fused to obtain a fused feature map corresponding to each distinguishing digital number; According to the sampling interval, the fused feature maps corresponding to all distinguishing digital numbers are stacked and stitched to obtain a final feature map with the same scale as the original medical image; The final feature map is input into a preset convolution layer for convolution processing to obtain an image feature vector corresponding to the visual feature of the medical image.

4. The image information recognition method according to claim 1, characterized in that: Before executing the step of mapping the visual features of the medical image to the target language embedding space via a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image, the method further includes: Acquire a batch of medical images annotated with language description texts to generate a training atlas, wherein the language description texts include medical lesion representation texts; Preprocessing all medical images in the training atlas to obtain preprocessed medical images; Inputting the preprocessed medical image into a visual feature extraction model to be trained, performing learning training on the visual feature extraction model to obtain a pretrained visual feature extraction model, and extracting image feature vectors of all medical images in the training atlas; Inputting the language description texts corresponding to all the medical images into the language feature extraction component to be trained, performing learning and training on the language feature extraction component, obtaining the trained language feature extraction component, and extracting the language feature vectors corresponding to all the language description texts respectively; Deploy the paired image feature vectors and language feature vectors as prior knowledge to the linear projection layer according to the annotation relationship; The step of mapping the medical image visual features to the target language embedding space via a preset linear projection layer to obtain the language embedding vector representation corresponding to the original medical image specifically includes: Mapping the image feature vector corresponding to the visual feature of the medical image to the target language embedding space through the linear projection layer to convert the image feature vector into the corresponding language feature vector; The image feature vector and the language feature vector are fused by using a multimodal fusion technique, and a fusion result is obtained as a language embedding vector representation corresponding to the original medical image.

5. The image information recognition method according to claim 4, characterized in that: Before executing the step of inputting the language embedding vector representation into a pre-trained multimodal medical information generation model, and encoding and generating target medical information corresponding to the original medical image according to pre-deployed medical information knowledge, the method further includes: Deploy all medical images in the training atlas and the language description texts corresponding to all medical images as medical information knowledge into the multimodal medical information generation model to be trained; According to the annotation relationship, the image feature vectors and language feature vectors of all the medical images in the training atlas are fused using multimodal fusion technology to obtain language embedding vector representations corresponding to all the medical images in the training atlas; Inputting the language embedding vector representations corresponding to all medical images in the training atlas into the multimodal medical information generation model to be trained, performing learning and training on the multimodal medical information generation model, and obtaining a pre-trained multimodal medical information generation model; The step of inputting the language embedding vector representation into a pre-trained multimodal medical information generation model, and encoding and generating target medical information corresponding to the original medical image according to the pre-deployed medical information knowledge, specifically includes: Inputting the language embedding vector representation corresponding to the original medical image into a pre-trained multimodal medical information generation model; According to the language embedding vector representation corresponding to the original medical image, all language description texts covered by the original medical image are screened out from the pre-deployed medical information knowledge; All the language description texts are encoded to generate a medical information encoding result corresponding to the original medical image.

6. The image information recognition method according to claim 5, characterized in that: The multimodal medical information generation model includes a medical image generation component, a language description text generation component, a first comparison component, a second comparison component, a learning and training adjustment component, and a loss verification component. The step of inputting the language embedding vector representations corresponding to all medical images in the training atlas into the multimodal medical information generation model to be trained, learning and training the multimodal medical information generation model, and obtaining a pre-trained multimodal medical information generation model specifically includes: Step 601, encoding the language embedding vector representations corresponding to all medical images in the training atlas to obtain encoding processing results; Step 602, inputting the encoding processing result into the medical image generation component for decoding processing, generating medical images corresponding to all language embedding vector representations as training generated images; Step 603, inputting the encoding processing result into the language description text generation component for decoding processing, generating language description texts corresponding to all language embedding vector representations respectively as training generated texts; Step 604, comparing the training generated image with all medical images in the training atlas through the first comparison component to obtain a first comparison result; Step 605, comparing the training generated text with the language description texts corresponding to all medical images in the training atlas through the second comparison component to obtain a second comparison result; Step 606: input the first comparison result and the second comparison result into the loss verification component to obtain a learning training loss value; Step 607, if the loss value exceeds the preset loss threshold, the multimodal medical information generation model is adjusted by the learning and training adjustment component, and steps 601 to 606 are repeatedly performed until the loss value does not exceed the preset loss threshold, and the learning and training of the multimodal medical information generation model is completed; Step 608: If the loss value does not exceed the preset loss threshold, the multimodal medical information generation model that has been learned and trained is obtained as the pre-trained multimodal medical information generation model.

7. The image information recognition method according to claim 6, characterized in that: The step of outputting the target medical information through the multimodal medical information generation model to complete the image information recognition of the original medical image specifically includes: Using the language description text generation component to decode the medical information encoding result corresponding to the original medical image to obtain the target medical information; The target medical information is used as the image information recognition result of the original medical image.

8. An image information recognition device, characterized in that: include: The original medical image acquisition module is used to acquire the original medical image that needs to be identified; A medical image preprocessing module, used to preprocess the original medical image to obtain a preprocessed medical image; A visual feature extraction module, used to input the preprocessed medical image into a pre-trained visual feature extraction model to extract visual features of the medical image; A linear mapping module, used to map the visual features of the medical image to the target language embedding space through a preset linear projection layer, and obtain a language embedding vector representation corresponding to the original medical image; A multimodal encoding processing module, used for inputting the language embedding vector representation into a pre-trained multimodal medical information generation model, and encoding and generating target medical information corresponding to the original medical image according to pre-deployed medical information knowledge; The target medical information output module is used to output the target medical information through the multimodal medical information generation model to complete the image information recognition of the original medical image.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the image information recognition method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image information recognition method according to any one of claims 1 to 7 are implemented.

Images