Medical visual question answering methods, devices, and electronic devices based on transfer learning

Abstract

This invention relates to the field of intelligent decision-making, and discloses a medical visual question-answering method, device, and electronic device based on transfer learning. The method includes: acquiring a user's medical question, which includes a question image and question text; extracting textual features from the question text; segmenting the question image into blocks; performing a random masking operation on the segmented images and marking the mask position information of the masked segments; extracting features from the visible segments; reconstructing the masked segments to obtain masked reconstructed images; fusing the masked reconstructed images with the visible feature images; normalizing the target image features and text features; calculating the cosine similarity between the normalized image features and the normalized text features; fusing the normalized image features and normalized text features when the cosine similarity is not less than a preset threshold; and querying the answer to the user's medical question from a pre-constructed medical question database. This invention can improve the accuracy of answers to user questions.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making, and in particular to a medical visual question-answering method, device, and electronic device based on transfer learning. Background Technology

[0002] Transfer learning is a machine learning method that uses a model developed for task A as an initial step in developing a model for task B, thereby reducing the learning cost of the model for task B and improving its performance. Transfer learning can be categorized by its learning method into sample-based transfer, feature-based transfer, model-based transfer, and relation-based transfer. This invention applies a model-based transfer learning method.

[0003] Currently, traditional visual question answering tasks typically train models on user question-text pair datasets to achieve the intended message. However, the datasets for commonly used medical visual question answering tasks are often very small. For example, the most commonly used VQA-RAD and SLAKE datasets only contain a few hundred to a few thousand images and corresponding question-answer pairs. If the model is trained directly on a small dataset, it is impossible to obtain a high-performance model, resulting in low accuracy in answering user questions. Summary of the Invention

[0004] This invention provides a medical visual question answering method, device, and electronic device based on transfer learning, the main purpose of which is to improve the accuracy of user question answers.

[0005] To achieve the above objectives, this invention provides a medical visual question answering method based on transfer learning, comprising:

[0006] The user's medical question is obtained, which includes a question image and a question text. The text features of the question text are extracted, and the question image is processed into image blocks to obtain block images.

[0007] A random masking operation is performed on the segmented image to obtain a visible segmented image and a masked segmented image. The mask position information of the masked segmented image is marked, and features are extracted from the visible segmented image to obtain a visible feature image.

[0008] Based on the visible feature image and the mask position information, image reconstruction is performed on the mask block image to obtain a mask reconstruction image. Then, based on the mask reconstruction image, the visible feature image, and the mask position information, the mask reconstruction image and the visible feature image are fused to obtain the target image features.

[0009] The target image features and the text features are normalized separately to obtain normalized image features and normalized text features. The cosine similarity between the normalized image features and the normalized text features is calculated. When the cosine similarity is not less than a preset threshold, the normalized image features and the normalized text features are fused to obtain question fusion features. Based on the question fusion features, the answer to the user's medical question is queried from a pre-built medical question database.

[0010] Optionally, the step of extracting features from the visible segmented image to obtain a visible feature image includes:

[0011] The visible block image is dimensionally mapped using the linear projection layer in the preset image feature extraction model to obtain the projection mapping vector;

[0012] Obtain the block sequence of the segmented image, query the position information of the visible segmented image in the block sequence, and obtain the visible position information;

[0013] The projection mapping vector and the visible location information are fused using the fusion layer in the preset image feature extraction model to obtain the visible feature image.

[0014] Optionally, the step of reconstructing the mask block image based on the visible feature image and the mask position information to obtain a mask reconstructed image includes:

[0015] Obtain the block image corresponding to the visible feature image, and calculate the frequency domain estimation function of the block image using a preset Wiener filtering algorithm;

[0016] Based on the frequency domain estimation function, the block-reconstructed image of the block image is calculated using inverse Fourier transform;

[0017] Based on the segmented reconstructed image and the mask position information, construct the mask reconstruction candidate image of the mask segmented image;

[0018] The quality index score of the mask reconstruction candidate image is calculated using a quality index function;

[0019] When the quality index score is less than a preset threshold, the candidate image for mask reconstruction is used as the mask reconstruction image.

[0020] Optionally, the Wiener filtering algorithm includes:

[0021]

[0022] in, Let H(u,v) represent the frequency domain estimation function corresponding to the spatial domain problem image, and let S represent the degradation function.η (u,v) represents the power spectrum of the noise, S f (u,v) represents the power spectrum of the spatial domain problem image, G(u,v) represents the Fourier transform function of the block image after masking, and u and v represent the frequency domain variables after Fourier transform of the spatial domain function.

[0023] Optionally, the quality indicator function includes:

[0024]

[0025] MSE represents the quality score of the mask-reconstructed image. y represents the value of the j-th reconstructed pixel in the i-th masked reconstructed image. ij The value of the j-th pixel in the original i-th block image corresponding to the mask position information is represented by m, where m represents the number of blocks in the mask reconstruction image corresponding to the block image, and n represents the number of pixels in each mask reconstruction image.

[0026] Optionally, the normalization processing of the target image features and the text features to obtain normalized image features and normalized text features includes:

[0027] The target image features and the text features are normalized using a normalization method to obtain the normalized image features and the normalized text features.

[0028] The normalization method includes:

[0029]

[0030]

[0031]

[0032] Among them, y k x represents the k-th normalized image feature value or the k-th normalized text feature value. l μ represents the k-th target image feature value or the k-th text feature value. k σ represents the mean of the k-th target image feature value or the k-th text feature value. l Let x represent the standard deviation of the k-th target image feature value or text feature value, and N represent the number of samples of the target image feature or text feature. ki This represents the i-th sample value of the k-th target image feature or the k-th text feature.

[0033] Optionally, the step of querying the answer to the user's medical question from a pre-built medical question database based on the question fusion features includes:

[0034] Obtain the fusion features to be matched from the pre-built medical question database, and calculate the cosine similarity between the question fusion features and the fusion features to be matched;

[0035] When the cosine similarity is not less than a preset threshold, the mapping relationship table between the fusion features to be matched and the question answers in the pre-built medical question database corresponding to the cosine similarity is queried to obtain the question answer of the user's medical question.

[0036] To address the aforementioned problems, the present invention also provides a medical visual question-answering device based on transfer learning, the device comprising:

[0037] The text feature extraction module is used to obtain the user's medical questions, which include a question image and question text. The module extracts the text features of the question text and performs image segmentation on the question image to obtain segmented images.

[0038] The mask processing module is used to perform random masking operations on the segmented image to obtain visible segmented images and masked segmented images, and to mark the mask position information of the masked segmented images, and to extract features from the visible segmented images to obtain visible feature images;

[0039] The image reconstruction module is used to reconstruct the mask block image based on the visible feature image and the mask position information to obtain a mask reconstructed image, and to fuse the mask reconstructed image and the visible feature image based on the mask reconstructed image, the visible feature image and the mask position information to obtain the target image features;

[0040] The question-answer generation module is used to normalize the target image features and the text features respectively to obtain normalized image features and normalized text features, calculate the cosine similarity between the normalized image features and the normalized text features, and when the cosine similarity is not less than a preset threshold, fuse the normalized image features and the normalized text features to obtain question fusion features. Based on the question fusion features, the module queries the answer to the user's medical question from a pre-built medical question database.

[0041] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0042] At least one processor; and,

[0043] A memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores a computer program that can be executed by the at least one processor to implement the medical visual question answering method based on transfer learning described above.

[0045] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the aforementioned transfer learning-based medical visual question answering method.

[0046] As can be seen, this embodiment of the invention first obtains the user's medical question, which includes a question image and question text, providing the operation object for subsequent method implementation. Extracting the textual features of the question text is a preprocessing step to obtain the semantic information of the user's medical question. The question image is then processed into image blocks, providing a standard data format for subsequent masking operations. Random masking is performed on the image blocks to obtain visible and masked image blocks. These blocks can mask certain areas of the image blocks, preventing them from participating in the processing, thereby improving image processing efficiency. The method of reconstructing the mask block image by marking its position information can provide a location basis for subsequent image reconstruction of the mask block image. Furthermore, feature extraction of the visible block image can remove redundant information and extract key information, thereby improving the model's execution efficiency. Secondly, based on the visible feature image and the mask position information, the embodiment of the invention reconstructs the mask block image, obtaining a reconstructed mask image. This reconstructed mask image can provide a guarantee for the subsequent reconstruction of the original problem image and the generation of target image features. The image, the visible feature image, and the mask position information are used to fuse the mask-reconstructed image and the visible feature image to obtain the target image features, which can yield the final question features to support subsequent searches for the answer to the question based on the question features. Furthermore, in this embodiment, the target image features and the text features are normalized respectively to obtain normalized image features and normalized text features. This eliminates the dimensional limitations of the image feature and text feature data and maps the vector components of the target image features and the text features to values ​​in the interval [0, 1], facilitating subsequent calculations. The normalized features are then calculated. The cosine similarity between image features and normalized text features can determine the degree of correlation between the normalized image features and the normalized text features. When the cosine similarity is not less than a preset threshold, the normalized image features and the normalized text features are fused to obtain a question fusion feature. This allows for the generation of a comprehensive feature of the question, i.e., a question fusion feature, when the normalized image features and the normalized text features represent the same question (the cosine similarity is not less than the preset threshold). Based on the question fusion feature, the answer to the user's medical question can be retrieved from a pre-built medical question database to achieve the final question intent. Therefore, the medical visual question answering method, device, and electronic device based on transfer learning proposed in this invention can improve the accuracy of user question answers. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating a medical visual question-answering method based on transfer learning provided in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of a module of a medical visual question-answering device based on transfer learning provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the internal structure of an electronic device that implements a medical visual question answering method based on transfer learning, according to an embodiment of the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0052] This invention provides a medical visual question-answering method based on transfer learning. The executing entity of the transfer learning-based medical visual question-answering method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this invention: a server, a terminal, etc. In other words, the transfer learning-based medical visual question-answering method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0053] Reference Figure 1 The diagram shown is a flowchart illustrating a medical visual question-answering method based on transfer learning according to an embodiment of the present invention. In this embodiment, the medical visual question-answering method based on transfer learning includes:

[0054] S1. Obtain the user's medical question, which includes a question image and a question text. Extract the text features of the question text and perform image segmentation on the question image to obtain segmented images.

[0055] This invention provides an operational object for subsequent methods by acquiring user medical questions, which include question images and question text. The user medical questions refer to the questioning conditions and content upon which the computer performs visual question answering, and include question images and question text. These questions can be acquired through a data script, which can be compiled using the JavaScript scripting language.

[0056] Furthermore, in this embodiment of the invention, extracting the textual features of the question text is a preprocessing of the question text to obtain the textual semantic information of the user's medical question.

[0057] Furthermore, in an optional embodiment of the present invention, the extraction of text features from the problem text can be achieved using a BERT pre-trained model.

[0058] Furthermore, embodiments of the present invention provide a standard data format for subsequent masking operations by performing image block processing on the problem image.

[0059] Furthermore, in an optional embodiment of the present invention, the problem image is processed by image block segmentation to obtain a block image. The problem image can be divided into non-overlapping image blocks of the same size according to the length and width of the problem image, and the image blocks are used as the block image.

[0060] S2. Perform a random masking operation on the segmented image to obtain a visible segmented image and a masked segmented image, and mark the mask position information of the masked segmented image, and extract features from the visible segmented image to obtain a visible feature image;

[0061] This invention improves image processing efficiency by performing a random masking operation on the segmented image, which can shield certain areas of the segmented image from processing. The mask is a binary image composed of 0s and 1s, used to occlude (fully or partially) the image to be processed, thereby controlling the area or process of image processing.

[0062] Furthermore, in an optional embodiment of the present invention, the random masking operation performed on the segmented image to obtain the visible segmented image and the masked segmented image can be achieved by performing a bitwise AND operation between each pixel in the segmented image and each corresponding pixel in the pre-constructed masked image, such that some image blocks in the segmented image are completely obscured by black, the image blocks obscured by black are the masked segmented image, and the image blocks in the segmented image that are not obscured are the visible segmented image.

[0063] Furthermore, in an optional embodiment of the present invention, the pre-construction of the mask image includes: obtaining the image size and segmentation method of the segmented image; constructing a mask segmented image according to the image size and the segmentation method; and performing pixel reset processing on the mask segmented image to obtain the mask image.

[0064] Optionally, configuring the mask block image to obtain the mask image can be done by: initializing all pixels of the mask block image to 0, randomly selecting an interest image block in the mask block image according to a preset mask ratio threshold, and setting all pixels in the interest image block to 1 to obtain the mask image.

[0065] The mask ratio threshold can be selected as 0.75, or it can be set according to the actual business scenario.

[0066] Furthermore, in this embodiment of the invention, by marking the mask position information of the mask block image, a positional basis can be provided for subsequent image reconstruction of the mask block image.

[0067] Furthermore, in an optional embodiment of the present invention, marking the mask position information of the mask block image includes: dividing the block image into blocks and sorting them to obtain a block sequence; querying the arrangement number of the mask block image in the block sequence, and using the arrangement number as the mask position information.

[0068] Optionally, the step of dividing and sorting the segmented images to obtain a segmented sequence can be: sorting the segmented images in order from top to bottom and from left to right to obtain a segmented sequence.

[0069] Furthermore, by performing feature extraction on the visible block image, the present invention obtains a visible feature image, which can remove redundant information in the visible block image and extract key information from the image, thereby improving the execution efficiency of the model.

[0070] Further, in an optional embodiment of the present invention, the step of extracting features from the visible segmented image to obtain a visible feature image includes: using a linear projection layer in a preset image feature extraction model to perform dimensional mapping on the projection of the visible segmented image to obtain a projection mapping vector; obtaining the segmented sequence of the segmented image, querying the position information of the visible segmented image in the segmented sequence to obtain visible position information; and using a fusion layer in the preset image feature extraction model to fuse the projection mapping vector and the visible position information to obtain the visible feature image.

[0071] The linear projection layer is used to convert the input image into a vector with a pre-set dimension.

[0072] S3. Based on the visible feature image and the mask position information, perform image reconstruction on the mask block image to obtain a mask reconstruction image. Then, based on the mask reconstruction image, the visible feature image, and the mask position information, fuse the mask reconstruction image and the visible feature image to obtain the target image features.

[0073] In this embodiment of the invention, the mask block image is reconstructed based on the visible feature image and the mask position information. The resulting mask reconstructed image can provide a guarantee for the subsequent reconstruction of the original problem image to generate target image features.

[0074] Further, in an optional embodiment of the present invention, the step of reconstructing the mask block image based on the visible feature image and the mask position information to obtain a mask reconstructed image includes: obtaining the block image corresponding to the visible feature image; calculating the frequency domain estimation function of the block image using a preset Wiener filtering algorithm; calculating the block reconstructed image of the block image using inverse Fourier transform based on the frequency domain estimation function; constructing a mask reconstructed candidate image of the mask block image based on the block reconstructed image and the mask position information; calculating the quality index score of the mask reconstructed candidate image using a quality index function; and using the mask reconstructed candidate image as the mask reconstructed image when the quality index score is less than a preset threshold.

[0075] The Wiener filtering algorithm is an algorithm that minimizes the mean square value of the estimation error (defined as the difference between the expected response and the actual output of the filter). The quality index function refers to a method that objectively reflects and evaluates the quality of image reconstruction, such as MSE, PSNR, SSIM, MSSIM, FSIM, LPIPS, etc.

[0076] Furthermore, in an optional embodiment of the present invention, the Wiener filtering algorithm includes:

[0077]

[0078] in, Let H(u,v) represent the frequency domain estimation function corresponding to the spatial domain problem image, and let S represent the degradation function. η (u,v) represents the power spectrum of the noise, S f (u,v) represents the power spectrum of the spatial domain problem image, G(u,v) represents the Fourier transform function of the block image after masking, and u and v represent the frequency domain variables after Fourier transform, respectively.

[0079] Furthermore, in an optional embodiment of the present invention, the quality index function includes:

[0080]

[0081] MSE represents the quality score of the mask-reconstructed image. y represents the value of the j-th reconstructed pixel in the i-th masked reconstructed image. ijThe value of the j-th pixel in the original i-th block image corresponding to the mask position information is represented by m, where m represents the number of blocks in the mask reconstruction image corresponding to the block image, and n represents the number of pixels in each mask reconstruction image.

[0082] Furthermore, in this embodiment of the invention, by fusing the mask-reconstructed image and the visible feature image based on the mask reconstruction image, the visible feature image, and the mask position information, the target image features can be obtained to obtain the final problem features, thereby providing support for subsequent search for the answer to the problem features.

[0083] Furthermore, in an optional embodiment of the present invention, the step of fusing the mask-reconstructed image and the visible feature image based on the mask-reconstructed image, the visible feature image, and the mask position information to obtain target image features includes: extracting features from the mask-reconstructed image to obtain a mask feature image; and concatenating the visible feature image with the mask position information based on the mask feature image to obtain the target image features.

[0084] S4. Normalize the target image features and the text features respectively to obtain normalized image features and normalized text features. Calculate the cosine similarity between the normalized image features and the normalized text features. When the cosine similarity is not less than a preset threshold, fuse the normalized image features and the normalized text features to obtain question fusion features. Based on the question fusion features, query the answer to the user's medical question from a pre-built medical question database.

[0085] This invention, through normalization of the target image features and the text features respectively, obtains normalized image features and normalized text features. This eliminates the dimensional limitations of the image and text feature data and maps the target image features and the text features to values ​​in the interval [0, 1], facilitating subsequent calculations. Normalization is a method to simplify calculations, transforming dimensional expressions into dimensionless expressions, i.e., scalars.

[0086] Furthermore, in an optional embodiment of the present invention, the normalization processing of the target image features and the text features to obtain normalized image features and normalized text features can be achieved using normalization methods, such as linear function method, logarithmic function method, inverse cotangent function method, norm method, mean method, and Z-score method.

[0087] Furthermore, in an optional embodiment of the present invention, the normalization method includes:

[0088]

[0089]

[0090]

[0091] Among them, y k x represents the k-th normalized image feature value or the k-th normalized text feature value. k μ represents the k-th target image feature value or the k-th text feature value. k σ represents the mean of the k-th target image feature value or the k-th text feature value. k Let x represent the standard deviation of the k-th target image feature value or text feature value, and N represent the number of samples of the target image feature or text feature. ki This represents the i-th sample value of the k-th target image feature or the k-th text feature.

[0092] Furthermore, in this embodiment of the invention, the degree of correlation between the normalized image features and the normalized text features can be determined by calculating the cosine similarity between the normalized image features and the normalized text features.

[0093] Furthermore, in an optional embodiment of the present invention, the cosine similarity between the normalized image features and the normalized text features is calculated using the following formula:

[0094]

[0095] Where sim(u,v) represents the cosine similarity between normalized image features and normalized text features, and cos(u,v) represents the cosine of the angle between normalized image feature u and normalized text feature v. Let represent the sum of the products of the i-th normalized image feature component u and the i-th normalized text feature component v. Let represent the arithmetic square root of the sum of squares of the i-th normalized image feature component u. Let represent the arithmetic square root of the sum of squares of the i-th normalized text feature component v.

[0096] Furthermore, in this embodiment of the invention, when the cosine similarity is not less than a preset threshold, the normalized image features and the normalized text features are fused to obtain a question fusion feature. This can generate a comprehensive feature of the question, i.e., a question fusion feature, when the normalized image features and the normalized text features represent the same question (the cosine similarity is not less than the preset threshold).

[0097] Furthermore, in an optional embodiment of the present invention, when the cosine similarity is not less than a preset threshold, fusing the normalized image features and the normalized text features to obtain the problem fusion features can be achieved by calculating the arithmetic mean of the normalized image features and the normalized text features.

[0098] Furthermore, in this embodiment of the invention, the final question intent can be achieved by querying the answer to the user's medical question from a pre-built medical question database based on the question fusion features.

[0099] It should be understood that, in this embodiment of the invention, a medical question database should be constructed before querying the answer to the user's medical question from a pre-built medical question database based on the question fusion features.

[0100] Furthermore, in an optional embodiment of the present invention, the construction of the medical question database includes: identifying medical questions and their answers, wherein the medical questions include question text and question images; constructing fusion features of the medical questions based on the question text and question images; establishing a mapping relationship table between the fusion features and the question answers based on the fusion features; and constructing the medical question database based on the medical questions, the fusion features, and the mapping relationship table.

[0101] Further, in an optional embodiment of the present invention, the step of querying the answer to the user's medical question from a pre-built medical question database based on the question fusion features includes: obtaining the fusion features to be matched in the pre-built medical question database, calculating the cosine similarity between the question fusion features and the fusion features to be matched; when the cosine similarity is not less than a preset threshold, querying the mapping relationship table between the fusion features to be matched corresponding to the cosine similarity and the question answers in the pre-built medical question database, to obtain the answer to the user's medical question.

[0102] As can be seen, this embodiment of the invention obtains a user's medical question, which includes a question image and question text, providing an operational object for subsequent method implementation. Extracting the textual features of the question text is a preprocessing step to obtain the semantic information of the user's medical question. The question image is then segmented into blocks, providing a standard data format for subsequent masking operations. Random masking is performed on the segmented images to obtain visible and masked segmented images. These masked segmented images can block certain areas of the segmented images, preventing them from participating in processing, thereby improving image processing efficiency. The labeling of the mask position information of the mask block image provides a location basis for subsequent image reconstruction of the mask block image, and the extraction of features from the visible block image to obtain a visible feature image can remove redundant information in the visible block image and extract key information from the image, thereby improving the execution efficiency of the model. Secondly, according to the visible feature image and the mask position information, the embodiment of the present invention performs image reconstruction on the mask block image, and the obtained mask reconstruction image can provide a guarantee for the subsequent reconstruction of the original problem image to generate target image features. The visible feature image and the mask position information are used to fuse the mask-reconstructed image and the visible feature image to obtain the target image features, which can be used to obtain the final question features to support the subsequent search for the answer to the question based on the question features. Furthermore, in this embodiment of the invention, the target image features and the text features are normalized respectively to obtain normalized image features and normalized text features. This eliminates the dimensional limitations of the image feature and text feature data, and maps the vector components of the target image features and the text features to values ​​in the interval [0, 1], facilitating subsequent calculations. The normalized features are then calculated. The cosine similarity between image features and normalized text features can determine the correlation between the normalized image features and the normalized text features. When the cosine similarity is not less than a preset threshold, the normalized image features and the normalized text features are fused to obtain a question fusion feature. This allows for the generation of a comprehensive feature of the question, i.e., a question fusion feature, when the normalized image features and the normalized text features represent the same question (the cosine similarity is not less than the preset threshold). Based on the question fusion feature, the answer to the user's medical question can be retrieved from a pre-built medical question database to achieve the final question intent. Therefore, the medical visual question answering method, device, electronic device, and storage medium based on transfer learning proposed in this invention can improve the accuracy of user question answers.

[0103] like Figure 2 The diagram shown is a functional block diagram of the medical visual question-answering device based on transfer learning of the present invention.

[0104] The medical visual question-answering device 100 based on transfer learning described in this invention can be installed in an electronic device. Depending on the functions implemented, the medical visual question-answering device based on transfer learning may include a text feature extraction module 101, a mask processing module 102, an image reconstruction module 103, and a question-answer generation module 104. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0105] In this embodiment, the functions of each module / unit are as follows:

[0106] The text feature extraction module 101 is used to obtain the user's medical question, which includes a question image and question text. The module extracts the text features of the question text and performs image segmentation on the question image to obtain segmented images.

[0107] The mask processing module 102 is used to perform random masking operation on the block image to obtain visible block image and masked block image, and to mark the mask position information of the masked block image, and to extract features from the visible block image to obtain visible feature image;

[0108] The image reconstruction module 103 is used to reconstruct the mask block image based on the visible feature image and the mask position information to obtain a mask reconstructed image, and to fuse the mask reconstructed image and the visible feature image based on the mask reconstructed image, the visible feature image and the mask position information to obtain the target image features;

[0109] The question-answer generation module 104 is used to normalize the target image features and the text features respectively to obtain normalized image features and normalized text features, calculate the cosine similarity between the normalized image features and the normalized text features, and when the cosine similarity is not less than a preset threshold, fuse the normalized image features and the normalized text features to obtain question fusion features. Based on the question fusion features, the module queries the answer to the user's medical question from a pre-built medical question database.

[0110] In detail, the modules in the medical visual question-answering device 100 based on transfer learning described in this embodiment of the invention employ the same methods as described above during use. Figure 1 The method uses the same techniques as the medical visual question answering method based on transfer learning described above and can produce the same technical effects, so it will not be repeated here.

[0111] like Figure 3The diagram shown is a schematic representation of the structure of an electronic device 1 that implements a medical visual question-answering method based on transfer learning according to the present invention.

[0112] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a medical vision question answering program based on transfer learning.

[0113] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device 1, connecting various components of the electronic device 1 through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing a medical vision question-answering program based on transfer learning) and calls data stored in the memory 11 to perform various functions and process data of the electronic device 1.

[0114] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as the portable hard drive of the electronic device 1. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a medical visual question-answering program based on transfer learning, but also to temporarily store data that has been output or will be output.

[0115] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0116] The communication interface 13 is used for communication between the aforementioned electronic device 1 and other devices, including a network interface and an employee interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish a communication connection between the electronic device 1 and other electronic devices 1. The employee interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual employee interface.

[0117] Figure 3 Only electronic device 1 with components is shown; those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0118] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0119] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in terms of the scope of the patent invention.

[0120] The medical visual question-answering program based on transfer learning stored in the memory 11 of the electronic device 1 is a combination of multiple computer programs. When run in the processor 10, it can achieve the following:

[0121] The user's medical question is obtained, which includes a question image and a question text. The text features of the question text are extracted, and the question image is processed into image blocks to obtain block images.

[0122] A random masking operation is performed on the segmented image to obtain a visible segmented image and a masked segmented image. The mask position information of the masked segmented image is marked, and features are extracted from the visible segmented image to obtain a visible feature image.

[0123] Based on the visible feature image and the mask position information, image reconstruction is performed on the mask block image to obtain a mask reconstruction image. Then, based on the mask reconstruction image, the visible feature image, and the mask position information, the mask reconstruction image and the visible feature image are fused to obtain the target image features.

[0124] The target image features and the text features are normalized separately to obtain normalized image features and normalized text features. The cosine similarity between the normalized image features and the normalized text features is calculated. When the cosine similarity is not less than a preset threshold, the normalized image features and the normalized text features are fused to obtain question fusion features. Based on the question fusion features, the answer to the user's medical question is queried from a pre-built medical question database.

[0125] Specifically, the processor 10's implementation method of the above-mentioned computer program can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0126] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0127] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device 1, can perform the following:

[0128] The user's medical question is obtained, which includes a question image and a question text. The text features of the question text are extracted, and the question image is processed into image blocks to obtain block images.

[0129] A random masking operation is performed on the segmented image to obtain a visible segmented image and a masked segmented image. The mask position information of the masked segmented image is marked, and features are extracted from the visible segmented image to obtain a visible feature image.

[0130] Based on the visible feature image and the mask position information, image reconstruction is performed on the mask block image to obtain a mask reconstruction image. Then, based on the mask reconstruction image, the visible feature image, and the mask position information, the mask reconstruction image and the visible feature image are fused to obtain the target image features.

[0131] The target image features and the text features are normalized separately to obtain normalized image features and normalized text features. The cosine similarity between the normalized image features and the normalized text features is calculated. When the cosine similarity is not less than a preset threshold, the normalized image features and the normalized text features are fused to obtain question fusion features. Based on the question fusion features, the answer to the user's medical question is queried from a pre-built medical question database.

[0132] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0133] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0135] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0136] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0137] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0138] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A medical visual question answering method based on transfer learning, characterized in that, The method includes: The system acquires user medical questions, which include a question image and question text. It extracts text features from the question text and performs image segmentation on the user question image to obtain segmented images. A random masking operation is performed on the segmented image to obtain a visible segmented image and a masked segmented image. The mask position information of the masked segmented image is marked, and features are extracted from the visible segmented image to obtain a visible feature image. Based on the visible feature image and the mask position information, image reconstruction is performed on the mask block image to obtain a mask reconstruction image. Then, based on the mask reconstruction image, the visible feature image, and the mask position information, the mask reconstruction image and the visible feature image are fused to obtain the target image features. The target image features and the text features are normalized separately to obtain normalized image features and normalized text features. The cosine similarity between the normalized image features and the normalized text features is calculated. When the cosine similarity is not less than a preset threshold, the normalized image features and the normalized text features are fused to obtain question fusion features. Based on the question fusion features, the answer to the user's medical question is queried from a pre-built medical question database.

2. The medical visual question answering method based on transfer learning as described in claim 1, characterized in that, The step of extracting features from the visible segmented image to obtain a visible feature image includes: The visible block image is dimensionally mapped using the linear projection layer in the preset image feature extraction model to obtain the projection mapping vector; Obtain the block sequence of the segmented image, query the position information of the visible segmented image in the block sequence, and obtain the visible position information; The projection mapping vector and the visible location information are fused using the fusion layer in the preset image feature extraction model to obtain the visible feature image.

3. The medical visual question answering method based on transfer learning as described in claim 1, characterized in that, The step of reconstructing the mask block image based on the visible feature image and the mask position information to obtain the mask reconstructed image includes: Obtain the block image corresponding to the visible feature image, and calculate the frequency domain estimation function of the block image using a preset Wiener filtering algorithm; Based on the frequency domain estimation function, the block-reconstructed image of the block image is calculated using inverse Fourier transform; Based on the segmented reconstructed image and the mask position information, construct the mask reconstruction candidate image of the mask segmented image; The quality index score of the mask reconstruction candidate image is calculated using a quality index function; When the quality index score is less than a preset threshold, the candidate image for mask reconstruction is used as the mask reconstruction image.

4. The medical visual question answering method based on transfer learning as described in claim 3, characterized in that, The Wiener filtering algorithm includes: in, Let H(u,v) represent the frequency domain estimation function corresponding to the spatial domain problem image, and let S represent the degradation function. η (u,v) represents the power spectrum of the noise, S f (u,v) represents the power spectrum of the spatial domain problem image, G(u,v) represents the Fourier transform function of the block image after masking, and u and v represent the frequency domain variables after Fourier transform of the spatial domain function.

5. The medical visual question answering method based on transfer learning as described in claim 3, characterized in that, The quality indicator function includes: MSE represents the quality score of the mask-reconstructed image. y represents the value of the j-th reconstructed pixel in the i-th masked reconstructed image. ij The value of the j-th pixel in the original i-th block image corresponding to the mask position information is represented by m, where m represents the number of blocks in the mask reconstruction image corresponding to the block image, and n represents the number of pixels in each mask reconstruction image.

6. The medical visual question answering method based on transfer learning as described in claim 1, characterized in that, The step of normalizing the target image features and the text features respectively to obtain normalized image features and normalized text features includes: The target image features and the text features are normalized using a normalization method to obtain the normalized image features and the normalized text features. The normalization method includes: Among them, y k x represents the k-th normalized image feature value or the k-th normalized text feature value. k μ represents the k-th target image feature value or the k-th text feature value. k σ represents the mean of the k-th target image feature value or the k-th text feature value. k Let x represent the standard deviation of the k-th target image feature value or text feature value, and N represent the number of samples of the target image feature or text feature. ki This represents the i-th sample value of the k-th target image feature or the k-th text feature.

7. The medical visual question answering method based on transfer learning as described in claim 1, characterized in that, The step of querying the answer to the user's medical question from a pre-built medical question database based on the question fusion features includes: Obtain the fusion features to be matched from the pre-built medical question database, and calculate the cosine similarity between the question fusion features and the fusion features to be matched; When the cosine similarity is not less than a preset threshold, the mapping relationship table between the fusion features to be matched and the question answers in the pre-built medical question database corresponding to the cosine similarity is queried to obtain the question answer of the user's medical question.

8. A medical visual question-answering device based on transfer learning, characterized in that, The device includes: The text feature extraction module is used to obtain the user's medical questions, which include a question image and question text. The module extracts the text features of the question text and performs image segmentation on the user's question image to obtain segmented images. The mask processing module is used to perform random masking operations on the segmented image to obtain visible segmented images and masked segmented images, and to mark the mask position information of the masked segmented images, and to extract features from the visible segmented images to obtain visible feature images; The image reconstruction module is used to reconstruct the mask block image based on the visible feature image and the mask position information to obtain a mask reconstructed image, and to fuse the mask reconstructed image and the visible feature image based on the mask reconstructed image, the visible feature image and the mask position information to obtain the target image features; The question-answer generation module is used to normalize the target image features and the text features respectively to obtain normalized image features and normalized text features, calculate the cosine similarity between the normalized image features and the normalized text features, and when the cosine similarity is not less than a preset threshold, fuse the normalized image features and the normalized text features to obtain question fusion features. Based on the question fusion features, the module queries the answer to the user's medical question from a pre-built medical question database.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the medical visual question answering method based on transfer learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the medical visual question answering method based on transfer learning as described in any one of claims 1 to 7.

Images