Image processing method and device based on multiple modes, electronic equipment and storage medium
Through the multimodal image processing method, feature extraction and fusion are combined with text and image information, the problem of insufficient image processing accuracy in traditional medical image analysis is solved, and higher accuracy and robustness are achieved.
Patent Information
- Application Number
- CN202411063890.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-08
AI Technical Summary
In traditional medical picture analysis, computer vision modal processing is relatively single, which affects the accuracy of picture analysis. How to improve the accuracy of image processing has become an urgent problem.
A multimodal-based image processing method is proposed. By obtaining text information and image information, using pre-trained image encoder and text encoder for encoding, extracting features and performing mean-based processing, combining text and image information for stitching and Fusion, thereby improving the accuracy of image processing.
Through the deep fusion of multimodal information, the information representation range is broadened, the accuracy of image classification is significantly improved, and the model's robustness and adaptability are enhanced.
Smart Images

Figure CN118918381A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of pharmaceutical computer vision technology, and in particular to a multimodal image processing method and device, electronic device, and storage medium. Background Art
[0002] With the development of computer technology, more and more technologies are applied in the medical field, and traditional medicine is gradually transforming into medical technology. Among them, the use of artificial intelligence technology to analyze medical images has become an important branch. In related technologies, image analysis tasks are usually performed on computer vision modalities through models, and the processing is relatively simple, which affects the accuracy of image analysis. Therefore, how to improve the accuracy of image processing has become a technical problem that needs to be solved urgently. Summary of the invention
[0003] The main purpose of the embodiments of the present application is to propose a multi-modal image processing method and device, an electronic device and a storage medium, aiming to improve the accuracy of image processing.
[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes a multi-modal image processing method, the method comprising:
[0005] Acquire text information and image information; the image information includes a number of objects to be classified, and the text information includes associated information for indicating classification of the objects to be classified;
[0006] Encoding the image information by a pre-trained image encoder to obtain an image vector, and encoding the text information by a pre-trained text encoder to obtain a text vector;
[0007] Extracting features from the image vector and the text vector respectively, and performing mean processing on the obtained features to obtain mean features;
[0008] The mean feature and the text vector are spliced to obtain a text splicing feature, and the mean feature and the image vector are spliced to obtain an image splicing feature;
[0009] The text splicing features and the image splicing features are fused, and the features obtained by the fusion processing are classified to obtain a target classification result corresponding to the object to be classified.
[0010] The embodiment of the present application extracts multi-level features by splicing and fusing text information and image information, thereby improving the accuracy of image processing.
[0011] In some embodiments, the extracting features from the image vector and the text vector respectively, and performing mean processing on the obtained features to obtain mean features, comprises:
[0012] Feature extraction is performed on the image vector and the text vector respectively through multiple feature extractors, and the obtained features are averaged to obtain mean features; the initial number of the feature extractors is a first number, and the length value of the mean feature in the image dimension is equal to the first number.
[0013] In some embodiments, the fusing process of the text stitching feature and the image stitching feature includes:
[0014] Fusing the text splicing feature and the image vector to obtain a text fusion feature;
[0015] Fusing the image splicing feature and the text vector to obtain an image fusion feature;
[0016] The text fusion feature is used as a new text vector, and the image fusion feature is used as a new image vector, and the features are returned to the process of performing feature extraction on the image vector and the text vector respectively through multiple feature extractors; wherein the ratio of the number of feature extractors in the current cycle round to the number of feature extractors in the previous cycle round is a first ratio, and the first ratio is less than 1.
[0017] In some embodiments, the image vector includes an image part vector and a hidden layer vector; the method further includes:
[0018] Detecting a first length value of the image portion vector and a second length value of the hidden layer vector;
[0019] The length value of the image dimension in the obtained image splicing feature is determined to be the sum of the first length value and the first quantity, and the length value of the hidden layer vector in the image splicing feature is determined to be the second length value.
[0020] In some embodiments, classifying the features obtained by the fusion process includes:
[0021] Performing pooling processing on the features obtained by the fusion processing to obtain pooled features;
[0022] The pooled features are classified and output through a fully connected layer.
[0023] In some embodiments, performing feature extraction on the image vector includes:
[0024] Determining the correlation of each position with other positions in the image vector;
[0025] Normalizing the correlation to obtain the weight of each correlation;
[0026] According to the correlation and the weight, a feature corresponding to the image vector is obtained.
[0027] In some embodiments, the fusing the image stitching feature and the text vector to obtain the image fusion feature includes:
[0028] Determining the degree of association between each element in the image stitching feature and each element in the text vector;
[0029] A weighted sum process is performed on the image splicing feature and each element in the text vector according to the association degree to obtain an image fusion feature.
[0030] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a multi-modal image processing device, the device comprising:
[0031] The first module is used to obtain text information and image information; the image information includes a number of objects to be classified, and the text information contains associated information for indicating the classification of the objects to be classified;
[0032] The second module is used to encode the image information through a pre-trained image encoder to obtain an image vector, and to encode the text information through a pre-trained text encoder to obtain a text vector;
[0033] The third module is used to extract features from the image vector and the text vector respectively, and perform mean processing on the obtained features to obtain mean features;
[0034] The fourth module is used to perform splicing processing on the mean feature and the text vector to obtain a text splicing feature, and to perform splicing processing on the mean feature and the image vector to obtain an image splicing feature;
[0035] The fifth module is used to fuse the text splicing features and the image splicing features, and classify the features obtained by the fusion processing to obtain the target classification result corresponding to the object to be classified.
[0036] In some embodiments, the third module is also used to: perform feature extraction on the image vector and the text vector respectively through multiple feature extractors, and average the obtained features to obtain mean features; the initial number of the feature extractors is the first number, and the length value of the mean feature in the image dimension is equal to the first number.
[0037] In some embodiments, the third module is further used to: fuse the text splicing feature and the image vector to obtain a text fusion feature;
[0038] Fusing the image splicing feature and the text vector to obtain an image fusion feature;
[0039] The text fusion feature is used as a new text vector, and the image fusion feature is used as a new image vector, and the features are returned to the process of performing feature extraction on the image vector and the text vector respectively through multiple feature extractors; wherein the ratio of the number of feature extractors in the current cycle round to the number of feature extractors in the previous cycle round is a first ratio, and the first ratio is less than 1.
[0040] In some embodiments, the third module is further used to: detect a first length value of the image portion vector and a second length value of the hidden layer vector;
[0041] The length value of the image dimension in the obtained image splicing feature is determined to be the sum of the first length value and the first quantity, and the length value of the hidden layer vector in the image splicing feature is determined to be the second length value.
[0042] In some embodiments, the fifth module is further used to: perform pooling processing on the features obtained by the fusion processing to obtain pooled features;
[0043] The pooled features are classified and output through a fully connected layer.
[0044] In some embodiments, the third module is further used to: determine the correlation between each position in the image vector and other positions;
[0045] Normalizing the correlation to obtain the weight of each correlation;
[0046] According to the correlation and the weight, a feature corresponding to the image vector is obtained.
[0047] In some embodiments, the third module is further used to: determine the correlation between each element in the image stitching feature and each element in the text vector;
[0048] A weighted sum process is performed on the image splicing feature and each element in the text vector according to the association degree to obtain an image fusion feature.
[0049] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.
[0050] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.
[0051] The multimodal image processing method and device, electronic device and storage medium proposed in the present application include: obtaining text information and image information; encoding the image information through a pre-trained image encoder to obtain an image vector; encoding the text information through a pre-trained text encoder to obtain a text vector; extracting features from the image vector and the text vector respectively, and averaging the obtained features to obtain mean features; splicing the mean features with the text vector and the image vector respectively to obtain text splicing features and image splicing features; fusing the text splicing features and the image splicing features, and classifying and outputting the fused features to obtain target classification results of the text information and the image information. The embodiments of the present application are conducive to improving the accuracy of image processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 is a flowchart of a multi-modal image processing method provided in an embodiment of the present application;
[0053] Figure 2 yes Figure 1 Flow chart of step S300 in FIG.
[0054] Figure 3 is a picture schematic diagram of picture recognition provided by an embodiment of the present application;
[0055] Figure 4 is a flowchart of the image recognition process provided by an embodiment of the present application;
[0056] Figure 5 is a schematic structural diagram of a multi-modal image processing device provided in an embodiment of the present application;
[0057] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0061] First, some nouns involved in this application are analyzed:
[0062] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0063] Self-Attention is a widely used technology in natural language processing and computer vision. During the model processing, the correlation between different positions is learned to capture rich contextual information. Compared with traditional neural networks, it can achieve parallel computing, greatly improving efficiency; it realizes parameter sharing and reduces the amount of model calculation.
[0064] Encoder: Converts input data into different forms, compresses input data into less information, and maintains the semantic content of the data; reduces the dimension of the data to facilitate subsequent processing. Encoders have many applications in machine learning, such as image compression, feature extraction, dimensionality reduction, etc. Among them, autoencoders, as a common encoder, are widely used in image, speech, text and other fields.
[0065] Image caption generates a natural language description for an image and uses the generated description to help applications understand the semantics expressed in the image visual scene. For example, image captioning can convert image retrieval into text retrieval, which is used to classify images and improve image retrieval results. People usually only need to take a quick glance to describe the details of the image visual scene, while automatically adding descriptions to images is a comprehensive and arduous computer vision task that requires converting the complex information contained in the image into a natural language description. Compared with ordinary computer vision tasks, image captioning not only requires identifying objects from images, but also requires associating the identified objects with natural semantics and describing them in natural language. Therefore, image description requires people to extract deep features of images, associate them with semantic features, and convert them for generating descriptions.
[0066] Medical image analysis is a very important problem in the field of artificial intelligence today, and it is also a very important technology in the medical field. Deep learning is used to detect and analyze medical images, such as X-rays, CT scans, etc.
[0067] Traditional deep learning model analysis methods have many advantages, but still have some defects, including:
[0068] Limitations of the model: Traditional learning models mostly use computer vision (CV) to complete tasks and lack interaction with text information.
[0069] Based on this, the embodiments of the present application provide a multi-modal image processing method and device, an electronic device and a storage medium, aiming to improve the accuracy of image classification processing.
[0070] The multimodal image processing method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the multimodal image processing method in the embodiments of the present application is described.
[0071] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0072] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0073] The multimodal image processing method provided in the embodiment of the present application relates to the field of pharmaceutical computer vision technology. The multimodal image processing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a multimodal image processing method, etc., but is not limited to the above forms.
[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0075] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.
[0076] Figure 1 is an optional flowchart of a multimodal image processing method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S500.
[0077] Step S100, obtaining text information and image information; the image information includes a number of objects to be classified, and the text information includes associated information for indicating the classification of the objects to be classified;
[0078] Step S200, encoding the image information by a pre-trained image encoder to obtain an image vector, and encoding the text information by a pre-trained text encoder to obtain a text vector;
[0079] Step S300, extracting features from the image vector and the text vector respectively, and performing mean processing on the obtained features to obtain mean features;
[0080] Step S400, performing splicing processing on the mean feature and the text vector to obtain a text splicing feature, and performing splicing processing on the mean feature and the image vector to obtain an image splicing feature;
[0081] Step S500 , fusing the text splicing features and the image splicing features, and classifying the features obtained by the fusion process to obtain a target classification result corresponding to the object to be classified.
[0082] The embodiment of the present application processes the image information in combination with the text information, processes the image, and determines the classification result based on the image. It can be understood that the image information in the embodiment of the present application includes several objects to be classified. For different application fields, the images can be of different types, and the objects to be classified can also be objects in different fields. This application is not specifically limited. For example, in the medical field, if it is necessary to classify drug pictures, the image information can be drug pictures, the text information can be description information of the drug, and the associated information for classifying the objects to be classified can be information points describing the shape, size, color, etc. of the drug. The embodiment of the present application encodes the text information and image information through an encoder, and extracts features from the encoded text vector and image vector to obtain the mean feature. After the mean feature is spliced with the text vector and the image vector, a fusion process is performed, so that the text and the image are deeply fused, which broadens the representation range of the information and improves the accuracy of image classification.
[0083] It should be noted that multimodal fusion is usually used to combine data from different modalities (such as vision, hearing, text, etc.) to improve information processing and understanding capabilities. It is understandable that multimodal data usually have different physical properties and information characteristics. By fusing these multimodal information, a more comprehensive and accurate understanding can be obtained. Specifically, this fusion process can occur at the data layer, feature layer, and decision layer. Data layer fusion: directly fuse data of different modalities. Feature layer fusion: extract data features of different modalities and then fuse them. Decision layer fusion: fuse the processing results of different modalities. Multimodal fusion is of great significance in the field of artificial intelligence and can improve information understanding capabilities. Single modal data often cannot provide comprehensive information. For example, relying solely on visual data may not accurately classify a picture of Chinese herbal medicine, but combining voice and text data can significantly improve the accuracy of picture classification. Multimodal fusion can enhance the robustness of the model, and multimodal fusion can enhance the adaptability of the model to different environments and scenarios. For example, in the field of autonomous driving, combining visual, radar and lidar data can improve the accuracy and safety of environmental perception. Multimodal fusion can provide rich contextual information. By fusing multimodal data, richer contextual information can be obtained, thereby improving the execution effect of tasks. For example, in human-computer interaction, combining voice and gesture information can provide a more natural and intuitive interactive experience. Multimodal fusion can improve the quality of generation tasks. In generation tasks (such as image and text generation), multimodal fusion can generate more realistic and consistent content. For example, combining text descriptions and image data can generate high-quality images that meet the description. Multimodal fusion can be used in a variety of application scenarios. Exemplarily, multimodal fusion can be used for image and text generation and understanding; it can also be used for sentiment analysis combining voice and vision; it can also be used for autonomous driving and human-computer interaction. Of course, in the field of medical image analysis, multimodal fusion has important uses. In this regard, the embodiments of the present application do not limit specific application scenarios. Those skilled in the art can build and process models according to actual needs through the methods provided in the above steps S100 to S500 to solve specific problems. In addition, the embodiments of the present application do not limit the number of modalities used for image processing. Steps S100 to S500 provide an image processing process that combines text information and image information. Those skilled in the art can expand the number of modalities used for processing in actual application scenarios based on the above logic.
[0084] The image processing method provided by the embodiment of the present application includes: obtaining text information and image information; encoding the image information through a pre-trained image encoder to obtain an image vector; encoding the text information through a pre-trained text encoder to obtain a text vector; extracting features from the image vector and the text vector respectively, and averaging the obtained features to obtain mean features; splicing the mean features with the text vector and the image vector respectively to obtain text splicing features and image splicing features; fusing the text splicing features and the image splicing features, and classifying and outputting the fused features to obtain target classification results of the text information and the image information. The embodiment of the present application is conducive to improving the accuracy of image processing.
[0085] In some embodiments, feature extraction is performed on the image vector and the text vector respectively, and the obtained features are averaged to obtain mean features; including:
[0086] Feature extraction is performed on image vectors and text vectors respectively through multiple feature extractors, and the obtained features are averaged to obtain mean features; the initial number of feature extractors is a first number, and the length value of the mean feature in the image dimension is equal to the first number.
[0087] In an embodiment of the present application, feature extraction can be performed on image vectors and text vectors by multiple feature extractors, and features of multiple images and texts can be obtained, and the mean of multiple features can be taken. Specifically, the number of feature extractors is a first number, and the length value of the mean feature in the image dimension / text dimension is equal to the first number. Through the averaging process, subsequent feature splicing is facilitated, and deep fusion between information is facilitated. In some embodiments, feature extraction can be performed on image vectors, and the obtained features can be averaged to obtain image mean features; and feature extraction can be performed on text vectors, and the obtained features can be averaged to obtain text mean features. In this regard, in the subsequent processing process, the text mean feature is spliced with the text vector to obtain a text splicing feature, and the image mean feature is spliced with the image vector to obtain an image splicing feature. The present application does not limit the specific source of the mean feature, and those skilled in the art can adjust it according to actual conditions.
[0088] See also Figure 2 In some embodiments, the text splicing features and the image splicing features are fused, including:
[0089] Step S311, fusing the text splicing feature and the image vector to obtain a text fusion feature;
[0090] Step S312, fusing the image splicing feature and the text vector to obtain an image fusion feature;
[0091] Step S313, using the text fusion feature as a new text vector and the image fusion feature as a new image vector, and returning to perform feature extraction on the image vector and the text vector respectively through multiple feature extractors; wherein the ratio of the number of feature extractors in the current cycle round to the number of feature extractors in the previous cycle round is a first ratio, and the first ratio is less than 1.
[0092] The embodiment of the present application realizes the fusion of multimodal information through a fusion layer. Specifically, the image splicing features are fused with the text vector, and the text splicing features are fused with the image vector, so as to fully fuse the multimodal features. The fused features are used as new vectors to perform splicing and fusion operations in the next cycle. Through multiple cycles, the fusion of text and image is strengthened, which facilitates the extraction of multi-level features and improves the accuracy of image processing. It is understandable that during the loop process, the number of feature extractors is gradually reduced until the number of cycles is greater than or equal to the preset number of cycles; or the similarity between the fused features obtained from two consecutive cycles is less than or equal to the preset similarity. In some embodiments, the first ratio can be a fixed value, or the first ratio can be a variable. The present application does not limit the specific value and change trend of the first ratio.
[0093] In some embodiments, the image vector includes an image part vector and a hidden layer vector; and the method further includes:
[0094] Detecting a first length value of the image portion vector and a second length value of the hidden layer vector;
[0095] The length value of the image dimension in the obtained image splicing feature is determined to be the sum of the first length value and the first quantity, and the length value of the hidden layer vector in the image splicing feature is determined to be the second length value.
[0096] When encoding information, the embodiment of the present application can encode image information into image vectors and hidden vectors, and encode text information into text vectors and hidden vectors. By setting the hidden vectors, the classification accuracy of the model is improved. Of course, it can be understood that for the text dimension, the text vector includes a text part vector and a hidden vector; the third length value of the text part vector is detected; the length value of the text dimension in the obtained text splicing feature is determined to be the sum of the third length value and the first number, and the length value of the hidden vector in the text splicing feature is determined to be the second length value.
[0097] In some embodiments, the features obtained by the fusion process are classified, including:
[0098] Performing pooling processing on the features obtained by the fusion processing to obtain pooled features;
[0099] The pooled features are classified and output through the fully connected layer.
[0100] After performing multiple cycles of fusion operations, the embodiment of the present application performs splicing and fusion processing on the obtained text fusion features and image fusion features, performs pooling processing on the splicing features of the image features / text features in the features obtained by the fusion processing, and then performs a full connection operation to obtain a classification result.
[0101] It is understandable that pooling can reduce the dimension of the feature map in a convolutional neural network or a recurrent neural network. In an embodiment of the present application, it is used to reduce the dimension of the feature. Specifically, the basic idea of the pooling operation is to divide the feature (map) into several sub-regions (generally rectangular) and perform statistical summary on each sub-region. In general, there are many ways of pooling operation, such as maximum pooling, average pooling, etc. Among them, the maximum pooling operation selects the maximum value in each sub-region as output, and the average pooling operation calculates the average value in each sub-region as output. The present application does not limit the specific pooling operation.
[0102] In some embodiments, performing feature extraction on the image vector includes:
[0103] Determine the correlation of each position in the image vector with respect to other positions;
[0104] Normalize the correlations to get the weight of each correlation;
[0105] According to the correlation and weight, the features corresponding to the image vector are obtained.
[0106] The embodiment of the present application can realize feature extraction through the self-attention module, that is, the feature extractor can be set through the self-attention mechanism. Specifically, by modeling the relationship between each element / position in the input vector, the weight and representation of each element / position are calculated, and then feature extraction is realized. In some embodiments, the process includes calculating correlation, calculating weight, and weighted sum. Specifically, for text, determine the correlation between each word in the text vector and other words; normalize the correlation to obtain the weight of each correlation; according to the correlation and weight, obtain the feature corresponding to the text vector.
[0107] In some embodiments, the image splicing feature and the text vector are fused to obtain the image fusion feature, including:
[0108] Determine the correlation between each element in the image stitching feature and each element in the text vector;
[0109] The image splicing features and each element in the text vector are weighted and summed according to the correlation to obtain the image fusion features.
[0110] The embodiments of the present application can realize text / image fusion through a cross-attention layer, realize the fusion of multimodal learning and attention mechanism through cross-attention, establish deep connections between different modules, promote the exchange and fusion of different modal information, and enhance the information processing capability of the model.
[0111] It is understandable that fusing features of different modalities and scales can improve the accuracy of information processing. In some embodiments, feature fusion can be performed by means of feature pyramid (FPN), fully convolutional neural network (FCN), etc., and the present application does not limit the specific method of feature fusion. After multiple splicing and fusion operations, the embodiment of the present application obtains image fusion features and text fusion features; after splicing / fusion processing of the image fusion features and the text fusion features, classification output is performed to improve the processing capability of the model.
[0112] The image processing method provided by the embodiment of the present application includes: obtaining text information and image information; encoding the image information through a pre-trained image encoder to obtain an image vector; encoding the text information through a pre-trained text encoder to obtain a text vector; extracting features from the image vector and the text vector respectively, and averaging the obtained features to obtain mean features; splicing the mean features with the text vector and the image vector respectively to obtain text splicing features and image splicing features; fusing the text splicing features and the image splicing features, and classifying and outputting the fused features to obtain target classification results of the text information and the image information. The embodiment of the present application is conducive to improving the accuracy of image processing.
[0113] The image processing method provided by this application is described below with a specific embodiment:
[0114] Reference Figure 3 The drug picture shown in a specific embodiment is shown. Specifically, for a solution to multimodal drug picture analysis, this application proposes a multimodal model, referring to Figure 4 The model structure and processing flow shown in the figure, the model task description is as follows:
[0115] Task: Use a multimodal model to perform multimodal image and text classification tasks based on text prompts. For example, input text:
[0116] The medicinal material of Bupleurum is in the form of irregular thick slices. The root head is swollen, with 3-15 stem bases or short fibrous leaf bases remaining at the top. Mint, also known as silver dan grass and night fragrance, is a perennial herbaceous plant of the genus Mentha in the family Lamiaceae. The leaves are opposite, the flowers are small and lavender, and after flowering, small dark purple-brown fruits are produced.
[0117] Input picture: Figure 3Output: Which Chinese medicinal material does the image belong to? This task is a classification task, and the specific processing process is as follows:
[0118] Step S41: Use pre-trained text encoders and image encoders, such as a multimodal model (Contrastive Language-Image Pre-training, CLIP), as input information encoders. Encode text information into a hidden layer vector of (sequence_length, hidden_size), where sequence_length is the length of the text and hidden_size is the length of the hidden vector. Similarly, the input image information is encoded into (channel_length, hidden_size).
[0119] Step S42, clustering: The encoded image and text information are input into n feature extractors respectively, and the feature extractors include text feature extractors and image feature extractors. The feature extractor is composed of a self-attention module, and the mean of the output text and image features in the sequence_length (channel_length) dimension is taken to obtain a vector of (n, hidden_size) dimensions. The output value is concatenated with the encoding in step S41 to obtain a text vector of size (sequence_length+n, hidden_size) and an image vector of (channel_length+n, hidden_size).
[0120] Step S43, multimodal fusion layer: add multiple layers of cross-attention layers. For the visual encoder, the content of cross-attention is text features, and for the text encoder, the content of cross-attention is visual features.
[0121] Step S44, repeat the clustering and multimodal fusion operations of step S42 and step S43, and the number of classifiers each time is 1 / 4 of the original. For example, in the second loop, the input dimension of the text and image is (sequence_length+n, hidden_size), and the output is (sequence_length+1.25*n, hidden_size). The role of this loop is to strengthen the fusion of text and image and extract multi-level features.
[0122] Step S45, access the output layer. The final visual features and text are concatenated to obtain an output vector of (channel_length+sequence_length, hidden_size) dimension. After performing maxpooling on the dimension of channel_length+sequence_length, multiple classification output is performed through two layers of fully connected layers.
[0123] Through the image processing method provided by this application, according to Figure 3 Provided pictures and Figure 4 The text description information provided is used to classify the pictures of Chinese medicinal materials, and can accurately identify the important categories of Chinese medicinal materials pictures, greatly improving the precision and accuracy of Chinese medicinal materials category identification based on pictures of Chinese medicinal materials. Of course, it is understandable that for the field of intelligent driving, feature extraction, feature splicing, and feature fusion can also be performed according to the information of different models through the methods provided in the embodiments of this application, thereby realizing data processing. This application does not limit specific application fields and application scenarios.
[0124] In order to provide a solution for multimodal medical image analysis, the embodiment of this application proposes a multimodal model. This technology can meet the needs of medical image analysis from multiple aspects, including text and vision. The model uses multimodal input to broaden the scope of information representation; the deep fusion of text and image information makes the information combination more accurate and improves the processing capability of the model.
[0125] The image processing method provided by the embodiment of the present application includes: obtaining text information and image information; encoding the image information through a pre-trained image encoder to obtain an image vector; encoding the text information through a pre-trained text encoder to obtain a text vector; extracting features from the image vector and the text vector respectively, and averaging the obtained features to obtain mean features; splicing the mean features with the text vector and the image vector respectively to obtain text splicing features and image splicing features; fusing the text splicing features and the image splicing features, and classifying and outputting the fused features to obtain target classification results of the text information and the image information. The embodiment of the present application is conducive to improving the accuracy of image processing.
[0126] See also Figure 5 The embodiment of the present application further provides a multi-modal image processing device, which can implement the above-mentioned multi-modal image processing method, and the device includes:
[0127] The first module is used to obtain text information and image information; the image information includes a number of objects to be classified, and the text information contains associated information for indicating the classification of the objects to be classified;
[0128] The second module is used to encode the image information through a pre-trained image encoder to obtain an image vector, and to encode the text information through a pre-trained text encoder to obtain a text vector;
[0129] The third module is used to extract features from the image vector and the text vector respectively, and perform mean processing on the obtained features to obtain mean features;
[0130] The fourth module is used to perform splicing processing on the mean feature and the text vector to obtain the text splicing feature, and to perform splicing processing on the mean feature and the image vector to obtain the image splicing feature;
[0131] The fifth module is used to fuse the text splicing features and the image splicing features, and classify the features obtained by the fusion processing to obtain the target classification result corresponding to the object to be classified.
[0132] In some embodiments, the third module is also used to: perform feature extraction on the image vector and the text vector respectively through multiple feature extractors, and average the obtained features to obtain mean features; the initial number of the feature extractors is the first number, and the length value of the mean feature in the image dimension is equal to the first number.
[0133] In some embodiments, the third module is further used to: fuse the text splicing feature and the image vector to obtain a text fusion feature;
[0134] Fusing the image splicing feature and the text vector to obtain an image fusion feature;
[0135] The text fusion feature is used as a new text vector, and the image fusion feature is used as a new image vector, and the features are returned to the process of performing feature extraction on the image vector and the text vector respectively through multiple feature extractors; wherein the ratio of the number of feature extractors in the current cycle round to the number of feature extractors in the previous cycle round is a first ratio, and the first ratio is less than 1.
[0136] In some embodiments, the third module is further used to: detect a first length value of the image portion vector and a second length value of the hidden layer vector;
[0137] The length value of the image dimension in the obtained image splicing feature is determined to be the sum of the first length value and the first quantity, and the length value of the hidden layer vector in the image splicing feature is determined to be the second length value.
[0138] In some embodiments, the fifth module is further used to: perform pooling processing on the features obtained by the fusion processing to obtain pooled features;
[0139] The pooled features are classified and output through a fully connected layer.
[0140] In some embodiments, the third module is further used to: determine the correlation between each position in the image vector and other positions;
[0141] Normalizing the correlation to obtain the weight of each correlation;
[0142] According to the correlation and the weight, a feature corresponding to the image vector is obtained.
[0143] In some embodiments, the third module is further used to: determine the correlation between each element in the image stitching feature and each element in the text vector;
[0144] A weighted sum process is performed on the image splicing feature and each element in the text vector according to the association degree to obtain an image fusion feature.
[0145] The specific implementation of the multimodal image processing device is substantially the same as the specific implementation of the multimodal image processing method described above, and will not be described in detail herein.
[0146] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned multi-modal image processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0147] See also Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0148] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0149] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the multi-modal image processing method of the embodiment of this application;
[0150] Input / output interface 903, used to implement information input and output;
[0151] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0152] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0153] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0154] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the multi-modality-based image processing method is implemented.
[0155] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0156] The embodiment of the present application provides a multimodal image processing method, a multimodal image processing device, an electronic device and a storage medium, the method comprising obtaining text information and image information; encoding the image information by a pre-trained image encoder to obtain an image vector; encoding the text information by a pre-trained text encoder to obtain a text vector; extracting features from the image vector and the text vector respectively, and averaging the obtained features to obtain mean features; splicing the mean features with the text vector and the image vector respectively to obtain text splicing features and image splicing features; fusing the text splicing features and the image splicing features, and classifying and outputting the fused features to obtain target classification results of the text information and the image information. The embodiment of the present application is conducive to improving the accuracy of image processing.
[0157] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0158] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0159] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0160] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0161] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0162] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0163] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0164] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0166] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0167] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A multi-modal image processing method, It is characterized in that The method comprises: Acquire text information and image information; the image information includes a number of objects to be classified, and the text information includes associated information for indicating classification of the objects to be classified; Encoding the image information by a pre-trained image encoder to obtain an image vector, and encoding the text information by a pre-trained text encoder to obtain a text vector; Extracting features from the image vector and the text vector respectively, and performing mean processing on the obtained features to obtain mean features; The mean feature and the text vector are spliced to obtain a text splicing feature, and the mean feature and the image vector are spliced to obtain an image splicing feature; The text splicing features and the image splicing features are fused, and the features obtained by the fusion processing are classified to obtain a target classification result corresponding to the object to be classified.
2. The method according to claim 1, It is characterized in that The step of extracting features from the image vector and the text vector respectively and performing mean processing on the obtained features to obtain mean features comprises: Feature extraction is performed on the image vector and the text vector respectively through multiple feature extractors, and the obtained features are averaged to obtain mean features; the initial number of the feature extractors is a first number, and the length value of the mean feature in the image dimension is equal to the first number.
3. The method according to claim 2, It is characterized in that The fusing process of the text splicing feature and the image splicing feature includes: Fusing the text splicing feature and the image vector to obtain a text fusion feature; Fusing the image splicing feature and the text vector to obtain an image fusion feature; The text fusion feature is used as a new text vector, and the image fusion feature is used as a new image vector, and the features are returned to the process of performing feature extraction on the image vector and the text vector respectively through multiple feature extractors; wherein the ratio of the number of feature extractors in the current cycle round to the number of feature extractors in the previous cycle round is a first ratio, and the first ratio is less than 1.
4. The method according to claim 2, It is characterized in that The image vector includes an image part vector and a hidden layer vector; the method further includes: Detecting a first length value of the image portion vector and a second length value of the hidden layer vector; The length value of the image dimension in the obtained image splicing feature is determined to be the sum of the first length value and the first quantity, and the length value of the hidden layer vector in the image splicing feature is determined to be the second length value.
5. The method according to claim 1, It is characterized in that The classifying of the features obtained by the fusion process includes: Performing pooling processing on the features obtained by the fusion processing to obtain pooled features; The pooled features are classified and output through a fully connected layer.
6. The method according to any one of claims 1 to 5, It is characterized in that Performing feature extraction on the image vector includes: Determining the correlation of each position with other positions in the image vector; Normalizing the correlation to obtain the weight of each correlation; According to the correlation and the weight, a feature corresponding to the image vector is obtained.
7. The method according to claim 3, It is characterized in that The fusing process of the image splicing feature and the text vector to obtain the image fusion feature includes: Determining the degree of association between each element in the image stitching feature and each element in the text vector; A weighted sum process is performed on the image splicing feature and each element in the text vector according to the association degree to obtain an image fusion feature.
8. A multi-modal image processing device, It is characterized in that The device comprises: The first module is used to obtain text information and image information; the image information includes a number of objects to be classified, and the text information contains associated information for indicating the classification of the objects to be classified; The second module is used to encode the image information through a pre-trained image encoder to obtain an image vector, and to encode the text information through a pre-trained text encoder to obtain a text vector; The third module is used to extract features from the image vector and the text vector respectively, and perform mean processing on the obtained features to obtain mean features; The fourth module is used to perform splicing processing on the mean feature and the text vector to obtain a text splicing feature, and to perform splicing processing on the mean feature and the image vector to obtain an image splicing feature; The fifth module is used to fuse the text splicing features and the image splicing features, and classify the features obtained by the fusion processing to obtain the target classification result corresponding to the object to be classified.
9. An electronic device, It is characterized in that The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the multi-modality-based image processing method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the multi-modality-based image processing method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Emotion recognition method and device, computer equipment and storage medium
CN114863949A
Image classification method and device, electronic equipment and computer readable storage medium
CN116469110A
Social media image-text multi-modal data classification method based on multi-level coding
CN117496315A
Case classification method and device, electronic equipment and computer readable storage medium
CN117992826A
Text-guided image detection method, system, device, medium and program product
CN118314148A
Cited By
Multi-modal identification method and apparatus for hazardous liquid, and computer program product
WO2026137768A1
