Method for processing multi-modal data and electronic device

Abstract

The application discloses a multi-modal data processing method and an electronic device, and relates to the technical field of data processing. The method comprises the following steps: determining text data and image data from received multi-modal data, and respectively pre-processing the text data and the image data to obtain a semantic feature vector corresponding to the text data and a visual feature vector corresponding to the image data; generating a multi-modal feature vector based on the semantic feature vector and the visual feature vector; inputting the multi-modal feature vector and a prompt word template into an entity extraction model to obtain an entity extraction result output by the entity extraction model, wherein the entity extraction model is a model obtained by pre-training a basic large language model based on an open source; obtaining an evaluation result for evaluating the entity extraction result; and in the case where the evaluation result is used to indicate that the entity extraction result meets an evaluation condition, sending the entity extraction result to a target object.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and more specifically, to a method and electronic device for processing multimodal data. Background Technology

[0002] Currently, entity extraction in specialized technical fields such as medicine and finance often relies on models like CRF (Conditional Random Field) and BiLSTM-CRF (Bidirectional Long Short-Term Memory-Conditional Random Field) to process single-modal data. For example, processing only text data fails to comprehensively utilize the effective information in image data, involves simple feature concatenation, and ignores the dynamic weights of semantic relationships between modalities, easily leading to information redundancy and interference. Furthermore, this isolated single-modal processing method results in a lack of entity correlation, and these general models lack a deep understanding of terminology, making it difficult to handle terminology variations and standardized expressions, leading to high false negative rates and ultimately inaccurate extraction results.

[0003] Therefore, no effective solution has yet been proposed for the technical problem of improving the accuracy of entity extraction results in professional technical fields such as medicine and finance. Summary of the Invention

[0004] This application provides a method and electronic device for processing multimodal data, which at least solves the technical problem of how to improve the accuracy of entity extraction results in professional technical fields such as medicine and finance.

[0005] According to one embodiment of this application, a method for processing multimodal data is provided, comprising: determining text data and image data from received multimodal data, and preprocessing the text data and the image data respectively to obtain a semantic feature vector corresponding to the text data and a visual feature vector corresponding to the image data; generating a multimodal feature vector based on the semantic feature vector and the visual feature vector; inputting the multimodal feature vector and a prompt word template into an entity extraction model to obtain an entity extraction result output by the entity extraction model, wherein the entity extraction model is a model obtained by pre-training based on an open-source basic large language model; obtaining an evaluation result for evaluating the entity extraction result, and sending the entity extraction result to a target object if the evaluation result indicates that the entity extraction result meets the evaluation conditions.

[0006] According to another embodiment of this application, a multimodal data processing apparatus is provided, comprising: a preprocessing module, configured to determine text data and image data from received multimodal data, and preprocess the text data and the image data respectively to obtain a semantic feature vector corresponding to the text data and a visual feature vector corresponding to the image data; a generation module, configured to generate a multimodal feature vector based on the semantic feature vector and the visual feature vector; an obtaining module, configured to input the multimodal feature vector and a prompt word template into an entity extraction model to obtain an entity extraction result output by the entity extraction model, wherein the entity extraction model is a model obtained after pre-training based on an open-source basic large language model; and a sending module, configured to obtain an evaluation result for evaluating the entity extraction result, and, if the evaluation result indicates that the entity extraction result meets the evaluation conditions, send the entity extraction result to a target object.

[0007] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0008] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0009] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0010] This application utilizes a method to separate text and image data from received multimodal data. By preprocessing the text and image data separately to obtain semantic feature vectors for the text data and visual feature vectors for the image data, text and image data can be isolated from the received multimodal data. Semantic and visual feature vectors are extracted separately, and a unified multimodal feature vector is generated by fusing the two. This overcomes the technical limitation of traditional methods where single modalities are difficult to model collaboratively. Furthermore, a multimodal feature vector is generated based on the semantic and visual feature vectors. The multimodal feature vector and prompt word templates are then input into an entity extraction model. The entity extraction results output by the entity extraction model can then be fed into the entity extraction model pre-trained based on an open-source large language model. This allows the model to accurately identify and extract entities by combining cross-modal semantic association and domain guidance information, based on its strong language understanding capabilities. Finally, an evaluation mechanism is introduced to verify the quality of the entity extraction results. The results are only output to the target object when they meet the preset evaluation conditions. This technical solution solves the technical problem of how to improve the accuracy of entity extraction results in professional technical fields such as medicine and finance, thereby improving the accuracy of entity information in the entity extraction results. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a hardware structure block diagram of a server device for a multimodal data processing method according to an embodiment of this application;

[0013] Figure 2 This is a flowchart of a multimodal data processing method according to an embodiment of this application;

[0014] Figure 3 This is a schematic diagram of the framework of a multimodal data processing system according to an embodiment of this application;

[0015] Figure 4 This is a flowchart illustrating a method for processing multimodal data according to an embodiment of this application;

[0016] Figure 5 This is a structural block diagram of a multimodal data processing apparatus according to an embodiment of this application;

[0017] Figure 6This is a computer system architecture block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0020] It should be noted that the relevant technical terms in this application are explained.

[0021] LLM, or Large Language Models, refers to artificial intelligence models with a large number of parameters that can process and generate natural language and are adaptable to multimodal data processing.

[0022] NER, or Named Entity Recognition, refers to the identification of entities with specific meanings from multimodal data such as text and images, including disease names and drug names in the medical field, and company names and stock codes in the financial field.

[0023] MM-LLM, or Multimodal Large Language Models, are large language models capable of receiving and processing various types of data, such as text, images, audio, and video, and performing corresponding analysis and generation tasks.

[0024] CNN, Convolutional Neural Network, is a deep learning model commonly used in image processing that can effectively extract local features from images.

[0025] The Transformer model, a deep learning model based on self-attention mechanism, is widely used in natural language processing and multimodal data processing, and can capture long-distance dependencies in data.

[0026] CRF, or Conditional Random Field, is a discriminative probabilistic graphical model used for sequence labeling. It is commonly used in named entity recognition to improve the accuracy of sequence prediction by modeling the dependencies between labels.

[0027] BiLSTM-CRF, or Bidirectional Long Short-Term Memory-Conditional RandomField, is a deep learning model that combines bidirectional long short-term memory networks with CRF. It is widely used in text named entity extraction tasks and can effectively model long-distance dependencies and label constraints.

[0028] ICD-10, International Classification of Diseases, 10th Revision, is an internationally unified standard for classifying diseases, injuries, and health problems, developed by the World Health Organization.

[0029] FINBERT, Financial Bidirectional Encoder Representations from Transformers, is a pre-trained language model for the financial domain. It is pre-trained on financial texts and performs exceptionally well in natural language processing tasks related to finance.

[0030] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] As an optional implementation, the method embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a multimodal data processing method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a microprocessor MPU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multimodal data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0034] This embodiment provides a method for processing multimodal data, applied to the aforementioned server device. Figure 2 This is a flowchart of a multimodal data processing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0035] Step S202: Determine text data and image data from the received multimodal data, and preprocess the text data and image data respectively to obtain the semantic feature vector corresponding to the text data and the visual feature vector corresponding to the image data.

[0036] In this step, the text and image data within the multimodal data are first independently identified and separated. Then, preprocessing operations are performed on the text data to extract semantic feature vectors. Simultaneously, preprocessing operations are performed on the image data to extract visual feature vectors. Through these preprocessing steps, the initial structuring and characterization of the multimodal data at the input stage are achieved.

[0037] Step S204: Generate a multimodal feature vector based on the semantic feature vector and the visual feature vector.

[0038] In this step, generating a multimodal feature vector based on semantic and visual feature vectors means that the semantic feature vector extracted from preprocessed text data and the visual feature vector extracted from preprocessed image data are unified into the same dimensional space through feature mapping. Then, the cross-modal attention weight is calculated by combining the degree of correlation between the two in the named entity extraction task. The semantic and visual feature vectors are then weighted and summed according to the weights to form a multimodal feature vector that integrates the information of the two. This multimodal feature vector carries the collaborative representation of text semantics and visual content, which is used to support the comprehensive recognition of named entities by subsequent large-scale language models.

[0039] Step S206: Input the multimodal feature vector and prompt word template into the entity extraction model to obtain the entity extraction result output by the entity extraction model. The entity extraction model is a model obtained by pre-training based on an open-source basic large language model.

[0040] In this step, the multimodal feature vector and the prompt word template are input into the entity extraction model. The entity extraction model integrates and analyzes the cross-modal semantic information carried by the input multimodal feature vector based on its existing language understanding and generation capabilities, and generates named entity recognition results that conform to the template format according to the entity extraction task requirements explicitly guided in the prompt word template.

[0041] Step S208: Obtain the evaluation result of the entity extraction result. If the evaluation result indicates that the entity extraction result meets the evaluation conditions, send the entity extraction result to the target object.

[0042] In this step, the named entity extraction results are evaluated to determine whether they meet the preset quality standards. When the evaluation results indicate that the entity extraction results meet the evaluation conditions (e.g., in terms of accuracy, recall, F1 score, cross-modal entity association accuracy, or domain-specific entity accuracy), it means that the extracted named entities have reached the preset threshold requirements. Under this premise, the entity extraction results are sent to the target object. This means that the output and transmission of the entity extraction results are only triggered when the evaluation confirms that the results are qualified, thereby ensuring that the delivered content is the final valid information that has been quality verified.

[0043] The above method identifies text and image data from received multimodal data, and preprocesses them separately to obtain semantic feature vectors for the text data and visual feature vectors for the image data. This allows for the separation of text and image data from the received multimodal data, and the extraction of semantic and visual feature vectors. By fusing these two vectors to generate a unified multimodal feature vector, the technical limitation of traditional methods—the difficulty of co-modeling single modalities—is overcome. Furthermore, a multimodal feature vector is generated based on the semantic and visual feature vectors. This multimodal feature vector, along with a prompt word template, is then input into the entity extraction model. The entity extraction results output by the entity extraction model can then be fed into the entity extraction model pre-trained based on an open-source large language model. This allows the model to accurately identify and extract entities by combining cross-modal semantic association and domain guidance information, based on its strong language understanding capabilities. Finally, an evaluation mechanism is introduced to verify the quality of the entity extraction results. The results are only output to the target object when they meet the preset evaluation conditions. This technical solution solves the technical problem of how to improve the accuracy of entity extraction results in professional technical fields such as medicine and finance, thereby improving the accuracy of entity information in the entity extraction results.

[0044] In an exemplary embodiment, preprocessing the text data to obtain a semantic feature vector corresponding to the text data includes: performing a cleaning operation on the text data to obtain cleaned data; obtaining the word segmentation result of the cleaned data, and performing part-of-speech tagging on the words in the word segmentation result to obtain the part-of-speech type of the words; obtaining candidate words corresponding to the words and the part-of-speech type from a preset dictionary, and obtaining the semantic feature vector of the candidate words.

[0045] In this embodiment, when preprocessing text data to generate high-quality semantic feature vectors, noise content is first removed through a cleaning operation to obtain clean data with a clear structure. Then, word segmentation is performed on the clean data to obtain semantic units formed by reasonable segmentation. Part-of-speech tagging technology is combined to identify the grammatical category of each word, thereby selecting words with entity semantic potential. Subsequently, based on a preset dictionary, candidate words related to entity recognition are matched by word and part-of-speech type, and the semantic feature vectors corresponding to these candidate words are extracted. This can improve the relevance of text features input to the multimodal model, reduce the interference of non-entity candidate information, and reduce the probability of irrelevant words, noise words, or non-entity words mixed into semantic representations due to lack of cleaning, word segmentation, or part-of-speech constraints. This improves the accuracy and domain adaptability of text semantic vectors and facilitates the improvement of the reliability of subsequent multimodal feature vectors.

[0046] In an exemplary embodiment, preprocessing the image data to obtain a visual feature vector corresponding to the image data includes: performing an image enhancement operation on the image data to obtain enhanced data; locating a region to be detected in the enhanced image of the enhanced data, and extracting the target to be detected in the region to be detected to obtain a visual feature vector corresponding to the target to be detected.

[0047] In this embodiment, image enhancement operations are performed on the original image data to improve image clarity and structural discernibility, generating enhanced data. Then, based on the enhanced image in the enhanced data, the detection region related to named entities is located, and extraction operations are performed on the target within the detection region to selectively capture key visual information, thereby generating a more representative and robust visual feature vector. This facilitates the synergistic fusion of the optimized visual feature vector and semantic feature vector to form a high-quality multimodal feature vector, which is then input into an entity extraction model fine-tuned based on an open-source large language model. This improves the accuracy and domain adaptability of entity recognition, reduces feature distortion and cross-modal association failure caused by poor image quality, and achieves the technical effect of improving entity extraction performance.

[0048] In an exemplary embodiment, the extraction operation of the target to be detected in the region to be detected to obtain the visual feature vector corresponding to the target to be detected includes: identifying a text object from the target to be detected using optical character recognition technology for the coordinate axis region in the region to be detected; extracting change trend data corresponding to the text object from the target to be detected for the data annotation region in the region to be detected; and generating the visual feature vector based on the text object and the change trend data.

[0049] In this embodiment, by analyzing the region to be detected in the enhanced image and applying optical character recognition technology to the coordinate axis region, text objects such as units, dimensions, or time scales are extracted. Simultaneously, trend data corresponding to the text objects, such as curve increase / decrease direction and numerical fluctuation patterns, are extracted from the data annotation region. These two are then fused to generate a visual feature vector rich in chart semantic structure. This allows the numerical meaning and trend relationship carried by the image to be transformed into a semantic representation that is aligned with the text semantics. This effectively compensates for the shortcomings of traditional visual feature vectors, which rely solely on visual appearance and lack semantic understanding capabilities. It improves the semantic consistency between multimodal feature vectors and entities, enhances the entity extraction model's ability to accurately identify entities in cross-modal contexts, improves the accuracy of deep semantic associations between text and images, and enhances the accuracy of multimodal named entity extraction.

[0050] In an exemplary embodiment, generating a multimodal feature vector based on the semantic feature vector and the visual feature vector includes: setting the vector dimension of the semantic feature vector and the vector dimension of the visual feature vector to the same dimension, so as to map the semantic feature vector and the visual feature vector to the same dimensional space; calculating a similarity matrix between the semantic feature vector and the visual feature vector, normalizing the similarity matrix to obtain an attention weight matrix; and using the attention weight matrix to perform a weighted summation of the semantic feature vector and the visual feature vector to obtain the fused multimodal feature vector.

[0051] In this embodiment, by mapping the semantic feature vector generated from text data and the visual feature vector generated from image data to the same dimensional space, the comparability and alignment of the two modal features in terms of numerical dimensions are ensured. Then, a similarity matrix is ​​calculated between the two to quantify the strength of cross-modal semantic association. The similarity matrix is ​​then normalized to generate a dynamically adjusted attention weight matrix, reflecting the interdependence of semantic and visual information in the entity extraction task. Subsequently, the attention weight matrix is ​​used to weight and fuse the original semantic feature vector and visual feature vector, achieving fine-grained multimodal feature integration guided by semantic association. This improves the ability of the fused features to represent entity boundaries and semantic meanings, ultimately enabling the entity extraction model based on the open-source large language model to more accurately capture the implicit cross-modal entity associations in text and images. This reduces information loss and semantic distortion caused by dimensional mismatch or simple splicing, improving the accuracy of entity extraction results.

[0052] In an exemplary embodiment, before inputting the multimodal feature vector and prompt word template into the entity extraction model to obtain the entity extraction result output by the entity extraction model, the method further includes: fine-tuning the open-source basic large language model using preset training data and fine-tuning parameters until the loss value of the fine-tuning loss function of the open-source basic large language model meets the fine-tuning completion condition, and obtaining the fine-tuned open-source basic large language model, wherein the fine-tuning parameters include at least: model learning rate, model training epochs, and the amount of data in each training epoch; performing a verification operation on the fine-tuned open-source basic large language model using preset verification data to obtain a verification result; and determining the entity extraction model based on the fine-tuned open-source basic large language model if the verification result indicates that the verification of the fine-tuned open-source basic large language model has passed.

[0053] In this embodiment, the open-source basic large language model is fine-tuned using preset training data and fine-tuning parameters, so that the model gradually converges to a state that meets the preset loss threshold in the multimodal named entity extraction task. This is beneficial for the model to fully learn the domain semantics and cross-modal association rules. Subsequently, the effect of the fine-tuned model is evaluated using preset validation data. Only when the validation results clearly indicate that the model's ability meets the standard is it confirmed as a usable entity extraction model. This reduces the extraction bias and instability caused by directly applying the original model without adaptation and validation to multimodal scenarios, and improves the model's extraction accuracy under multimodal data.

[0054] In one exemplary embodiment, the open-source basic large language model that has been fine-tuned is validated using preset validation data to obtain a validation result. This includes: determining input sample data and output sample data from the preset validation data; inputting the input sample data into the fine-tuned open-source basic large language model to obtain the validation result output by the fine-tuned open-source basic large language model; calculating the index value of an evaluation index based on the validation result and the output sample data; determining that the validation result is valid if the index value is greater than or equal to a preset index value; and determining that the validation result is invalid if the index value is less than the preset index value, and continuing to fine-tune the open-source basic large language model using updated fine-tuning parameters.

[0055] In this embodiment, input sample data and corresponding output sample data are split from preset verification data. The input sample data is then imported into a finely tuned open-source basic large language model to obtain the verification results of the predicted output. Based on the semantic consistency between the verification results and the actual output sample data, a quantitative evaluation index is calculated. When the index value reaches or exceeds a preset threshold, the verification is deemed successful and the model performance meets the standard. Otherwise, the verification is deemed unsuccessful and the dynamic update of the fine-tuning parameters is automatically triggered, driving the model to re-enter the fine-tuning iteration process. This embodiment constructs a closed-loop verification and optimization mechanism of "data splitting → model inference → index quantification → threshold judgment → parameter feedback", which reduces the probability of the original verification process relying on subjective judgment, causing the model to fail to adapt and adjust, and improves the stability and reliability of the entity extraction model.

[0056] In an exemplary embodiment, obtaining an evaluation result for evaluating the entity extraction result includes: obtaining evaluation parameters for evaluating the entity extraction result, wherein the evaluation parameters include basic evaluation metrics, cross-modal evaluation metrics, and domain terminology evaluation metrics; calculating parameter values ​​corresponding to the evaluation parameters based on the comparison results between the entity extraction result and a standard dataset, wherein the parameter values ​​include: a first parameter value corresponding to the basic evaluation metric, a second parameter value corresponding to the cross-modal evaluation metric, and a third parameter value corresponding to the domain terminology evaluation metric; and determining the evaluation result based on the parameter values.

[0057] In this embodiment, three dimensions—basic evaluation metrics, cross-modal evaluation metrics, and domain terminology evaluation metrics—are introduced to quantitatively evaluate the entity extraction results from multiple dimensions. Based on the comparison between the entity extraction results and the standard dataset, the corresponding first, second, and third parameter values ​​are calculated respectively. This enables independent measurement of entity recognition accuracy, the completeness of entity association between text and images, and the accuracy of domain-specific terminology recognition. It effectively identifies the cross-modal association loss or domain terminology omission problems that are masked by the single evaluation dimension in traditional methods. The results are determined based on the comprehensive parameter values ​​to determine whether the results meet the preset conditions. The results are only output to the target object when the evaluation meets the standards. This solves the technical problems of ambiguous evaluation process, difficulty in defect location, and unclear optimization direction, and achieves the effect of improving the reliability, accuracy, and domain adaptability of multimodal entity extraction results.

[0058] In one exemplary embodiment, the parameter values ​​corresponding to the evaluation parameters are calculated based on the comparison results between the entity extraction results and the standard dataset, including at least one of the following schemes.

[0059] Option 1: Determine the first number of correctly identified entity words in the entity extraction results and the second number of all entity words in the entity extraction results from the comparison results; determine the parameter value corresponding to the accuracy index in the basic evaluation index based on the first ratio of the first number and the second number; and determine the first parameter value based on the parameter value corresponding to the accuracy index.

[0060] Option 2: Determine the first number of correctly identified entity words in the entity extraction results and the third number of all entity words in the standard dataset from the comparison results; determine the parameter value corresponding to the recall index in the basic evaluation index based on the second ratio of the first number and the third number; and determine the first parameter value based on the parameter value corresponding to the recall index.

[0061] Option 3 involves determining, from the comparison results, the first number of correctly identified entity words in the entity extraction results, the second number of all entity words in the entity extraction results, and the third number of all entity words in the standard dataset; determining, based on the first ratio, the parameter value corresponding to the accuracy indicator in the basic evaluation metrics; determining, based on the second ratio, the parameter value corresponding to the recall indicator in the basic evaluation metrics; and determining, based on the parameter values ​​corresponding to the accuracy indicator and the recall indicator in the basic evaluation metrics, the first parameter value.

[0062] Option 4: Determine the fourth number of correctly identified cross-modal associated paired entity words in the entity extraction results and the fifth number of all cross-modal associated paired entity words in the entity extraction results from the comparison results; determine the second parameter value corresponding to the cross-modal evaluation index based on the third ratio of the fourth number and the fifth number.

[0063] Option 5: Determine the sixth number of correctly identified entity words belonging to the domain term in the entity extraction results and the seventh number of all entity words belonging to the domain term in the entity extraction results from the comparison results; determine the third parameter value corresponding to the domain term evaluation index based on the fourth ratio of the sixth number and the seventh number.

[0064] In the above embodiments, based on the comparison results between the entity extraction results and the standard dataset, the performance is quantitatively evaluated from three dimensions: basic recognition, cross-modal association, and domain terminology. The accuracy parameter (i.e., the first parameter value) in the basic evaluation metrics is determined by calculating the first ratio of the number of correctly identified entities to the total number of entities in the extraction results; or the recall parameter is determined by calculating the second ratio of the number of correctly identified entities to the total number of entities in the standard dataset, and the first parameter value is calculated based on a combination of the accuracy and recall parameters. Alternatively, the third ratio is calculated by calculating the third ratio of the number of correctly identified cross-modal paired entities to the total number of cross-modal paired entities in the extraction results. The second parameter value is calculated to reflect the accuracy of entity association between text and image; or, the third parameter value is calculated by the fourth ratio of the number of correctly identified domain term entities to the total number of domain term entities in the extraction results to measure the model's recognition specificity for specific domain terms; the above three types of parameter values ​​are combined to form a multi-dimensional evaluation result, and the entity extraction result is only output to the target object when the evaluation result meets the preset conditions, thereby effectively reducing the model's erroneous output in terms of cross-modal association ambiguity or insufficient domain adaptation. Compared with the technical problem of the original evaluation method relying on only a single indicator and being unable to locate model defects, this improves the reliability and accuracy of entity extraction quality.

[0065] Optionally, it should be noted that correctly identified entity words in the entity extraction results indicate that the entity words in the entity extraction results are consistent with the entity words in the standard dataset.

[0066] In an exemplary embodiment, the first parameter value, the second parameter value, and the third parameter value are obtained from the evaluation result; if the first parameter value is greater than a first threshold, the second parameter value is greater than a second threshold, and the third parameter value is greater than a third threshold, it is determined that the entity extraction result satisfies the evaluation condition.

[0067] Optionally, the thresholds (first threshold, second threshold, and third threshold) mentioned in this embodiment can be determined by historical experience, or they can utilize default parameters from engineering practice. For example, the first threshold is 0.9, the second threshold is 0.85, and the third threshold is 0.88.

[0068] In this embodiment, the first, second, and third parameter values ​​are obtained by quantifying evaluation indicators (such as accuracy, cross-modal association accuracy, and domain-specific accuracy). When all three exceed preset thresholds (such as F1 ≥ 0.9, cross-modal association accuracy ≥ 0.85, and domain entity accuracy ≥ 0.88), the entity extraction result is determined to meet the high-quality standard, and the result can be output.

[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0070] This embodiment also provides a multimodal data processing system corresponding to the above-described multimodal data processing method. This system includes functional modules for implementing the above embodiments and preferred embodiments, such as... Figure 3 As shown, the details are as follows.

[0071] The multimodal data input module performs format verification and preliminary classification on the received input data such as text, images, and charts, and assigns different types of data to the corresponding preprocessing sub-modules in the multimodal data preprocessing module.

[0072] The multimodal data preprocessing module includes a text preprocessing submodule and an image preprocessing submodule.

[0073] The text preprocessing submodule can perform the following functions: 1. Text formatting: Remove special characters, garbled text, headers and footers, and other irrelevant content from the text; standardize the expression of terms such as disease names and drug names; correct spelling errors; and standardize data format. 2. Word segmentation and part-of-speech tagging: Use a domain-adaptive word segmentation tool to segment the cleaned text, and then use a part-of-speech tagging model to tag the segmentation results, distinguishing between nouns, verbs, adjectives, etc., with a focus on nouns related to entities. 3. Matching domain dictionaries: Construct a medical domain dictionary (including ICD-10 disease names, etc.) and a financial domain dictionary (including listed company names, stock codes, financial product names, financial indicator names, etc.). Match the word segmentation results with the domain dictionaries to initially screen potential named entity candidate words, providing a foundation for subsequent entity extraction.

[0074] The image preprocessing submodule can perform the following functions: 1. Convert image formats or perform image enhancement, remove irrelevant background information in charts, adjust image resolution to a uniform standard, and use edge detection algorithms to highlight lines, text, and data annotation areas in charts. 2. Locate key regions and extract features.

[0075] Pre-trained object detection models are used to locate key regions in medical images, such as those of organs like the lungs, heart, and liver, or lesions. Then, a CNN model is used to extract visual features from these key regions, resulting in region feature vectors. Alternatively, a chart classification model can be used to distinguish between line charts, bar charts, pie charts, and other chart types. For different chart types, appropriate image segmentation techniques are used to extract data annotation areas such as numerical labels and legend labels, or to extract coordinate axis areas containing scale and unit information. Then, optical character recognition (OCR) technology is used to identify the text information in the extracted image regions. Simultaneously, data trend features such as upward / downward trends in line charts and height differences in bar charts are extracted. Finally, both are converted into structured feature vectors.

[0076] By performing preprocessing operations on different types of multimodal data, we can lay the foundation for subsequent cross-modal feature fusion and improve the effectiveness and usability of each modality of data.

[0077] The cross-modal feature fusion module specifically refers to building a cross-modal attention fusion model based on an attention mechanism to achieve deep fusion of text features and image features. First, the text features obtained after text preprocessing (such as semantic feature vectors extracted by the BERT model) and the image features obtained after image preprocessing (such as visual feature vectors extracted by CNN) are mapped to the same dimensional space (e.g., both mapped to 1024-dimensional vectors) to ensure feature fusion. Then, the attention weights between text features and image features are calculated; these attention weights reflect the importance of different modal features for the named entity extraction task.

[0078] For example, in the task of "identifying lung cancer entities" in medical data, if the text features include symptom descriptions such as "long-term smoking history, coughing and hemoptysis" and the image features include nodule features in lung CT images, the attention model will assign corresponding weights based on the degree of association between these features and the "lung cancer" entity.

[0079] For the above fusion process, firstly, the similarity matrix between text features and image features is calculated. Then, the similarity matrix is ​​normalized using the Softmax function to obtain the attention weight matrix. Finally, based on the attention weight matrix, the text features and image features are weighted and summed to obtain the fused multimodal feature vector. Next, the fused multimodal feature vector can be normalized to remove feature redundancy, while dropout technology is used to prevent model overfitting, ensuring the effectiveness and generalization ability of the fused features.

[0080] The aforementioned cross-modal feature fusion module, by introducing intermodal association modeling, captures the semantic and feature associations between modalities such as text and images. It can automatically learn the importance weights of different modal features in named entity extraction tasks, achieving dynamic feature fusion. In multimodal data processing in the medical field, when extracting disease-related entities, if the text data contains clear disease symptoms and diagnostic results, such as the association between "lung shadow" in medical text and abnormal features of a specific lung region in medical images, the weight of text modal features will automatically increase; if the image data (medical images) contains typical lesion features, the weight of image modal features will increase accordingly. Similarly, in multimodal data processing in the financial field, if the text data "revenue growth" is associated with the upward trend of the revenue curve in a financial statement line chart, the weight of text modal features will automatically increase, forming a more comprehensive and accurate multimodal feature representation, providing sufficient information support for LLM entity extraction.

[0081] The LLM entity extraction module specifically includes the LLM domain fine-tuning submodule and the entity extraction submodule.

[0082] The LLM domain fine-tuning submodule can construct datasets, for example, by building training, validation, and test sets in an 8:1:1 ratio. For the medical domain, datasets can include publicly available medical corpora, annotated medical named entity datasets, and manually annotated medical multimodal entity data. For the financial domain, datasets can include listed company annual reports, financial news corpora, annotated financial named entity datasets, and manually annotated financial multimodal entity data. The constructed datasets are then cleaned, formatted, and partitioned. Next, a basic LLM model (such as GPT-4 or LLaMA2-7B / 13B) is selected as a pre-trained model and fine-tuned using the domain dataset. The fused multimodal feature vectors are converted into an LLM-understandable input format and fine-tuned using training prompts.

[0083] In the medical field, training prompts might be something like, "Based on the following medical record text and lung CT image features, extract the disease name, symptom name, and drug name entities: [multimodal feature information]", guiding LLM to perform named entity recognition. Through comprehensive analysis of multimodal features, LLM can not only identify entities in text but also verify and supplement entity information by combining image features. For example, if "lung nodules" are found in medical images, the corresponding disease entity name can be confirmed by combining the medical record text; similarly, if "a company's 2024 revenue data" is identified in a financial chart, the corresponding financial indicator entity category can be determined by combining the annual report text.

[0084] During fine-tuning, a mini-batch gradient descent optimization algorithm is employed, with appropriate learning rate, batch size, and training epochs set. A domain-specific fine-tuning loss function (such as a joint loss function combining entity recognition loss and cross-modal association loss) is also used to ensure that LLM learns domain-specific entity recognition capabilities while simultaneously mastering multimodal information association patterns. After fine-tuning, model performance is evaluated using a validation set. If performance is below target (e.g., F1 score below 0.9), fine-tuning parameters are adjusted (e.g., increasing training epochs, adjusting the learning rate) or supplementing the dataset, and fine-tuning is repeated. It's important to note that the F1 score is a crucial metric in machine learning and statistics for evaluating classification model performance; it represents the harmonic mean of precision and recall, combining the performance of both metrics.

[0085] The entity extraction submodule takes the multimodal feature vectors output by the cross-modal feature fusion module and combines them with domain cue word templates, then inputs them into the LLM. The constructed input information is then fed into the domain-fine-tuned LLM, appropriate inference parameters are set, and the LLM is started for named entity extraction inference. Guided by the input multimodal features and cue words, the LLM outputs results containing named entities and their types (i.e., entity extraction results).

[0086] The above inference parameters, such as temperature=0.1, ensure the stability and accuracy of the output results, and max_length=2048, meet the requirements for long text entity extraction.

[0087] For example, the prompt word template in the medical field is "Based on the following medical text features and medical image features, extract the entities of disease name, symptom name, drug name, and organ name, with the entity type annotation format being [entity name, entity type]: [multimodal feature information]"; the prompt word template in the financial field is "Based on the following financial text features and financial chart features, extract the entities of company name, stock code, financial indicator name, and financial product name, with the entity type annotation format being [entity name, entity type]: [multimodal feature information]".

[0088] The evaluation and optimization module includes an evaluation submodule and a parameter optimization submodule.

[0089] The evaluation submodule can comprehensively evaluate the results output by the LLM entity extraction module by combining basic evaluation metrics, cross-modal evaluation metrics, and domain terminology evaluation metrics.

[0090] The basic evaluation metrics include accuracy, recall, and F1 score.

[0091] Accuracy = Number of correctly identified entities / Total number of identified entities.

[0092] Recall = Number of correctly identified entities / Total number of entities actually existing in the dataset.

[0093] F1 score = 2 × precision × recall / (precision + recall).

[0094] Cross-modal evaluation metrics include cross-modal entity association accuracy. Specifically, it measures the accuracy of cross-modal entity association by calculating the ratio of correctly associated entity pairs to the total number of associated entity pairs for entities in multimodal data (such as the entity corresponding to the lung nodule region in medical text and medical image).

[0095] Domain terminology evaluation metrics include domain entity specificity accuracy, which is specifically calculated as the ratio of the number of correctly identified domain-specific entities to the total number of identified domain-specific entities (such as rare disease names in the medical field or specific financial derivative names in the financial field). This can assess the accuracy of domain entity identification.

[0096] During the evaluation process, the extracted results are compared with a manually labeled standard dataset, and the values ​​of each evaluation indicator are calculated.

[0097] The parameter optimization submodule automatically adjusts the parameters of each module in the system based on the evaluation results output by the evaluation submodule, forming an optimization closed loop. Specifically: if the cross-modal entity association accuracy is lower than a preset threshold (e.g., 0.85), the calculation strategy of the attention weight in the cross-modal feature fusion module is adjusted (e.g., increasing the weight coefficient of inter-modal association features), and feature fusion and entity extraction are re-performed. If the domain-specific entity accuracy is lower than a preset threshold (e.g., 0.88), domain-specific entity annotation data is supplemented, and the LLM is fine-tuned to optimize its ability to recognize domain-specific entities. If the recall is lower than a preset threshold (e.g., 0.85), the reasons for the low recall are analyzed and optimization strategies are provided. If the low recall is for text entities, the word segmentation rules of the text preprocessing submodule are optimized (e.g., adding word segmentation exception rules for domain-specific vocabulary) or the domain dictionary is expanded. If the low recall is for image-related entities, the key region localization algorithm of the image preprocessing submodule is optimized (e.g., adjusting the confidence threshold of the object detection model). If the accuracy is lower than a preset threshold (e.g., 0.88), adjust LLM inference parameters such as the temperature value or optimize the prompt word template to enhance the accuracy of the LLM output. Then iteratively execute the above evaluation and optimization process until each evaluation indicator reaches the preset standard such as F1 value ≥ 0.9, cross-modal entity association accuracy ≥ 0.85, and domain entity specificity accuracy ≥ 0.88.

[0098] A multi-dimensional evaluation index system was established through the evaluation and optimization module, which can form a closed-loop evaluation process of "preprocessing-fusion-extraction-evaluation-adjustment". Based on the evaluation results, the parameters of each link are optimized in reverse to dynamically optimize the model's extraction capability.

[0099] The results output module can output evaluated and optimized entity extraction results that meet the standards in multiple formats. It also supports users in viewing, exporting, and providing feedback on the output results. Users can mark erroneous entity extraction results, and this feedback information will serve as reference data for subsequent optimization. These multiple formats include, but are not limited to, structured files such as charts and graphs, and visual reports. Entity extraction results can include entity name, entity type, entity source, and entity location.

[0100] Furthermore, in combination Figure 4 The processing flow for the above multimodal data is further explained, and the implementation steps are as follows.

[0101] Step 1: Input multimodal data. Users input multimodal data such as text and images, which are then formatted and categorized by the system described above.

[0102] Step 2: Next, multimodal data preprocessing is performed. For text data, cleaning, word segmentation and part-of-speech tagging, and domain dictionary matching are performed to obtain text features (corresponding semantic feature vectors) and potential entity candidate words. For image data, format conversion and enhancement, key region localization and feature extraction are performed to obtain image features (corresponding visual feature vectors).

[0103] Step 3: Cross-modal feature fusion. Text features and image features are mapped to the same dimensional space, cross-modal attention weights are calculated, and the features are weighted and fused to generate a unified multimodal feature vector.

[0104] Step 4: LLM Entity Extraction. Input information is constructed by combining multimodal feature vectors and domain cue word templates. This input information is then fed into the domain-fine-tuned LLM for named entity extraction, yielding preliminary extraction results.

[0105] Step 5: Multi-dimensional evaluation. Using basic evaluation metrics, cross-modal entity association accuracy, domain entity specificity accuracy, and other metrics, compare the preliminary extraction results with the standard dataset to evaluate the quality of the extraction results.

[0106] Step 6: Indicator Judgment. Determine whether each evaluation indicator meets the preset standards. If it meets the standards, proceed to Step 9; if it does not meet the standards, proceed to Step 7.

[0107] Step 7, Parameter Optimization: The parameter optimization submodule adjusts the parameters of each module of the system (such as cross-modal fusion weights, LLM fine-tuning parameters, preprocessing rules, etc.) based on the evaluation results.

[0108] Step 8: Return to Step 2 and repeat the preprocessing to evaluation process until the indicators meet the standards.

[0109] Step 9, Results Output: The results output module outputs the extracted results that meet the criteria to the user in formats such as structured tables, JSON files, and visual reports.

[0110] The above steps effectively reduce data noise interference through collaborative preprocessing of multimodal data, and can effectively capture entity relationships between modal data such as text and images. In addition, domain-adjusted LLM is used for entity extraction to accurately identify entities in multimodal data. A dynamic optimization closed loop is introduced to adaptively adjust the LLM based on the evaluation results, thereby improving the entity extraction performance of the LLM.

[0111] It should be noted that the above embodiments and preferred embodiments have already been described and will not be repeated.

[0112] Furthermore, as used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0113] Figure 5 This is a structural block diagram of a multimodal data processing apparatus according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0114] The preprocessing module 52 is used to determine text data and image data from the received multimodal data, and to preprocess the text data and the image data respectively to obtain the semantic feature vector corresponding to the text data and the visual feature vector corresponding to the image data.

[0115] The generation module 54 is used to generate a multimodal feature vector based on the semantic feature vector and the visual feature vector.

[0116] The module 56 is used to input the multimodal feature vector and prompt word template into the entity extraction model to obtain the entity extraction result output by the entity extraction model, wherein the entity extraction model is a model obtained by pre-training based on an open-source basic large language model.

[0117] The sending module 58 is used to obtain the evaluation result of the entity extraction result, and when the evaluation result indicates that the entity extraction result meets the evaluation conditions, the entity extraction result is sent to the target object.

[0118] Using the aforementioned apparatus, text data and image data are determined from received multimodal data. The text data and image data are preprocessed separately to obtain semantic feature vectors corresponding to the text data and visual feature vectors corresponding to the image data. This allows for the separation of text data and image data from the received multimodal data, and the extraction of semantic and visual feature vectors. By fusing these two vectors to generate a unified multimodal feature vector, the technical shortcomings of traditional methods, such as the difficulty in co-modeling single modalities, are overcome. Furthermore, a multimodal feature vector is generated based on the semantic and visual feature vectors. The multimodal feature vector and the prompt word template are then input into the entity extraction model. The entity extraction results output by the entity extraction model can then be fed into the entity extraction model pre-trained based on an open-source large language model. This allows the model to accurately identify and extract entities by combining cross-modal semantic association and domain guidance information, based on its strong language understanding capabilities. Finally, an evaluation mechanism is introduced to verify the quality of the entity extraction results. The results are only output to the target object when they meet the preset evaluation conditions. This technical solution solves the technical problem of how to improve the accuracy of entity extraction results in professional technical fields such as medicine and finance, thereby improving the accuracy of entity information in the entity extraction results.

[0119] In an exemplary embodiment, the above-described apparatus is further configured to: perform a cleaning operation on the text data to obtain cleaned data; obtain the word segmentation result of the cleaned data, and perform part-of-speech tagging on the words in the word segmentation result to obtain the part-of-speech type of the words; obtain candidate words corresponding to the words and the part-of-speech type from a preset dictionary, and obtain the semantic feature vector of the candidate words.

[0120] In an exemplary embodiment, the above-described apparatus is further configured to: perform an image enhancement operation on the image data to obtain enhanced data; locate the region to be detected in the enhanced image of the enhanced data, and extract the target to be detected in the region to be detected to obtain the visual feature vector corresponding to the target to be detected.

[0121] In an exemplary embodiment, the above-described apparatus is further configured to: identify a text object from the target to be detected using optical character recognition technology for a coordinate axis region in the region to be detected; extract change trend data corresponding to the text object from the target to be detected for a data annotation region in the region to be detected; and generate the visual feature vector based on the text object and the change trend data.

[0122] In an exemplary embodiment, the above apparatus is further configured to: set the vector dimension of the semantic feature vector and the vector dimension of the visual feature vector to the same dimension, so as to map the semantic feature vector and the visual feature vector to the same dimensional space; calculate the similarity matrix between the semantic feature vector and the visual feature vector, normalize the similarity matrix to obtain an attention weight matrix; and use the attention weight matrix to perform a weighted summation of the semantic feature vector and the visual feature vector to obtain the fused multimodal feature vector.

[0123] In an exemplary embodiment, before inputting the multimodal feature vector and prompt word template into the entity extraction model to obtain the entity extraction result output by the entity extraction model, the above-mentioned apparatus is further configured to: fine-tune the open-source basic large language model using preset training data and fine-tuning parameters until the loss value of the fine-tuning loss function of the open-source basic large language model meets the fine-tuning completion condition, and obtain the fine-tuned open-source basic large language model, wherein the fine-tuning parameters include at least: model learning rate, model training epochs, and data volume in each training epoch; perform a verification operation on the fine-tuned open-source basic large language model using preset verification data to obtain a verification result; and, if it is determined that the verification result is used to indicate that the fine-tuned open-source basic large language model has passed verification, determine the entity extraction model based on the fine-tuned open-source basic large language model.

[0124] In an exemplary embodiment, the above-described apparatus is further configured to: determine input sample data and output sample data from the preset verification data; input the input sample data into the fine-tuned open-source basic large language model to obtain the verification result output by the fine-tuned open-source basic large language model; calculate the index value of the evaluation index based on the verification result and the output sample data; determine that the verification result is verified as passed if the index value is greater than or equal to the preset index value; determine that the verification result is verified as failed if the index value is less than the preset index value, and continue to fine-tune the open-source basic large language model using updated fine-tuning parameters.

[0125] In an exemplary embodiment, the apparatus is further configured to: obtain evaluation parameters for evaluating the entity extraction result, wherein the evaluation parameters include a basic evaluation index, a cross-modal evaluation index, and a domain terminology evaluation index; calculate parameter values ​​corresponding to the evaluation parameters based on the comparison results of the entity extraction result and a standard dataset, wherein the parameter values ​​include: a first parameter value corresponding to the basic evaluation index, a second parameter value corresponding to the cross-modal evaluation index, and a third parameter value corresponding to the domain terminology evaluation index; and determine the evaluation result based on the parameter values.

[0126] In an exemplary embodiment, the above-described apparatus is further configured to at least implement one of the following: determining, from the comparison result, a first number of correctly identified entity words in the entity extraction result and a second number of all entity words in the entity extraction result; determining, based on a first ratio of the first number and the second number, a parameter value corresponding to the accuracy index in the basic evaluation metric; and determining, based on the parameter value corresponding to the accuracy index, the first parameter value; determining, from the comparison result, a first number of correctly identified entity words in the entity extraction result and a third number of all entity words in the standard dataset; determining, based on a second ratio of the first number and the third number, a parameter value corresponding to the recall index in the basic evaluation metric; and determining, based on the parameter value corresponding to the recall index, the first parameter value; and determining, from the comparison result, a first number of correctly identified entity words in the entity extraction result, a second number of all entity words in the entity extraction result, and all entity words in the standard dataset. The third quantity is determined based on the first ratio to obtain the parameter value corresponding to the accuracy index in the basic evaluation index, the second ratio to obtain the parameter value corresponding to the recall index in the basic evaluation index, and the first parameter value is determined based on the parameter values ​​corresponding to the accuracy index and the recall index in the basic evaluation index. From the comparison results, the fourth quantity of correctly identified cross-modal associated paired entity words in the entity extraction results and the fifth quantity of all cross-modal associated paired entity words in the entity extraction results are determined. The second parameter value corresponding to the cross-modal evaluation index is determined based on the third ratio of the fourth quantity and the fifth quantity. From the comparison results, the sixth quantity of correctly identified entity words belonging to domain terms in the entity extraction results and the seventh quantity of all entity words belonging to domain terms in the entity extraction results are determined. The third parameter value corresponding to the domain term evaluation index is determined based on the fourth ratio of the sixth quantity and the seventh quantity.

[0127] In an exemplary embodiment, the above-described apparatus is further configured to: obtain the first parameter value, the second parameter value, and the third parameter value from the evaluation result; and determine that the entity extraction result satisfies the evaluation condition if the first parameter value is greater than a first threshold, the second parameter value is greater than a second threshold, and the third parameter value is greater than a third threshold.

[0128] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0129] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0130] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0131] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0132] Optionally, Figure 6 This is a computer system architecture block diagram of an electronic device according to an embodiment of this application. For example... Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM). The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output interface 605 (I / O interface) is also connected to the bus 604.

[0133] The following components are connected to the input / output interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a local area network card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0134] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0135] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0136] The embodiments described herein also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0137] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0138] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0140] The foregoing has provided a detailed description of a multimodal data processing method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for processing multimodal data, characterized in that, include: Text data and image data are determined from the received multimodal data, and the text data and image data are preprocessed respectively to obtain the semantic feature vector corresponding to the text data and the visual feature vector corresponding to the image data; A multimodal feature vector is generated based on the semantic feature vector and the visual feature vector; The multimodal feature vector and prompt word template are input into the entity extraction model to obtain the entity extraction result output by the entity extraction model. The entity extraction model is a model obtained by pre-training based on an open-source basic large language model. Obtain an evaluation result for evaluating the entity extraction result, and if the evaluation result indicates that the entity extraction result meets the evaluation conditions, send the entity extraction result to the target object.

2. The method for processing multimodal data according to claim 1, characterized in that, The text data is preprocessed to obtain the semantic feature vector corresponding to the text data, including: The text data is cleaned to obtain cleaned data. Obtain the word segmentation results of the cleaned data, and perform part-of-speech tagging on the words in the word segmentation results to obtain the part-of-speech type of the words; Candidate words corresponding to the given word and part-of-speech type are obtained from a preset dictionary, and the semantic feature vectors of the candidate words are obtained.

3. The method for processing multimodal data according to claim 1, characterized in that, The image data is preprocessed to obtain the visual feature vector corresponding to the image data, including: The image data is subjected to image enhancement operation to obtain enhanced data; For the enhanced image in the enhanced data, the region to be detected in the enhanced image is located, and the target to be detected in the region to be detected is extracted to obtain the visual feature vector corresponding to the target to be detected.

4. The method for processing multimodal data according to claim 3, characterized in that, The target to be detected in the region to be detected is extracted to obtain the visual feature vector corresponding to the target to be detected, including: For the coordinate axis region in the area to be detected, optical character recognition technology is used to identify text objects from the target to be detected; For the data annotation area in the region to be detected, extract the change trend data corresponding to the text object from the target to be detected; The visual feature vector is generated based on the text object and the trend data.

5. The method for processing multimodal data according to claim 1, characterized in that, Generating a multimodal feature vector based on the semantic feature vector and the visual feature vector includes: Set the vector dimension of the semantic feature vector and the vector dimension of the visual feature vector to the same dimension so that the semantic feature vector and the visual feature vector are mapped to the same dimensional space. Calculate the similarity matrix between the semantic feature vector and the visual feature vector, and normalize the similarity matrix to obtain the attention weight matrix; The semantic feature vector and the visual feature vector are weighted and summed using the attention weight matrix to obtain the fused multimodal feature vector.

6. The method for processing multimodal data according to claim 1, characterized in that, Before inputting the multimodal feature vector and prompt word template into the entity extraction model to obtain the entity extraction result output by the entity extraction model, the method further includes: The open-source basic large language model is fine-tuned using preset training data and fine-tuning parameters until the loss value of the fine-tuning loss function of the open-source basic large language model meets the fine-tuning completion condition, and the fine-tuned open-source basic large language model is obtained. The fine-tuning parameters include at least: model learning rate, model training epochs, and data volume in each training epoch. The open-source basic large language model, which has been fine-tuned, is used to perform a verification operation using preset verification data to obtain the verification results. If the verification result indicates that the fine-tuned open-source basic large language model has passed verification, the entity extraction model is determined based on the fine-tuned open-source basic large language model.

7. The method for processing multimodal data according to claim 6, characterized in that, The finely tuned open-source basic language model is validated using preset validation data to obtain validation results, including: The input sample data and output sample data are determined from the preset verification data; The input sample data is input into the fine-tuned open-source basic large language model, and the verification results output by the fine-tuned open-source basic large language model are obtained. The evaluation index values ​​are calculated based on the verification results and the output sample data. If the indicator value is determined to be greater than or equal to the preset indicator value, the verification result is determined to be a successful verification. If the value of the indicator is determined to be less than the preset value, the verification result is determined to be a failure, and the open-source basic large language model is further fine-tuned using updated fine-tuning parameters.

8. The method for processing multimodal data according to claim 1, characterized in that, Obtaining the evaluation results of the entity extraction results includes: Obtain evaluation parameters for evaluating the entity extraction results, wherein the evaluation parameters include basic evaluation metrics, cross-modal evaluation metrics, and domain terminology evaluation metrics; The parameter values ​​corresponding to the evaluation parameters are calculated based on the comparison results between the entity extraction results and the standard dataset. The parameter values ​​include: the first parameter value corresponding to the basic evaluation index, the second parameter value corresponding to the cross-modal evaluation index, and the third parameter value corresponding to the domain terminology evaluation index. The evaluation result is determined based on the parameter values.

9. The method for processing multimodal data according to claim 8, characterized in that, The evaluation parameters are calculated based on the comparison between the entity extraction results and the standard dataset, and include at least one of the following: From the comparison results, determine the first number of correctly identified entity words in the entity extraction results and the second number of all entity words in the entity extraction results. Based on the first ratio of the first number and the second number, determine the parameter value corresponding to the accuracy index in the basic evaluation index, and determine the first parameter value based on the parameter value corresponding to the accuracy index. From the comparison results, determine the first number of correctly identified entity words in the entity extraction results and the third number of all entity words in the standard dataset. Based on the second ratio of the first number and the third number, determine the parameter value corresponding to the recall index in the basic evaluation index, and determine the first parameter value based on the parameter value corresponding to the recall index. From the comparison results, determine the first number of correctly identified entity words in the entity extraction results, the second number of all entity words in the entity extraction results, and the third number of all entity words in the standard dataset. Based on the first ratio, determine the parameter value corresponding to the accuracy index in the basic evaluation metrics. Based on the second ratio, determine the parameter value corresponding to the recall index in the basic evaluation metrics. And based on the parameter value corresponding to the accuracy index and the parameter value corresponding to the recall index in the basic evaluation metrics, determine the first parameter value. From the comparison results, determine the fourth number of correctly identified cross-modal associated paired entity words in the entity extraction results and the fifth number of all cross-modal associated paired entity words in the entity extraction results; and determine the second parameter value corresponding to the cross-modal evaluation index based on the third ratio of the fourth number and the fifth number. From the comparison results, determine the sixth number of entity words correctly identified as belonging to the domain term in the entity extraction results and the seventh number of all entity words belonging to the domain term in the entity extraction results. Based on the fourth ratio of the sixth number and the seventh number, determine the third parameter value corresponding to the domain term evaluation index.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the multimodal data processing method as described in any one of claims 1 to 9.

Images