Multi-modal data processing method and device, equipment, storage medium and program product

Abstract

The invention provides a multi-modal data processing method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, in particular to the technical field related to data processing. The method comprises the steps of analyzing text data in multi-modal data corresponding to a first classification task based on a zero sample cue word to obtain a first classification result, and constructing an initial database based on the first classification result; based on the initial database and the first few-sample cue word, analyzing text data in the multi-modal data corresponding to the second classification task to obtain a second classification result; based on an intermediate database obtained by updating the initial database by adopting the second classification result, processing text data and image data in multi-modal data corresponding to a third classification task to obtain a third classification result; and updating the intermediate database based on the third classification result to obtain a target database. According to the method, the cost can be saved, and effective classification processing of the multi-modal data can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the field of data processing related technology, and more particularly to a multimodal data processing method, a multimodal data processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of artificial intelligence, computers have made great progress in their ability to understand images and texts. Deep learning technology has been widely and successfully used in tasks that process data such as images and texts. Generally, it is necessary to collect a large amount of high-quality annotated data in advance as sample data for model training. Only models that have been trained and meet the training completion standards will be applied and deployed. Summary of the invention

[0003] The embodiments of the present disclosure provide a multimodal data processing method, device, multimodal data processing system, server, computer-readable storage medium, and computer program product.

[0004] In a first aspect, an embodiment of the present disclosure proposes a multimodal data processing method, including: based on a zero-sample prompt word, analyzing the text data in the multimodal data corresponding to a first classification task to obtain a first classification result, and constructing an initial database based on the first classification result; based on the initial database and the first few-sample prompt word, analyzing the text data in the multimodal data corresponding to the second classification task to obtain a second classification result; based on an intermediate database obtained by updating the initial database with the second classification result, processing the text data and image data in the multimodal data corresponding to the third classification task to obtain a third classification result; and updating the intermediate database based on the third classification result to obtain a target database.

[0005] In the second aspect, the embodiment of the present disclosure proposes a multimodal data processing device, including: a first processing module, a second processing module, a third processing module and an update module. The first processing module is configured to analyze the text data in the multimodal data corresponding to the first classification task based on the zero-sample prompt word to obtain a first classification result, and to build an initial database based on the first classification result; the second processing module is configured to analyze the text data in the multimodal data corresponding to the second classification task based on the initial database and the first few-sample prompt word to obtain a second classification result; the third processing module is configured to process the text data and image data in the multimodal data corresponding to the third classification task based on the intermediate database obtained by updating the initial database with the second classification result to obtain a third classification result; the update module is configured to update the intermediate database based on the third classification result to obtain a target database.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor, when executing, can implement the multimodal data processing method described in any implementation method in the first aspect.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement a multimodal data processing method as described in any implementation method of the first aspect when executed.

[0008] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the multimodal data processing method as described in any implementation manner in the first aspect.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0011] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0012] Figure 2 A flowchart of a multimodal data processing method provided by an embodiment of the present disclosure;

[0013] Figure 3 A flowchart of another multimodal data processing method provided by an embodiment of the present disclosure;

[0014] Figure 4 An overall framework diagram for performing multimodal data processing in an application scenario provided by an embodiment of the present disclosure;

[0015] Figure 5 A structural block diagram of a multimodal data processing device provided in an embodiment of the present disclosure;

[0016] Figure 6 A schematic diagram of the structure of an electronic device suitable for executing a multimodal data processing method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0018] It should be pointed out that in the technical solution of the present invention, the collection, acquisition, storage, processing, transmission, provision, disclosure and application of user personal information (such as file sharing records, etc.) are all carried out with the user's knowledge and explicit authorization, comply with the relevant laws and regulations, and do not violate public order and good morals.

[0019] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the multimodal data processing scheme of the present disclosure may be applied.

[0020] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0021] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 and the server 105 can be installed with various applications for realizing information communication between the two, such as multimodal data processing applications, data processing applications, instant messaging applications, etc.

[0022] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0023] The server 105 can provide various services through various built-in applications. Taking the provision of multimodal data processing related applications as an example, the server 105 can achieve the following effects when running the multimodal data processing related applications: by flexibly connecting the stage of analyzing and processing the text data in the received multimodal data based on the zero-sample prompt method, the stage of analyzing and processing the text data in the received multimodal data based on the few-sample prompt method, and the stage of analyzing and processing both the text data and the image data in the received multimodal data, without the need to collect annotated data in advance and without explicitly training the model, especially without explicitly training the model for data of different modalities, the multimodal data processing solution can be quickly deployed in the business environment, and by continuously collecting data during business operations and gradually improving the data analysis and processing performance, costs can be saved while achieving effective classification processing of multimodal data.

[0024] It should be noted that, in addition to being obtained from the terminal devices 101, 102, 103 through the network 104, the multimodal related information or data input by the user can also be pre-stored locally in the server 105 in various ways. Therefore, when the server 105 detects that the data has been stored locally (for example, the multimodal related information or data input by the user will be regularly backed up or stored in the server 105 before starting the processing), it can choose to directly obtain the data locally. In this case, the exemplary system architecture 100 may also not include the terminal devices 101, 102, 103 and the network 104.

[0025] Since the analysis of text data, image data, etc. to realize multimodal data processing requires more computing resources and stronger computing power, the multimodal data processing methods provided in the subsequent embodiments of the present disclosure are generally performed by the server 105 with stronger computing power and more computing resources, and accordingly, the multimodal data processing device is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned operations that are handed over to the server 105 through the multimodal data processing applications installed thereon, and then output the same results as the server 105. Especially in the case where there are multiple terminal devices with different computing powers at the same time, but when the multimodal data processing application determines that the terminal device where it is located has stronger computing power and more remaining computing resources, it can allow the terminal device to perform the above-mentioned operations, thereby appropriately reducing the computing pressure of the server 105, and accordingly, the multimodal data processing device can also be set in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104 .

[0026] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0027] Please refer to Figure 2 , Figure 2 This is a flowchart of a multimodal data processing method provided by an embodiment of the present disclosure. The multimodal data processing method is executed by an execution subject server, wherein process 200 includes the following steps:

[0028] Step 201: Based on the zero-sample prompt word, the text data in the multimodal data corresponding to the first classification task is analyzed to obtain a first classification result, and an initial database is constructed based on the first classification result.

[0029] This step is intended to be performed by the above-mentioned execution subject of the multimodal data processing method (for example Figure 1The server 105 shown in the figure can adopt the zero-shot prompting (ZSP) method in the initial stage, that is, based on the zero-sample prompt word, analyze the text data in the multimodal data corresponding to the first classification task received in the current stage to obtain the corresponding classification results, collect the return data, and store the classification results in the corresponding database to realize the construction of the initial database and the dynamic update of the initial database. Among them, the first classification task is the data analysis and processing task received in the stage of processing the text data in the multimodal data based on the zero-sample prompt word. The number of the first classification tasks and the standard for completing the construction of the initial database are not specifically limited here, depending on the specific situation of the specific application scenario. For example, when the number of sample data accumulated in the initial database reaches the corresponding set value or the time length for processing the text data in the multimodal data based on the zero-sample prompt word reaches the corresponding set time length, it can be characterized that the initial database construction is completed, that is, it can also be characterized that the initial stage of multimodal data processing based on the zero-sample prompt word is over.

[0030] Among them, the zero-shot prompt method, which can be applied to a variety of tasks (including question answering, text generation, text classification, etc.), is a technique that uses prior knowledge or contextual information to guide the Large Language Model (LLM) to make inferences and decisions when performing zero-shot tasks. In this technique, prompt words or phrases are used to guide the model to understand the task or problem without providing any relevant text examples or model training. In this way, the large language model can use prior knowledge to derive the correct answer or solution.

[0031] In some optional implementations of the disclosed embodiments, the data in the above-mentioned initial database may be stored in the form of "text data + classification / category label". Furthermore, the image data in the multimodal data corresponding to the first classification task may also be stored in the initial database; of course, the initial database may also receive and store other image data uploaded by the user through a terminal device, etc.

[0032] Step 202: Based on the initial database and the first few sample prompt words, the text data in the multimodal data corresponding to the second classification task is analyzed to obtain a second classification result.

[0033] On the basis of step 201, this step is intended to be performed by the above-mentioned execution subject, in the case where the initial database is constructed, in the latter stage of the initial stage, the Few-Shot Prompting (FSP) method can be used, that is, based on the sample data in the initial database and the first few-sample prompt words, to analyze the text data in the multimodal data corresponding to the second classification task received in the current stage to obtain the corresponding classification result. Among them, the second classification task is the data analysis and processing task received in the stage of processing the text data in the multimodal data based on the first few-sample prompt words. The number of the second classification tasks is not specifically limited here, and depends on the specific situation of the specific application scenario.

[0034] Step 203: Based on the intermediate database obtained by updating the initial database with the second classification result, the text data and image data in the multimodal data corresponding to the third classification task are processed to obtain a third classification result.

[0035] On the basis of step 202, this step is intended to enable the execution subject to collect the return data after obtaining the second classification result corresponding to the second classification task, and store the classification result in the initial database to update the initial database, obtain the intermediate database, and dynamically update the intermediate database, and then analyze the text data and image data in the multimodal data corresponding to the third classification task received in the current stage based on the intermediate database in the current multimodal data processing stage to obtain the corresponding classification result. Among them, the third classification task is the data analysis and processing task received in the stage of processing both the text data and the image data in the multimodal data. The number of the third classification tasks is not specifically limited here, and depends on the specific situation of the specific application scenario.

[0036] It should be noted that the second classification task is a data analysis and processing task received in the stage of processing text data in multimodal data based on the few-sample prompt words, and the third classification task is a data analysis and processing task received in the stage of processing text data and image data in multimodal data based on the intermediate database obtained in the previous stage. Among them, the number of second classification tasks, the number of third classification tasks, the standard for completing the construction of the intermediate database, etc. are not specifically limited here, and depend on the specific circumstances of the specific application scenario. For example, when the number of sample data accumulated in the intermediate database reaches the corresponding set value or the duration of processing text data in multimodal data based on the few-sample prompt words reaches the corresponding set duration, it can be indicated that the intermediate database is completed, that is, it can also be indicated that the stage of processing multimodal data based on the few-sample prompt words is over.

[0037] In some optional implementations of the disclosed embodiments, the data in the intermediate database may be stored in the form of "text data + classification / category label". Furthermore, the image data in the multimodal data corresponding to the second classification task may also be stored in the intermediate database; of course, the initial database may also receive and store other image data uploaded by the user through a terminal device, etc.

[0038] Step 204: Update the intermediate database based on the third classification result to obtain a target database.

[0039] On the basis of step 203, this step aims to enable the execution subject to collect the return data after obtaining the third classification result corresponding to the third classification task, and store the classification result in the intermediate database to dynamically update the intermediate database and obtain the target database. The data in the target database can be stored in the form of "text data + image data + classification / category label".

[0040] It should be noted that any of the above classification tasks may correspond to a specific field, and the above multimodal data may be data generated during the specific production or operation process of the field.

[0041] In a specific application scenario, take the multimodal data as text and image (or video) data information generated in the medical process as an example. Figure 4 As shown, the user can input multimodal data through the user terminal 401, including text input (which can correspond to the text data in the multimodal data corresponding to the above-mentioned first classification task, the second classification task and the third classification task respectively) and image input (which can correspond to the image data in the multimodal data corresponding to the above-mentioned first classification task, the second classification task and the third classification task respectively). In stage 4-1, there is no data required to solve the corresponding classification task (corresponding to the first classification task mentioned above). In this stage, we can focus on the medical text data analysis task. Specifically, we can use ZSP technology. In process 4-1-1, the data corresponding to the text input and the result of splicing the ZSP prompt word template are input into the large language model, and the text is analyzed with the help of prior knowledge and prompts. In this way, the intention contained in the text can be identified and analyzed without the traditional model training, gradient descent, and parameter adjustment process, supporting cold start from scratch. Further, in process 4-1-2, the text classification result output by the large language model can be adjusted, and the accurate classification result can be stored in the database 402 of stage 4-1 (corresponding to the above-mentioned initial database), that is, the reflux data is collected, wherein the data can be stored in the form of "text data n / / category label", where the value of n can be a positive integer, representing the number of the text data.

[0042] In one example, the classification task may be: classify the intent contained in the following content into the "image interpretation", "diagnosis triage", "case understanding" or "unknown" module, and the specific text input may be "radiology, whether this patient's bones are abnormal", and the corresponding classification result is that the specific text input is classified into "image interpretation". In another example, the specific text input may be "which department should I go to for the removal of black hairy nevus", and the corresponding classification result is that the specific text input is classified into "image interpretation", which is obviously an incorrect recognition result. At this time, it needs to be adjusted and its corresponding classification result is adjusted to "diagnosis triage".

[0043] As data collection progresses, the large-scale language model will better understand the semantics contained in the text and can resolve possible misclassifications in stage 4-1. Then, in stage 4-2, the FSP method can be used to analyze and process the text input of this stage, and the classification results obtained can be updated to database 402 (see process 4-2-3). Further, in stage 4-3, based on database 402 updated in stage 2 (corresponding to the above-mentioned intermediate database), the text input and image input of this stage can be analyzed and processed, and the classification results obtained can be updated to database 402 (see process 4-3-11), to obtain an updated database 402 (corresponding to the above-mentioned target database).

[0044] The multimodal data processing method provided by the embodiments of the present disclosure flexibly connects the stage of analyzing and processing the text data in the received multimodal data based on the zero-sample prompting method, the stage of analyzing and processing the text data in the received multimodal data based on the few-sample prompting method, and the stage of analyzing and processing both the text data and the image data in the received multimodal data. Without the need to collect annotated data in advance and without explicitly training the model, especially without explicitly training the model for data of different modalities, the multimodal data processing solution can be quickly deployed in the business environment. By continuously collecting data during business operations and gradually improving data analysis and processing performance, costs can be saved while achieving effective classification processing of multimodal data.

[0045] Please refer to Figure 3 , Figure 3 A flowchart of another multimodal data processing method provided by an embodiment of the present disclosure, wherein process 300 includes the following steps:

[0046] Step 301: Based on the zero-sample prompt word, the text data in the multimodal data corresponding to the first classification task is analyzed to obtain a first classification result, and an initial database is constructed based on the first classification result.

[0047] Step 302: Based on the initial database and the first few sample prompt words, the text data in the multimodal data corresponding to the second classification task is analyzed to obtain a second classification result.

[0048] Step 303: Determine a second target text vector based on the intermediate database obtained by updating the initial database using the second classification result and the text data in the multimodal data corresponding to the third classification task.

[0049] On the basis of step 302, this step aims to enable the above-mentioned execution subject to collect the backflow data after obtaining the second classification result corresponding to the second classification task, store the classification result in the initial database, so as to update the initial database and obtain the intermediate database. Then, in the current multimodal data processing stage, the sample data in the intermediate database and the text data in the multimodal data corresponding to the third classification task received in the current stage can be combined to obtain the corresponding text vector.

[0050] Step 304: extracting sample image description prompt words from the sample image description data in the intermediate database.

[0051] Step 305: Based on the second target text vector and the sample image description prompt words, determine a plurality of candidate text features corresponding to the text data in the multimodal data corresponding to the third classification task.

[0052] On the basis of steps 303 and 304, this step is intended to enable the above-mentioned execution subject to perform feature extraction on the text data in the multimodal data corresponding to the third classification task received in the current multimodal data processing stage based on the above-mentioned second target text vector and the sample image description prompt words extracted from the intermediate database, and obtain multiple candidate text features corresponding to the text data in the multimodal data corresponding to the third classification task.

[0053] Step 306: Obtain image features to be analyzed corresponding to the image data in the multimodal data corresponding to the third classification task.

[0054] Step 307: Determine a third classification result based on the second target text vector, the plurality of candidate text features and the image features to be analyzed.

[0055] On the basis of steps 305 and 306, this step aims to enable the above-mentioned execution subject to achieve the classification result of the multimodal data corresponding to the third classification task based on obtaining the above-mentioned second target text vector, multiple candidate text features corresponding to the text data in the multimodal data corresponding to the third classification task, and the image features to be analyzed corresponding to the image data in the multimodal data corresponding to the third classification task.

[0056] Step 308: Update the intermediate database based on the third classification result to obtain a target database.

[0057] The above steps 301-302 and 308 are similar to Figure 2 The steps 201-202 and 204 shown are consistent. For the same contents, please refer to the corresponding parts of the previous embodiment and will not be repeated here.

[0058] The multimodal data processing method provided by the embodiment of the present disclosure flexibly connects the stage of analyzing and processing the text data in the received multimodal data based on the zero-sample prompt method, the stage of analyzing and processing the text data in the received multimodal data based on the few-sample prompt method, and the stage of analyzing and processing both the text data and the image data in the received multimodal data, and in the stage of analyzing and processing the text data and the image data in the multimodal data, fully utilizes the data accumulated in the database in the previous stage to analyze and process the text data and the image data in the multimodal data corresponding to the third classification task received in the current stage to obtain the classification result for the multimodal data. In this way, without the need to collect annotated data in advance and without explicitly training the model, especially without explicitly training the model for data of different modalities, the multimodal data processing solution can be quickly deployed in the business environment, and by continuously collecting data in business operations and gradually improving the data analysis and processing performance, while saving costs, effective processing of multimodal data is achieved.

[0059] In some optional implementations of the embodiments of the present disclosure, the above step 303 can be specifically executed as follows: for the second candidate category label in the intermediate database, based on the sample text data in the intermediate database, determine the second candidate text vector corresponding to the second candidate category label; encode the text data in the multimodal data corresponding to the third classification task into a second text vector to be analyzed; based on the second text vector to be analyzed, determine the second target text vector from the second candidate text vector; wherein the similarity between the second target text vector and the second text vector to be analyzed is greater than the second similarity threshold.

[0060] In this embodiment, the corresponding second candidate text vector can be determined for each candidate category label in the intermediate database, and the text data in the multimodal data corresponding to the third classification task received in the current stage can be encoded and converted into the corresponding second text vector to be analyzed. Then, based on the similarity between the second text vector to be analyzed and each second candidate text vector, the second target text vector having a similarity value greater than the second similarity threshold can be screened out to achieve accurate analysis of the multimodal data corresponding to the third classification task.

[0061] In some optional implementations of the embodiments of the present disclosure, the similarity between vectors may be calculated using cosine similarity, Pearson similarity, or other calculation methods.

[0062] It should be noted that the value of the second similarity threshold may depend on the specific situation of the specific application scenario and is not specifically limited here. The above-mentioned tools for encoding the text string corresponding to the text data into a vector may include but are not limited to word embedding, the open source CLIP text encoder (Contrastive Language-Image Pretraining Text Encoder), the Embedding interface of Generative Pre-Trained (GPT), etc.

[0063] In any of the above embodiments of the present disclosure, the above step of determining the second candidate text vector corresponding to the second candidate category label in the intermediate database based on the sample text data in the intermediate database can be implemented by different specific implementation methods, which can be flexibly selected according to the specific situation:

[0064] Implementation method 1: For the second candidate category label, obtain the second sample text data corresponding to the second candidate category label from the intermediate database, and encode the second sample text data into a second candidate text vector. In this implementation method, the corresponding sample text data can be first screened from the intermediate database for each second candidate category label, and then converted into the corresponding candidate text vector, so that the efficiency of text vector conversion can be ensured. Among them, the second sample text data can be encoded into a second candidate text vector using a tool that encodes the text string corresponding to the text data into a vector.

[0065] Implementation method 2: Encode the sample text data in the intermediate database into corresponding sample text vectors, and obtain the second candidate text vector corresponding to the second candidate category label from all the sample text vectors corresponding to the intermediate database. In this implementation method, all the sample text data in the intermediate database can be pre-encoded into corresponding sample text vectors, and then the corresponding candidate text vectors are determined for each second candidate category label, so that the efficiency of obtaining the candidate text vectors can be ensured. Among them, a tool that encodes the text string corresponding to the text data into a vector can be used to encode the sample text data in the intermediate database into the corresponding sample text vector.

[0066] Continuing with the above specific application scenario as an example, Figure 4As shown, in the process 4-3-1 of stage 4-3, based on the sample data stored in the database 402 in stages 4-1 and 4-2, screening is performed. Specifically, for each of the C2 categories (corresponding to the second candidate category labels) currently in the database 402, K2 pieces of known sample data belonging to the category can be screened respectively. Wherein, K2 and C2 are both positive integers.

[0067] In this embodiment, all text data in the database can be encoded in advance to obtain corresponding text vectors. The encoding method can use Word Embedding parameters, open source CLIP Text Encoder, GPT's Embedding interface or any other tool that can encode text strings into vectors. Then, the text vector to be analyzed corresponding to the current text input, that is, the text to be analyzed, is calculated, and the similarity between the text vector to be analyzed and each vector corresponding to all text data in the database (corresponding to the above-mentioned second candidate text vector) is calculated, and the K2 vectors with the highest similarity are selected for each category (corresponding to the above-mentioned K2 known sample data belonging to the category, and corresponding to the above-mentioned second target text vector).

[0068] In some optional implementations of the embodiments of the present disclosure, the above-mentioned step 305 is specifically performed as follows: the result data obtained by concatenating the sample text data and the sample image description prompt words corresponding to the second target text vector is input into the large language model for analysis to output multiple candidate image description data; the multiple candidate image description data are encoded to obtain multiple candidate text features.

[0069] In this embodiment, by analyzing the second target text vector corresponding to the text data in the multimodal data corresponding to the third classification task received at the current stage and the sample image description prompt words extracted based on the intermediate database with the help of a large language model, a plurality of candidate image description data corresponding to the text data in the multimodal data corresponding to the third classification task received at the current stage can be extracted, and then the plurality of candidate image description data can be encoded to obtain a plurality of candidate text features for accurately analyzing the multimodal data corresponding to the third classification task, thereby realizing the amplification of the text data in the multimodal data received at the current stage.

[0070] Continuing with the above specific application scenario as an example, Figure 4As shown, in process 4-3-2 of stage 4-3, based on the image data stored in database 402 in stages 4-1 and 4-2, the image description prompt words are summarized and extracted, and the text data corresponding to the vectors with high similarity screened out in the above process 4-3-1 are spliced ​​and input into the large language model, and the corresponding N image description data (corresponding to the above multiple candidate image description data) are output (see process 4-3-4), so as to realize the amplification of the text input, and further perform CLIP text encoding processing on it in process 4-3-5, and output N text features (corresponding to the above multiple candidate text features) (see process 4-3-6). Wherein, the value of N is a positive integer.

[0071] In some optional implementations of the embodiments of the present disclosure, the above-mentioned step 307 is specifically performed as follows: determining the target text features based on the image features to be analyzed and multiple candidate text features; wherein the similarity between the target text features and the image features to be analyzed is greater than a third similarity threshold; the sample text data corresponding to the second target text vector and the second few sample prompt words are concatenated to obtain the result data and the target text features, and the result data are input into the large language model for analysis to output a third classification result.

[0072] In this embodiment, similarity calculation is performed on the image features to be analyzed corresponding to the image data in the multimodal data corresponding to the third classification task and each of the multiple candidate text features obtained by screening the text data in the multimodal data corresponding to the third classification task, so as to screen out the target text features that are highly similar to the above-mentioned image features to be analyzed, and then the concatenation result of the second target text vector and the second few sample prompt word corresponding to the text data in the multimodal data corresponding to the third classification task received at the current stage can be combined, and with the help of the large language model, accurate analysis of the multimodal data corresponding to the third classification task can be achieved.

[0073] It should be noted that the value of the third similarity threshold may be determined according to the specific situation of the specific application scenario and is not specifically limited here.

[0074] Continuing with the above specific application scenario as an example, Figure 4As shown, in process 4-3-7 of stage 4-3, the image input in the multimodal data is subjected to CLIP image encoding processing, and the corresponding image features are output (see process 4-3-8); further in process 4-3-9, the similarity between the image feature (corresponding to the above-mentioned image feature to be analyzed) and each of the N text features can be calculated based on the following formula (1), and K3 image descriptions that best match the image input are output, that is, K3 text features are selected from the N text features (see process 4-3-10), and further, the text data corresponding to the above-selected K2*C2 vectors can be concatenated with the result of splicing the FSP prompt word template 2 and the K3 text features, and input into the large language model for analysis (see process 4-3-11).

[0075] argmax c (f · g c ) (1)

[0076] Among them, argmax represents the identifier of the category of the text feature that best matches the image feature (for example, it can be the ID of the category label), f represents the image feature to be analyzed, and g c Represents the filtered text features.

[0077] In some optional implementations of the multimodal data processing method in any of the above embodiments of the present disclosure, the above step 201 or step 301 may be specifically performed as follows:

[0078] For the first candidate category label in the initial database, based on the sample text data in the initial database, determine the first candidate text vector corresponding to the first candidate category label; encode the text data in the multimodal data corresponding to the second classification task into a first text vector to be analyzed; according to the first text vector to be analyzed, determine the first target text vector from the first candidate text vector; wherein the similarity between the first target text vector and the first text vector to be analyzed is greater than a first similarity threshold; the sample text data corresponding to the first target text vector and the first few sample prompt word are concatenated to obtain the result data, and input them into the large language model for analysis to output the second classification result.

[0079] Among them, the above-mentioned first candidate category labels may include part or all of the category labels already stored in the above-mentioned constructed initial database, and the number of first candidate category labels may be multiple, the number of first candidate text vectors screened out for each first candidate category label may also be multiple, and the number of first target text vectors determined from the first candidate text vectors may also be multiple.

[0080] In this embodiment, the corresponding first candidate text vector can be determined for each candidate category label in the initial database, and the text data in the multimodal data corresponding to the second classification task received in the current stage can be encoded and converted into the corresponding first text vector to be analyzed. Then, based on the similarity between the first text vector to be analyzed and each first candidate text vector, the first target text vector with a similarity value greater than the first similarity threshold can be screened out, and the result of splicing the screened first target text vector and the first few sample prompt word can be efficiently analyzed with the help of the large language model to obtain the corresponding classification result.

[0081] In any of the above embodiments of the present disclosure, the above step of determining the first candidate text vector corresponding to the first candidate category label based on the sample text data in the initial database for the first candidate category label in the initial database can be implemented by different specific implementation methods, which can be flexibly selected according to the specific situation:

[0082] Implementation method 1: For the first candidate category label, obtain the first sample text data corresponding to the first candidate category label from the initial database, and encode the first sample text data into a first candidate text vector. In this implementation method, the corresponding sample text data can be first screened from the initial database for each first candidate category label, and then converted into the corresponding candidate text vector, so that the efficiency of text vector conversion can be ensured. Among them, a tool that encodes the text string corresponding to the text data into a vector can be used to encode the first sample text data into a first candidate text vector.

[0083] Implementation method 2: Encode the sample text data in the initial database into corresponding sample text vectors, and obtain the first candidate text vector corresponding to the first candidate category label from all the sample text vectors corresponding to the initial database. In this implementation method, all the sample text data in the initial database can be pre-encoded into corresponding sample text vectors, and then the corresponding candidate text vectors are determined for each first candidate category label, so that the efficiency of obtaining the candidate text vectors can be ensured. Among them, a tool that encodes the text string corresponding to the text data into a vector can be used to encode the sample text data in the initial database into the corresponding sample text vector.

[0084] Among them, the few-shot prompt method, which can be used for various natural language processing tasks (including question answering, text classification, text generation, etc.), is a technology that guides large language models to reason and make decisions in the case of few samples. Unlike zero-shot prompt technology, few-shot prompt technology requires a small number of sample examples to provide task information and guide the decision-making process of the model. In the few-shot prompt technology, a small number of examples are used as prompts to help the model understand the requirements and goals of the task. These examples usually contain relevant questions, example answers or texts to guide the large language model to generate the correct answer or solution. In this way, the few-shot prompt technology can improve the performance and generalization ability of the large language model in the case of few samples, so that by providing a small number of examples to guide the large language model to reason and make decisions, it can reduce the dependence on a large amount of training data.

[0085] It should be noted that, in order to ensure the accuracy of the classification results of the text data in the multimodal data stored in the intermediate database, it can be calibrated before storing it in the database. The value of the first similarity threshold can be determined according to the specific situation of the specific application scenario and is not specifically limited here.

[0086] Continuing with the above specific application scenario as an example, Figure 4 As shown, in the process 4-2-1 of stage 4-2, based on the sample data stored in the database 402 in stage 4-1, screening is performed. Specifically, for each category in the C1 categories currently existing in the database 402 (corresponding to the first candidate category label mentioned above), K1 known sample data belonging to the category can be screened respectively. Wherein, K1 and C1 are both positive integers.

[0087] In this embodiment, all text data in the database can be encoded in advance to obtain corresponding text vectors. The encoding method can use Word Embedding parameters, open source CLIP Text Encoder, GPT Embedding interface or any other tool that can encode text strings into vectors. Then, the text vector to be analyzed corresponding to the current text input, that is, the text to be analyzed, is calculated, and the similarity between the text vector to be analyzed and each vector corresponding to all text data in the database (corresponding to the above-mentioned first candidate text vector) is calculated, and the text data corresponding to the K1 vectors with the highest similarity (and K1 known sample data belonging to the category, corresponding to the above-mentioned first target text vector) are selected for each category, and the text data corresponding to the vectors is spliced ​​with the FSP prompt word template 1 and input into the large language model (see process 4-2-2), and the corresponding text classification result is output. Wherein, m1 is a positive integer.

[0088] In some optional implementations of the multimodal data processing method in any of the above embodiments of the present disclosure, the process of constructing an initial database based on the first classification result in the above step 201 or step 301 can be specifically performed as follows: in response to receiving an adjustment instruction for the first classification result, calibrating the first classification result; and constructing an initial database based on the calibrated first classification result.

[0089] In this embodiment, in order to ensure the accuracy of the classification results of the text data in the multimodal data stored in the initial database, it can be calibrated before storing it in the database.

[0090] In any of the above embodiments of the present disclosure, any of the above prompt words can be a single word, a phrase or a whole sentence, which can provide additional information for the large language model to better understand and process the input text. It should be noted that the above zero-sample prompt words, few-sample prompt words, etc. can be acquired or set according to the actual scenario and actual needs of the multimodal data processing application depending on the specific situation, and are not specifically limited here. For example, the above first few-sample prompt word can be obtained based on sampling of sample data in the initial database, and of course it can also be obtained by adopting other sample data; and the above second few-sample prompt word can be obtained based on sampling of sample data in the intermediate database, and of course it can also be obtained by adopting other sample data; all are not specifically limited here.

[0091] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a multimodal data processing device. Figure 2-3 Corresponding to the method embodiment shown, the device can be specifically applied to various servers.

[0092] like Figure 5As shown, the multimodal data processing device 500 of the embodiment of the present disclosure may include: a first processing module 501, a second processing module 502, a third processing module 503 and an updating module 504. The first processing module 501 is configured to analyze the text data in the multimodal data corresponding to the first classification task based on the zero-sample prompt word to obtain a first classification result, and to construct an initial database based on the first classification result; the second processing module 502 is configured to analyze the text data in the multimodal data corresponding to the second classification task based on the initial database and the first few-sample prompt word to obtain a second classification result; the third processing module 503 is configured to process the text data and image data in the multimodal data corresponding to the third classification task based on the intermediate database obtained by updating the initial database with the second classification result to obtain a third classification result; the updating module 504 is configured to update the intermediate database based on the third classification result to obtain a target database. In the embodiment of the present disclosure, the specific processing of the first processing module 501, the second processing module 502, the third processing module 503 and the updating module 504 and the technical effects thereof can be referred to in the following sections. Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0093] In some optional implementations of the embodiments of the present disclosure, the second processing module 502 may specifically include: a screening submodule 5011, an encoding submodule 5012, a first determination submodule 5013, and an analysis submodule 5014. The screening submodule 5011 is configured to determine the first candidate text vector corresponding to the first candidate category label based on the sample text data in the initial database for the first candidate category label in the initial database; the encoding submodule 5012 is configured to encode the text data in the multimodal data corresponding to the second classification task into the first text vector to be analyzed; the first determination submodule 5013 is configured to determine the first target text vector from the first candidate text vector according to the first text vector to be analyzed; wherein the similarity between the first target text vector and the first text vector to be analyzed is greater than the first similarity threshold; the analysis submodule 5014 is configured to input the result data obtained by splicing the sample text data corresponding to the first target text vector and the first few sample prompt word into the large language model for analysis and output of the second classification result.

[0094] In some optional implementations of the embodiments of the present disclosure, the above-mentioned screening submodule 5011 is specifically configured as follows: for the first candidate category label, obtaining the first sample text data corresponding to the first candidate category label from the initial database, and encoding the first sample text data into a first candidate text vector; or encoding the sample text data in the initial database into a corresponding sample text vector, and obtaining the first candidate text vector corresponding to the first candidate category label from all the sample text vectors corresponding to the initial database.

[0095] In some optional implementations of the embodiments of the present disclosure, the third processing module 503 may specifically include: a second determination submodule 5031, an extraction submodule 5032, a third determination submodule 5033, an acquisition submodule 5034, and a processing submodule 5035. The second determination submodule 5031 is configured to determine a second target text vector based on the text data in the intermediate database and the multimodal data corresponding to the third classification task; the extraction submodule 5032 is configured to extract sample image description prompt words from the sample image description data in the intermediate database; the third determination submodule 5033 is configured to determine a plurality of candidate text features corresponding to the text data in the multimodal data corresponding to the third classification task based on the second target text vector and the sample image description prompt words; the acquisition submodule 5034 is configured to acquire the image features to be analyzed corresponding to the image data in the multimodal data corresponding to the third classification task; and the processing submodule 5035 is configured to determine the third classification result based on the second target text vector, the plurality of candidate text features, and the image features to be analyzed.

[0096] In the embodiment of the present disclosure, the specific processing of the first processing module 501, the second processing module 502, the second determination submodule 5031, the extraction submodule 5032, the third determination submodule 5033, the acquisition submodule 5034, the processing submodule 5035 and the updating module 504 and the technical effects thereof can be referred to respectively. Figure 3 The relevant descriptions of steps 301 - 308 in the corresponding embodiment are not repeated here.

[0097] In some optional implementations of the embodiments of the present disclosure, the above-mentioned second determination submodule 5031 can be specifically configured as: for the second candidate category label in the intermediate database, based on the sample text data in the intermediate database, determine the second candidate text vector corresponding to the second candidate category label; encode the text data in the multimodal data corresponding to the third classification task into a second text vector to be analyzed; based on the second text vector to be analyzed, determine the second target text vector from the second candidate text vector; wherein the similarity between the second target text vector and the second text vector to be analyzed is greater than the second similarity threshold.

[0098] In some optional implementations of the embodiments of the present disclosure, the above-mentioned second determination submodule 5031 can also be specifically configured as: for the second candidate category label, obtaining the second sample text data corresponding to the second candidate category label from the intermediate database, and encoding the second sample text data into a second candidate text vector; or encoding the sample text data in the intermediate database into a corresponding sample text vector, and obtaining the second candidate text vector corresponding to the second candidate category label from all the sample text vectors corresponding to the intermediate database.

[0099] In some optional implementations of the embodiments of the present disclosure, the third determination submodule 5033 is specifically configured as follows: the result data obtained by concatenating the sample text data and the sample image description prompt words corresponding to the second target text vector is input into the large language model for analysis to output multiple candidate image description data; the multiple candidate image description data are encoded to obtain multiple candidate text features.

[0100] In some optional implementations of the embodiments of the present disclosure, the above-mentioned processing submodule 5035 is specifically configured to: determine the target text features based on the image features to be analyzed and multiple candidate text features; wherein the similarity between the target text features and the image features to be analyzed is greater than a third similarity threshold; the sample text data corresponding to the second target text vector and the second few sample prompt words are concatenated to obtain the result data and the target text features, and input them into the large language model for analysis to output a third classification result.

[0101] In some optional implementations of the embodiments of the present disclosure, the first processing module 501 is specifically configured to: calibrate the first classification result in response to receiving an adjustment instruction for the first classification result; and construct an initial database based on the calibrated first classification result.

[0102] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. It flexibly connects the stage of analyzing and processing the text data in the received multimodal data based on the zero-sample prompting method, the stage of analyzing and processing the text data in the received multimodal data based on the few-sample prompting method, and the stage of analyzing and processing both the text data and the image data in the received multimodal data. Without the need to collect annotated data in advance and without explicitly training the model, especially without explicitly training the model for data of different modalities, the multimodal data processing solution can be quickly deployed in the business environment. By continuously collecting data during business operations and gradually improving data analysis and processing performance, costs can be saved while achieving effective classification processing of multimodal data.

[0103] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, the electronic device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions are executed by the at least one processor, so that the at least one processor can implement the multimodal data processing method described in any of the above embodiments when executing. The electronic device may be a server.

[0104] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the multimodal data processing method described in any of the above embodiments when executed.

[0105] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product including a computer program, which, when executed by a processor, can implement the multimodal data processing method described in any of the above embodiments.

[0106] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0107] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0108] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as multimodal data processing methods. For example, in some embodiments, the multimodal data processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 1003 and executed by the computing unit 601, one or more steps of the multimodal data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the multimodal data processing method in any other appropriate manner (eg, by means of firmware).

[0110] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0112] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0114] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0115] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain. Among them, a cloud server, also known as a cloud computing server or a cloud host, is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS, Virtual Private Server) services.

[0116] According to the technical solution of the embodiments of the present disclosure, by flexibly connecting the stage of analyzing and processing the text data in the received multimodal data based on the zero-sample prompting method, the stage of analyzing and processing the text data in the received multimodal data based on the few-sample prompting method, and the stage of analyzing and processing both the text data and the image data in the received multimodal data, the multimodal data processing solution can be quickly deployed in the business environment without the need to collect annotated data in advance and without the need to explicitly train the model, especially without the need to explicitly train the model for data of different modalities respectively. By continuously collecting data during business operations and gradually improving data analysis and processing performance, costs can be saved while achieving effective classification processing of multimodal data.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit them here; and the terms "first", "second", "third", "fourth" and the like (if any) in the disclosure are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor do they constitute specific limitations.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multimodal data processing method, comprising: Based on the zero-sample prompt word, analyzing the text data in the multimodal data corresponding to the first classification task to obtain a first classification result, and constructing an initial database based on the first classification result; Based on the initial database and the first few sample prompt words, analyzing the text data in the multimodal data corresponding to the second classification task to obtain a second classification result; Based on the intermediate database obtained by updating the initial database using the second classification result, the text data and the image data in the multimodal data corresponding to the third classification task are processed to obtain a third classification result; The intermediate database is updated based on the third classification result to obtain a target database.

2. The method according to claim 1, wherein: The step of analyzing the text data in the multimodal data corresponding to the second classification task based on the initial database and the first few sample prompt words to obtain a second classification result includes: For a first candidate category label in the initial database, based on sample text data in the initial database, determining a first candidate text vector corresponding to the first candidate category label; Encoding text data in the multimodal data corresponding to the second classification task into a first text vector to be analyzed; Determine the first target text vector from the first candidate text vectors according to the first text vector to be analyzed; wherein the similarity between the first target text vector and the first text vector to be analyzed is greater than a first similarity threshold; The result data obtained by concatenating the sample text data corresponding to the first target text vector and the first few sample prompt words is input into the large language model for analysis to output the second classification result.

3. The method according to claim 2, wherein: The determining, for the first candidate category label in the initial database, a first candidate text vector corresponding to the first candidate category label based on sample text data in the initial database, comprises: For the first candidate category label, obtaining first sample text data corresponding to the first candidate category label from the initial database, and encoding the first sample text data into the first candidate text vector; or The sample text data in the initial database is encoded into corresponding sample text vectors, and the first candidate text vector corresponding to the first candidate category label is obtained from all sample text vectors corresponding to the initial database.

4. The method according to any one of claims 1 to 3, wherein: The method of processing the text data and the image data in the multimodal data corresponding to the third classification task to obtain a third classification result based on the intermediate database obtained by updating the initial database using the second classification result includes: Determining a second target text vector based on the text data in the intermediate database and the multimodal data corresponding to the third classification task; Extracting sample image description prompt words from the sample image description data in the intermediate database; Determining, based on the second target text vector and the sample image description prompt word, a plurality of candidate text features corresponding to the text data in the multimodal data corresponding to the third classification task; Acquire image features to be analyzed corresponding to image data in the multimodal data corresponding to the third classification task; The third classification result is determined based on the second target text vector, the multiple candidate text features and the image features to be analyzed.

5. The method according to claim 4, wherein: The determining a second target text vector based on the text data in the intermediate database and the multimodal data corresponding to the third classification task includes: For a second candidate category label in the intermediate database, based on sample text data in the intermediate database, determining a second candidate text vector corresponding to the second candidate category label; Encoding the text data in the multimodal data corresponding to the third classification task into a second text vector to be analyzed; According to the second text vector to be analyzed, the second target text vector is determined from the second candidate text vectors; wherein the similarity between the second target text vector and the second text vector to be analyzed is greater than a second similarity threshold.

6. The method according to claim 5, wherein: The determining, for the second candidate category label in the intermediate database, based on the sample text data in the intermediate database, a second candidate text vector corresponding to the second candidate category label comprises: For the second candidate category label, obtaining second sample text data corresponding to the second candidate category label from the intermediate database, and encoding the second sample text data into the second candidate text vector; or The sample text data in the intermediate database is encoded into corresponding sample text vectors, and the second candidate text vector corresponding to the second candidate category label is obtained from all sample text vectors corresponding to the intermediate database.

7. The method according to claim 4, wherein: The determining, based on the second target text vector and the sample image description prompt word, a plurality of candidate text features corresponding to the text data in the multimodal data corresponding to the third classification task includes: The result data obtained by splicing the sample text data corresponding to the second target text vector and the sample image description prompt word is input into the large language model for analysis to output multiple candidate image description data; The plurality of candidate image description data are encoded to obtain the plurality of candidate text features.

8. The method according to claim 4, wherein: The determining the third classification result based on the second target text vector, the plurality of candidate text features and the image features to be analyzed includes: Determine a target text feature according to the image feature to be analyzed and the plurality of candidate text features; wherein the similarity between the target text feature and the image feature to be analyzed is greater than a third similarity threshold; The sample text data corresponding to the second target text vector and the second few sample prompt word are concatenated to obtain result data and the target text features, and are input into the large language model for analysis to output the third classification result.

9. The method according to claim 1, wherein: The constructing an initial database based on the first classification result includes: In response to receiving an adjustment instruction for the first classification result, calibrating the first classification result; The initial database is constructed based on the calibrated first classification result.

10. A multimodal data processing device, comprising: A first processing module is configured to analyze text data in the multimodal data corresponding to the first classification task based on the zero-sample prompt word to obtain a first classification result, and to construct an initial database based on the first classification result; A second processing module is configured to analyze text data in the multimodal data corresponding to the second classification task based on the initial database and the first few sample prompt words to obtain a second classification result; A third processing module is configured to process text data and image data in the multimodal data corresponding to a third classification task to obtain a third classification result based on an intermediate database obtained by updating the initial database using the second classification result; An updating module is configured to update the intermediate database based on the third classification result to obtain a target database.

11. The device according to claim 10, wherein: The second processing module specifically includes: A screening submodule is configured to extract, for a first candidate category label in the initial database, first sample text data corresponding to the first candidate category label from the initial database, and determine a first candidate text vector corresponding to the encoding of the first sample text data; an encoding submodule, configured to encode text data in the multimodal data corresponding to the second classification task into a first text vector to be analyzed; A first determination submodule is configured to determine the first target text vector from the first candidate text vectors according to the first text vector to be analyzed; wherein the similarity between the first target text vector and the first text vector to be analyzed is greater than a first similarity threshold; The analysis submodule is configured to input the result data obtained by concatenating the sample text data corresponding to the first target text vector and the first few sample prompt words into the large language model for analysis to output the second classification result.

12. The device according to claim 11, wherein The screening submodule is specifically configured as follows: For the first candidate category label, obtaining first sample text data corresponding to the first candidate category label from the initial database, and encoding the first sample text data into the first candidate text vector; or The sample text data in the initial database is encoded into corresponding sample text vectors, and the first candidate text vector corresponding to the first candidate category label is obtained from all sample text vectors corresponding to the initial database.

13. The device according to any one of claims 10 to 12, wherein: The third processing module specifically includes: A second determination submodule is configured to determine a second target text vector based on the text data in the intermediate database and the multimodal data corresponding to the third classification task; An extraction submodule is configured to extract sample image description prompt words from the sample image description data in the intermediate database; A third determination submodule is configured to determine, based on the second target text vector and the sample image description prompt word, a plurality of candidate text features corresponding to the text data in the multimodal data corresponding to the third classification task; an acquisition submodule, configured to acquire image features to be analyzed corresponding to the image data in the multimodal data corresponding to the third classification task; The processing submodule is configured to determine the third classification result based on the second target text vector, the multiple candidate text features and the image features to be analyzed.

14. The device according to claim 13, wherein: The second determining submodule is specifically configured as follows: For a second candidate category label in the intermediate database, extract second sample text data corresponding to the second candidate category label from the intermediate database, and determine a second candidate text vector corresponding to the encoding of the second sample text data; Encoding the text data in the multimodal data corresponding to the third classification task into a second text vector to be analyzed; According to the second text vector to be analyzed, the second target text vector is determined from the second candidate text vectors; wherein the similarity between the second target text vector and the second text vector to be analyzed is greater than a second similarity threshold.

15. The device according to claim 14, wherein: The second determining submodule is further specifically configured as follows: For the second candidate category label, obtaining second sample text data corresponding to the second candidate category label from the intermediate database, and encoding the second sample text data into the second candidate text vector; or The sample text data in the intermediate database is encoded into corresponding sample text vectors, and the second candidate text vector corresponding to the second candidate category label is obtained from all sample text vectors corresponding to the intermediate database.

16. The device according to claim 13, wherein: The third determining submodule is specifically configured as follows: The result data obtained by splicing the sample text data corresponding to the second target text vector and the sample image description prompt word is input into the large language model for analysis to output multiple candidate image description data; The plurality of candidate image description data are encoded to obtain the plurality of candidate text features.

17. The device according to claim 13, wherein: The processing submodule is specifically configured as follows: Determine a target text feature according to the image feature to be analyzed and the plurality of candidate text features; wherein the similarity between the target text feature and the image feature to be analyzed is greater than a third similarity threshold; The sample text data corresponding to the second target text vector and the second few sample prompt word are concatenated to obtain result data and the target text features, and are input into the large language model for analysis to output the third classification result.

18. The device according to claim 10, wherein: The first processing module is specifically configured as follows: In response to receiving an adjustment instruction for the first classification result, calibrating the first classification result; The initial database is constructed based on the calibrated first classification result.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal data processing method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the multimodal data processing method according to any one of claims 1 to 9.

21. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the multimodal data processing method according to any one of claims 1 to 9 are implemented.

Images