A method for processing multimodal data input and related equipment

Abstract

The present application discloses a method for processing multimodal data input and related equipment, and belongs to the field of data processing. The present application extracts features from multimodal input data, inputs the multimodal input features into an intent recognition model, obtains the user's operation intention, and obtains a target intelligent entity that is adapted to the user's operation intention from a preset intelligent entity library, wherein the intelligent entity library pre-stores a number of intelligent entities generated based on large language model training, and uses the target intelligent entity to generate a service call instruction corresponding to the user operation based on the multimodal input features, responds to the service call instruction, and generates a data processing result. The present application integrates a large language model into an existing system at a relatively low cost, avoids the problem that the rigid system interface design is difficult to cope with multimodal data input, improves system flexibility, reduces the cost of interface modification or redesign without changing the existing architecture, and reduces technical debt accumulation and system maintenance costs.

Description

Technical Field

[0001] The present application belongs to the field of data processing, and specifically relates to a method for processing multimodal data input and related equipment. Background Art

[0002] Multimodal user requests refer to input requests made by users through a variety of different methods and channels, with a wide range of data sources and in various forms. In today's context of expanding data acquisition channels, multimodal user requests include not only traditional structured text requests, but also unstructured data such as user text evaluations, video images, and sound and audio. Due to the complexity and diversity of needs, a single request input method is often difficult to fully reflect user needs, opinions, and behaviors. Multimodal user requests can provide richer and more comprehensive information from multiple angles and in different forms, helping the system to better understand user intentions and analyze user behavior.

[0003] In response to multimodal user requests, in current computer systems, various types of input data often interact with the Application Programming Interface (API) through preset formats and interfaces to achieve corresponding functions. The general workflow of existing computer software is: among the available services provided by the server, the user selects a specified service on the client side, and sends an API request in a specified format to the server in accordance with its data format requirements. After receiving the request, the server immediately starts running the specified service. However, the API interface design within this processing system is usually based on fixed and pre-set requirements and specific data formats. When faced with business model updates caused by the continuous emergence of new user requirements, the API interface needs to be modified or even redesigned, which will cause a substantial increase in system development and maintenance costs, and technical debt will gradually accumulate, and it will pose a potential threat to the stability of the system. In addition, this fixed-mode internal call process cannot timely understand the implicit user requirements contained in multimodal inputs, and lacks the ability to quickly adapt to the market. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a method, apparatus, computer device and storage medium for processing multimodal data input to solve the technical problems of poor adaptability, lack of flexibility and high development and maintenance costs in existing multimodal user request processing.

[0005] In order to solve the above technical problems, the present application provides a method for processing multimodal data input, which adopts the following technical solution:

[0006] A method for processing multimodal data input, comprising:

[0007] Receiving a user input instruction and obtaining user input data, wherein the user input data is multimodal input data;

[0008] Perform feature extraction on multimodal input data to obtain multimodal input features;

[0009] Input the multimodal input features into the pre-trained intent recognition model to obtain the user's operation intention;

[0010] Acquire a target intelligent entity adapted to the user's operation intention from a preset intelligent entity library, wherein the intelligent entity library pre-stores a number of intelligent entities generated based on large language model training;

[0011] Based on the multimodal input features, the target intelligent entity is used to generate the service call instructions corresponding to the user operation;

[0012] Respond to service call instructions, generate data processing results, and output data processing results.

[0013] Furthermore, the multimodal input data includes voice data, text data and image data, and the step of extracting features from the multimodal input data to obtain multimodal input features specifically includes:

[0014] Performing audio-to-text processing on the voice data in the multimodal input data to obtain the voice text;

[0015] Perform text feature extraction on the speech text to obtain speech text features;

[0016] Performing text feature extraction on text data in the multimodal input data to obtain text data features;

[0017] Performing image feature extraction on image data in the multimodal input data to obtain image data features;

[0018] The speech text features, text data features and image data features are fused to obtain multimodal input features.

[0019] Furthermore, the multimodal input data also includes video data, and the step of extracting features from the multimodal input data to obtain multimodal input features specifically includes:

[0020] Parse the video data to obtain the video frame sequence and video voice data;

[0021] Identify key frame images from the video frame sequence, and extract image features from the key frame images to obtain key frame image features;

[0022] Perform audio-to-text processing on the video and voice data to obtain the video and voice text;

[0023] Extracting text features from the video speech text to obtain video speech text features;

[0024] The speech-to-text features, text data features, image data features, video-to-speech-to-text and video-to-speech-to-text features are fused to obtain multimodal input features.

[0025] Furthermore, before the step of acquiring the target intelligent entity adapted to the user's operation intention from the preset intelligent entity library, the method further includes:

[0026] Acquire pre-collected historical input data, wherein the historical input data is historical multimodal input data;

[0027] Obtain historical user intentions and historical service call instructions corresponding to historical input data;

[0028] A training data set is constructed based on historical multimodal input data and historical service call instructions, and a large language model is used to perform feature learning based on the training data set to generate a number of intelligent entities, wherein each type of historical multimodal input data corresponds to one intelligent entity;

[0029] For each type of historical multimodal input data, there is one intelligent entity, which is annotated with the corresponding historical user intent;

[0030] Several fully labeled intelligent entities are stored in a preset database to obtain an intelligent entity library.

[0031] Furthermore, a training data set is constructed based on historical multimodal input data and historical service call instructions, and a large language model is made to perform feature learning based on the training data set to generate a number of intelligent entities, specifically including:

[0032] For each type of historical multimodal input data, feature extraction is performed on the historical multimodal input data and the historical service call instruction to obtain historical multimodal input features and historical service call instruction features;

[0033] Performing feature association on historical multimodal input features and historical service call instruction features, and enabling the large language model to perform feature learning based on the feature association results, and outputting a first predicted service call instruction;

[0034] Calculating a first prediction error of the large language model based on the first prediction service call instruction and the history service call instruction;

[0035] Iteratively updating the large language model based on the first prediction error until the model is fitted, thereby obtaining a pre-trained first large language model;

[0036] Each of the first language models that have completed pre-training is encapsulated to obtain several intelligent entities.

[0037] Furthermore, the step of acquiring a target intelligent entity adapted to the user's operation intention from a preset intelligent entity library specifically includes:

[0038] Perform feature vector conversion on the user's operation intention to obtain the intention feature vector;

[0039] Calculate the similarity between the intent feature vector and the intent label of each smart entity in the smart entity library to obtain a similarity sequence, where each smart entity corresponds to an intent label, and the intent label is generated when the smart entity is labeled using historical user intent;

[0040] Sorting the similarity values ​​in the similarity sequence and determining the maximum similarity value;

[0041] The intelligent entity corresponding to the maximum similarity value is identified, and the intelligent entity corresponding to the maximum similarity value is used as the target intelligent entity.

[0042] Furthermore, after the steps of sorting the similarity values ​​in the similarity sequence and determining the maximum similarity value, the method further includes:

[0043] Determine whether the maximum similarity value is less than a preset similarity threshold;

[0044] If the maximum similarity value is less than a preset similarity threshold, the multimodal input feature is input into the first language model, and an output second prediction service call instruction is obtained;

[0045] Obtaining the standard service calling instruction uploaded by the user, and calculating the second prediction error of the large language model based on the second prediction service calling instruction and the standard service calling instruction;

[0046] Iteratively updating the first largest language model based on the second prediction error until the model is fitted, thereby obtaining a second largest language model;

[0047] The second largest language model is encapsulated to obtain new intelligent entities, and the intelligent entity library is updated based on the new intelligent entities.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a multimodal data input processing device, which adopts the following technical solution:

[0049] A processing device for multimodal data input, comprising:

[0050] A data acquisition module, used to receive user input instructions and acquire user input data, wherein the user input data is multimodal input data;

[0051] A feature extraction module is used to extract features from multimodal input data to obtain multimodal input features;

[0052] The intention recognition module is used to input multimodal input features into the pre-trained intention recognition model to obtain the user's operation intention;

[0053] An intelligent entity adaptation module is used to obtain a target intelligent entity adapted to the user's operation intention from a preset intelligent entity library, wherein the intelligent entity library pre-stores a number of intelligent entities generated based on large language model training;

[0054] An instruction generation module is used to generate service call instructions corresponding to user operations using a target intelligent entity based on multimodal input features;

[0055] The service response module is used to respond to service call instructions, generate data processing results, and output data processing results.

[0056] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0057] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for processing multimodal data input as described in any one of the above items are implemented.

[0058] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0059] A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the method for processing multimodal data input as described in any one of the above.

[0060] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0061] First, by leveraging the ability of large language models to cope with multimodal data, the system can pre-process, analyze, extract, and integrate user multimodal demand data, breaking the limitations of existing technologies that are difficult to cope with new demands and multimodal data due to fixed presets and specific formats, and enhancing the flexibility of the system.

[0062] Secondly, by using a large language model to complete tasks such as splitting user intent, intelligent entity mapping, and generating service call instructions, the manpower, time, and resource investment in interface modification or redesign is greatly reduced, solving the problem of high maintenance costs.

[0063] Third, mapping user multimodal requests to intelligent entities trained based on large language models can promptly process and respond to changing multimodal business needs, making up for the shortcomings of existing technologies that are unable to understand implicit needs and adapt to changes, and improving the system's adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0066] Figure 2 A flowchart showing an embodiment of a method for processing multimodal data input according to the present application is shown;

[0067] Figure 3 A schematic structural diagram of an embodiment of a multimodal data input processing device according to the present application is shown;

[0068] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0070] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0071] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0072] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0073] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0074] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0075] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0076] It should be noted that the method for processing multimodal data input provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the processing device for multimodal data input is generally arranged in the server / terminal device.

[0077] It should be understood that Figure 1 The numbers of terminal devices, networks and servers in the embodiment are merely illustrative. The above system may have any number of terminal devices, networks and servers according to implementation requirements.

[0078] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for processing multimodal data input according to the present application. The method for processing multimodal data input comprises the following steps:

[0079] S201, receiving a user input instruction and acquiring user input data, wherein the user input data is multimodal input data.

[0080] Specifically, the system first receives input instructions and data from the user, where the user input data is multimodal input data, which refers to data containing multiple forms of information, such as text, voice, image, video, etc. The system is not limited to receiving data in a single mode, but supports users to input in various forms to more comprehensively express their needs or intentions. Through the integration of multimodal data, the system can be made richer in terms of data sources. For example, a user may speak certain instructions through voice, upload a picture to supplement the explanation, or embed an image in the text input to express more complex instruction requirements. The implementation of this process requires the system to have strong data reception and recognition capabilities, be able to identify data in different modes, and correctly pass it to subsequent steps to ensure that various types of input information can be effectively integrated together.

[0081] S202, extracting features from the multimodal input data to obtain multimodal input features.

[0082] Specifically, the system will conduct an in-depth analysis of the received multimodal input data and extract key information features from it so that it can reflect the user's true intention. For example, text input involves processing such as keyword extraction, sentiment analysis, and syntactic parsing; image input may extract content features through methods such as target detection and image classification; voice input may include voice recognition and extraction of audio sentiment features. The purpose of multimodal feature extraction is to convert various types of data in different modalities into a unified feature representation so that consistency can be maintained during processing. This feature extraction process requires the comprehensive use of natural language processing, image processing, audio signal processing and other technologies, and the fusion of data in different modalities to extract the features that are most helpful for judging user intentions. For example, by fusing text keywords, object labels in images, voice intonation, etc., the system can obtain a more comprehensive feature set.

[0083] For example, for the user's <voice message> "Report this ticket" + <invoice photo>, for voice messages, the system can use OpenAI's Whisper to automatically recognize the user's voice and convert it into text, and then pre-process it with the multimodal model GPT-4o, and then use the embedding model to obtain vectorized voice features; for invoice images, the system also uses GPT-4o to describe the input image with text and vectorize it, and then uses OpenAI's CLIP model to process the obtained image feature vector. Through the above steps, the user's multimodal input can be processed and analyzed, and the semantic understanding and image analysis capabilities of LLM are fully utilized to enhance robustness.

[0084] S203, input the multimodal input features into the pre-trained intention recognition model to obtain the user operation intention.

[0085] Specifically, the system inputs the multimodal features after feature extraction into a pre-trained intent recognition model. The intent recognition model is a deep learning model that is usually trained on a large-scale data set so that it can accurately recognize user intent. In practical applications, this model can be a complex neural network model based on the LLM model (Large Language Model l), which can understand and infer the potential intent of the user input. Through the analysis of the input features, the model will comprehensively process various modal information to derive one or more possible user intentions. For example, the user inputs a picture and a descriptive text. The intent recognition model will fuse the image features and text features to determine whether the user wants to obtain information about an object, wants to perform an operation, or express an emotion. In this way, the system can understand the user's true intention more comprehensively.

[0086] S204, obtaining a target intelligent entity adapted to the user's operation intention from a preset intelligent entity library, wherein the intelligent entity library pre-stores a plurality of intelligent entities generated based on large language model training.

[0087] In this solution, LLM intelligent entities refer to intelligent entities built based on large language model technology that can perform specific tasks or process specific types of data. These intelligent entities can understand and generate human language through deep learning and natural language processing technology, and can reason, judge and perform corresponding operations based on the input information.

[0088] There are multiple pre-set LLM intelligent entities in the system that have been fine-tuned for specific tasks. Each of these intelligent entities is responsible for processing different types of input data or performing different tasks. For example, some intelligent entities specialize in handling customer service requests, while others focus on tasks such as data analysis or text generation.

[0089] One of the routing LLM intelligent entities is responsible for determining whether the input falls within the scope of the system design based on the intent of the input data, that is, determining the target intelligent entity that is adapted to the user's operation intention. The routing LLM intelligent entity needs to decide to forward the input data to a specific LLM intelligent entity in the preset intelligent entity library for processing. This process is similar to routing selection in the network, ensuring that the data can accurately reach the correct processing node.

[0090] Specifically, the system selects the target intelligent entity that matches the user's operation intention from the intelligent entity library. The intelligent entity library stores many different intelligent entities, which are generated through training based on a large language model. Each intelligent entity has its own specialized processing capabilities and functions to meet the needs of different user operation intentions. For example, the intelligent entity library contains an intelligent assistant specifically responsible for scheduling, an intelligent question-and-answer system responsible for information retrieval, or an intelligent document generator responsible for generating documents in a specific format. When the intent recognition model recognizes that the user wants to schedule a schedule, the system will automatically select a schedule assistant entity from the intelligent entity library. Each intelligent entity contains predefined processing logic and rich background knowledge, thereby ensuring that it can quickly enter the execution phase after the user's intention is determined, providing users with professional services.

[0091] In the pre-preparation stage, the system has pre-configured a variety of LLM intelligent entities that have been fine-tuned for targeted tasks, and the description documents and other materials of these LLM intelligent entities (including the functions supported by each sub-agent) have been vectorized and indexed with the help of an embedded model, and integrated into the intelligent entity library; in the subsequent routing judgment stage, the routing LLM intelligent entity uses the obtained user operation intention, the Retrieval Augmented Generation (RAG) technology, and the semantic information and contextual information of the multimodal input data to match and correspond in the intelligent entity library to determine the intelligent entity corresponding to the input data, and the intelligent entity performs the processing task for the input data.

[0092] Retrieval-augmented generation technology improves the accuracy and reliability of AI systems in answering natural language questions by combining traditional retrieval-based question-answering systems with natural language generation-based technologies. The RAG model uses a retrieval module to obtain relevant data from an external knowledge base, and then combines this information with a generation module (such as large language models such as GPT and BERT) to synthesize this information into a coherent and accurate response.

[0093] S205, based on the multimodal input features, using the target intelligent entity to generate a service call instruction corresponding to the user operation.

[0094] Specifically, the system will use the target intelligent entity selected previously and generate specific service call instructions based on the multimodal input features. The target intelligent entity decides how to respond and generate instructions based on the user's intention and the detailed characteristics of the input. These service call instructions can be requests to external systems, devices, or service interfaces, instructing them to perform specific tasks. For example, if the user input instruction is a financial data risk prediction, the system will select the corresponding financial-risk prediction intelligent entity after intent recognition, and then generate specific service call instructions based on the multimodal input features of the user input, such as API requests to complete the financial data risk prediction. The generation of service call instructions will adjust parameters based on the details of the multimodal input to ensure that the generated instructions can accurately reflect user needs, thereby providing users with high-quality personalized services.

[0095] For example, for the input data of <voice message> "Report this ticket" + <invoice photo> in the previous text, the system determines that the user needs to complete the invoice reimbursement business based on the intent recognition result, and the routing LLM intelligent entity determines that the target intelligent entity is the financial service LLM intelligent entity based on the intent recognition result. Then, combined with the user's personal financial information, travel application and other information obtained, the financial service LLM intelligent entity is used to select the invoice reimbursement business within the system, and the corresponding reimbursement business API call instructions are generated based on the financial service LLM intelligent entity. At the specific implementation level: First, use OpenAI's function call (Function Calling) to connect the model with external tools, and obtain additional information required to execute the target task from the system server, such as current user information; then, the financial service LLM intelligent entity uses the GPT-4o small model to select the target task based on the input; finally, use the GPT-4 large model to aggregate data from various places, and generate API call instructions according to the requirements of the target task API paradigm OpenAPISchema.

[0096] S206, responding to the service call instruction, generating a data processing result, and outputting the data processing result.

[0097] Specifically, the system will respond to the generated service call instructions, perform corresponding operations, and generate data processing results. This result can be data feedback obtained from an external system, or output generated within the system according to user needs. For example, assuming that the service call instruction requests financial data risk prediction, the system will process the input financial data, output the financial data risk prediction results, and convert them into intuitive feedback forms, such as financial risk warning diagrams or related text descriptions. After the processing results are generated, the system will output them to the user interface to provide the user with the final service or feedback. In this way, the system can completely close the service loop from user input to result output, while ensuring the accuracy and timeliness of the response. The final data processing results must not only meet user expectations, but also be easy for users to understand and use through appropriate presentation methods.

[0098] Furthermore, the multimodal input data includes voice data, text data and image data, and the step of extracting features from the multimodal input data to obtain multimodal input features specifically includes:

[0099] Performing audio-to-text processing on the voice data in the multimodal input data to obtain the voice text;

[0100] Perform text feature extraction on the speech text to obtain speech text features;

[0101] Performing text feature extraction on text data in the multimodal input data to obtain text data features;

[0102] Performing image feature extraction on image data in the multimodal input data to obtain image data features;

[0103] The speech text features, text data features and image data features are fused to obtain multimodal input features.

[0104] In this embodiment, first, the voice data is processed into text, and the voice content is converted into text data; then, the converted voice text is subjected to text feature extraction to mine the key information and semantic features in the text to obtain voice text features. For text data directly input by the user, the text feature extraction technology is applied to ensure that the system can identify important features such as keywords in the text to obtain text data features. In addition, for the image input by the user, the image data is processed by image feature extraction to extract the subject, background information or specific targets (such as objects, scenes, etc.) in the image, thereby supplementing the visual information of the user's intention and obtaining image data features. Finally, through feature fusion technology, the voice text features, text data features and image data features are integrated to form a complete multimodal input feature.

[0105] Through the above steps, the user's multimodal information can be obtained more comprehensively, which enhances the system's understanding of complex instructions, makes intent recognition more accurate, and effectively improves the user experience.

[0106] Furthermore, the multimodal input data also includes video data, and the step of extracting features from the multimodal input data to obtain multimodal input features specifically includes:

[0107] Parse the video data to obtain the video frame sequence and video voice data;

[0108] Identify key frame images from the video frame sequence, and extract image features from the key frame images to obtain key frame image features;

[0109] Perform audio-to-text processing on the video and voice data to obtain the video and voice text;

[0110] Extracting text features from the video speech text to obtain video speech text features;

[0111] The speech-to-text features, text data features, image data features, video-to-speech-to-text and video-to-speech-to-text features are fused to obtain multimodal input features.

[0112] In another embodiment, the user's multimodal input data also includes video data. First, the system parses the video data, extracts the video frame sequence and video voice data, and divides the video content into two parts, image and audio, for independent processing. Next, key frame images are identified from the video frame sequence. These key frames represent important moments or scene information in the video content. By extracting image features of the key frame images, the system can extract visual information such as the subject, background, and scene changes in the image, so as to capture the core visual information in the video content. In the processing of the video voice data, the system first converts it into text data to obtain the video voice text, and performs text feature extraction on it to analyze the key content in the video audio. Finally, the voice text features, text data features, image data features, video voice text, and video voice text features are integrated through feature fusion to form a unified multimodal input feature representation.

[0113] Through the above steps, multimodal information such as video, voice, text, and images are effectively integrated, which greatly improves the system's understanding of complex content and further enhances the user interaction experience and system intelligence.

[0114] Furthermore, before the step of acquiring the target intelligent entity adapted to the user's operation intention from the preset intelligent entity library, the method further includes:

[0115] Acquire pre-collected historical input data, wherein the historical input data is historical multimodal input data;

[0116] Obtain historical user intentions and historical service call instructions corresponding to historical input data;

[0117] A training data set is constructed based on historical multimodal input data and historical service call instructions, and a large language model is used to perform feature learning based on the training data set to generate a number of intelligent entities, wherein each type of historical multimodal input data corresponds to one intelligent entity;

[0118] For each type of historical multimodal input data, there is one intelligent entity, which is annotated with the corresponding historical user intent;

[0119] Several fully labeled intelligent entities are stored in a preset database to obtain an intelligent entity library.

[0120] In this embodiment, first, the system extracts the user's past input content from the pre-collected historical multimodal input data, including multimodal information such as text, voice, and image, and obtains the user intention and service call instructions corresponding to these historical input data. By collecting and organizing the historical input data, the system establishes a training data set covering multimodal data and service responses. In the training stage, the LLM model performs deep learning based on these historical data to form an LLM intelligent entity that can understand and handle similar tasks. Each type of historical multimodal input data is trained to generate an intelligent entity, which can correspond to specific input features, reflect specific user needs, and accurately generate service call instructions. During the training process, the intelligent entity is annotated with historical user intentions to generate intention labels so that user needs can be responded to more accurately in actual applications. All annotated LLM intelligent entities are uniformly stored in the database to form an intelligent entity library, so that the system can quickly call different intelligent entities in subsequent applications, respond to the intentions of different user inputs, and perform corresponding tasks.

[0121] Through the above steps, an efficient intelligent entity library is built. The system can use historical data for accurate intent recognition and service matching, achieving rapid response to user needs and enhancing adaptive interaction capabilities.

[0122] Furthermore, a training data set is constructed based on historical multimodal input data and historical service call instructions, and a large language model is made to perform feature learning based on the training data set to generate a number of intelligent entities, specifically including:

[0123] For each type of historical multimodal input data, feature extraction is performed on the historical multimodal input data and the historical service call instruction to obtain historical multimodal input features and historical service call instruction features;

[0124] Performing feature association on historical multimodal input features and historical service call instruction features, and enabling the large language model to perform feature learning based on the feature association results, and outputting a first predicted service call instruction;

[0125] Calculating a first prediction error of the large language model based on the first prediction service call instruction and the history service call instruction;

[0126] Iteratively updating the large language model based on the first prediction error until the model is fitted, thereby obtaining a pre-trained first large language model;

[0127] Each of the first language models that have completed pre-training is encapsulated to obtain several intelligent entities.

[0128] In this embodiment, first, for each historical multimodal input data, features are extracted from the input data and the corresponding service call instructions, respectively, to obtain historical multimodal input features and historical service call instruction features. Through the feature association step, the input features and instruction features are associated with each other so that the LLM model can better learn the corresponding relationship between user input and system response. Next, based on these feature association results, feature learning is performed on the LLM model to generate a first predicted service call instruction, that is, the instruction predicted by the model based on the input data. In order to improve the accuracy of the model, the system compares the first predicted service call instruction output by the model with the historical service call instruction and calculates the first prediction error. By continuously adjusting and optimizing the model, the error between the predicted instruction and the historical instruction is gradually reduced, and iterative updates are performed until the model can fit the historical data with high precision. Finally, after the model fitting is completed, each trained LLM model is encapsulated to generate a corresponding intelligent entity. In this way, each intelligent entity has the ability to process specific multimodal inputs and generate service call instructions, which is convenient for direct calling in practical applications.

[0129] Through the above steps, a highly accurate and responsive intelligent entity is achieved, ensuring that the system can accurately identify the user's multimodal input intentions and provide efficient personalized service instruction generation.

[0130] Furthermore, the step of acquiring a target intelligent entity adapted to the user's operation intention from a preset intelligent entity library specifically includes:

[0131] Perform feature vector conversion on the user's operation intention to obtain the intention feature vector;

[0132] Calculate the similarity between the intent feature vector and the intent label of each smart entity in the smart entity library to obtain a similarity sequence, where each smart entity corresponds to an intent label, and the intent label is generated when the smart entity is labeled using historical user intent;

[0133] Sorting the similarity values ​​in the similarity sequence and determining the maximum similarity value;

[0134] The intelligent entity corresponding to the maximum similarity value is identified, and the intelligent entity corresponding to the maximum similarity value is used as the target intelligent entity.

[0135] In this embodiment, first, the user operation intention is converted into an intention feature vector. This vectorization process facilitates the conversion of abstract user needs into numerical representations, and can be compared with the intent labels in the smart entity library for similarity. In the smart entity library, each LLM smart entity is labeled with historical user intentions and given a unique intent label, which is also represented in the form of a feature vector. Next, the system calculates the similarity between the user intention feature vector and the intent label of each LLM smart entity to generate a similarity sequence. By sorting the sequence, the smart entity with the highest similarity value can be found, that is, the smart entity that best meets the current user intention. This smart entity will be regarded as the target smart entity, responsible for responding to user requests and performing corresponding service operations.

[0136] Through the above steps, user intentions and intelligent entities can be accurately matched, which significantly improves the system's automation and response accuracy, allowing users to obtain a more efficient and intelligent service experience.

[0137] Furthermore, after the steps of sorting the similarity values ​​in the similarity sequence and determining the maximum similarity value, the method further includes:

[0138] Determine whether the maximum similarity value is less than a preset similarity threshold;

[0139] If the maximum similarity value is less than a preset similarity threshold, the multimodal input feature is input into the first language model, and an output second prediction service call instruction is obtained;

[0140] Obtaining the standard service calling instruction uploaded by the user, and calculating the second prediction error of the large language model based on the second prediction service calling instruction and the standard service calling instruction;

[0141] Iteratively updating the first largest language model based on the second prediction error until the model is fitted, thereby obtaining a second largest language model;

[0142] The second largest language model is encapsulated to obtain new intelligent entities, and the intelligent entity library is updated based on the new intelligent entities.

[0143] In this embodiment, first, the system determines whether the maximum similarity value is lower than a preset threshold to ensure the matching quality of the target LLM smart entity. If the similarity value is lower than the threshold, it indicates that there are not enough matching LLM smart entities in the current smart entity library to accurately respond to user intentions. In this case, the system inputs the current multimodal input features into the first LLM model trained above, generates a second prediction service call instruction, and attempts to build a new response instruction. Next, the system compares the standard service call instruction provided by the user with the second prediction service call instruction, calculates the second prediction error of the model, and analyzes the gap between the model output and user needs. Through multiple rounds of iterative updates, the prediction error of the model is gradually reduced until the model can fit user needs well. After the model training is completed, the trained second LLM model is encapsulated as a new LLM smart entity and updated to the smart entity library.

[0144] Through the above steps, the system's adaptability and expansion capabilities are enhanced, enabling the system to respond to new user needs, continuously improve the accuracy of user experience and intent matching, and ensure the system's efficient service capabilities in dynamic scenarios.

[0145] In the above embodiments, the present application discloses a method for processing multimodal data input, which belongs to the field of data processing. The present application integrates LLM into the existing system at a relatively low cost, avoids the problem that the traditional system interface design is rigid and difficult to deal with multimodal request data, improves the flexibility of the system, and enables the system to understand the implicit needs of users, meet business needs in a timely manner, and adapt to market changes; without changing the existing architecture, it reduces the cost of interface modification or redesign, reduces the accumulation of technical debt and system maintenance costs.

[0146] In this embodiment, the method for processing multimodal data input is executed on an electronic device (eg, Figure 1 The server shown in the figure) can receive instructions or obtain data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0148] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0149] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a multi-modal data input processing device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0150] like Figure 3 As shown, the multimodal data input processing device 300 described in this embodiment includes:

[0151] The data acquisition module 301 is used to receive a user input instruction and acquire user input data, wherein the user input data is multi-modal input data;

[0152] A feature extraction module 302 is used to extract features from multimodal input data to obtain multimodal input features;

[0153] The intention recognition module 303 is used to input the multimodal input features into the pre-trained intention recognition model to obtain the user operation intention;

[0154] The intelligent entity adaptation module 304 is used to obtain a target intelligent entity adapted to the user's operation intention from a preset intelligent entity library, wherein the intelligent entity library pre-stores a number of intelligent entities generated based on large language model training;

[0155] The instruction generation module 305 is used to generate a service call instruction corresponding to the user operation using the target intelligent entity based on the multimodal input feature;

[0156] The service response module 306 is used to respond to the service call instruction, generate data processing results, and output the data processing results.

[0157] Furthermore, the multimodal input data includes voice data, text data and image data, and the feature extraction module 302 is specifically used for:

[0158] The speech data in the multimodal input data is processed into text to obtain speech text; text feature extraction is performed on the speech text to obtain speech text features; text feature extraction is performed on the text data in the multimodal input data to obtain text data features; image feature extraction is performed on the image data in the multimodal input data to obtain image data features; feature fusion is performed on the speech text features, text data features and image data features to obtain multimodal input features.

[0159] Furthermore, the multimodal input data also includes video data, and the feature extraction module 302 is further used to:

[0160] Parse the video data to obtain video frame sequence and video voice data; identify key frame images from the video frame sequence, and extract image features of the key frame images to obtain key frame image features; perform audio-to-text processing on the video voice data to obtain video voice text; extract text features from the video voice text to obtain video voice text features; perform feature fusion on voice text features, text data features, image data features, video voice text and video voice text features to obtain multimodal input features.

[0161] Furthermore, the multimodal data input processing device 300 further includes an agent training module, which is specifically used for:

[0162] Acquire pre-collected historical input data, wherein the historical input data is historical multimodal input data; acquire historical user intentions and historical service call instructions corresponding to the historical input data; construct a training data set based on the historical multimodal input data and the historical service call instructions, and enable a large language model to perform feature learning based on the training data set to generate a number of intelligent entities, wherein each type of historical multimodal input data corresponds to an intelligent entity; for each type of historical multimodal input data corresponding to an intelligent entity, annotate with the corresponding historical user intention; store the fully annotated intelligent entities in a preset database to obtain an intelligent entity library.

[0163] Furthermore, the agent training module is further used to:

[0164] For each type of historical multimodal input data, feature extraction is performed on the historical multimodal input data and the historical service call instruction respectively to obtain historical multimodal input features and historical service call instruction features; feature association is performed on the historical multimodal input features and the historical service call instruction features, and the large language model is made to perform feature learning based on the feature association results to output a first predicted service call instruction; based on the first predicted service call instruction and the historical service call instruction, a first prediction error of the large language model is calculated; based on the first prediction error, the large language model is iteratively updated until the model is fitted to obtain a pre-trained first large language model; each pre-trained first large language model is encapsulated to obtain a number of intelligent entities.

[0165] Furthermore, the agent adaptation module 304 is specifically used for:

[0166] The user operation intention is transformed into a feature vector to obtain an intention feature vector; the similarity value between the intention feature vector and the intention label of each smart entity in the smart entity library is calculated to obtain a similarity sequence, wherein each smart entity corresponds to an intention label, and the intention label is generated when the smart entity is labeled using historical user intentions; the similarity values ​​in the similarity sequence are sorted and the maximum similarity value is determined; the smart entity corresponding to the maximum similarity value is identified, and the smart entity corresponding to the maximum similarity value is used as the target smart entity.

[0167] Furthermore, the multimodal data input processing device 300 further includes an intelligent entity library updating module, and the intelligent entity library updating module is specifically used for:

[0168] Determine whether the maximum similarity value is less than a preset similarity threshold; if the maximum similarity value is less than the preset similarity threshold, input the multimodal input feature into the first large language model, and obtain the output second prediction service call instruction; obtain the standard service call instruction uploaded by the user, and calculate the second prediction error of the large language model based on the second prediction service call instruction and the standard service call instruction; iteratively update the first large language model based on the second prediction error until the model is fitted to obtain the second large language model; encapsulate the second large language model to obtain a new intelligent entity, and update the intelligent entity library based on the new intelligent entity.

[0169] In the above embodiment, the present application discloses a processing device for multimodal data input, which belongs to the field of data processing. The present application integrates LLM into the existing system at a relatively low cost, avoids the problem that the traditional system interface design is rigid and difficult to deal with multimodal request data, improves the flexibility of the system, and enables the system to understand the implicit needs of users, meet business needs in a timely manner, and adapt to market changes; without changing the existing architecture, it reduces the cost of interface modification or redesign, reduces the accumulation of technical debt and system maintenance costs.

[0170] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0171] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0172] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0173] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a processing method for multimodal data input, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0174] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the multimodal data input processing method.

[0175] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0176] In the above embodiment, the present application discloses a computer device, which belongs to the field of data processing. The present application integrates LLM into the existing system at a relatively low cost, avoids the problem that the traditional system interface design is rigid and difficult to deal with multi-modal request data, improves the flexibility of the system, and enables the system to understand the implicit needs of users, meet business needs in a timely manner, and adapt to market changes; on the basis of not changing the existing architecture, it reduces the cost of interface modification or redesign, reduces the accumulation of technical debt and system maintenance costs.

[0177] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the multimodal data input processing method as described above.

[0178] In the above embodiments, the present application discloses a computer-readable storage medium, which belongs to the field of data processing. The present application integrates LLM into the existing system at a relatively low cost, avoids the problem that the traditional system interface design is rigid and difficult to deal with multi-modal request data, improves the flexibility of the system, and enables the system to understand the implicit needs of users, meet business needs in a timely manner, and adapt to market changes; on the basis of not changing the existing architecture, it reduces the cost of interface modification or redesign, reduces the accumulation of technical debt and system maintenance costs.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0180] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for processing multimodal data input, characterized in that: include: Receiving a user input instruction and acquiring user input data, wherein the user input data is multimodal input data; Performing feature extraction on the multimodal input data to obtain multimodal input features; Inputting the multimodal input features into a pre-trained intention recognition model to obtain the user operation intention; Acquire a target intelligent entity adapted to the user's operation intention from a preset intelligent entity library, wherein the intelligent entity library pre-stores a plurality of intelligent entities generated based on large language model training; Based on the multimodal input feature, using the target intelligent entity to generate a service call instruction corresponding to the user operation; In response to the service call instruction, a data processing result is generated and outputted.

2. The method for processing multimodal data input according to claim 1, characterized in that: The multimodal input data includes voice data, text data and image data, and the step of extracting features from the multimodal input data to obtain multimodal input features specifically includes: Performing voice-to-text processing on the voice data in the multimodal input data to obtain voice text; Performing text feature extraction on the speech text to obtain speech text features; Performing text feature extraction on the text data in the multimodal input data to obtain text data features; Performing image feature extraction on image data in the multimodal input data to obtain image data features; The speech text feature, the text data feature and the image data feature are subjected to feature fusion to obtain the multimodal input feature.

3. The method for processing multimodal data input according to claim 2, characterized in that: The multimodal input data also includes video data, and the step of extracting features from the multimodal input data to obtain multimodal input features specifically includes: Parsing the video data to obtain a video frame sequence and video voice data; Identifying key frame images from the video frame sequence, and performing image feature extraction on the key frame images to obtain key frame image features; Performing audio-to-text processing on the video and voice data to obtain video and voice text; Extracting text features from the video speech text to obtain video speech text features; The voice-text feature, the text data feature, the image data feature, the video-voice-text and the video-voice-text feature are subjected to feature fusion to obtain the multimodal input feature.

4. The method for processing multimodal data input according to claim 1, characterized in that: Before the step of acquiring the target intelligent entity adapted to the user operation intention from the preset intelligent entity library, the method further includes: Acquire pre-collected historical input data, wherein the historical input data is historical multimodal input data; Obtaining historical user intentions and historical service call instructions corresponding to the historical input data; Building a training data set based on the historical multimodal input data and the historical service call instructions, and enabling the large language model to perform feature learning based on the training data set to generate a plurality of intelligent entities, wherein each type of the historical multimodal input data corresponds to one intelligent entity; For each type of the historical multimodal input data corresponding to an intelligent entity, the intelligent entity is annotated using the corresponding historical user intention; The fully labeled intelligent entities are stored in a preset database to obtain the intelligent entity library.

5. The method for processing multimodal data input according to claim 4, characterized in that: The step of constructing a training data set based on the historical multimodal input data and the historical service call instructions, and enabling the large language model to perform feature learning based on the training data set to generate a plurality of intelligent entities specifically includes: For each type of the historical multimodal input data, feature extraction is performed on the historical multimodal input data and the historical service call instruction to obtain historical multimodal input features and historical service call instruction features; Performing feature association on the historical multimodal input features and the historical service call instruction features, and causing the large language model to perform feature learning based on the feature association results, and outputting a first predicted service call instruction; Calculating a first prediction error of the large language model based on the first prediction service calling instruction and the history service calling instruction; Iteratively updating the large language model based on the first prediction error until the model is fitted, thereby obtaining a pre-trained first large language model; Each of the first large language models that have completed pre-training is encapsulated to obtain a plurality of the intelligent entities.

6. The method for processing multimodal data input according to claim 5, characterized in that: The step of acquiring a target intelligent entity adapted to the user operation intention from a preset intelligent entity library specifically includes: Performing feature vector conversion on the user operation intention to obtain an intention feature vector; Calculate the similarity value between the intention feature vector and the intention label of each smart entity in the smart entity library to obtain a similarity sequence, wherein each smart entity corresponds to an intention label, and the intention label is generated when the smart entity is labeled using the historical user intention; Sorting the similarity values ​​in the similarity sequence and determining the maximum similarity value; The intelligent entity corresponding to the maximum similarity value is identified, and the intelligent entity corresponding to the maximum similarity value is used as the target intelligent entity.

7. The method for processing multimodal data input according to claim 6, characterized in that: After the steps of sorting the similarity values ​​in the similarity sequence and determining the maximum similarity value, the method further includes: Determine whether the maximum similarity value is less than a preset similarity threshold; If the maximum similarity value is less than a preset similarity threshold, inputting the multimodal input feature into the first language model, and obtaining an output second prediction service calling instruction; Obtaining a standard service calling instruction uploaded by a user, and calculating a second prediction error of the large language model based on the second prediction service calling instruction and the standard service calling instruction; Iteratively updating the first largest language model based on the second prediction error until the model is fitted, thereby obtaining a second largest language model; The second largest language model is encapsulated to obtain a new intelligent entity, and the intelligent entity library is updated based on the new intelligent entity.

8. A processing device for multimodal data input, characterized in that: include: A data acquisition module, used to receive a user input instruction and acquire user input data, wherein the user input data is multimodal input data; A feature extraction module, used to extract features from the multimodal input data to obtain multimodal input features; An intention recognition module, used to input the multimodal input features into a pre-trained intention recognition model to obtain the user's operation intention; An intelligent entity adaptation module, used to obtain a target intelligent entity adapted to the user's operation intention from a preset intelligent entity library, wherein the intelligent entity library pre-stores a number of intelligent entities generated based on large language model training; An instruction generation module, used to generate a service call instruction corresponding to a user operation using the target intelligent entity based on the multimodal input feature; The service response module is used to respond to the service call instruction, generate data processing results, and output the data processing results.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for processing multimodal data input as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for processing multimodal data input according to any one of claims 1 to 7 are implemented.

Images