LLM-based multi-model scheduling architecture method

Abstract

The invention relates to the technical field of big data processing, in particular to an LLM-based multi-model scheduling architecture method. The method is used for application programs of multi-modal input, and particularly relates to the related field of artificial intelligence AI large models, and processing is carried out by judging various modal data input by a user. According to the method, application and mutual scheduling of the multi-modal model are achieved in a text mode, and the main process comprises recognition and processing of multi-modal input, text processing and associated problem output, resource type processing, interactive experience optimization and the like. A user can interact with the system in various natural modes, behavior habits do not need to be changed, various types of input are processed at the same time, and the interaction efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a multi-model scheduling architecture method based on LLM; an application program for multi-modal input, especially in the field of artificial intelligence (AI) big models, which processes multiple modal data input by users by judging them. Background Art

[0002] In the current technical solutions, there are some challenges in multimodal data processing, mainly due to the lack of effective interaction and scheduling mechanisms between various models. Usually, different types of data input (such as speech, image, text, etc.) require separate models for processing. For example, the speech recognition model is used to process speech input, the image recognition model is used to process image input, and the text processing model is used to process text input. However, this separate processing method has some problems in practical applications.

[0003] First, there is a lack of effective information exchange mechanism between models, and the relevant information between different modalities cannot be fully utilized. For example, when processing user input data, the voice model may extract the voice content in the video, and the image model may extract the visual information in the video, but the current technical solution cannot effectively integrate and utilize this information, resulting in information redundancy and waste.

[0004] Secondly, there is a lack of effective scheduling mechanism between models, which makes it impossible to achieve dynamic resource allocation and task scheduling. Since the processing tasks of each model may have different complexity and priority, a flexible scheduling mechanism is needed to reasonably allocate and schedule the processing tasks of different models according to actual conditions to ensure the overall performance of the system and user experience.

[0005] In addition, the current technical solutions also have some technical problems, the most important of which is how to achieve effective fusion and processing of multimodal data. Since multimodal data have different characteristics and representation methods, how to effectively fuse and process these data is a challenging problem. Current technical solutions often use simple and crude methods to fuse data, such as directly splicing data from different modes together or using simple weighted averages, but this method often cannot fully utilize the relevant information between different modes, resulting in inaccurate processing results and low efficiency.

[0006] Therefore, the main technical problem in the current technical solution is how to design an effective multimodal data processing framework to achieve information interaction and task scheduling between different modalities to improve the overall performance of the system and user experience. This requires in-depth research in model design and computing power optimization, and the exploration of new data fusion and processing methods to meet the challenges in multimodal data processing. Summary of the invention

[0007] This application aims to realize the application and mutual scheduling of multimodal models. The main processes include recognition and processing of multimodal input, text processing and output of related questions, resource type processing, and optimization of interactive experience. The main point of the invention lies in the comprehensive processing of multimodal input and intelligent management of output. Taking speech recognition as an example, ASR is used to identify the intention of user voice input, and image recognition is combined to obtain more comprehensive information. Keyword recognition further refines user needs and generates related questions to improve the convenience of user interaction. This invention solves the problem of incoordination between information processing and output in multimodal applications and improves the user interaction experience. The interface supports mutual calls of different models, effectively utilizes the cache mechanism to reduce computing power consumption, and ensures efficient response of the system.

[0008] A multi-model scheduling architecture method based on LLM includes the following steps: S1. User inputs multimodal data: User questions are input into multimodal data through voice commands, text input, and picture taking, and the input multimodal data is converted into text format data; S2. Extract user intent and entities: User questions are saved in text format and the text is analyzed and understood using Neuro-Linguistic Programming (NLP) technology. S3. Multimodal management using text: Use text format to unify scheduling and control AI model corpus training to optimize user problems; S4. Call the large language model to generate answer text: Optimize the text format of user questions and generate corresponding answer texts through data models; optimize the training of answer text AI model corpus; S5. Call the TTS model to generate explanation voice: The answer text uses the voice TTS model to convert the generated text into voice output; S6. Return using text and voice: The text and voice results are integrated and then output to the user.

[0009] S7. Generate cache based on user intent and entity: The text and voice output to the user generates cache data. When a request for the same user question is received again, the data in the cache is directly used to quickly respond.

[0010] Beneficial effects of the present invention: 1. Improve interaction efficiency: Users can interact with the system in a variety of natural ways without changing their behavioral habits, and process multiple types of input at the same time to improve interaction efficiency. 2. Improve user experience: Select the most appropriate feedback method based on user needs and scenarios to make information transmission more intuitive and easy to understand. 3. Powerful processing capabilities: Integrate multiple model processing capabilities to more accurately understand user intentions and provide more accurate feedback. 4. Save resources: The cache mechanism reduces computing power consumption, improves response speed, and no longer repeats large model reasoning for the same or similar problems. 5. Wide range of application scenarios: It can be applied to a variety of interactive scenarios such as intelligent explanations, online education, and intelligent customer service, and has broad market application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Flow chart of the scheduling architecture of the present invention. DETAILED DESCRIPTION

[0012] A multi-model scheduling architecture method based on LLM includes the following steps: S1. User inputs multimodal data: User questions are input into multimodal data through voice commands, text input, and picture taking, and the input multimodal data is converted into text format data; S2. Extract user intent and entities: User questions are saved in text format and the text is analyzed and understood using Neuro-Linguistic Programming (NLP) technology. S3. Multimodal management using text: Use text format to unify scheduling and control AI model corpus training to optimize user problems; S4. Call the large language model to generate answer text: Optimize the text format of user questions and generate corresponding answer texts through data models; optimize the training of answer text AI model corpus; S5. Call the TTS model to generate explanation voice: The answer text uses the voice TTS model to convert the generated text into voice output; S6. Return using text and voice: The text and voice results are integrated and then output to the user.

[0013] S7. Generate cache based on user intent and entity: The text and voice output to the user generates cache data. When a request for the same user question is received again, the data in the cache is directly used to quickly respond.

[0014] The AI ​​model corpus training in S3 and S4 is a conventional technology in this field. Here, it is only necessary to standardize the user questions and answer texts to facilitate the execution of subsequent solution steps, and no further details will be given.

[0015] Data of various models: speech recognition Qwen-Audio-Chat module, image recognition Yolo module, speech generation GPT-Sovits module, large language Chat GLM3 model.

[0016] Multimodal interaction system: The system mainly includes input modules (speech recognition, image recognition and plain text input), processing modules (user intent recognition and keyword recognition) and output modules (associated question output, streaming plain text output and voice streaming output). The system supports mutual calls between models through interfaces, including multimodal models calling text processing models, and text model outputs calling voice output models at the same time. In addition, the system also supports a caching mechanism to reduce the computing power consumption of large models and improve C-end response. The specific implementation method is: for the same or similar questions asked by users, the model will identify their intent, and cache the same intent so that questions with similar intents are no longer subject to large model reasoning.

[0017] Usage scenarios: This technical solution can be applied to various scenarios that require multimodal interaction, such as intelligent explanation, online education, intelligent customer service, etc. In these scenarios, users can interact with the system through voice, photo taking and input. The system processes the user's input through a multimodal model and then feedbacks information to the user in the most suitable way (such as voice, picture, text, etc.). Compared with the existing technology, the main advantages of the multimodal interaction system are: Improve interaction efficiency: The multimodal interaction system can process multiple types of inputs at the same time, such as voice, image and text, which allows users to interact with the system in the most natural way without changing their behavior habits. Improve user experience: The system can select the most appropriate feedback method, such as voice, text, picture, etc., according to the specific needs and scenarios of the user, making the transmission of information more intuitive and easy to understand. Powerful processing power: By integrating the processing power of multiple models, the multimodal interaction system can understand the user's intention more accurately and provide more accurate feedback. Save resources: Through the caching mechanism, the system will cache the same or similar questions asked by the user so that questions with similar intentions will no longer be inferred by large models, thereby reducing computing power consumption and improving response speed. Wide range of application scenarios: Multimodal interaction systems can be applied to various scenarios that require interaction, such as intelligent explanations, online education, intelligent customer service, etc., and have broad market application prospects. Example

[0018] In this step, users interact with the system in a variety of ways, such as voice commands, text input, and picture taking. The system first needs to convert these multimodal input data into a unified text format. This includes not only transcribing speech into text, but also parsing the text information in the picture or understanding its visual content and converting it into a text description. This conversion process requires a high degree of accuracy and adaptability to different data types to ensure accurate transmission of information.

[0019] Once the user's input is converted into text format, the next step is to use neuro-linguistic programming (NLP) technology to analyze and understand the text. This includes but is not limited to intent recognition and entity extraction. Intent recognition helps the system understand what the user's goal is, such as "introducing cultural relics", "finding addresses" or "learning history". Entity extraction involves identifying specific information in the text, such as the name of the cultural relic, the name of the person, etc., which helps the system process contextual requests more accurately.

[0020] In order to efficiently handle these complex tasks, the system uses a dedicated multimodal management tool to uniformly schedule and control all relevant AI models and algorithms. This tool is not only responsible for the coordination and communication between models, but also for the allocation and optimization of resources to ensure the smooth operation of the system and avoid resource waste. For example, some models only focus on optimization problems, while some models focus on answering questions.

[0021] Based on the optimized user questions in the previous step, the system will then call a powerful text processing model to generate the corresponding answers or explanation text. This text model is trained based on a large-scale corpus and can generate accurate and fluent natural language text to provide detailed explanations or meet user query needs.

[0022] Calling the TTS model to generate explanation voice: In order to provide a more humanized interactive experience, the system will also call the text-to-speech (TTS) model to convert the generated text into voice output. This step allows users to obtain information by listening, which is especially important for users with visual impairments or who prefer auditory learning.

[0023] Using a multimodal framework to combine text and voice for feedback: Finally, the system will integrate the results of text and voice through a multimodal framework and then output it to the user. This multimodal feedback method allows users to choose the most appropriate receiving method under different situations and preferences, improving the flexibility and satisfaction of the user experience.

[0024] Generate cache based on user intent and entity: During processing, the system will intelligently generate cache data based on user intent and entity recognition results. When the system receives a request with similar intent again, it can directly use the data in the cache to respond quickly, greatly improving efficiency and response speed. This not only optimizes system performance, but also saves valuable computing resources.

[0025] The current case is the explanation of museum cultural relics. In order to illustrate the implementation scheme of this patent, the event content currently submitted by this patent is marked as {Type} and {data}. {Type} is a multimodal type parameter input by the C-end, which is 01, 02 or 03, indicating that the multimodal type currently used may be voice, picture or plain text. {data} is its corresponding content, which may be a segment of audio stream data, picture data or text data. In addition, different data types will be integrated according to different businesses, such as location coordinates, and different content responses can be made according to the location. After sending the request, the system will first perform data parsing, and schedule different models according to the data type to extract its intent. Then the system will schedule the multimodal alignment module according to the intent. This module will determine whether the intent exists in the cache. If it exists, it will return the cache content and its corresponding voice stream. If it does not exist, it will schedule a large language model for question and answer. At the same time, the voice cloning model will be scheduled to generate a real-time voice stream through text. Finally, the system will return the response text and voice stream to the C-end and store the intent content in the cache to achieve the effect of reducing computing power and responding quickly.

[0026] Detailed description of each module: Input module: mainly includes speech recognition model (Qwen-Audio-Chat), which is a large-scale audio language model. It is a multi-task language model conditioned on audio and text input, which expands the large dialogue model and effectively perceives audio signals by connecting a single audio encoder. Unlike other speech recognition models, it is a model similar to voice dialogue. This architecture makes full use of this dialogue function with the principle of prompt engineering, so that the model can accurately and quickly recognize uncommon words. Image recognition (YOLO) is a deep learning algorithm for object detection. Its main feature is the ability to simultaneously detect and classify objects in one forward propagation, thereby achieving fast and accurate detection. It is mainly used in the system to quickly process incoming images and achieve real-time detection goals, so as to achieve the effect of identifying the subject through image information.

[0027] Processing module: mainly includes user intent recognition and keyword recognition (FastGPT), a knowledge base question-answering system based on the LLM large language model, providing out-of-the-box data processing, model calling and other capabilities. At the same time, workflow orchestration can be performed through Flow visualization to achieve complex question-answering scenarios. This system uses FastGPT to parse and process user-input questions, identify and extract keywords and intents, perform business processing based on keywords, and call the large dialogue model (ChatGLM3) for intents. It is a dialogue model based on the Generative Pre-trained Transformer (GPT) architecture, specifically used for natural language processing tasks such as dialogue generation and text understanding, and can provide intelligent text interaction in a variety of application scenarios.

[0028] Output module: including the output of related questions, which FastGPT obtains by parsing the answer content of the large dialogue model ChatGLM3 and encapsulating it into the returned content. There are two cases for streaming plain text output. If the ChatGLM3 model is intended to be called for the first time, the built-in streaming output interface is used. Otherwise, the cache content is streamed and returned by calling the cache. The voice stream is realized by segmenting the text stream, calling the voice cloning model (GPT-SoVits) for each segmented text in turn, and finally encapsulating the save path of the voice file into the text stream.

[0029] Fusion of multimodal scheduling technology: Integrate multiple technologies such as the speech recognition model Qwen-Audio-Chat, the image recognition algorithm YOLO, the user intent recognition and keyword recognition tool FastGPT, and the large dialogue model ChatGLM3 to give full play to their respective advantages and achieve more comprehensive and accurate information processing and interaction.

[0030] Optimize workflow arrangement: Through Flow visualization, workflow arrangement can be carried out to enable complex question-and-answer scenarios to be handled in an orderly manner, improving the overall operating efficiency of the system and its ability to cope with complex tasks. For example, when dealing with multiple rounds of dialogue or interactions involving multiple topics, the working order and coordination methods of each module can be reasonably arranged.

[0031] Improve the diversity and flexibility of output: It not only realizes the output of related questions, but also provides multiple processing methods for streaming plain text output and innovative voice stream output. Since the current voice cloning model (GPT-SoVits) does not support true streaming output, this system innovatively uses text segmentation to simulate streaming output, and uses cluster deployment + multi-threaded processing. It does not affect the output of the original text stream, but also responds to the voice in real time, ensuring that the text stream and audio stream are output at the same time.

[0032] Strengthening model synergy: When processing user input, modules can work closely together to achieve accurate recognition and processing of keywords and intent, fully call on corresponding models and resources, and provide answers that better meet user needs. For example, when users ask complex technical questions, the system can accurately identify intent and call on professional knowledge models to answer.

[0033] Strengthening model synergy: When processing user input, modules can work closely together to achieve accurate recognition and processing of keywords and intents, fully call on corresponding models and resources, and provide answers that better meet user needs. Taking users asking complex technical questions as an example, the system can accurately identify intents, innovatively add a caching mechanism, and match different intents for rapid response to ensure the normal use of the system in a high-concurrency environment.

[0034] Technical points: 1. The architecture of the multimodal interaction system consists of input modules (speech recognition, image recognition and plain text input), processing modules (user intent recognition and keyword recognition) and output modules (associated question output, streaming plain text output and speech streaming output).

[0035] 2. The system supports interface calls between various models, such as the multimodal model calling the text processing model, and the text model output calling the speech output model at the same time.

[0036] 3. Implementation of the cache mechanism: After identifying the intent of the same or similar questions asked by the user, the same intent is cached to avoid large-model reasoning for similar questions again.

[0037] Glossary 1. Automatic Speech Recognition: ASR for short, this is a technology that can convert sound signals into text form. This can be compared to the human auditory system, or "ears", which captures sounds and transmits them to the brain for processing. In this process, the computer uses signal processing technology to extract features, match patterns, and recognize speech signals, thereby achieving the purpose of converting speech into text.

[0038] 2. Text To Speech: TTS for short, this is a technology that converts text information into audible speech. This process can be seen as the human speaking process, that is, the "mouth", which converts text into sound output, allowing computers to not only process text, but also "speak".

[0039] 3.LLM (Large Language Model): Short for Large Language Model, an AI system that can understand and generate human language by processing large amounts of text data.

[0040] 4.NLP (Neuro Linguistic Programming): short for neuro-linguistic programming, is a technology that enables computers to understand, interpret and generate natural language. It covers aspects such as grammatical analysis, semantic understanding, and machine translation, helping computers process and interact with human language.

[0041] Multimodal: involves fusing data from different types (such as text, voice, and images) to gain a more comprehensive understanding. This approach can provide a richer interactive experience and improve the system's ability to handle complex situations.

Claims

1. A multi-model scheduling architecture method based on LLM, characterized in that The steps include: S1. User inputs multimodal data: User questions are input into multimodal data through voice commands, text input, and picture taking, and the input multimodal data is converted into text format data; S2. Extract user intent and entities: User questions are saved in text format and the text is analyzed and understood using Neuro-Linguistic Programming (NLP) technology. S3. Multimodal management using text: Use text format to unify scheduling and control AI model corpus training to optimize user problems; S4. Call the large language model to generate answer text: Optimize the text format of user questions and generate corresponding answer texts through data models; optimize the training of answer text AI model corpus; S5. Call the TTS model to generate explanation voice: The answer text uses the voice TTS model to convert the generated text into voice output; S6. Return using text and voice: The text and voice results are integrated and then output to the user.

2. According to claim 1, a multi-model scheduling architecture method based on LLM is characterized in that Also includes S7. Generate cache based on user intent and entity: The text and voice output to the user generates cache data. When a request for the same user question is received again, the data in the cache is directly used to quickly respond.

Images