Multi-mode large model auxiliary problem solving system
Through a multimodal large model assisted problem-solving system, combining voice and image input, and using a multimodal large language model to process data, the limitations of the existing single input mode of the problem-solving system are solved, and more intelligent and efficient problem-solving is achieved Auxiliary services.
Patent Information
- Application Number
- CN202411167040.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-26
AI Technical Summary
The existing problem-solving systems mainly rely on a single text input or image input mode, which limits the system's application scenarios and problem-solving efficiency. Moreover, special characters are in trouble when inputting text, and a system that simplifies the input method is needed.
Design a multimodal large-modal model assisted problem-solving system, combining voice input and image input, obtain data through cameras and microphones, and use multimodal large-language models to process and display answers, including language recognition, visual recognition and answer pre-examination Training the model to support diverse input methods and powerful semantic understanding.
It realizes multi-way input of the problem-solving system, provides more intelligent and efficient problem-solving auxiliary services, simplifies the user input process, especially the input of special characters, and improves the application scenarios and efficiency of the system.
Smart Images

Figure CN119024967A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of data processing, and in particular relates to a multi-modal large model assisted problem-solving system. Background Art
[0002] Most current problem-solving systems rely on a single text input or image input mode, which to a certain extent limits the application scenarios and problem-solving efficiency of the system. In addition, when text input is required, the input of many special characters is extremely troublesome. Therefore, there is an urgent need to design a problem-solving system that can simplify text input and provide users with more options. Summary of the invention
[0003] In order to solve the above problems, this application designs a multimodal large model assisted problem-solving system, which aims to supplement the single question input method of the existing assisted problem-solving system through image input and voice input, and provide users with more intelligent and efficient problem-solving assistance services.
[0004] .A multimodal large model assisted problem-solving system, comprising an input module, a multimodal large language model, and an output module;
[0005] The input module is used to upload the acquired voice data and image data to the multimodal large language model;
[0006] The multimodal large language model is used to process voice data and image data, obtain answer data, and transmit the answer data through an output module for user display.
[0007] Preferably, the input module includes a camera module, a microphone and a front data transmission module;
[0008] The camera module is used to obtain image information of the test questions and transmit it to the multimodal large language model through the front data transmission module;
[0009] The microphone is used to obtain the user's voice input information and transmit it to the multimodal large language model through the front data transmission module.
[0010] Preferably, the camera module also includes an image preprocessing module for performing denoising and edge detection processing on the image.
[0011] Preferably, the multimodal large language model includes a language recognition module, a visual recognition module and an answer pre-training model;
[0012] The language recognition module is connected to the front-end data transmission module, converts the voice input information obtained by the microphone into question data, and inputs the converted question data into the answer pre-training model;
[0013] The visual recognition module is connected to the front data transmission module, converts the image information obtained by the camera module into text, and inputs the converted question data into the answer pre-training model;
[0014] The answer pre-training model is used to answer the converted question data, obtain answer data, and display the answer data through an output module.
[0015] Preferably, the output module includes a screen display module and a speaker, so that the screen display module is used to display the answer data, and the speaker is used to give a voice explanation of the answer data.
[0016] Preferably, the multimodal large language model also includes a voice wake-up word recognition module, which is used to open the language recognition module.
[0017] The advantages and effects of this application are as follows:
[0018] A multimodal large model-assisted problem-solving system designed in this application realizes multi-mode input of the problem-solving system by combining voice input and image input. Combined with a multimodal large language model, this application can provide users with more intelligent and efficient problem-solving assistance services through diversified input methods and powerful semantic understanding capabilities.
[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application so that it can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings as follows.
[0020] Based on the detailed description of the specific embodiments of the present application in combination with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale.
[0022] Figure 1 Framework diagram of a multimodal large model-assisted problem-solving system designed for this application. DETAILED DESCRIPTION
[0023] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, rather than all of the embodiments. In the following description, specific details such as specific configuration and components are provided only to help fully understand the embodiments of the present application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, for clarity and brevity, the description of known functions and structures is omitted in the embodiment.
[0024] It should be understood that the references to "one embodiment" or "this embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "one embodiment" or "this embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0025] In addition, the present application may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity, and does not in itself indicate the relationship between the various embodiments and / or settings discussed.
[0026] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" in this article describes another type of association object relationship, indicating that there can be two relationships. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship.
[0027] The term "at least one" in this article is merely a description of the association relationship of associated objects, indicating that there may be three relationships. For example, at least one of A and B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0028] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusions.
[0029] Example 1
[0030] This embodiment mainly introduces a multi-modal large model assisted problem-solving system, please refer to Figure 1 , which includes an input module, a multimodal large language model, and an output module;
[0031] The input module is used to upload the acquired voice data and image data to the multimodal large language model;
[0032] The multimodal large language model is used to process voice data and image data, obtain answer data, and transmit the answer data through an output module for user display.
[0033] Further, the input module includes a camera module, a microphone and a front data transmission module;
[0034] The camera module is used to obtain image information of the test questions and transmit it to the multimodal large language model through the front data transmission module;
[0035] The microphone is used to obtain the user's voice input information and transmit it to the multimodal large language model through the front data transmission module.
[0036] Furthermore, the camera module also includes an image preprocessing module for performing denoising and edge detection on the image.
[0037] Furthermore, the multimodal large language model includes a language recognition module, a visual recognition module and an answer pre-training model;
[0038] The language recognition module is connected to the front-end data transmission module, converts the voice input information obtained by the microphone into question data, and inputs the converted question data into the answer pre-training model;
[0039] The visual recognition module is connected to the front data transmission module, converts the image information obtained by the camera module into text, and inputs the converted question data into the answer pre-training model;
[0040] The answer pre-training model is used to answer the converted question data, obtain answer data, and display the answer data through an output module.
[0041] Furthermore, the output module includes a screen display module and a speaker, so that the screen display module is used to display the answer data, and the speaker is used to give a voice explanation of the answer data.
[0042] Furthermore, the multimodal large language model also includes a voice wake-up word recognition module, which is used to open the language recognition module.
[0043] Furthermore, the input module also includes text input, and the screen display module is designed as a touch screen. The screen display module is also connected to the front data transmission module to input text data into the multimodal large language model to achieve data interaction.
[0044] A multimodal large model-assisted problem-solving system designed in this application realizes multi-mode input of the problem-solving system by combining voice input and image input. Combined with a multimodal large language model, this application can provide users with more intelligent and efficient problem-solving assistance services through diversified input methods and powerful semantic understanding capabilities.
[0045] The above description is only the preferred embodiment of the present invention, which does not limit the protection scope of the present invention. For those skilled in the art, the present invention can be modified and varied in various ways. Within the spirit and principle of the present invention, any change, modification, replacement, integration and parameter change of these embodiments by conventional substitution or capable of achieving the same function without departing from the principle and spirit of the present invention shall fall within the protection scope of the present invention.
Claims
1. A multi-modal large model assisted problem-solving system, characterized in that: It includes an input module, a multimodal large language model, and an output module; The input module is used to upload the acquired voice data and image data to the multimodal large language model; The multimodal large language model is used to process voice data and image data, obtain answer data, and transmit the answer data through an output module for user display.
2. A multi-modal large model assisted problem-solving system according to claim 1, characterized in that: The input module includes a camera module, a microphone and a front data transmission module; The camera module is used to obtain image information of the test questions and transmit it to the multimodal large language model through the front data transmission module; The microphone is used to obtain the user's voice input information and transmit it to the multimodal large language model through the front data transmission module.
3. A multi-modal large model assisted problem-solving system according to claim 1, characterized in that: The camera module also includes an image preprocessing module for performing denoising and edge detection processing on the image.
4. A multi-modal large model assisted problem-solving system according to claim 1, characterized in that: The multimodal large language model includes a language recognition module, a visual recognition module and an answer pre-training model; The language recognition module is connected to the front-end data transmission module, converts the voice input information obtained by the microphone into question data, and inputs the converted question data into the answer pre-training model; The visual recognition module is connected to the front data transmission module, converts the image information obtained by the camera module into text, and inputs the converted question data into the answer pre-training model; The answer pre-training model is used to answer the converted question data, obtain answer data, and display the answer data through an output module.
5. A multi-modal large model assisted problem-solving system according to claim 1, characterized in that: The output module includes a screen display module and a speaker, so the screen display module is used to display the answer data, and the speaker is used to give a voice explanation of the answer data.
6. A multi-modal large model assisted problem-solving system according to claim 1, characterized in that: The multimodal large language model also includes a voice wake-up word recognition module, which is used to open the language recognition module.
