Control system and equipment for realizing multi-modal interaction by using AI large model

Abstract

The invention belongs to the technical field of multi-modal interaction, and particularly relates to a control system and equipment for realizing multi-modal interaction by using an AI large model. According to the method, intelligent judgment can be made according to the use habit and the input accuracy of the user through the obtained multi-modal input, the user can interact with the system more naturally and smoothly in a complex environment, the control system can process a plurality of input sources at the same time, judgment is made according to the completion degree, the accuracy and historical data, and the user experience is improved. When the input is inconsistent, the input preference of the user can be gradually learned through reasonable priority selection, accurate and reliable response of the system and historical data of the emphasis degree module, better customization support is provided in future interaction, and the individuation degree of user experience is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal interaction, and specifically relates to a control system and equipment for realizing multimodal interaction by utilizing an AI large model. Background Art

[0002] With the rapid development of artificial intelligence technology, AI big models have become a research hotspot in the current scientific and technological field. These models are based on deep learning algorithms and are obtained through large-scale training data. They have excellent generalization performance and expression capabilities and can be widely used in image recognition, natural language processing, speech recognition and other fields. The rise of AI big model technology is due to the explosive growth of data volume and the significant improvement of computing power. In the era of big data, people hope to extract more useful information from massive data and solve various complex problems through AI big model technology. AI big models achieve accurate modeling and efficient solution of complex tasks through complex network structures and a large amount of training data.

[0003] Multimodal interaction refers to the interaction with the system by integrating multiple information input methods (such as voice, text, images, gestures, etc.). This interaction method can provide a more natural and intuitive user experience and is widely used in smart homes, smart customer service, autonomous driving and other fields. However, there are many challenges in achieving multimodal interaction, such as information fusion, semantic understanding, and real-time response. The emergence of AI big models provides new ideas and methods for solving these problems.

[0004] Although AI big models have shown great potential in multimodal interaction, existing technologies still have some shortcomings. Existing systems lack a reasonable judgment mechanism for the priority of commands from various input sources, and may not be able to select the command that best meets the user's intention for execution, and cannot make intelligent adjustments based on user preferences, resulting in insufficient personalization and unsatisfactory interaction results. Summary of the invention

[0005] The purpose of the present invention is to provide a control system that uses an AI big model to achieve multimodal interaction, which can achieve efficient and intelligent control of multimodal interaction through the combination of big data analysis, can adapt to various application scenarios and continuously optimize user experience.

[0006] The technical solution adopted by the present invention is as follows: A control system for realizing multimodal interaction using an AI big model, including a multimodal module, a completion module, an accuracy module, a focus module and a matching execution module The multimodal module is used to obtain multimodal data input by the user and obtain the modal instruction of the corresponding input source according to the multimodal data; A completion module, used for obtaining instruction similarity according to modal instructions of multiple input sources, determining whether the instruction similarity meets a first preset condition, and if not, constructing a first time period and obtaining the instruction execution completion of each input source in the first time period; An accuracy module, used to obtain the input accuracy of multiple modal instructions of each input source in the first time period, and obtain a corresponding input accuracy weight according to the input accuracy of each input source; The emphasis module is used to obtain the input period of the multimodal data input by the user, obtain multiple modal historical instructions corresponding to each input source within the input period, and obtain the instruction history emphasis of each input source according to the multiple modal historical instructions of each input source; A matching execution module is used to obtain the instruction priority value of each input source according to the instruction execution completion, input accuracy weight and instruction history emphasis of each input source, obtain the modal instruction corresponding to the instruction priority value that meets the second preset condition, and execute the corresponding modal instruction.

[0007] In a preferred embodiment, the multimodal module includes an execution process unit, an execution vector unit, and an instruction similarity unit; An execution process unit, used for obtaining a corresponding instruction execution process according to the modal instruction of each input source; An execution vector unit, used to convert multiple instruction execution processes into instruction execution vectors of uniform specifications; The instruction similarity unit is used to obtain the instruction similarity between every two instruction execution processes according to multiple instruction execution vectors of the modal instructions of each input source.

[0008] In a preferred embodiment, the multimodal module further includes a similarity threshold unit and a similarity judgment unit; A similarity threshold unit, used for obtaining an instruction similarity threshold; A similarity judgment unit, used to judge whether the instruction similarity exceeds an instruction similarity threshold; If the instruction similarity exceeds the instruction similarity threshold, it is determined that the modal instructions of the multiple input sources are the same; If the instruction similarity does not exceed the instruction similarity threshold, it is determined that there are differences in the modal instructions of the multiple input sources, and the instruction similarity that does not exceed the instruction similarity threshold is marked as an instruction difference value.

[0009] In a preferred embodiment, the multimodal module further includes a standard duration unit, an end time unit, a time period weight unit, a first duration unit, a start time unit, a time period unit, and a completion degree unit; Standard duration unit, used to obtain standard duration; An end time unit, used to obtain the time marked as the instruction difference value and mark it as the end time; A time period weight unit, used to obtain the time period marked as the instruction difference value, and obtain the corresponding time period weight according to the time period; A first duration unit, used to obtain a first duration according to a standard duration, a time period weight, an instruction similarity threshold, and an instruction difference value; A start time unit, used to obtain a start time according to the first duration and the end time; A time period unit, for obtaining a first time period according to a start time and a time; The completion unit is used to obtain the instruction execution completion of each input source in the first time period.

[0010] In a preferred embodiment, the completion degree unit includes an input quantity subunit, a completed quantity subunit and a completion degree acquisition subunit; An input quantity subunit, used to obtain and summarize multiple modal instructions corresponding to each input source in the first period to obtain the modal instruction quantity; The completion number subunit is used to obtain the number of modal instructions executed corresponding to each input source in the first period and mark it as the number of completed instructions; The completion degree acquisition subunit is used to acquire the corresponding instruction execution completion degree according to the number of modal instructions and the number of completed instructions corresponding to each input source.

[0011] In a preferred embodiment, the accuracy module includes a valid input unit, an accuracy unit, a weight table unit, an accurate interval matching unit, and an accurate weight matching unit; An effective input unit, used to obtain effective input information of multiple modal instructions of each input source within a first time period; An accuracy unit, used for obtaining the input accuracy of a corresponding input source according to valid input information; A weight table unit, used to obtain an accurate weight table, wherein the accurate weight table includes a plurality of input accurate intervals and an input accurate weight corresponding to each input accurate interval; An accuracy interval matching unit, used for obtaining a corresponding target input accuracy interval according to the input accuracy of each input source; The accurate weight matching unit is used to obtain the corresponding input accurate weight from the accurate weight table according to the target input accurate interval.

[0012] In a preferred embodiment, the emphasis module includes a time input unit, a time period input unit, a history instruction unit, a multiple time unit, a time quantity unit, a modal execution instruction unit, an execution quantity unit and an emphasis unit; A time input unit, used to obtain the input instruction time of the multimodal data input by the user; A time period input unit, used to obtain a corresponding input time period according to an input instruction time; A history instruction unit, used to obtain multiple modal history instructions corresponding to each input source within an input period; A multi-time unit, used to obtain multiple input times of multi-modal data input by a user within an input period; The time quantity unit is used to obtain the input time quantity corresponding to each input source based on multiple input times. A modal execution instruction unit, used to obtain the modal history instructions executed by the control system within the same input time according to multiple modal history instructions corresponding to each input source, and mark them as modal execution instructions; An execution quantity unit, used for obtaining the execution quantity of instructions corresponding to the input source according to the multiple modal execution instructions; The emphasis unit is used to obtain the instruction history emphasis of each input source according to the instruction execution quantity and input time quantity of each input source.

[0013] In a preferred solution, the matching execution module includes a function acquisition unit and a calculation unit; A function acquisition unit, used for acquiring a priority function; The computing unit is used to input the instruction execution completion degree, input accuracy weight and instruction history emphasis of each input source into the priority function, and mark the output result as the instruction priority value.

[0014] In a preferred embodiment, the system comprises a priority table unit, a maximum value unit and an execution unit; A priority table unit is used to sort the instruction priority values ​​corresponding to the multiple input sources in descending order to obtain an instruction priority table; The maximum value unit is used to select the modal instruction of the input source corresponding to the first instruction priority value in the instruction priority table in descending order according to the instruction priority values, and mark it as the maximum value modal instruction; The execution unit is used to execute the maximum value mode instruction.

[0015] And, a control device for realizing multimodal interaction using an AI big model, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the control system that uses the AI ​​big model to achieve multimodal interaction.

[0016] The technical effects achieved by the present invention are: The present invention, through the acquired multimodal input, can make intelligent judgments based on the user's usage habits and input accuracy, so that the user can interact with the system more naturally and smoothly in a complex environment. Since the control system can process multiple input sources at the same time and make judgments based on completion, accuracy and historical data, it can ensure the accuracy and reliability of the system's response through reasonable priority selection when the input is inconsistent. By focusing on the historical data of the module, it can gradually learn the user's input preferences, provide better customized support in future interactions, and enhance the personalization of the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a system module diagram provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0019] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0020] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The term "in a preferred embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0021] Secondly, the present invention is described in detail in conjunction with schematic diagrams. When describing the embodiments of the present invention in detail, for the convenience of explanation, the schematic diagrams are only examples and should not limit the scope of protection of the present invention.

[0022] Please see attached Figure 1 As shown, a control system for realizing multimodal interaction using an AI large model is provided, including: Multimodal module, completion module, accuracy module, emphasis module and matching execution module The multimodal module is used to obtain multimodal data input by the user and obtain the modal instruction of the corresponding input source according to the multimodal data; A completion module, used for obtaining instruction similarity according to modal instructions of multiple input sources, determining whether the instruction similarity meets a first preset condition, and if not, constructing a first time period and obtaining the instruction execution completion of each input source in the first time period; An accuracy module, used to obtain the input accuracy of multiple modal instructions of each input source in the first time period, and obtain a corresponding input accuracy weight according to the input accuracy of each input source; The emphasis module is used to obtain the input period of the multimodal data input by the user, obtain multiple modal historical instructions corresponding to each input source within the input period, and obtain the instruction history emphasis of each input source according to the multiple modal historical instructions of each input source; A matching execution module is used to obtain the instruction priority value of each input source according to the instruction execution completion, input accuracy weight and instruction history emphasis of each input source, obtain the modal instruction corresponding to the instruction priority value that meets the second preset condition, and execute the corresponding modal instruction.

[0023] As mentioned above, the multimodal module is responsible for obtaining multimodal data (such as voice, vision, touch, etc.) input by the user, and extracting corresponding modal instructions according to different input sources. It can process different types of input information and flexibly respond to complex interactive scenarios. When the control system receives instructions from multiple input sources, the completion module obtains the similarity of these instructions to determine whether they meet the set conditions. If there are differences between the instructions, the execution completion of the instructions of each input source in the specified time period is obtained through the specified time period, so as to evaluate the performance of different input sources in the task execution process. When determining the execution accuracy of each input source, the accuracy module will evaluate the accuracy of multiple modal instructions in the time period, and assign corresponding accuracy weights to each input source according to the accuracy results. The emphasis module analyzes the user's preferences in historical input according to the input time period, and obtains the historical emphasis of each input source. For example, some users may use voice input more frequently in specific scenarios, while others may not. Users prefer touch input. Based on these historical emphasis data, the emphasis on different input sources is adjusted. The matching execution module combines the command execution completion, input accuracy weight and command historical emphasis to calculate the command priority value of each input source, and selects the command with a higher priority value for execution according to the preset conditions to ensure that the modal command finally executed can reflect the user's main intentions and preferences. Through the obtained multi-modal input, it can make intelligent judgments based on the user's usage habits and input accuracy, so that users can interact with the system more naturally and smoothly in complex environments. Since the control system can process multiple input sources at the same time and make judgments based on completion, accuracy and historical data, it can ensure that the system responds accurately and reliably through reasonable priority selection when the input is inconsistent. Through the historical data of the emphasis module, it can gradually learn the user's input preferences, provide better customization support in future interactions, and enhance the personalization of user experience.

[0024] In a specific embodiment, the multimodal module includes an execution process unit, an execution vector unit, and an instruction similarity unit; An execution process unit, used for obtaining a corresponding instruction execution process according to the modal instruction of each input source; An execution vector unit, used to convert multiple instruction execution processes into instruction execution vectors of uniform specifications; The instruction similarity unit is used to obtain the instruction similarity between every two instruction execution processes according to multiple instruction execution vectors of the modal instructions of each input source.

[0025] As mentioned above, the execution process unit is responsible for obtaining the corresponding execution process according to the modal instruction of each input source. In order to unify the modal instruction execution processes of different input sources, the execution vector unit converts these instruction execution processes into instruction execution vectors of uniform specifications so that they can be compared and analyzed on the same dimension. After obtaining the instruction execution vectors of uniform specifications, the instruction similarity unit determines whether the instruction execution processes of different input sources are similar by calculating the similarity between every two instruction execution vectors, that is, comparing all the modal instructions one by one. The calculation formula of instruction similarity is: , where S represents the instruction similarity, k represents the number of instruction similarity between each two instruction execution processes, k=1,2,3…g, where, , f represents the number of input instruction execution processes, i represents the instruction execution vector number of every two instruction execution processes, i=1,2,3…n, It is represented as the i-th instruction execution vector of one of every two instruction execution processes. It is represented as the i-th instruction execution vector of another instruction execution process in every two instruction execution processes. By converting the execution processes of instructions of different modalities into a unified vector, it can effectively process heterogeneous data from different input sources. Whether it is voice input or gesture input, it can eventually be converted into a standardized vector representation, thereby improving data fusion and processing efficiency, and being able to accurately analyze the differences or similarities of instructions between different input sources.

[0026] In a specific embodiment, the multimodal module further includes a similarity threshold unit and a similarity judgment unit; A similarity threshold unit, used for obtaining an instruction similarity threshold; A similarity judgment unit, used to judge whether the instruction similarity exceeds an instruction similarity threshold; If the instruction similarity exceeds the instruction similarity threshold, it is determined that the modal instructions of the multiple input sources are the same; If the instruction similarity does not exceed the instruction similarity threshold, it is determined that there are differences in the modal instructions of the multiple input sources, and the instruction similarity that does not exceed the instruction similarity threshold is marked as an instruction difference value.

[0027] As mentioned above, the similarity threshold unit is responsible for setting a similarity threshold for the instruction similarity calculation. This threshold is used to determine whether the modal instructions of the input source are similar enough. The setting of the similarity threshold may be based on pre-training or dynamically adjusted by the system to ensure that it can adapt to user needs in different scenarios. The similarity judgment unit uses the similarity threshold to evaluate the instruction similarity. If the similarity of the modal instructions of multiple input sources exceeds the threshold, the instructions are determined to be the same, which means that the instructions from different input sources can be classified as the same operation, thereby improving processing efficiency. If the similarity does not reach the threshold, it is determined that there are differences between the instructions, and these differences are recorded as instruction difference values, which can flexibly respond to the differences in instructions from different input sources and ensure that they are merged only when they are similar enough, thereby improving the accuracy and reliability of instruction recognition.

[0028] In a specific embodiment, the multimodal module further includes a standard duration unit, an end time unit, a time period weight unit, a first duration unit, a start time unit, a time period unit, and a completion degree unit; Standard duration unit, used to obtain standard duration; An end time unit, used to obtain the time marked as the instruction difference value and mark it as the end time; A time period weight unit, used to obtain the time period marked as the instruction difference value, and obtain the corresponding time period weight according to the time period; A first duration unit, used to obtain a first duration according to a standard duration, a time period weight, an instruction similarity threshold, and an instruction difference value; A start time unit, used to obtain a start time according to the first duration and the end time; A time period unit, for obtaining a first time period according to a start time and a time; The completion unit is used to obtain the instruction execution completion of each input source in the first time period.

[0029] As mentioned above, the standard duration unit is used to obtain the standard duration preset by the system as a benchmark for subsequent analysis of the input time period. This refers to a reference time period that the control system defaults to during the multimodal interaction process. When the modal instruction of the input source is marked as "instruction difference value", the end time unit will capture this moment and mark it as the end time of the time period. This end time is used to divide the processing time range. The time period weight unit obtains the time period weight of the time period according to the time period where the instruction difference value is located. Different time periods may have different weights. The first duration unit calculates the first duration by comprehensively considering the standard duration, time period weight, instruction similarity threshold and instruction difference value. The calculation formula for the first duration is: , where Represents the first duration, Expressed as standard duration, Expressed as instruction difference value, It is represented as the instruction similarity threshold, a is represented as the time period weight, this duration defines the time span for the control system to evaluate the input within this specific time period, according to the calculated first duration and end time, the start time unit obtains the corresponding start time, which defines the time starting point of the system backtracking analysis, the time period unit determines the first time period of the system analysis according to the start time and end time, and the completion unit is responsible for obtaining the instruction execution completion of each input source within the first time period. Through the comprehensive calculation of the standard duration, time period weight and instruction difference value, the time window for input data analysis can be flexibly defined and adjusted, making the processing of multimodal instructions more refined and adaptable.

[0030] In a specific embodiment, the completion degree unit includes an input quantity subunit, a completion quantity subunit, and a completion degree acquisition subunit; An input quantity subunit, used to obtain and summarize multiple modal instructions corresponding to each input source in the first period to obtain the modal instruction quantity; The completion number subunit is used to obtain the number of modal instructions executed corresponding to each input source in the first period and mark it as the number of completed instructions; The completion degree acquisition subunit is used to acquire the corresponding instruction execution completion degree according to the number of modal instructions and the number of completed instructions corresponding to each input source.

[0031] As mentioned above, the input quantity subunit is responsible for acquiring all modal instructions of each input source within the first period, summarizing them, and calculating the total number of modal instructions of each input source within the period. Within the first period, the completion quantity subunit will identify the number of modal instructions successfully executed by each input source, and mark these successfully executed instructions as "instruction completion number". The completion degree acquisition subunit calculates the instruction execution completion degree of each input source based on the number of modal instructions summarized by the input quantity subunit and the number of instruction completions acquired by the completion quantity subunit. The completion degree is the ratio of the number of instructions actually completed to the total number of input instructions. The calculation formula for the instruction execution completion degree is: , where W represents the instruction execution completion degree, Expressed as the number of modal instructions, Expressed as the number of instructions completed, it accurately measures the execution performance of each input source.

[0032] In a specific embodiment, the accuracy module includes a valid input unit, an accuracy unit, a weight table unit, an accurate interval matching unit, and an accurate weight matching unit; An effective input unit, used to obtain effective input information of multiple modal instructions of each input source within a first time period; An accuracy unit, used for obtaining the input accuracy of a corresponding input source according to valid input information; A weight table unit, used to obtain an accurate weight table, wherein the accurate weight table includes a plurality of input accurate intervals and an input accurate weight corresponding to each input accurate interval; An accuracy interval matching unit, used for obtaining a corresponding target input accuracy interval according to the input accuracy of each input source; The accurate weight matching unit is used to obtain the corresponding input accurate weight from the accurate weight table according to the target input accurate interval.

[0033] As mentioned above, the effective input unit is used to obtain which modal instructions of each input source in the first time period are valid inputs, and filter out those instruction information that meet the requirements. According to the valid input information filtered out by the effective input unit, the accuracy unit calculates the input accuracy of each input source. The weight table unit is provided with an accurate weight table, which defines different input accuracy intervals and the weight values ​​corresponding to each interval. The accurate interval matching unit corresponds the input accuracy of each input source to the target input accuracy interval in the weight table. Once the accurate weight matching unit determines the target accuracy interval of the input source, it obtains the input accuracy weight corresponding to the interval from the weight table, and can accurately evaluate the input quality of each input source, and perform graded processing according to the performance of different input sources, so that high-accuracy inputs can obtain more weights, ensuring that these high-quality inputs are executed first.

[0034] In a specific embodiment, the emphasis module includes a time input unit, a time period input unit, a history instruction unit, a multiple time unit, a time quantity unit, a modal execution instruction unit, an execution quantity unit and an emphasis unit; A time input unit, used to obtain the input instruction time of the multimodal data input by the user; A time period input unit, used to obtain a corresponding input time period according to an input instruction time; A history instruction unit, used to obtain multiple modal history instructions corresponding to each input source within an input period; A multi-time unit, used to obtain multiple input times of multi-modal data input by a user within an input period; The time quantity unit is used to obtain the input time quantity corresponding to each input source based on multiple input times. A modal execution instruction unit, used to obtain the modal history instructions executed by the control system within the same input time according to multiple modal history instructions corresponding to each input source, and mark them as modal execution instructions; An execution quantity unit, used for obtaining the execution quantity of instructions corresponding to the input source according to the multiple modal execution instructions; The emphasis unit is used to obtain the instruction history emphasis of each input source according to the instruction execution quantity and input time quantity of each input source.

[0035] As mentioned above, the time input unit is used to obtain the specific input time of each input instruction in the multimodal data input by the user. The time period input unit classifies the instruction time provided by the time input unit into the corresponding input time period. The historical instruction unit obtains multiple modal historical instructions of each input source in the input time period to understand the input behavior and execution status of each input source in the past time period. The multi-time unit is used to obtain multiple input times of the multimodal data input by the user in the input time period. The time quantity unit calculates the number of input times of each input source in the time period according to the multiple input times in the multi-time unit. The modal execution instruction unit filters out the modal historical instructions executed by the control system within the same input time according to the multiple modal historical instructions obtained by the historical instruction unit, and marks them as modal execution instructions to distinguish between actually executed instructions and unexecuted instructions. The execution quantity unit calculates the actual instruction execution quantity of each input source in the time period according to the execution instructions filtered by the modal execution instruction unit. The emphasis unit obtains the instruction history emphasis of the input source by comparing the instruction execution quantity and the input time quantity of each input source. The calculation formula of the instruction history emphasis is: , where D represents the historical emphasis of instructions, Expressed as the number of instructions executed, It is expressed as the number of input times. Through the detailed division of time and time periods, it is possible to understand the user's operating habits and preferences in a specific time period. The emphasis unit derives the user's input tendency by combining historical instruction data and input time data.

[0036] In a specific embodiment, the matching execution module includes a function acquisition unit and a calculation unit; A function acquisition unit, used for acquiring a priority function; The computing unit is used to input the instruction execution completion degree, input accuracy weight and instruction history emphasis of each input source into the priority function, and mark the output result as the instruction priority value.

[0037] As mentioned above, the function acquisition unit is used to obtain the priority function set in the control system. The priority function is a mathematical model that comprehensively considers the instruction execution completion, input accuracy weight and instruction history emphasis. The priority function is A=(W / D)*P, where A represents the instruction priority value, W represents the instruction execution completion, P represents the input accuracy weight, and D represents the instruction history emphasis. The calculation unit inputs the instruction execution completion, input accuracy weight and instruction history emphasis of each input source into the priority function, and calculates the instruction priority value of each input source. The instruction priority value represents the relative importance or execution priority of the input source in the current control system operation, and can automatically balance the instruction completion, input accuracy and history emphasis of different input sources, thereby dynamically adjusting the priority of each input source to ensure that the control system resources are reasonably allocated. The priority function combines multiple key factors such as execution completion, accuracy and history emphasis, making the evaluation of the control system more comprehensive and able to cope with complex multimodal data interaction scenarios.

[0038] In a specific embodiment, the method comprises a priority table unit, a maximum value unit and an execution unit; A priority table unit is used to sort the instruction priority values ​​corresponding to the multiple input sources in descending order to obtain an instruction priority table; The maximum value unit is used to select the modal instruction of the input source corresponding to the first instruction priority value in the instruction priority table in descending order according to the instruction priority values, and mark it as the maximum value modal instruction; Execution unit, used to execute the maximum value mode instruction As mentioned above, the priority table unit is responsible for sorting the instruction priority values ​​of multiple input sources in order from large to small to form an "instruction priority table". This process ensures that the instructions of each input source are reasonably sorted according to their priority, and more important instructions are processed first. From the instruction priority table, the maximum value unit will automatically select the modal instruction with the highest priority value and mark it as the "maximum value modal instruction". This is the key instruction that the current control system needs to execute first. The execution unit receives the maximum value modal instruction and executes it immediately to ensure that the highest priority user input is responded to quickly. By sorting the instruction priority values, it can ensure that the most critical instructions are executed first in complex multi-modal input scenarios, thereby improving the overall response efficiency. Through the dynamic sorting and maximum value selection of the priority table, the control system is adaptive and can adjust the instruction execution order in real time as the input source changes, meeting the diverse needs in complex interactive scenarios.

[0039] And, a control device for realizing multimodal interaction using an AI big model, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the control system that uses the AI ​​big model to achieve multimodal interaction.

[0040] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A control system that uses a large AI model to achieve multimodal interaction, characterized in that: Includes multimodal module, completion module, accuracy module, emphasis module and matching execution module The multimodal module is used to obtain multimodal data input by the user and obtain the modal instruction of the corresponding input source according to the multimodal data; A completion module, used for obtaining instruction similarity according to modal instructions of multiple input sources, determining whether the instruction similarity meets a first preset condition, and if not, constructing a first time period and obtaining the instruction execution completion of each input source in the first time period; An accuracy module, used to obtain the input accuracy of multiple modal instructions of each input source in the first time period, and obtain a corresponding input accuracy weight according to the input accuracy of each input source; The emphasis module is used to obtain the input period of the multimodal data input by the user, obtain multiple modal historical instructions corresponding to each input source within the input period, and obtain the instruction history emphasis of each input source according to the multiple modal historical instructions of each input source; A matching execution module is used to obtain the instruction priority value of each input source according to the instruction execution completion, input accuracy weight and instruction history emphasis of each input source, obtain the modal instruction corresponding to the instruction priority value that meets the second preset condition, and execute the corresponding modal instruction.

2. According to claim 1, the control system for realizing multimodal interaction using an AI large model is characterized in that: The multimodal module includes an execution process unit, an execution vector unit, and an instruction similarity unit; An execution process unit, used for obtaining a corresponding instruction execution process according to a modal instruction of each input source; An execution vector unit, used to convert multiple instruction execution processes into instruction execution vectors of uniform specifications; The instruction similarity unit is used to obtain the instruction similarity between every two instruction execution processes according to multiple instruction execution vectors of the modal instructions of each input source.

3. The control system for realizing multimodal interaction using an AI large model according to claim 2 is characterized in that: The multimodal module also includes a similarity threshold unit and a similarity judgment unit; A similarity threshold unit, used for obtaining an instruction similarity threshold; A similarity judgment unit, used to judge whether the instruction similarity exceeds an instruction similarity threshold; If the instruction similarity exceeds the instruction similarity threshold, it is determined that the modal instructions of the multiple input sources are the same; If the instruction similarity does not exceed the instruction similarity threshold, it is determined that there are differences in the modal instructions of the multiple input sources, and the instruction similarity that does not exceed the instruction similarity threshold is marked as an instruction difference value.

4. The control system for realizing multimodal interaction using an AI large model according to claim 3 is characterized in that: The multimodal module also includes a standard duration unit, an end time unit, a time period weight unit, a first duration unit, a start time unit, a time period unit, and a completion degree unit; Standard duration unit, used to obtain standard duration; An end time unit, used to obtain the time marked as the instruction difference value and mark it as the end time; A time period weight unit, used to obtain the time period where the time marked as the instruction difference value is located, and obtain the corresponding time period weight according to the time period; A first duration unit, used to obtain a first duration according to a standard duration, a time period weight, an instruction similarity threshold, and an instruction difference value; A start time unit, used to obtain a start time according to the first duration and the end time; A time period unit, for obtaining a first time period according to a start time and a time; The completion unit is used to obtain the instruction execution completion of each input source in the first time period.

5. The control system for realizing multimodal interaction using an AI large model according to claim 4 is characterized in that: The completion degree unit includes an input quantity subunit, a completed quantity subunit and a completion degree acquisition subunit; An input quantity subunit, used to obtain and summarize multiple modal instructions corresponding to each input source in the first period to obtain the modal instruction quantity; The completion number subunit is used to obtain the number of modal instructions executed corresponding to each input source in the first period, and mark it as the number of completed instructions; The completion degree acquisition subunit is used to acquire the corresponding instruction execution completion degree according to the number of modal instructions and the number of completed instructions corresponding to each input source.

6. The control system for realizing multimodal interaction using an AI big model according to claim 1, characterized in that: The accuracy module includes a valid input unit, an accuracy unit, a weight table unit, an accurate interval matching unit and an accurate weight matching unit; An effective input unit, used to obtain effective input information of multiple modal instructions of each input source within a first time period; An accuracy unit, used for obtaining the input accuracy of a corresponding input source according to valid input information; A weight table unit, used to obtain an accurate weight table, wherein the accurate weight table includes a plurality of input accurate intervals and an input accurate weight corresponding to each input accurate interval; An accuracy interval matching unit, used for obtaining a corresponding target input accuracy interval according to the input accuracy of each input source; The accurate weight matching unit is used to obtain the corresponding input accurate weight from the accurate weight table according to the target input accurate interval.

7. The control system for realizing multimodal interaction using an AI big model according to claim 1, characterized in that: The emphasis module includes a time input unit, a time period input unit, a history instruction unit, a multiple time unit, a time quantity unit, a modal execution instruction unit, an execution quantity unit and an emphasis unit; A time input unit, used to obtain the input instruction time of the multimodal data input by the user; A time period input unit, used to obtain a corresponding input time period according to an input instruction time; A history instruction unit, used to obtain multiple modal history instructions corresponding to each input source within an input period; A multi-time unit, used to obtain multiple input times of multi-modal data input by a user within an input period; The time quantity unit is used to obtain the input time quantity corresponding to each input source based on multiple input times. A modal execution instruction unit, used to obtain the modal history instructions executed by the control system within the same input time according to multiple modal history instructions corresponding to each input source, and mark them as modal execution instructions; An execution quantity unit, used for obtaining the execution quantity of instructions corresponding to the input source according to the multiple modal execution instructions; The emphasis unit is used to obtain the instruction history emphasis of each input source according to the instruction execution quantity and input time quantity of each input source.

8. The control system for realizing multimodal interaction using an AI big model according to claim 1, characterized in that: The matching execution module includes a function acquisition unit and a calculation unit; A function acquisition unit, used for acquiring a priority function; The computing unit is used to input the instruction execution completion degree, input accuracy weight and instruction history emphasis of each input source into the priority function, and mark the output result as the instruction priority value.

9. The control system for realizing multimodal interaction using an AI big model according to claim 8, characterized in that: The method comprises a priority table unit, a maximum value unit and an execution unit; A priority table unit is used to sort the instruction priority values ​​corresponding to the multiple input sources in descending order to obtain an instruction priority table; The maximum value unit is used to select the modal instruction of the input source corresponding to the first instruction priority value in the instruction priority table in descending order according to the instruction priority values, and mark it as the maximum value modal instruction; The execution unit is used to execute the maximum value mode instruction.

10. A control device for realizing multimodal interaction using a large AI model, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the control system for realizing multimodal interaction using an AI big model as described in any one of claims 1 to 9.

Images