Multi-modal data processing method and device based on thinking chain, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a multi-modal data processing method and device based on a thinking chain, equipment and a medium. On one hand, a planning agent is called to perform thinking chain splitting according to preference data and input data so as to reasonably split tasks according to user requirements and preferences; on the other hand, other secondary agents are called in sequence to carry out context analysis based on all the feedback data and the user input data to obtain analysis results, corresponding task nodes are executed according to the analysis results and the preference data, and when one task node is executed, the user feedback is waited and then the response is continued; the problems that a user cannot modify, too many steps are executed at a time, waiting time is too long, consumed computing power is too much, result deviation is caused and the like are solved, meanwhile, user requirements are fully considered due to context analysis, and the final multi-modal data processing result better meets the actual processing requirements of the user.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a multimodal data processing method, device, equipment and medium based on a chain of thought. Background Art

[0002] Currently, whether in the field of graphic design, game design, or architectural design, designers need to operate corresponding software according to the requirements put forward by customers to gradually process multimodal data, so as to meet the design requirements of customers.

[0003] However, a design task often includes multiple fine-grained subtasks, and the design process is complex. However, traditional design software can only complete multimodal data processing at one time according to the information input by users. If customers want to make modifications, they need to operate on the basis of the already processed multimodal data, which not only wastes time but also has a high operation difficulty. Summary of the Invention

[0004] In view of the above, it is necessary to provide a multimodal data processing method, device, equipment and medium based on a chain of thought, aiming to solve the problems of high operation difficulty, low processing efficiency, and poor processing results in multimodal data processing.

[0005] A multimodal data processing method based on a chain of thought, the multimodal data processing method based on a chain of thought includes:

[0006] In response to a processing instruction for target multimodal data triggered by a target user, obtain the preference data and input data of the target user;

[0007] Call a planning agent to perform a chain of thought split according to the preference data and the input data to obtain a target chain of thought; wherein, the target chain of thought includes at least one task node and a sub-agent corresponding to each task node;

[0008] Obtain the first task node to be executed in the target chain of thought, and call the sub-agent matching the first task node as the first agent;

[0009] Use the first agent to process the target multimodal data based on the input data and the preference data to obtain a processing result, and feedback the processing result to the target user based on the preference data, and receive feedback data of the target user on the processing result;

[0010] Call other sub-agents in sequence to perform context analysis based on all the feedback data and the input data to obtain an analysis result, and execute the corresponding task node according to the analysis result and the preference data;

[0011] When all the task nodes in the target thought chain are executed, obtain the current processing result and feedback the current processing result to the target user;

[0012] When receiving the confirmation signal of the target user for the current processing result, determine the current processing result as the processing result of the target multimodal data.

[0013] According to a preferred embodiment of the present invention, the method further includes:

[0014] When detecting a registration instruction initiated by a user to a specified multimodal data processing platform, send a prompt message for the user to select user preferences.

[0015] According to a preferred embodiment of the present invention, before the call planning agent splits the thought chain according to the preference data and the input data, the method further includes:

[0016] Obtain a large language model as an initial model;

[0017] Construct a training set; wherein each training sample in the training set is marked with a corresponding task node and agent;

[0018] Use the training set to train the initial model until the accuracy of the initial model reaches the configured accuracy, and then stop training;

[0019] Determine the currently trained model as the planning agent.

[0020] According to a preferred embodiment of the present invention, the planning agent and the sub-agent are agents accumulated through continuous calls and optimized training in the corresponding industry background;

[0021] The sub-agent includes a large language model, an algorithm model matching the required functions of the task, and a third-party model called through a third-party API interface.

[0022] According to a preferred embodiment of the present invention, the feedback of the processing result to the target user based on the preference data includes:

[0023] Feedback the processing result to the target user according to the language style, language type, and feedback method marked in the preference data.

[0024] According to a preferred embodiment of the present invention, the method further includes:

[0025] For any agent among the planning agent and the sub-agent, during the execution process, perform intent recognition on the input data and / or all current feedback data to obtain a recognition result;

[0026] When the recognition result is clear and meets the execution requirements, call any of the agents to execute the corresponding task node; or

[0027] When the recognition result is unclear and / or does not meet the execution requirements, call any of the agents to send a guiding prompt to the target user to supplement the missing information according to the information required when executing the corresponding task node; or

[0028] When the recognition result shows that the target thought chain is incorrect, call the planning agent to adjust the target thought chain according to the recognition result.

[0029] According to a preferred embodiment of the present invention, after determining the current processing result as the processing result of the target multimodal data, the method further includes:

[0030] Perform a tagging process on the processing result;

[0031] Wherein, the tag of the processing result is used to indicate that the processing result is generated based on the specified multimodal data processing platform.

[0032] A multimodal data processing device based on a thought chain, the multimodal data processing device based on a thought chain includes:

[0033] An acquisition unit, configured to acquire the preference data and input data of the target user in response to a processing instruction for target multimodal data triggered by the target user;

[0034] A splitting unit, configured to call a planning agent to split a thought chain according to the preference data and the input data to obtain a target thought chain; wherein, the target thought chain includes at least one task node and a secondary agent corresponding to each task node;

[0035] A calling unit, configured to obtain the first task node to be executed in the target thought chain and call the secondary agent matching the first task node as the first agent;

[0036] A processing unit, configured to use the first agent to process the target multimodal data based on the input data and the preference data to obtain a processing result, and feedback the processing result to the target user based on the preference data, and receive feedback data of the target user on the processing result;

[0037] An execution unit, configured to sequentially call other secondary agents to perform context analysis based on all the feedback data and the input data to obtain an analysis result, and execute corresponding task nodes according to the analysis result and the preference data;

[0038] A feedback unit, configured to obtain a current processing result when all task nodes in the target thought chain are executed, and feedback the current processing result to the target user;

[0039] A determination unit, configured to determine the current processing result as the processing result of the target multimodal data when receiving a confirmation signal of the target user for the current processing result.

[0040] A computer device, comprising:

[0041] A memory storing at least one instruction; and

[0042] A processor, configured to execute the instructions stored in the memory to implement the multimodal data processing method based on a thought chain.

[0043] A computer-readable storage medium storing at least one instruction, where the at least one instruction is executed by a processor in a computer device to implement the multimodal data processing method based on a thought chain.

[0044] It can be seen from the above technical solutions that, on the one hand, the planning agent is called to split the thought chain according to the preference data and the input data, so as to reasonably split tasks according to the user's needs and preferences; on the other hand, other secondary agents are called in turn to perform context analysis based on all the feedback data and the user input data to obtain an analysis result, and execute the corresponding task nodes according to the analysis result and the preference data. Wait for the user's feedback before continuing to respond for each executed task node, avoiding problems such as the user being unable to modify, too many steps being executed at one time resulting in too long waiting time, excessive consumption of computing power, and deviation of the result. At the same time, since the context analysis fully considers the user's needs, the final multimodal data processing result better meets the actual processing needs of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flowchart of a preferred embodiment of the multimodal data processing method based on a thought chain of the present invention.

[0046] Figure 2 is a functional module diagram of a preferred embodiment of the multimodal data processing device based on a thought chain of the present invention.

[0047] Figure 3 is a structural schematic diagram of a computer device of a preferred embodiment for implementing the multimodal data processing method based on a thought chain of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] As shown Figure 1 in the figure, it is a flowchart of a preferred embodiment of the multi-modal data processing method based on the chain of thought of the present invention. According to different requirements, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0050] The multi-modal data processing method based on the chain of thought is applied to one or more computer devices. The computer device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0051] The computer device can be any electronic product that can interact with users, such as personal computers, tablets, smart phones, personal digital assistants (PDAs), game consoles, Internet Protocol Televisions (IPTVs), smart wearable devices, etc.

[0052] The computer device may also include network devices and / or user devices. Among them, the network devices include, but are not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing (Cloud Computing).

[0053] The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0054] Among them, artificial intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0055] The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0056] The network where the computer device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0057] S10. In response to a processing instruction for target multimodal data triggered by a target user, obtain the preference data and input data of the target user.

[0058] In this embodiment, the target user can be a user with design needs such as a designer or a customer.

[0059] In this embodiment, the target multimodal data can be line drawings, line graphs, basic images, 3D models, videos that need to be processed, or various types of data such as text and models.

[0060] In this embodiment, the processing instruction can be triggered by a designer on a specified multimodal data processing platform according to actual design needs.

[0061] In this embodiment, the preference data may include, but is not limited to: user background (such as game design background, graphic design background, architectural design background, etc.), language style preferred by the user (such as chat mode with emoticons, cute style chat mode, professional style chat mode, encouraging style chat mode, etc.), language preferred by the user (such as Chinese, English, etc.), chat mode preferred by the user (such as text chat, voice chat, etc.).

[0062] In this embodiment, the input data can reflect the specific processing requirements of the target user, such as "Please add a forest background to this picture".

[0063] In this embodiment, the method further includes:

[0064] When detecting a registration instruction initiated by a user to a specified multimodal data processing platform, send a prompt message for the user to select user preferences.

[0065] Through the above embodiments, it is possible to guide the user to input preference data during user registration for reference during subsequent multimodal data processing, facilitate the invocation of an intelligent agent that better conforms to the user's preferences and background, and thus make the processing result more in line with the user's preferences.

[0066] S11. Invoke the planning agent to perform a thought chain split according to the preference data and the input data, obtaining a target thought chain; wherein the target thought chain includes at least one task node and a sub-agent corresponding to each task node.

[0067] In this embodiment, before the planning agent performs the thought chain split according to the preference data and the input data, the method further includes:

[0068] Obtain a large language model as the initial model;

[0069] Construct a training set; wherein each training sample in the training set is labeled with a corresponding task node and an agent.

[0070] Use the training set to train the initial model until the accuracy rate of the initial model reaches the configured accuracy rate, and then stop the training.

[0071] Determine the currently trained model as the planning agent.

[0072] Among them, the configured accuracy rate can be determined according to actual needs, such as 95%.

[0073] Through the above embodiments, the planning agent can be accurately trained according to the labels for subsequent reasonable task planning based on the content input by the user.

[0074] In this embodiment, the planning agent and the sub-agents are agents accumulated through continuous invocation and optimization training in the corresponding industry background.

[0075] That is to say, through continuous use, each agent is also continuously improved, and each agent continuously accumulates knowledge in the corresponding industry background, so as to better meet the design requirements in the corresponding industry background, and the generated content will also more meet the user's needs.

[0076] In this embodiment, the sub-agents include a large language model, an algorithm model that matches the functions required by the task, and a third-party model called through a third-party API (Application Programming Interface).

[0077] For example: the sub-agents may include a chat model, an analysis model, an evaluation model, an interrogation model, a multi-modal data processing model, a search model, a picture enhancement model, a video generation model, an audio generation model, a picture generation model, a text generation model, a 3D generation model, and a third-party model called through a third-party API interface (such as a fine-tuned picture generation model, a 3D generation model), etc.

[0078] S12. Obtain the first task node to be executed in the target thought chain, and call the secondary agent that matches the first task node as the first agent.

[0079] For example, when the tasks in the target thought chain sequentially include: generating an image based on a line drawing, generating a 3D model, generating a topological surface, and overall processing, the first task node is to call a multi-modal data generation model as the first agent to generate an image based on the line drawing input by the user.

[0080] S13. Use the first agent to process the target multi-modal data based on the input data and the preference data to obtain a processing result, and feedback the processing result to the target user based on the preference data, and receive the feedback data of the target user on the processing result.

[0081] In this embodiment, the feedback of the processing result to the target user based on the preference data includes:

[0082] Feedback the processing result to the target user according to the language style, language type, and feedback method marked in the preference data.

[0083] For example, continuing with the above example, use a multi-modal data generation model as the first agent to generate a cute-style image based on the line drawing input by the user, and use emoticons as language embellishments, and feedback the multi-modal data processing result to the user in the form of Chinese text to ask the user whether they are satisfied with the generated image and what needs to be modified.

[0084] S14. Sequentially call other secondary agents to perform context analysis based on all the feedback data and the input data to obtain an analysis result, and execute the corresponding task nodes according to the analysis result and the preference data.

[0085] In this embodiment, through context analysis, the user's intention can be fully understood, and targeted multi-modal data processing can be performed, so that the multi-modal data processing result can better meet the user's needs.

[0086] In this embodiment, only one step is executed each time and the user's further feedback is awaited, rather than automatically completing all the multi-modal data processing processes. Since the generated image can be adjusted in a timely manner according to the user's feedback on the generation result, problems such as the user being unable to modify, waiting too long due to too many steps being executed at once, consuming too much computing power, and causing deviations in the result can be avoided.

[0087] S15. When all the task nodes in the target thought chain have been executed, obtain the current processing result and feedback the current processing result to the target user.

[0088] In this embodiment, the method further includes:

[0089] For any of the planning agent and the sub-agent, during the execution process, perform intent recognition on the input data and / or all current feedback data to obtain a recognition result;

[0090] When the recognition result is clear and meets the execution requirements, call the any agent to execute the corresponding task node; or

[0091] When the recognition result is unclear and / or does not meet the execution requirements, call the any agent to send a guiding prompt for supplementing missing information to the target user according to the information required when executing the corresponding task node; or

[0092] When the recognition result shows that the target thinking chain is incorrect, call the planning agent to adjust the target thinking chain according to the recognition result.

[0093] Through the above embodiments, it is possible to perform intent recognition based on all user input content (including all user feedback data), so as to judge whether to continue executing the remaining task nodes according to the user's intent, or to guide the user to input missing information to execute the current task node, or to adjust the thinking chain, so as to better respond according to the user's actual intent, improving the effect of multimodal data processing.

[0094] S16. When receiving the confirmation signal of the target user for the current processing result, determine the current processing result as the processing result of the target multimodal data.

[0095] For example: When receiving the processing instruction for multimodal data triggered by a designer, first, the planning agent splits the task chain according to the user input and preferences. For example, the multimodal data design task, as a creative process, may include multiple sub-task nodes such as finding references, analyzing, hand-drawing sketches, rendering, P-picture (an electronic picture processing method, including picture processing, beautification, and modification, etc.), and writing design descriptions. Then, perform context analysis based on the interaction with the user to sequentially call the corresponding sub-agents to execute each task node, and finally generate the multimodal data content required by the designer.

[0096] In this embodiment, after determining the current processing result as the processing result of the target multimodal data, the method further includes:

[0097] Perform tagging processing on the processing result;

[0098] Among them, the tag of the processing result is used to indicate that the processing result is generated based on the specified multimodal data processing platform.

[0099] Through the above embodiments, the processing results can be tagged to mark the processing sources of multimodal data, effectively avoiding security issues such as theft and misappropriation of multimodal data.

[0100] This embodiment can be used to support cross-stage and cross-modal creative activities, including but not limited to link such as reference material search, content analysis, sketch generation, image processing, visual rendering, copywriting, and creative summary. Considering that most current image generation or content creation platforms only focus on a certain link in the creation process (such as AI drawing or image restoration) and lack the understanding and support for the complete creative process. Therefore, by constructing an intelligent collaborative multimodal data processing system based on the chain of thought mechanism, this embodiment can identify the creation stage, understand the context intention, and assist users across modalities to complete multiple stages from reference retrieval, creative analysis to content generation and optimization, effectively improving the creation efficiency and creative quality.

[0101] It can be seen from the above technical solutions that, on the one hand, the planning agent is called to split the chain of thought according to the preference data and input data to reasonably split tasks according to the user's needs and preferences; on the other hand, other secondary agents are called in turn to perform context analysis based on all feedback data and user input data to obtain analysis results, and execute the corresponding task nodes according to the analysis results and preference data. Wait for the user's feedback before continuing to respond for each executed task node, avoiding problems such as the user being unable to modify, too many steps being executed at one time resulting in too long waiting time, excessive consumption of computing power, and deviation of the results. At the same time, because the context analysis fully considers the user's needs, the final multimodal data processing results better meet the actual processing needs of the user.

[0102] As Figure 2 shown, it is a functional module diagram of a preferred embodiment of the multimodal data processing device based on the chain of thought of the present invention. The multimodal data processing device 11 based on the chain of thought includes an acquisition unit 110, a splitting unit 111, a calling unit 112, a processing unit 113, an execution unit 114, a feedback unit 115, and a determination unit 116. The module / unit referred to in the present invention means a series of computer program segments that can be executed by a processor and can complete fixed functions, and are stored in a memory. In this embodiment, the functions of each module / unit will be described in detail in the subsequent embodiments.

[0103] Among them, the acquisition unit 110 is configured to obtain the preference data and input data of the target user in response to a processing instruction for target multimodal data triggered by the target user;

[0104] The splitting unit 111 is used to call a planning agent to perform a thought chain split according to the preference data and the input data to obtain a target thought chain; wherein, the target thought chain includes at least one task node and a sub-agent corresponding to each task node.

[0105] The calling unit 112 is used to obtain the first task node to be executed in the target thought chain and call the sub-agent matching the first task node as the first agent.

[0106] The processing unit 113 is used to use the first agent to process the target multimodal data based on the input data and the preference data to obtain a processing result, and feedback the processing result to the target user based on the preference data, and receive the feedback data of the target user on the processing result.

[0107] The execution unit 114 is used to sequentially call other sub-agents to perform context analysis based on all the feedback data and the input data to obtain an analysis result, and execute the corresponding task node according to the analysis result and the preference data.

[0108] The feedback unit 115 is used to obtain the current processing result when all the task nodes in the target thought chain are executed, and feedback the current processing result to the target user.

[0109] The determination unit 116 is used to determine the current processing result as the processing result of the target multimodal data when receiving the confirmation signal of the target user for the current processing result.

[0110] It can be seen from the above technical solutions that, on the one hand, a planning agent is called to perform a thought chain split according to the preference data and the input data to reasonably split tasks according to the user's needs and preferences; on the other hand, other sub-agents are sequentially called to perform context analysis based on all the feedback data and the user input data to obtain an analysis result, and the corresponding task node is executed according to the analysis result and the preference data. Each time a task node is executed, it waits for the user's feedback before continuing to respond, avoiding problems such as the user being unable to modify, too many steps being executed at one time resulting in too long waiting time, excessive consumption of computing power, and deviation of the result. At the same time, because context analysis fully considers the user's needs, the final processing result of the multimodal data better meets the actual processing needs of the user.

[0111] As Figure 3 shown, it is a schematic structural diagram of a computer device of a preferred embodiment for implementing the multimodal data processing method based on a thought chain of the present invention.

[0112] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrows in the figure represent the bus). It may also include a computer program stored in the memory 12 and executable on the processor 13, such as a multi-modal data processing program based on the chain of thought.

[0113] Those skilled in the art can understand that the schematic diagram is only an example of the computer device 1 and does not constitute a limitation on the computer device 1. The computer device 1 can be either a bus structure or a star structure. The computer device 1 may also include more or fewer other hardware or software than shown in the figure, or different component arrangements. For example, the computer device 1 may also include input / output devices, network access devices, etc.

[0114] It should be noted that the computer device 1 is only an example. Other existing or future electronic products that can be adapted to the present invention should also be included within the protection scope of the present invention and are hereby incorporated by reference.

[0115] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 may be an internal storage unit of the computer device 1, such as the mobile hard disk of the computer device 1. In other embodiments, the memory 12 may also be an external storage device of the computer device 1, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device 1. Further, the memory 12 may also include both an internal storage unit and an external storage device of the computer device 1. The memory 12 can be used not only to store application software installed on the computer device 1 and various types of data, such as the code of the multi-modal data processing program based on the chain of thought, but also to temporarily store data that has been output or will be output.

[0116] In some embodiments, the processor 13 may be composed of an integrated circuit. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips, etc. The processor 13 is the control core of the computer device 1. It uses various interfaces and circuits to connect all components of the entire computer device 1. By running or executing programs or modules stored in the memory 12 (such as executing a multi-modal data processing program based on the chain of thought, etc.), and by calling the data stored in the memory 12, it executes various functions of the computer device 1 and processes data.

[0117] The processor 13 executes the operating system of the computer device 1 and various installed application programs. The processor 13 executes the application programs to implement the steps in the above-mentioned embodiments of various multi-modal data processing methods based on the chain of thought. For example Figure 1 the steps shown.

[0118] Exemplarily, the computer program may be divided into one or more modules / units. The one or more modules / units are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an acquisition unit 110, a splitting unit 111, a calling unit 112, a processing unit 113, an execution unit 114, a feedback unit 115, and a determination unit 116.

[0119] The above-mentioned integrated units implemented in the form of software function modules can be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of the multi-modal data processing method based on the chain of thought described in each embodiment of the present invention.

[0120] If the modules / units integrated in the computer device 1 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware devices. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented.

[0121] Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory, etc.

[0122] Furthermore, the computer-readable storage medium mainly includes a storage program area and a storage data area. Among them, the storage program area can store an operating system, application programs required for at least one function, etc.; the storage data area can store data created according to the use of the blockchain node, etc.

[0123] The blockchain referred to in the present invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a string of data blocks generated by using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity (anti-counterfeiting) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, an application service layer, etc.

[0124] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, in Figure 3 it is only represented by a straight line, but it does not mean that there is only one bus or one type of bus. The bus is set to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0125] Although not shown, the computer device 1 may further include a power source (such as a battery) for powering each component. Preferably, the power source can be logically connected to the at least one processor 13 through a power management device, so as to implement functions such as charging management, discharging management, and power consumption management through the power management device. The power source may also include any components such as one or more DC or AC power sources, a recharge device, a power failure detection circuit, a power converter or inverter, and a power status indicator. The computer device 1 may also include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0126] Furthermore, the computer device 1 may further include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the computer device 1 and other computer devices.

[0127] Optionally, the computer device 1 may further include a user interface. The user interface may be a display (Display), an input unit (such as a keyboard), and optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display the information processed in the computer device 1 and to display a visual user interface.

[0128] It should be understood that the above embodiments are only for illustration purposes and are not limited by this structure in the scope of the patent application.

[0129] Those skilled in the art can understand that Figure 3 the shown structure does not limit the computer device 1, and it may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0130] In combination with Figure 1 , the memory 12 in the computer device 1 stores multiple instructions to implement a multi-modal data processing method based on a chain of thought, and the processor 13 can execute the multiple instructions to implement:

[0131] In response to a processing instruction for target multi-modal data triggered by a target user, obtain the preference data and input data of the target user;

[0132] Invoke the planning agent to perform a chain-of-thought split according to the preference data and the input data to obtain a target chain of thought; wherein, the target chain of thought includes at least one task node and a sub-agent corresponding to each task node.

[0133] Obtain the first task node to be executed in the target chain of thought, and invoke the sub-agent that matches the first task node as the first agent.

[0134] Use the first agent to process the target multimodal data based on the input data and the preference data to obtain a processing result, and feedback the processing result to the target user based on the preference data, and receive the feedback data of the target user on the processing result.

[0135] Successively invoke other sub-agents to perform context analysis based on all the feedback data and the input data to obtain an analysis result, and execute the corresponding task node according to the analysis result and the preference data.

[0136] When all the task nodes in the target chain of thought have been executed, obtain the current processing result and feedback the current processing result to the target user.

[0137] When receiving the confirmation signal of the target user for the current processing result, determine the current processing result as the processing result of the target multimodal data.

[0138] Specifically, for the specific implementation method of the above instructions by the processor 13, reference can be made to Figure 1 the description of the relevant steps in the corresponding embodiment, which will not be elaborated here.

[0139] It should be noted that all the data involved in this case are legally obtained. The non-company software tools or components appearing in the embodiments of the present application are only for illustrative introduction and do not represent actual use.

[0140] In several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0141] The present invention can be used in numerous general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0142] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, in each embodiment of the present invention, the various functional modules can be integrated in one processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of a combination of hardware and software functional modules.

[0144] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms.

[0145] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any reference signs in the claims should not be regarded as limiting the claimed rights.

[0146] In addition, it is obvious that the term "including" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described in the present invention can also be implemented by one unit or device through software or hardware. Words such as "first" and "second" are used to denote names and do not represent any specific order.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal data processing method based on the chain of thought, characterized in that The multimodal data processing method based on the chain of thought includes: In response to a processing instruction for target multimodal data triggered by a target user, obtain the preference data and input data of the target user; Call a planning agent to perform a chain of thought split according to the preference data and the input data to obtain a target chain of thought; wherein, the target chain of thought includes at least one task node and a sub-agent corresponding to each task node; Obtain the first task node to be executed in the target chain of thought, and call the sub-agent matching the first task node as the first agent; Use the first agent to process the target multimodal data based on the input data and the preference data to obtain a processing result, and feedback the processing result to the target user based on the preference data, and receive feedback data of the target user on the processing result; Call other sub-agents in sequence to perform context analysis on all feedback data and the input data to obtain an analysis result, and execute corresponding task nodes according to the analysis result and the preference data; When all task nodes in the target chain of thought are executed, obtain the current processing result and feedback the current processing result to the target user; When receiving a confirmation signal from the target user for the current processing result, determine the current processing result as the processing result of the target multimodal data.

2. The multimodal data processing method based on the chain of thought according to claim 1, wherein The method further includes: When detecting a registration instruction initiated by a user to a specified multimodal data processing platform, send a prompt message for the user to select user preferences.

3. The multimodal data processing method based on the chain of thought according to claim 1, characterized in that Before the call to the planning agent to perform a chain of thought split according to the preference data and the input data, the method further includes: Obtain a large language model as an initial model; Construct a training set; wherein each training sample in the training set is labeled with a corresponding task node and agent; Use the training set to train the initial model until the accuracy of the initial model reaches the configured accuracy, and then stop training; Determine the currently trained model as the planning agent.

4. The multimodal data processing method based on the chain of thought according to claim 1, wherein: The planning agent and the sub-agents are agents accumulated through continuous calls and optimized training in the corresponding industry background; The sub-agents include a large language model, an algorithm model matching the required functions of the task, and a third-party model called through a third-party API interface.

5. The multimodal data processing method based on the chain of thought according to claim 1, characterized in that, The feedback of the processing result to the target user based on the preference data includes: Feedback the processing result to the target user according to the language style, language, and feedback method marked in the preference data.

6. The multimodal data processing method based on the chain of thought according to claim 1, wherein The method further includes: For any agent among the planning agent and the sub-agents, during the execution process, perform intent recognition on the input data and / or all current feedback data to obtain a recognition result; When the recognition result is clear and meets the execution requirements, call the any agent to execute the corresponding task node; or When the recognition result is unclear and / or does not meet the execution requirements, call any of the agents to send a guiding prompt to the target user to supplement the missing information according to the information required when executing the corresponding task node; or When the recognition result shows that the target thought chain is incorrect, call the planning agent to adjust the target thought chain according to the recognition result.

7. The multimodal data processing method based on the chain of thought according to claim 2, characterized in that After determining the current processing result as the processing result of the target multimodal data, the method further includes: Perform tagging processing on the processing result; Among them, the tag of the processing result is used to indicate that the processing result is generated based on the specified multimodal data processing platform.

8. A multimodal data processing device based on a chain of thought, characterized in that The multimodal data processing device based on the thought chain includes: An acquisition unit, configured to acquire the preference data and input data of the target user in response to a processing instruction for target multimodal data triggered by the target user; A splitting unit, configured to call a planning agent to split the thought chain according to the preference data and the input data to obtain a target thought chain; wherein, the target thought chain includes at least one task node and a sub-agent corresponding to each task node; An invocation unit, configured to obtain the first task node to be executed in the target thought chain and call the sub-agent matching the first task node as the first agent; A processing unit, configured to use the first agent to process the target multimodal data based on the input data and the preference data to obtain a processing result, and feedback the processing result to the target user based on the preference data, and receive feedback data of the target user on the processing result; An execution unit, configured to sequentially call other sub-agents to perform context analysis on all feedback data and the input data to obtain an analysis result, and execute corresponding task nodes according to the analysis result and the preference data; A feedback unit, configured to obtain the current processing result when all task nodes in the target thought chain are executed, and feedback the current processing result to the target user; A determination unit, configured to determine the current processing result as the processing result of the target multimodal data when receiving a confirmation signal of the target user for the current processing result.

9. A computer device, characterized in that, The computer device includes: A memory, storing at least one instruction; and A processor, executing the instructions stored in the memory to implement the multimodal data processing method based on the thought chain according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: At least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in the computer device to implement the multimodal data processing method based on the thought chain according to any one of claims 1 to 7.

Images