Thought chain-based multi-modal data processing method and device, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, and provides a multi-modal data processing method and device based on a thinking chain, equipment and a medium. On one hand, a planning agent is called to split a thinking chain according to preference data and input data, so as to reasonably split tasks according to user requirements and preferences. On the other hand, other secondary agents are sequentially called to perform context analysis based on all feedback data and user input data to obtain analysis results, and corresponding task nodes are executed according to the analysis results and the preference data. After each task node is executed, the user feedback is waited for before the response is continued, so that the problems that the user cannot modify, the waiting time is too long due to too many one-time execution steps, the computing power is excessively consumed, and the result is deviated are avoided. Meanwhile, the context analysis fully considers the user requirements, so that the final multi-modal data processing result can better meet the actual processing requirements of the user.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal data processing method, apparatus, device, and medium based on thought chain. Background Technology

[0002] Currently, whether in the fields of graphic design, game design, or architectural design, designers are required to operate the corresponding software according to the client's requirements in order to process multimodal data step by step to meet the client's design needs.

[0003] However, a design task often includes multiple detailed sub-tasks, making the design process complex. Traditional design software can only complete multimodal data processing once based on the information input by the user. If the customer wants to make modifications, they need to operate on the already processed multimodal data, which is not only time-consuming but also difficult to operate. Summary of the Invention

[0004] In view of the above, it is necessary to provide a multimodal data processing method, device, equipment and medium based on thought chain, which aims to solve the problems of high operation difficulty, low processing efficiency and unsatisfactory processing results in multimodal data processing.

[0005] A multimodal data processing method based on thought chains, the multimodal data processing method based on thought chains includes:

[0006] In response to a processing instruction for target multimodal data triggered by a target user, the user's preference data and input data are obtained.

[0007] The planning agent is invoked to split the thought chain according to the preference data and the input data to obtain the target thought chain; wherein, the target thought chain includes at least one task node and a sub-agent corresponding to each task node;

[0008] Obtain the first task node that needs to be executed in the target thought chain, and call the secondary agent that matches the first task node as the first agent;

[0009] The first intelligent agent processes the target multimodal data based on the input data and the preference data to obtain the processing result, and feeds back the processing result to the target user based on the preference data, and receives feedback data from the target user on the processing result;

[0010] Other sub-agents are sequentially invoked to perform contextual analysis based on all feedback data and the input data to obtain analysis results, and the corresponding task nodes are executed according to the analysis results and the preference data;

[0011] When all task nodes in the target thought chain have been completed, the current processing result is obtained and the current processing result is fed back to the target user.

[0012] When the target user confirms the current processing result, the current processing result is determined as the processing result of the target multimodal data.

[0013] According to a preferred embodiment of the present invention, the method further includes:

[0014] When a user initiates a registration command to a designated multimodal data processing platform, a prompt message is sent to the user to select user preferences.

[0015] According to a preferred embodiment of the present invention, before the invocation planning agent performs thought chain splitting according to the preference data and the input data, the method further includes:

[0016] Obtain a large language model as the initial model;

[0017] Construct a training set; wherein each training sample in the training set is labeled with a corresponding task node and agent;

[0018] The initial model is trained using the training set until the accuracy of the initial model reaches the configured accuracy, at which point training stops.

[0019] The model obtained from the current training is determined as the planning agent.

[0020] According to a preferred embodiment of the present invention, the planning agent and the sub-agent are agents accumulated through continuous invocation and optimization training in the context of the corresponding industry.

[0021] The sub-agent includes a large language model and an algorithm model that match the functions required for the task, as well as a third-party model called through a third-party API interface.

[0022] According to a preferred embodiment of the present invention, the step of feeding back the processing result to the target user based on the preference data includes:

[0023] The processing results are fed back to the target user according to the language style, language, and feedback method marked in the preference data.

[0024] According to a preferred embodiment of the present invention, the method further includes:

[0025] For any agent among the planning agent and the sub-agent, during the execution process, intent recognition is performed on the input data and / or all current feedback data to obtain recognition results;

[0026] When the recognition result is clear and meets the execution requirements, the corresponding task node is executed by any intelligent agent; or

[0027] When the identification result is unclear and / or does not meet the execution requirements, the arbitrary intelligent agent is invoked to issue a guidance prompt to the target user to supplement the missing information based on the information required when executing the corresponding task node; or

[0028] When the identification result shows that the target thought chain is incorrect, the planning agent is invoked to adjust the target thought chain according to the identification result.

[0029] According to a preferred embodiment of the present invention, after determining the current processing result as the processing result of the target multimodal data, the method further includes:

[0030] The processing results are then labeled.

[0031] The label of the processing result is used to characterize that the processing result is generated based on the specified multimodal data processing platform.

[0032] A multimodal data processing device based on thought chain, the multimodal data processing device based on thought chain includes:

[0033] The acquisition unit is used to acquire the target user's preference data and input data in response to a processing instruction for target multimodal data triggered by the target user.

[0034] The splitting unit is used to call the planning agent to split the thought chain according to the preference data and the input data to obtain the target thought chain; wherein, the target thought chain includes at least one task node and a sub-agent corresponding to each task node;

[0035] The calling unit is used to obtain the first task node that needs to be executed in the target thought chain, and call the secondary agent that matches the first task node as the first agent;

[0036] The processing unit is configured to use the first intelligent agent to process the target multimodal data based on the input data and the preference data to obtain the processing result, and to feed back the processing result to the target user based on the preference data, and to receive feedback data from the target user on the processing result;

[0037] The execution unit is used to sequentially call other sub-agents to perform context analysis based on all feedback data and the input data to obtain analysis results, and execute the corresponding task nodes according to the analysis results and the preference data;

[0038] The feedback unit is used to obtain the current processing result when all task nodes in the target thinking chain have been completed, and to feed the current processing result back to the target user.

[0039] The determining unit is used to determine the current processing result as the processing result of the target multimodal data when it receives a confirmation signal from the target user regarding the current processing result.

[0040] A computer device, the computer device comprising:

[0041] Memory, storing at least one instruction; and

[0042] The processor executes the instructions stored in the memory to implement the thought chain-based multimodal data processing method.

[0043] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the thought chain-based multimodal data processing method.

[0044] As can be seen from the above technical solutions, on the one hand, the planning agent is invoked to split the thought chain according to preference data and input data, so as to reasonably divide the task according to user needs and preferences; on the other hand, other sub-agents are invoked in turn to perform context analysis based on all feedback data and user input data to obtain analysis results, and execute the corresponding task nodes according to the analysis results and preference data. Each task node is executed after waiting for user feedback before continuing to respond, which avoids problems such as users being unable to modify, too many steps being executed at once resulting in long waiting time, excessive consumption of computing power, and deviations in results. At the same time, because the context analysis fully considers user needs, the final multimodal data processing results better meet the user's actual processing needs. Attached Figure Description

[0045] Figure 1 This is a flowchart of a preferred embodiment of the multimodal data processing method based on the thought chain of the present invention.

[0046] Figure 2 This is a functional block diagram of a preferred embodiment of the multimodal data processing device based on the thought chain of the present invention.

[0047] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the multimodal data processing method based on thought chain according to the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the multimodal data processing method based on thought chains according to the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0050] The multimodal data processing method based on the thought chain is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0051] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0052] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0053] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0054] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0055] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0056] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0057] S10, in response to the processing instruction for target multimodal data triggered by the target user, obtain the target user's preference data and input data.

[0058] In this embodiment, the target user can be a designer or a client or other user with design needs.

[0059] In this embodiment, the target multimodal data can be line drawings, line charts, basic images, 3D models, videos, or various other types of data such as text and models that need to be processed.

[0060] In this embodiment, the processing instructions can be triggered by the designer on a designated multimodal data processing platform according to actual design requirements.

[0061] In this embodiment, the preference data may include, but is not limited to: user background (such as game design background, graphic design background, architectural design background, etc.), user preferred language style (such as chat mode with emoticons, cute style chat mode, professional style chat mode, encouraging chat mode, etc.), user preferred language (such as Chinese, English, etc.), and user preferred chat mode (such as text chat, voice chat, etc.).

[0062] In this embodiment, the input data can reflect the specific processing needs of the target user, such as "Please add a forest background to this image".

[0063] In this embodiment, the method further includes:

[0064] When a user initiates a registration command to a designated multimodal data processing platform, a prompt message is sent to the user to select user preferences.

[0065] Through the above embodiments, users can be guided to input preference data during user registration, which can be used as a reference for subsequent multimodal data processing. This makes it easier to call intelligent agents that are more in line with user preferences and background, thereby making the processing results more in line with user preferences.

[0066] S11, the planning agent is invoked to split the thought chain according to the preference data and the input data to obtain the target thought chain; wherein, the target thought chain includes at least one task node and a sub-agent corresponding to each task node.

[0067] In this embodiment, before the invocation planning agent performs thought chain splitting according to the preference data and the input data, the method further includes:

[0068] Obtain a large language model as the initial model;

[0069] Construct a training set; wherein each training sample in the training set is labeled with a corresponding task node and agent;

[0070] The initial model is trained using the training set until the accuracy of the initial model reaches the configured accuracy, at which point training stops.

[0071] The model obtained from the current training is determined as the planning agent.

[0072] The accuracy rate of the configuration can be determined according to actual needs, such as 95%.

[0073] Through the above embodiments, the planning agent can be accurately trained based on the labels, so that it can be used to rationally plan tasks based on user input.

[0074] In this embodiment, the planning agent and the sub-agent are agents accumulated through continuous calling and optimization training in the context of the corresponding industry.

[0075] In other words, through continuous use, each intelligent agent is constantly improving and accumulating knowledge in the corresponding industry context, thereby better meeting the design requirements of the corresponding industry context and generating content that is more in line with user needs.

[0076] In this embodiment, the sub-agent includes a large language model and an algorithm model that match the functions required for the task, as well as a third-party model that is called through a third-party API (Application Programming Interface).

[0077] For example, the sub-agent may include chat models, analysis models, evaluation models, query models, multimodal data processing models, search models, image enhancement models, video generation models, audio generation models, image generation models, text generation models, 3D generation models, as well as third-party models called through third-party API interfaces (such as fine-tuned raw image models and raw 3D models).

[0078] S12, obtain the first task node that needs to be executed in the target thought chain, and call the secondary agent that matches the first task node as the first agent.

[0079] For example, when the tasks in the target thought chain include, in sequence, generating an image based on a line drawing, generating a 3D model, generating a topological surface, and overall processing, the first task node is to call the multimodal data generation model as the first intelligent agent to generate an image based on the line drawing input by the user.

[0080] S13, the first intelligent agent processes the target multimodal data based on the input data and the preference data to obtain the processing result, and feeds back the processing result to the target user based on the preference data, and receives feedback data from the target user on the processing result.

[0081] In this embodiment, the step of feeding back the processing result to the target user based on the preference data includes:

[0082] The processing results are fed back to the target user according to the language style, language, and feedback method marked in the preference data.

[0083] For example, continuing from the previous example, a multimodal data generation model is used as the first agent to generate cute images based on the user's line drawing. Emojis are used as linguistic embellishments, and feedback on the multimodal data processing results is given to the user in Chinese text form to ask if the user is satisfied with the generated image and what parts need to be modified.

[0084] S14, sequentially call other sub-agents to perform context analysis based on all feedback data and the input data to obtain analysis results, and execute the corresponding task nodes according to the analysis results and the preference data.

[0085] In this embodiment, context analysis can fully understand user intent and perform targeted multimodal data processing, thereby making the multimodal data processing results better meet user needs.

[0086] In this embodiment, only one step is executed at a time and further feedback from the user is awaited, rather than completing all multimodal data processing processes fully automatically. Since the generated image can be adjusted in a timely manner based on the user's feedback on the generated result, problems such as the user being unable to modify it, too many steps being executed at once resulting in excessive waiting time, excessive consumption of computing power, and deviations in the result can be avoided.

[0087] S15, when all task nodes in the target thinking chain have been completed, obtain the current processing result and feed the current processing result back to the target user.

[0088] In this embodiment, the method further includes:

[0089] For any agent among the planning agent and the sub-agent, during the execution process, intent recognition is performed on the input data and / or all current feedback data to obtain recognition results;

[0090] When the recognition result is clear and meets the execution requirements, the corresponding task node is executed by any intelligent agent; or

[0091] When the identification result is unclear and / or does not meet the execution requirements, the arbitrary intelligent agent is invoked to issue a guidance prompt to the target user to supplement the missing information based on the information required when executing the corresponding task node; or

[0092] When the identification result shows that the target thought chain is incorrect, the planning agent is invoked to adjust the target thought chain according to the identification result.

[0093] Through the above embodiments, intent recognition can be performed based on all user input content (including all user feedback data), thereby determining whether to continue executing the remaining task nodes, guide the user to input missing information to execute the current task node, or adjust the thought process to better respond to the user's actual intent, thus improving the effectiveness of multimodal data processing.

[0094] S16, when the target user confirms the current processing result, the current processing result is determined as the processing result of the target multimodal data.

[0095] For example, when a designer triggers a multimodal data processing instruction, the planning agent first breaks down the task chain based on the user's input and preferences. For instance, a multimodal data design task, as a creative process, may include multiple sub-task nodes such as finding references, analysis, hand-drawing drafts, rendering, image editing (an electronic image processing method that includes image processing, enhancement, and modification), and writing design specifications. Then, based on the interaction with the user, the context is analyzed to sequentially call the corresponding sub-agents to execute each task node, ultimately generating the multimodal data content required by the designer.

[0096] In this embodiment, after determining the current processing result as the processing result of the target multimodal data, the method further includes:

[0097] The processing results are then labeled.

[0098] The label of the processing result is used to characterize that the processing result is generated based on the specified multimodal data processing platform.

[0099] Through the above embodiments, the processing results can be tagged to mark the processing source of multimodal data, which can effectively avoid security problems such as theft and misuse of multimodal data.

[0100] This embodiment can be used to support cross-stage and cross-modal creative activities, including but not limited to reference retrieval, content analysis, sketch generation, image processing, visual rendering, copywriting, and creative summarization. Considering that most current image generation or content creation platforms only focus on a single step in the creative process (such as AI drawing or image restoration), lacking understanding and support for the complete creative process, this embodiment constructs an intelligent collaborative multimodal data processing system based on a thought chain mechanism. This system can identify creative stages, understand contextual intent, and assist users across multiple stages from reference retrieval and creative analysis to content generation and optimization, effectively improving creative efficiency and quality.

[0101] As can be seen from the above technical solutions, on the one hand, the planning agent is invoked to split the thought chain according to preference data and input data, so as to reasonably divide the task according to user needs and preferences; on the other hand, other sub-agents are invoked in turn to perform context analysis based on all feedback data and user input data to obtain analysis results, and execute the corresponding task nodes according to the analysis results and preference data. Each task node is executed after waiting for user feedback before continuing to respond, which avoids problems such as users being unable to modify, too many steps being executed at once resulting in long waiting time, excessive consumption of computing power, and deviations in results. At the same time, because the context analysis fully considers user needs, the final multimodal data processing results better meet the user's actual processing needs.

[0102] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the multimodal data processing device based on the thought chain of the present invention. The multimodal data processing device 11 based on the thought chain includes an acquisition unit 110, a splitting unit 111, a calling unit 112, a processing unit 113, an execution unit 114, a feedback unit 115, and a determination unit 116. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0103] The acquisition unit 110 is used to acquire the target user's preference data and input data in response to a processing instruction for target multimodal data triggered by the target user.

[0104] The splitting unit 111 is used to call the planning agent to split the thought chain according to the preference data and the input data to obtain the target thought chain; wherein, the target thought chain includes at least one task node and a sub-agent corresponding to each task node;

[0105] The calling unit 112 is used to obtain the first task node that needs to be executed in the target thinking chain, and call the secondary agent that matches the first task node as the first agent;

[0106] The processing unit 113 is used to process the target multimodal data using the first intelligent agent based on the input data and the preference data to obtain a processing result, and to feed back the processing result to the target user based on the preference data, and to receive feedback data from the target user on the processing result;

[0107] The execution unit 114 is used to sequentially call other sub-agents to perform context analysis based on all feedback data and the input data to obtain analysis results, and execute the corresponding task nodes according to the analysis results and the preference data;

[0108] The feedback unit 115 is used to obtain the current processing result when all task nodes in the target thinking chain have been completed, and to feed the current processing result back to the target user.

[0109] The determining unit 116 is used to determine the current processing result as the processing result of the target multimodal data when it receives a confirmation signal from the target user regarding the current processing result.

[0110] As can be seen from the above technical solutions, on the one hand, the planning agent is invoked to split the thought chain according to preference data and input data, so as to reasonably divide the task according to user needs and preferences; on the other hand, other sub-agents are invoked in turn to perform context analysis based on all feedback data and user input data to obtain analysis results, and execute the corresponding task nodes according to the analysis results and preference data. Each task node is executed after waiting for user feedback before continuing to respond, which avoids problems such as users being unable to modify, too many steps being executed at once resulting in long waiting time, excessive consumption of computing power, and deviations in results. At the same time, because the context analysis fully considers user needs, the final multimodal data processing results better meet the user's actual processing needs.

[0111] like Figure 3 The diagram shown is a schematic representation of the structure of a computer device that implements a preferred embodiment of the multimodal data processing method based on thought chain according to the present invention.

[0112] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a multimodal data processing program based on thought chain.

[0113] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0114] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0115] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a multimodal data processing program based on a thought chain, but also to temporarily store data that has been output or will be output.

[0116] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing multimodal data processing programs based on thought chains) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0117] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various embodiments of the multimodal data processing method based on thought chains described above, for example... Figure 1 The steps are shown.

[0118] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an acquisition unit 110, a splitting unit 111, a calling unit 112, a processing unit 113, an execution unit 114, a feedback unit 115, and a determination unit 116.

[0119] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the multimodal data processing method based on thought chains described in the various embodiments of this invention.

[0120] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0121] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0122] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0123] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0124] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0125] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0126] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0127] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0128] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0129] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0130] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a multimodal data processing method based on thought chain, and the processor 13 can execute the multiple instructions to achieve:

[0131] In response to a processing instruction for target multimodal data triggered by a target user, the user's preference data and input data are obtained.

[0132] The planning agent is invoked to split the thought chain according to the preference data and the input data to obtain the target thought chain; wherein, the target thought chain includes at least one task node and a sub-agent corresponding to each task node;

[0133] Obtain the first task node that needs to be executed in the target thought chain, and call the secondary agent that matches the first task node as the first agent;

[0134] The first intelligent agent processes the target multimodal data based on the input data and the preference data to obtain the processing result, and feeds back the processing result to the target user based on the preference data, and receives feedback data from the target user on the processing result;

[0135] Other sub-agents are sequentially invoked to perform contextual analysis based on all feedback data and the input data to obtain analysis results, and the corresponding task nodes are executed according to the analysis results and the preference data;

[0136] When all task nodes in the target thought chain have been completed, the current processing result is obtained and the current processing result is fed back to the target user.

[0137] When the target user confirms the current processing result, the current processing result is determined as the processing result of the target multimodal data.

[0138] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0139] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0140] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0141] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0142] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0144] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0145] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0146] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for processing multi-modal data based on a thought chain, characterized in that, The method comprises the following steps: In response to a processing instruction triggered by a target user on target multi-modal data, obtaining preference data and input data of the target user; Calling a planning agent to perform mind chain splitting according to the preference data and the input data to obtain a target mind chain; wherein the target mind chain comprises at least one task node and a secondary agent corresponding to each task node; wherein, before the planning agent performs mind chain splitting according to the preference data and the input data, a large language model is obtained as an initial model; a training set is constructed; each training sample in the training set is labeled with a corresponding task node and an agent; the initial model is trained using the training set until the accuracy of the initial model reaches a configured accuracy, and the training is stopped; the model obtained by the current training is determined as the planning agent; Obtaining the first task node to be executed in the target mind chain, and calling a secondary agent matched with the first task node as a first agent; Using the first agent to process the target multi-modal data based on the input data and the preference data to obtain a processing result, and feeding back the processing result to the target user based on the preference data, and receiving feedback data of the target user on the processing result; Successively calling other secondary agents to perform context analysis based on all feedback data and the input data to obtain analysis results, and executing corresponding task nodes according to the analysis results and the preference data; When all task nodes in the target mind chain are executed, the current processing result is obtained, and the current processing result is fed back to the target user; When receiving a confirmation signal of the target user on the current processing result, the current processing result is determined as the processing result of the target multi-modal data.

2. The thought chain based multi-modal data processing method of claim 1, wherein, The method further comprises: When detecting a registration instruction initiated by a user to a specified multi-modal data processing platform, sending a prompt information to the user to select user preferences.

3. The method according to claim 1, wherein: The planning agent and the secondary agent are intelligent agents accumulated by continuously calling and optimizing training under corresponding industry backgrounds; The secondary agent comprises a large language model, an algorithm model, and a third-party model called through a third-party API interface, which are matched with the functions required by the task.

4. The thought chain based multi-modal data processing method of claim 1, wherein, The feedback of the processing result to the target user based on the preference data comprises: Feeding back the processing result to the target user according to the language style, language, and feedback mode marked in the preference data.

5. The thought chain based multi-modal data processing method as claimed in claim 1, wherein, The method further comprises: For any agent in the planning agent and the secondary agent, in the execution process, intent recognition is performed on the input data and / or all current feedback data to obtain a recognition result; When the recognition result is clear and meets the execution requirements, the agent is called to execute the corresponding task node; or When the identification result is unclear and / or does not meet the execution requirement, a guide prompt for supplementing the missing information is sent to the target user by the arbitrary agent according to the information required for executing the corresponding task node. When the identification result shows that the target thought chain is wrong, the planning agent is called to adjust the target thought chain according to the identification result.

6. The thought chain based multi-modal data processing method of claim 2, wherein, After determining the current processing result as the processing result of the target multi-modal data, the method further comprises: labeling the processing result; wherein the label of the processing result is used to represent that the processing result is generated based on the specified multi-modal data processing platform.

7. A multi-modal data processing apparatus based on a chain-of-thought, characterized by, The multi-modal data processing device based on a thought chain comprises: an acquisition unit configured to acquire preference data and input data of a target user in response to a processing instruction of target multi-modal data triggered by the target user; a splitting unit configured to call a planning agent to split a thought chain according to the preference data and the input data to obtain a target thought chain; wherein the target thought chain comprises at least one task node and a secondary agent corresponding to each task node; wherein, before calling the planning agent to split the thought chain according to the preference data and the input data, an initial model is acquired as the planning agent by using a large language model; a training set is constructed; wherein each training sample in the training set is marked with a corresponding task node and an agent; the initial model is trained by using the training set until the accuracy of the initial model reaches a configured accuracy, and the training is stopped; and a model obtained by the current training is determined as the planning agent; a calling unit configured to acquire a first task node to be executed in the target thought chain, and call a secondary agent matched with the first task node as a first agent; a processing unit configured to process the target multi-modal data based on the input data and the preference data by using the first agent to obtain a processing result, and feed back the processing result to the target user based on the preference data, and receive feedback data of the target user on the processing result; an execution unit configured to sequentially call other secondary agents to perform context analysis based on all the feedback data and the input data to obtain analysis results, and execute corresponding task nodes according to the analysis results and the preference data; a feedback unit configured to acquire a current processing result when all the task nodes in the target thought chain are executed, and feed back the current processing result to the target user; a determination unit configured to determine the current processing result as the processing result of the target multi-modal data when a confirmation signal of the target user on the current processing result is received.

8. A computer device, comprising: The computer device comprises: a memory configured to store at least one instruction; and a processor configured to execute the instruction stored in the memory to implement the multi-modal data processing method based on a thought chain according to any one of claims 1 to 6. The computer device comprises: a memory configured to store at least one instruction; and a processor configured to execute the instruction stored in the memory to implement the multi-modal data processing method based on a thought chain according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in the computer device to implement the thought chain-based multi-modal data processing method in any one of claims 1 to 6.

Images