Content generation method and multi-modal canvas system

Abstract

The embodiment of the invention provides a content generation method and a multi-modal canvas system, which are used for analyzing user intention in the same canvas scene and generating content required by a user. The method comprises the steps that a user interaction interface is displayed, wherein the user interaction interface comprises a canvas area; receiving a user interaction event triggered by a user in the canvas area, and identifying a user intention and event content of the user interaction event; generating a plurality of target agents for executing reasoning tasks of the user interaction event; and executing a reasoning task of the user intention and the event content of the user interaction event by utilizing the plurality of target intelligent agents, generating target content information, and displaying the target content information in the canvas area.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer technology, and particularly relate to a content generation method and a multi-modal canvas system. BACKGROUND

[0002] With the development of computer technology, people can improve the efficiency of content generation with the help of some automated auxiliary systems. For example, various content can be generated, and the content can include different formats of elements such as text, pictures, tables, symbols, etc.

[0003] However, in existing digital creation and collaboration tools, users usually rely on whiteboards, design software (such as Figma, Sketch), or presentation tools (such as PowerPoint, Miro) for content creation and editing. However, these tools have limitations in cross-modal interaction, intelligent intent understanding, and automated task execution. For example, current tools only take user gesture operations (such as arrows, crosses, handwritten annotations) as static visual elements, and cannot analyze the semantic intent behind them. At the same time, the AI agents in existing tools also have difficulty understanding the text / picture / video content in the canvas, and still need to rely on other functional blocks to generate pictures or videos.

[0004] Therefore, how to meet the content generation needs of users in the same canvas scenario is a technical problem that needs to be solved at present. SUMMARY

[0005] Embodiments of the present application provide a content generation method and a multi-modal canvas system for analyzing user intent and generating user required content in the same canvas scenario.

[0006] A first aspect of embodiments of the present application provides a content generation method, the method comprising: displaying a user interaction interface, the user interaction interface comprising a canvas area; receiving a user interaction event triggered by a user in the canvas area, identifying a user intent and event content of the user interaction event; generating a plurality of target agents for executing an inference task of the user interaction event; using the plurality of target agents, executing the inference task of the user intent and event content of the user interaction event, generating target content information, and displaying the target content information in the canvas area.

[0007] A second aspect of embodiments of the present application provides a multi-modal canvas system, the multi-modal canvas system comprising: a canvas area, a content carrier component, an interaction capturer, a task bus, an agent generator, an agent, an intent grabbing pipeline, and a canvas changing pipeline. The canvas area is configured to record a user interaction event triggered by a user on the canvas area and display target content information corresponding to the user interaction event. The content carrier component is configured to carry event content in the user interaction event and record a vector path drawn by the user for the event content, wherein the vector path is used to represent a user intention of the user interaction event. The interaction capturer is configured to capture the user interaction event and activate the task bus and the intention grasping pipeline. When the task bus is activated, the task bus is configured to activate an inference task queue, generate a target task flow, and control the agent generator to generate a plurality of agents corresponding to a plurality of inference subtasks in the target task flow. The agent is configured to execute an inference task of each inference subtask. The intention grasping pipeline is configured to grasp the user interaction event from the canvas area, identify a user intention and event content corresponding to the user interaction event, and send the user intention and the event content to a corresponding target agent. The target agent is further configured to generate the target content information and a canvas change event according to the user intention and the event content, and send the target content information and the canvas change event to the canvas change pipeline. The canvas change pipeline is configured to send the target content information to the canvas area through the canvas change event, so that the canvas area completes canvas change.

[0008] The multi-modal canvas system provided by the second aspect of the embodiments of the present application is used to execute the content generation method of the first aspect.

[0009] The third aspect of the embodiments of the present application provides a content generation device, comprising: A central processing unit, a memory, an input and output interface, a wired or wireless network interface, and a power supply; The memory is a transient storage memory or a persistent storage memory. The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to execute the content generation method of the first aspect.

[0010] The fourth aspect of the embodiments of the present application provides a computer readable storage medium, which includes instructions, when the instructions are run on a computer, the computer executes the content generation method of the first aspect.

[0011] The fifth aspect of the embodiment of the present application provides a computer program product, the computer program product comprising instructions which, when executed on a computer, cause the computer to perform the content generation method of the first aspect.

[0012] From the above technical solutions, the embodiment of the present application has the following advantages: by the content generation method disclosed in the embodiment of the present application, the operation of the user on the canvas is converted into a semantic intention, so that the interactive event triggered by the user on the canvas is mapped into executable content of the agent in real time, and the content required by the user is displayed on the canvas, and the content generation requirement of the user in the same canvas scene can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0014] Figure 1 The architecture schematic diagram of a multi-modal canvas system disclosed in the embodiment of the present application; Figure 2 The task flowchart of user intention recognition disclosed in the embodiment of the present application; Figure 3 The flowchart of agent task scheduling and push stream disclosed in the embodiment of the present application; Figure 4 The flowchart of generating a task disclosed in the embodiment of the present application; Figure 5 The flowchart of the content generation method disclosed in the embodiment of the present application; Figure 6 The flowchart of another content generation method disclosed in the embodiment of the present application; Figure 7 The flowchart of another content generation method disclosed in the embodiment of the present application; Figure 8 The flowchart of another content generation method disclosed in the embodiment of the present application; Figure 9 The flowchart of another content generation method disclosed in the embodiment of the present application; Figure 10A The schematic diagram of applying the content generation method disclosed in the embodiment of the present application; Figure 10B The schematic diagram of another application of the content generation method disclosed in the embodiment of the present application; Figure 10C The schematic diagram of another application of the content generation method disclosed in the embodiment of the present application; Figure 11 A structural schematic diagram of a content generation system disclosed by an embodiment of the present application; Figure 12 A structural schematic diagram of a content generation device disclosed by an embodiment of the present application. DETAILED DESCRIPTION

[0015] Currently, the existing digital creation and collaboration tools have the following three common problems: 1. Lack of cross-modal "what you see is what you get" semantic mapping function. 2. Lack of unified carrying and interaction environment for multi-modal data. 3. Failure of agent-human collaboration task chain. For example, for problem 1, existing whiteboard, design or PPT tools can only store arrows, crosses and handwritten annotations as "dead" layers, the system cannot understand the intention behind them, and cannot drive the agent to continue processing, but only manual reprocessing. When users draw an arrow or a cross on the canvas, they are essentially expressing commands such as "linking the two logical segments" or "removing this element"; past software did not establish a unified data model from gestures to semantics to action chains. When multiple people add annotations in natural language at the same time, the system cannot combine the annotations into a traceable and verifiable task chain, and still needs human to understand the spirit of the annotations and assign work, and then manually implement the modifications. For problem 2, current mainstream tools cannot natively carry multi-modal content such as text, images, videos, audio, and web pages in the same canvas, and lack semantic connection and unified operation model between modalities. Users need to frequently switch between multiple software (such as cutout, haishi AI, Photoshop, ChatGPT, PPT, etc.), leading to content fragmentation, context interruption, and greatly reducing the efficiency of creation and the feasibility of intelligent collaboration. Multi-modal elements cannot be uniformly managed and interacted, hindering high-density, multi-modal creation in a "what you see is what you get" manner. The lack of a unified carrying environment also makes it difficult for intelligent agents to obtain the full context and complete the cross-modal task chain execution with high quality. For problem 3, existing Chat products such as ChatGPT and Copilot only provide "single-step calling" and cannot manage multi-stage processes such as retrieval-generation-editing-layout-export. Existing Agent products such as Manus, Minimax, Coze Space, and Lovart have fixed task chain structures and single paths, making it difficult for users to intervene or modify them in real time through multi-modal methods such as graphics, annotations, and drag-and-drop, resulting in a lack of human feedback integration into the task process and a lack of negotiability and flexibility in intelligent agent execution, making the collaboration form highly rigid. Meanwhile, in traditional creation and collaboration tools such as Figma, Sketch, PowerPoint, and Miro, vector graphics such as arrows, rectangles, and scribbles are only saved as visual annotations; gestures themselves do not have any executable semantics. Copilot, ChatGPT, and NotionAI drive intelligent agents through natural language, but lack the ability to express instructions through "graphical gestures" and do not support direct operations on video, image or web fragments in the same canvas. XR and handwriting recognition applications can recognize body gestures or handwriting, but only control at the cursor level (such as zooming out and undoing), without mapping gestures to complex task chains at the semantic level.Take Figma FigJam, Lovart's "text-to-image" function as an example: gestures have no semantics, and the arrows drawn cannot tell the system that "two elements need to be connected and merged into one element"; the cross cannot trigger the "delete and reformat" action (i.e. problem 1). Modal island, AI cannot understand the text / picture / video content in the canvas, and still needs to rely on other separate image generation, video generation and other functions, which is cumbersome to operate (i.e. problem 2). No task chain, Figma has no task chain; arrows or annotations will not produce subsequent "review → reformat → export" automatic flow (i.e. problem 3). Task chain out of control, Lovart generates a series of pictures after generating a task chain, and humans can only watch. Eventually, not many pictures meet expectations, and only a new round of collaboration can be started, and the form of collaboration is highly rigid (i.e. problem 3).

[0016] As can be seen from the above description, the past software only solves "draw" and "see", and does not solve "draw as command, annotation as process". That is, in the prior art, there are mainly the following three defects. Defect A: the existing canvas only stores gestures such as arrows and crosses as static vectors; the system cannot analyze the semantics, and the agent cannot adjust the task accordingly; resulting in human-machine collaboration remaining at the "human reprocessing" level, which is low in efficiency. Defect B: multi-modal objects are scattered in various software; resulting in a fragmented context, and task materials need to be manually moved across tools; resulting in a long process, easy to make mistakes, and version out of control. Defect C: the existing agent task chain path is fixed; there is no real-time human feedback interface, and the task cannot be dynamically rearranged; resulting in the execution of the intelligent agent being rigid, and it is difficult to meet the creative iteration and collaborative flexibility.

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0018] Thus, to solve the various defects described in the above, the embodiments of the present application propose a set of "canvas gesture-semantics-task" integrated multi-modal canvas system, the core goal of which is to realize 1. semantic decoding (real-time mapping of intuitive gestures such as arrows, circle selection, etc. to machine executable instructions). 2. Unified bearing (natively bearing and linking multi-modal data such as text, image, video, code, etc. in a single canvas). 3. Dynamic arrangement (allowing users to insert, delete, rearrange task chains in real time through any modal feedback, and Agent self-adapting to adjust the execution process). 4. Continuous context (saving the entire canvas to generate reusable "workflow templates", supporting secondary loading and version tracking). Through the above technical solutions, the embodiments of the present application realize a new interaction paradigm of "drawing commands, annotations as processes", significantly improving multi-modal creation efficiency and human-machine collaboration flexibility.

[0019] For ease of understanding, please refer to Figure 1 , Figure 1 The architecture of a multi-modal canvas system disclosed by the embodiments of the present application is shown in the figure.

[0020] The multi-modal canvas system includes a multi-modal interaction layer 100, a semantic conversion layer 200, an Agent Engine 300, and a content generation layer 400. The multi-modal interaction layer 100 is used for multi-modal canvas, collects user input, and renders content. The semantic conversion layer 200 is used to decode the original natural input into task instructions. The Agent Engine 300 is used as a task arrangement and execution engine. The content generation layer 400 is used to generate text, images, audio, video, code, web pages, and other products.

[0021] Further, the multi-modal interaction layer 100 includes a content carrier component 101, a Canvas 102, and an interaction capturer 103. The content carrier component 101 is used to carry multimedia form carriers such as text, arrow marks, images, videos, websites, etc. The Canvas 102 is used as a real-time rendering visualization container for users. The interaction capturer 103 is used to capture the interaction intention of the user on the canvas.

[0022] The semantic conversion layer 200 includes an intention grabbing pipeline and a canvas changing pipeline. The intention grabbing pipeline 201 is used to grab content and interaction intention from the multi-modal interaction layer. The canvas changing pipeline 202 is used to write tasks or results of tasks back to the Canvas, and to operate the Canvas.

[0023] The agent engine 300 comprises a task bus 301, an agent generator 302 and an agent 303. The task bus 301 is configured to maintain a task context and an agent blueprint. The agent generator 302 is configured to generate an agent blueprint. The agent 303 is configured to execute a specific agent of the task.

[0024] The content production layer 400 comprises a generation request router 401, a generative model 402 and a content preprocessor 403. The generation request router 401 is configured to send a content generation request to different models. The generative model 402 is configured to be a Transformer multi-modal large model. The content preprocessor 403 is configured to compress and decompress different forms of content.

[0025] Referring to Figure 5 , Figure 5 A flowchart of a content generation method disclosed in an embodiment of the present application is shown. The method comprises steps 501-504.

[0026] 501, a user interaction interface is displayed.

[0027] In the multi-modal canvas system, to facilitate the user to output relevant intuitive gestures, a user interaction interface can be displayed on the multi-modal canvas system, wherein the user interaction interface displays a canvas area. The canvas area can be used for the user to interact with the agent, and the interaction information drawn by the user on the canvas area can be displayed. For example, the user can draw arrows, circles, and other intuitive gestures to map out the execution instructions of the agent in real time.

[0028] 502, a user interaction event triggered by the user on the canvas area is received, and the user intent and event content of the user interaction event are identified.

[0029] When the user outputs a relevant intuitive gesture on the canvas area, the canvas area can trigger a user interaction event associated with the intuitive gesture. Meanwhile, if there is relevant multi-modal data such as text, image, video or code on the canvas area, and the multi-modal data is associated with the intuitive gesture at this time, the multi-modal canvas system can also identify the user intent and event content in the user interaction event. It should be noted that the user intent can be information related to the content that the user expects to generate or the content that the user expects. For example, the user draws an arrow through an intuitive gesture between different images or texts, and the multi-modal canvas system identifies the user intent as the user needs to associate the event content (such as images or texts, etc.). Further, the user intent can also include other customized content, such as keywords related to the content that the user expects to generate or target content, sentences expressing the user intent, etc. Details are not described here.

[0030] 503. Generating a plurality of target agents for performing an inference task of the user interaction event.

[0031] Further, the multi-modal canvas system can activate and generate a target agent associated with the user interaction event through the user interaction event.

[0032] In one specific embodiment, the task bus in the multi-modal canvas system can also be activated after capturing the user interaction event. The task bus can schedule and generate a task flow of an inference task for the user interaction event, and then one or more agents can be generated through the task flow to execute the task flow, wherein the generated agent is the target agent.

[0033] Further, a plurality of task flows can also be set, and each task flow can be executed by a different agent to perform a different task.

[0034] 504. Using the plurality of target agents to perform an inference task of the user intent and event content of the user interaction event, generate target content information, and display the target content information on the canvas area.

[0035] Further, the multi-modal canvas system can execute the intent recognition and inference task through the target agent. Specifically, after the multi-modal canvas system captures the event content, the event content can be organized into a format for agent training and delivered to the target agent, and then the target agent can execute the inference task, that is, infer the user intent and event content, to generate target content information. Then, the target agent can transmit the target content information to the canvas area. Thus, the canvas area displays the target content information (which can also render images, etc.), so that the user sees the changed content of the canvas, that is, the inference task is completed.

[0036] wherein, for ease of understanding Figure 5 In Figure 1 The operation flow of the multi-modal canvas system is shown in Figure 2 , Figure 2 A task flow chart of user intent recognition disclosed in the embodiment of the present application.

[0037] As Figure 2 It can be seen that the embodiment mainly includes three steps. Specifically, in the S1 step, the Canvas 102 mainly implements intent capture. The user draws a label (such as an arrow, a guide, etc.) between element A and element B, the Canvas 102 receives the interaction and renders the screen, the content carrier component 101 records the vector path, the interaction capturer 103 captures the user's interaction event, and activates the subsequent process. In the S2 step, the main task is activated. Among them, the interaction capturer 103 activates the task bus 301; the task bus 301 executes the Agent generator 302 and generates the Agent Blueprint, which contains one or more Agents 303; the Agent 303 is responsible for specific reasoning tasks. In the S3 step, the main task is to execute the intent recognition and reasoning task. Among them, the intent capture pipeline 201 captures the content from the Canvas 102, reorganizes it into a unified format and delivers it to the Agent 303; the Agent 303 reasons the user's intent and generates content and canvas change events; the canvas change event acts on the Canvas 102 through the canvas change pipeline 202, and the Canvas 102 is changed, and the user sees the changed task.

[0038] Therefore, it can be seen that the prior art scheme can only activate a round of reasoning by inputting text through a keyboard or clicking to upload pictures, and the user's intent can only be expressed through text, and cannot be associated with the content of the canvas. By using the content generation method disclosed in the embodiment, the interaction event triggered by the user on the canvas is mapped into an executable content of an intelligent agent in real time, and multi-modal data is carried and rendered on the canvas, which can significantly improve the collaborative ability between users and improve efficiency. At the same time, in combination with Figure 2 As shown in the figure, in the embodiment, the multi-modal interaction layer 100 captures and activates tasks such as arrows, circle selection, and cross marks, the semantic conversion layer 200 sends the path features together with the context to the Agent Engine 300, outputs the change event, and the canvas change pipeline 202 writes the analysis result back to the Canvas 102. Thus, the "annotation is instruction" is realized, the "drawing→semantic→action chain" is connected, and the artificial secondary explanation is eliminated. Further, in the process of gesture / action mark capture-semantic analysis-content injection, the continuous data path and its triggering rules can be effectively determined (i.e., the problems 1 and defects A in the above can be solved).

[0039] Further, for the convenience Figure 5 For detailed description of step 504, please refer to Figure 6 , Figure 6 The flowchart of another content generation method disclosed by the embodiment of the present application is shown. It includes steps 601-602.

[0040] 601, reasoning the user intent by using the target agent, and generating a content reasoning instruction.

[0041] In one specific embodiment, in the process of reasoning the user intent by the target agent, a content reasoning instruction can be triggered and generated, wherein the content reasoning instruction is generated by capturing and identifying the user intent in the running operation in response to the user's running operation on the canvas area. Further, the content reasoning instruction is mainly a control instruction for controlling the target agent to perform the reasoning task.

[0042] 602, reasoning the event content according to the content reasoning instruction, and obtaining target content information.

[0043] Further, the target agent can be controlled to reason the event content according to the content reasoning instruction, so as to obtain the target content information.

[0044] In one specific embodiment, in response to the content reasoning instruction, the target agent associated with the content reasoning instruction is used to generate the target content information according to the event content and the content pointed to (or specified) in the content reasoning instruction. Further, the target content information can be displayed and rendered, etc. Meanwhile, the user can select the content that finally meets the requirement in the displayed target content information, which is not limited here.

[0045] By the content generation method disclosed in the embodiment, the user intent and the event content can be reasoned by the target agent, so as to complete the generation of the target content required by the user.

[0046] Further, in step 504, after the target content information is transmitted to the canvas area, the agent can interact with the canvas, which can be referred to Figure 7 , Figure 7 The flowchart of another content generation method disclosed by the embodiment of the present application is shown. It includes steps 701-702.

[0047] 701, triggering a canvas change event according to the target content information.

[0048] As described above, after the target content information is obtained, the target agent can trigger a canvas change event according to the target content information.

[0049] In one specific embodiment, after the target agent infers the user's intent and generates target content information, a canvas change event can be triggered after the inference task is completed. This canvas change event can be used to modify the canvas area.

[0050] 702. Send the canvas change event and target content information to the canvas change pipeline so that the canvas change pipeline can change the canvas area through the canvas change event to display the target content information on the canvas area.

[0051] Furthermore, through the target intelligent agent, the canvas change event is transmitted through the canvas change pipeline (see [reference]). Figure 2 The change is applied to the canvas area. Furthermore, the canvas change pipeline can be used as an intermediary to simultaneously send the target content information to the canvas area. Thus, after the canvas receives the target content information and the canvas change event, it can change the canvas area based on the event, allowing the user to see the required changes and complete the task.

[0052] Through the content generation method disclosed in this embodiment, the target intelligent agent completes canvas content recognition and can automatically operate the canvas through the canvas change pipeline, allowing users to access content in various ways (including but not limited to text, arrow markers, images, videos, etc.). Furthermore, the content carrier component 101 adopts a unified object model, supporting text, images, videos, web pages, etc. The Canvas 102 renders all objects in real time within the same coordinate system and supports semantic connections and action annotations. The canvas change pipeline 202 allows the Agent Engine 300 to perform modification operations on the real world. This enables scalable, connected, unified layout, and semantic association of multimodal data, providing the intelligent agent with a complete context and canvas operation interface, i.e., achieving unified management of cross-modal objects and interactive logic on the canvas (i.e., solving problem 2 and defect B mentioned above).

[0053] Furthermore, for a more detailed description of step 503 above, please refer to [reference needed]. Figure 8 , Figure 8 This is a flowchart illustrating another content generation method disclosed in an embodiment of this application. It includes steps 801-803.

[0054] 801. When a user interaction event is triggered, activate the inference task queue and generate the target task flow.

[0055] For easier understanding, please refer to Figure 3 , Figure 3 This is a flowchart illustrating the intelligent agent task orchestration and streaming disclosed in an embodiment of this application. Combined with... Figure 2It can be seen that after the interaction capture captures the user interaction event, the interaction capture can activate the inference task queue (i.e. the task bus in the above) in the multi-modal canvas system, so as to generate a target task flow corresponding to the user interaction event. It should be noted that the target task flow includes a plurality of inference sub-tasks, and there is a sequential execution relationship between the plurality of inference sub-tasks in the above. For example, the execution relationship of retrieval → generation → proofreading → typesetting → export, etc.

[0056] In one specific embodiment, the inference task queue can be activated to allow the Agent generator to arrange the task flow related to the user interaction event. For ease of execution, the task flow can be arranged into a plurality of inference sub-tasks, and each inference sub-task is executed by a separate Agent (intelligent agent). All inference sub-tasks can also be arranged according to the user intent identified in the user interaction event, which will not be described here. In combination Figure 1 As shown, the task arrangement can be completed by the Agent generator generating the Agent Blueprint.

[0057] 802. Generate a target intelligent agent according to the plurality of inference sub-tasks and the sequential execution relationship.

[0058] In one specific embodiment, each inference sub-task is associated with an intelligent agent, and the intelligent agent is set based on all inference sub-tasks. Therefore, it can be seen that each inference sub-task corresponds to at least one intelligent agent, and the set intelligent agent is described as a target intelligent agent Agent.

[0059] 803. Use the target intelligent agent corresponding to each inference sub-task in the target task flow to sequentially execute each inference sub-task in the target task flow according to the sequential execution relationship, execute the inference of the user intent and the event content, and generate target content information.

[0060] Further, in one specific embodiment, the target intelligent agent corresponding to each inference sub-task in the target task flow is used to capture the event content and the user intent, and perform intent recognition thereon, so that the inference task of the event content is completed according to the user intent in sequence according to the sequential execution relationship, so that the target intelligent agent can generate new content information, i.e. target content information, according to the intent. Further, in other implementable technical solutions, the target intelligent agent can also assemble the target content information into an event change (i.e. the canvas change event described above), and operate the canvas through the canvas change pipeline.

[0061] Further, for unified description, the inference task queue is described as a task bus in the following.

[0062] For further description of step 803, please refer to Figure 9 the embodiments shown.

[0063] In the content generation method disclosed in this embodiment, an intelligent agent can perform intent recognition on the canvas content and automate canvas operations through a canvas change pipeline. Furthermore, there are multiple ways to depict user intent, and association with the canvas content can also be achieved. Additionally, it can be combined with... Figure 3 ,Depend on Figure 3 As can be seen, task orchestration is accomplished by Agent Generator 302 generating AgentBlueprints. Then, Agent 303 captures content and performs intent recognition; Agent 303 generates new content based on the intent; Agent 303 assembles the new content into event changes and manipulates the canvas through the canvas change pipeline 202. Secondly, Agent 303 reads past inference trajectories through communication with the task bus 301 to ensure the stability and accuracy of inference generation. Agent Generator 302 can orchestrate very complex long chains of AgentBlueprints to handle complex tasks, such as: retrieval → generation → review → layout → export. Agent 303 ensures task continuity through repeated communication with the task summary. Simultaneously, the task bus 301 can listen for new instructions from the parsing layer. Agent Generator 302 incrementally maintains Blueprints, supporting insertion, deletion, rearrangement, and automatic redrawing for Agent 303. Therefore, user feedback at any time can instantly change the task's direction, and the agent executes "negotiable and rollback-able" actions. In other words, Agent Blueprint task flows can be mapped and incrementally rearranged online.

[0064] Furthermore, in the above Figure 8 In step 803, the reasoning process of user intent and inferred content, and the content generation process need to be completed sequentially. (See reference...) Figure 9 , Figure 9 This is a flowchart illustrating another content generation method disclosed in an embodiment of this application. It includes steps 901-907.

[0065] 901. Read historical reasoning trajectory information.

[0066] In one specific embodiment, combined with Figure 3As shown, after the target agent is generated, the target agent communicates with the task bus, so that the historical reasoning track information can be read from the task bus. It should be noted that the historical reasoning track information includes historical intention reasoning track information (user intention), and further includes historical content reasoning track information (event content). The historical reasoning track information is the reasoning track information obtained after any target agent reasons the historical user intention or historical reasoning content information, and the historical reasoning track information is generated by the target agent and saved in the task bus. At the same time, the historical reasoning track information can also be the reasoning track of the previous step of the current task, and can also be the reasoning track of other tasks, which is not limited here.

[0067] Further, when the target agent reads the reasoning track, the user intention and the event content can also be synchronized from the intention capture track (in Figure 3 As shown, the user intention can be captured first, and then the reasoning content is captured, that is, the user intention and the content related to the user interaction event (mainly the content drawn on the canvas).

[0068] 902、Using all target agents corresponding to the user intention, the order execution relationship and the historical reasoning track information are executed in sequence to reason the user intention one by one to obtain the target user intention.

[0069] In one specific embodiment, since multiple target agents related to the user interaction event have been generated in the task arrangement, the user intention is reasoned in sequence by all target agents related to the user intention according to the task order in the target task flow. At the same time, the historical reasoning track information (reasoning track related to intention recognition) can also be read to ensure the stability of this analysis reasoning process, and the target user intention is obtained.

[0070] Further, during the completion of the reasoning process by the target agent, the current reasoning track (corresponding to the reasoning process of intention recognition) can be saved in the task bus.

[0071] In other implementable technical solutions, all target agents corresponding to the event content can also be used to sequentially analyze the event content one by one according to the order execution relationship, and the target event content is obtained by reasoning the event content through the historical reasoning track information. Specifically, in one specific embodiment, since multiple target agents related to the user interaction event have been generated in the task arrangement, the event content is reasoned in sequence by all target agents related to the event content according to the task order in the target task flow. At the same time, the historical reasoning track information (reasoning track related to generating content) can also be read to ensure the stability of this analysis reasoning process, and the target event content is obtained.

[0072] Further, in the process of completing the inference of the target agent, the current inference track (corresponding to the inference process of generating content) can be saved to the task bus.

[0073] 903, obtain the current intention inference track information corresponding to the target user intention, and save the current intention inference track information.

[0074] Based on step 902, after obtaining the target user intention, the target agent can save the inference track generated in the process of inferring the target user intention, that is, the current intention inference track information, and save the current intention inference track information to the task bus through the communication channel between the target agent and the task bus as historical inference track information (inference track related to intention recognition) for the next inference task.

[0075] 904, read the current intention inference track information corresponding to the user intention, and grab the event content and user intention from the intention grabbing pipeline.

[0076] According to step 902, in other implementable technical solutions, reference can be made to Figure 4 , Figure 4 A flowchart for generating a task is disclosed in an embodiment of the present application. It can be known from Figure 4 that after generating the target agent, the target content information can also be generated by other means. Specifically, the target agent can synchronously read the inference track information associated with the user intention (described above as the current intention inference track information) in the task bus, and the content inference track information associated with the event content (both the intention inference track information and the content inference track information described above can be understood as historical inference track information), and at the same time, the user intention and the event content can also be grabbed from the intention grabbing pipeline.

[0077] 905, pre-process the current intention inference track information, the user intention and the event content to obtain pre-processed content information.

[0078] Therefore, based on step 904, the corresponding target agent communicates with the content production layer, so that the target agent sends the current intention inference track information, the event content and the user intention described above to the content pre-processor to make the content pre-processor complete the pre-processing work and obtain the pre-processed content information. In one specific embodiment, the content pre-processor can perform format conversion, normalization or unified processing of structuring.

[0079] As shown in the above Figure 1 , the content generation layer can be used to generate multi-modal content such as text, pictures or videos, and the specific embodiments are not limited here.

[0080] 906. The pre-processed content information is sent to the generative model to process the pre-processed content information into target content information meeting the canvas area by the generative model.

[0081] In this embodiment, the content pre-processor can send the pre-processed content information to the generation request router, and then the generation request router requests routing and sends the pre-processed content information to the appropriate generative model according to the task type of the pre-processed content information and the advantages of each type of model in the generative model, so as to process the pre-processed content information by the generative model to obtain target content information meeting the canvas area.

[0082] 907. The inference track information corresponding to the generation of the target content information is saved.

[0083] Further, after the generative model processes the pre-processed content information into target content information meeting the user's demand, the generative model can return the target content information to the content pre-processor and further send it to the target agent. Further, at this time, the target agent can also save the inference track of the target content information and save it to the task bus as a historical inference track for reference in the next task.

[0084] Further, after the content generation layer 400 completes the product, the Agent Engine 300 writes back to the Canvas 201 through the canvas change pipeline 202; the user can again make gesture annotations on the generation result to trigger a new round of analysis and Agent Blueprint adjustment. Thus, a high-frequency cycle of “generation → visual presentation → multi-modal feedback → task adjustment” is formed to realize real human-Agent co-creation. That is, the feedback collaboration link of human-canvas-Agent is realized.

[0085] Through the content generation method disclosed in this embodiment and in combination with the above embodiments, as long as the user's operation on the canvas (such as gestures, action markers, text descriptions, and providing reference pictures, etc.) changes, the entire system can automatically plan a new task according to the new user intention and complete the task. Moreover, for the same task, as long as the user provides different initial materials, the entire multi-modal canvas system can automatically process the new initial materials according to the original workflow and deliver the processed results.

[0086] Further, please refer to Figure 4The Agent 303 generates multi-modal content such as text, pictures, videos, and the like through the content generation layer 400. The content captured by the canvas from the intent capture pipeline 201 and the user intent saved in the reasoning track saved in the task bus 301 are sent to the content preprocessor 403 for unified processing of format conversion, normalization, and structuring; and then sent to the generation request router 401, which routes the request to the appropriate generative model 402 according to the task type and the advantages of various models. The generative model 402 returns a response through the generative model 402, and the generative model 402 processes the content to meet the requirements of the product project, and then sends it to the Agent 303 and finally saves it to the reasoning track of the task bus 301.

[0087] For the convenience of the above Figures 5 to 9 In the embodiment shown in the figure, 1. Interaction efficiency is improved, that is, intuitive gestures are immediately mapped into executable instructions, and menu search and secondary dictation are saved. The semantic analysis result directly reaches the task arrangement layer, eliminating the artificial translation link. 2. Cross-modal closed loop, that is, text, image, video, and code are uniformly generated by the same model and carried in the same canvas, and the agent can completely master the context, avoiding semantic loss caused by "modal island". 3. Iterative cost reduction, that is, the task Agent Blueprint supports real-time rearrangement, and any user annotation at any time can update the process locally rather than running in full, significantly shortening the cycle from creativity to delivery. 4. Reusable and extensible, that is, the canvas-intention-Agent Engine forms a workflow that can be reproduced or migrated to different tasks with one key. As long as the intention changes, the workflow can change; in the case where the intention does not change but the content changes, the original workflow can be reused. It has strong evolution ability, strong expansion ability, and high efficiency. Therefore, compared with the existing "first annotation and then manual execution" or "agent one-time output and cannot negotiate" solution, the application simultaneously improves the speed, accuracy, collaboration scale, and evolution ability by orders of magnitude, and can significantly improve the productivity of professional creators and multi-functional teams.

[0088] The process of changing the canvas is described with reference to 10A, Figure 10B and Figure 10C , Figure 10A is a schematic diagram of an application of the content generation method disclosed in the embodiment; Figure 10B is another schematic diagram of an application of the content generation method disclosed in the embodiment; Figure 10C is another schematic diagram of an application of the content generation method disclosed in the embodiment.

[0089] For example, in Figure 10AThis process falls under scene filling and semantic matting. Specifically, Step 1: Operation: Select the water surface area in the pool image and label it with the text "Add a flamingo". Step 2: Multimodal interaction layer 100 records the pool image, labeling the water surface area with paths and text. Step 3: Semantic parsing layer 200 captures content; changes the canvas; Agent Engine 300 inserts the intent understanding and image generation agent into the process, automatically executing the agent process. Step 4: Content generation layer 400 uses depth and lighting matching to synthesize the flamingo, generating its reflection in the water. Step 5: Result: The flamingo correctly lands in the water according to perspective and matches the overall color temperature.

[0090] For example, in Figure 10B This involves cross-modal style transfer. Specifically, the process is as follows: Step 1. Operation: The user draws a wireframe to represent the composition and drags in an anime-style reference image. Step 2. Multimodal interaction layer 100 records the wireframe drawing path, the anime-style reference image, and the text. Step 3. Semantic conversion layer 200 captures the content; changes the canvas; Agent Engine 300 inserts the intent understanding and image generation agent into the process, automatically executing the agent process. Step 4. Content generation layer 400 generates images of anime-style models and scenes, ensuring the composition conforms to the wireframe. Step 5. Result: The composition and art style meet the user's expectations.

[0091] For example, in Figure 10C In this process, we'll focus on partial deletion and automatic layout. Specifically, Step 1: Operation: The user circles the headphone area in the headphone marketing image, requesting that this part of the image be retained; they cross out the background, indicating they want to delete it. Step 2: Multimodal Interaction Layer 100 records the headphone marketing image, the area division path, the cross-shaped path, and the text. Step 3: Semantic Conversion Layer 200 captures the content; changes the canvas; Agent Engine 300 inserts the intent understanding and image generation agent into the process, automatically executing the agent process. Step 4: Content Generation Layer 400 deletes the specified background and recalculates the layout; renders the headphone product and places it in the appropriate position. Step 5: Result: The canvas displays the optimized pure product image in real time.

[0092] In other implementable application embodiments, there are other embodiments. For example, for intuitive "configuration replacement", step 1. operation: user drags in a black sunglasses image on the canvas; then draws an arrow from the sunglasses to the eyes of the portrait with a stylus. Step 2. multimodal interaction layer 100 captures the arrow path, text, black sunglasses image, and portrait. Step 3. semantic transformation layer 200 grabs the content; changes the canvas; Agent Engine 300 inserts the intent understanding and image generation Agent into the process, and automatically executes the Agent process. Step 4. content generation layer 400 calls the image understanding and generation model to perform occlusion detection and light matching on the original portrait, and outputs a new portrait wearing sunglasses. Step 5. result: the portrait has correctly worn sunglasses.

[0093] For example, for multi-role "mixed creation", step 1. operation: the user drags in the thumbnails of doll A and anime character B respectively, draws a "+" between the two images, and draws a "?" placeholder on the right side. Step 2. multimodal interaction layer 100 records the "+" and "?" two hand-drawn paths, and the thumbnails of doll A and anime character B. Step 3. semantic transformation layer 200 grabs the content; changes the canvas; Agent Engine 300 inserts the intent understanding and image generation Agent into the process, and automatically executes the agent process. Step 4. content generation layer 400 first performs style and topology fusion to generate mixed character C. Step 5. result: the canvas displays a new character C.

[0094] For example, for object replacement and automatic layout, step 1. operation: the user drags in a picture of a VR product, a portrait, and a magazine cover, then draws an arrow from the VR product picture to the portrait with a stylus and labels it to make the portrait wear the VR product; then the user draws another arrow from the portrait to the magazine cover and labels it as a requirement for using the magazine cover for layout. Step 2. multimodal interaction layer 100 records the VR product picture, portrait, and magazine cover image, arrow path, and text. Step 3. semantic transformation layer 200 grabs the content; changes the canvas; Agent Engine 300 inserts the intent understanding and image generation Agent into the process, and automatically executes the Agent process. Step 4. content generation layer 400 fuses the complex task of VR product + portrait + magazine cover layout, handles the angle, light and shadow of VR wear, the angle, color and atmosphere of the portrait, and predicts the dressing style and fabric material of the lower body not shown in the original portrait. Step 5. result: the portrait wearing the VR product is in the magazine cover.

[0095] Thus, in combination with the above Figure 10A 、 Figure 10B and Figure 10CThe above-mentioned embodiments can be further extended to other application fields or technical solutions. For the convenience of understanding, please refer to Table 1 and Table 2 below. Table 1 is an extension of the application field, and Table 2 is an extension of the technical solution.

[0096]

[0097] In other implementable technical solutions, the above-mentioned industry only needs to replace the content carrier component 101 (for example, CTDICOM, CAD IFC, IoT device, JSON), and the rest of the architecture remains unchanged, still in line with the "gesture / action mark-semantic-task" closed loop principle.

[0098]

[0099] It should be noted that any of the above alternatives maintains the core link of "interaction capture-semantic analysis-task arrangement-content generation-canvas rewriting", so it is considered as an equivalent embodiment of the present application and should be included in the protection scope.

[0100] It should be understood that although each step in the flowchart involved in each of the above-mentioned embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each of the above-mentioned embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0101] Please refer to Figure 11 , Figure 11 A structural schematic diagram of a content generation system disclosed by the embodiments of the present application is shown in FIG. 1.

[0102] The display unit 1101 is configured to display a user interaction interface, and the user interaction interface includes a canvas area. The receiving and identifying unit 1102 is configured to receive a user interaction event triggered by a user in the canvas area, and identify a user intent and event content of the user interaction event. The generation unit 1103 is configured to generate a plurality of target agents for executing an inference task of the user interaction event. The inference unit 1104 is configured to execute the inference task of the user intent and event content of the user interaction event by using the plurality of target agents, generate target content information, and display the target content information in the canvas area.

[0103] Please refer to Figure 12 The embodiment of the application discloses a structural schematic diagram of a content generation device, which comprises the following components: A central processor 1201, a memory 1205, an input and output interface 1204, a wired or wireless network interface 1203 and a power supply 1202; The memory 1205 is a volatile memory or a persistent storage memory; The central processor 1201 is configured to communicate with the memory 1205 and execute the instruction operation in the memory 1205 to perform the content generation method in any one of the foregoing Figures 5 to 9 Embodiments.

[0104] The embodiment of the application further provides a chip system, which comprises at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through a line, and the at least one processor is used for running a computer program or instruction to execute the content generation method in any one of the foregoing Figures 5 to 9 Embodiments.

[0105] The embodiment of the application further provides a computer readable storage medium, which comprises an instruction, when the instruction is run on a computer, the computer executes the content generation method in any one of the foregoing Figures 5 to 9 Embodiments.

[0106] The embodiment of the application further provides a computer program product comprising an instruction, when the computer program product is run on a computer, the computer executes the content generation method in any one of the foregoing Figures 5 to 9 Embodiments.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0108] In several embodiments provided in the application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0109] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0110] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0111] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical scheme of the present application essentially or the part that contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various program code storage media.

Claims

1. A content generation method characterized by, The method comprises: displaying a user interaction interface, the user interaction interface comprising a canvas area; receiving a user interaction event triggered by a user in the canvas area, identifying a user intention and event content of the user interaction event; generating a plurality of target agents for performing an inference task of the user interaction event; using the plurality of target agents, performing the inference task of the user intention and event content of the user interaction event, generating target content information, and displaying the target content information in the canvas area.

2. The content generation method of claim 1, wherein, The use of the plurality of target agents to perform the inference task of the user intention and event content of the user interaction event to generate target content information comprises: using the target agent to infer the user intention to generate content inference instructions; inference of the event content according to the content inference instructions to obtain the target content information.

3. The content generation method of claim 1, wherein, After displaying the target content information in the canvas area, the method further comprises: triggering a canvas change event according to the target content information; sending the canvas change event and the target content information to a canvas change pipeline to make the canvas change pipeline change the canvas area through the canvas change event to display the target content information on the canvas area.

4. The content generation method of claim 1, wherein, The generation of a plurality of target agents for performing an inference task of the user interaction event comprises: when the user interaction event is triggered, activating an inference task queue and generating a target task flow; wherein the target task flow comprises a plurality of inference subtasks and a sequential execution relationship between the plurality of inference subtasks; generating the target agent according to the plurality of inference subtasks and the sequential execution relationship; wherein each inference subtask corresponds to any one target agent; The use of the target agent to perform the inference task of the user intention and event content of the user interaction event to generate target content information comprises: using the target agent corresponding to each inference subtask in the target task flow, sequentially running each inference subtask in the target task flow according to the sequential execution relationship, performing inference on the user intention and the event content, and generating the target content information.

5. The content generation method of claim 4, wherein, The use of the target agent corresponding to each inference subtask in the target task flow, sequentially running each inference subtask in the target task flow according to the sequential execution relationship comprises: reading historical inference track information; wherein the historical inference track information comprises historical intention inference track information, the historical inference track information is the inference track information obtained after any one target agent infers a historical user intention, and the historical inference track information is generated by the target agent; using all target agents corresponding to the user intention to sequentially infer one by one the user intention according to the sequential execution relationship and the historical inference track information to obtain a target user intention; obtaining current intention reasoning track information corresponding to the target user intention, and storing the current intention reasoning track information; wherein the current intention reasoning track information is used as historical reasoning track information for next reading.

6. The content generation method of claim 5, wherein, The reasoning task of the user intention and the event content of the user interaction event is performed to generate target content information, including: reading the current intention reasoning track information corresponding to the user intention, and grabbing the event content and the user intention from an intention grabbing pipeline; preprocessing the current intention reasoning track information, the user intention, and the event content to obtain preprocessed content information; sending the preprocessed content information to a generative model to process the preprocessed content information into target content information meeting the canvas area by using the generative model.

7. The content generation method of claim 6, wherein, After the preprocessed content information is processed into target content information meeting the canvas area by using the generative model, the method further includes: storing reasoning track information corresponding to generation of the target content information.

8. A multi-modal canvas system, characterized by, The content generation method of any one of claims 1-7 is implemented, and the multi-modal canvas system includes a canvas area, a content carrier component, an interaction capturer, a task bus, an agent generator, an agent, an intention grabbing pipeline, and a canvas changing pipeline. The canvas area is configured to record a user interaction event triggered by a user on the canvas area, and display target content information corresponding to the user interaction event. The content carrier component is configured to carry event content in the user interaction event, and record a vector path drawn by the user for the event content; wherein the vector path is used to represent a user intention of the user interaction event. The interaction capturer is configured to capture the user interaction event, and activate the task bus and the intention grabbing pipeline. When the task bus is activated, the task bus is configured to activate a reasoning task queue, generate a target task flow, control the agent generator to generate a plurality of agents corresponding to a plurality of reasoning sub-tasks in the target task flow. The agent is configured to perform a reasoning task of each reasoning sub-task. The intention grabbing pipeline is configured to grab the user interaction event from the canvas area, and identify a user intention and event content corresponding to the user interaction event, to send the user intention and the event content to a corresponding target agent. The target agent is further configured to generate the target content information and a canvas changing event according to the user intention and the event content, and send the target content information and the canvas changing event to the canvas changing pipeline. The canvas changing pipeline is configured to send the target content information to the canvas area by using the canvas changing event, so that the canvas area completes canvas changing.

9. The multi-modal canvas system of claim 8, wherein The target agent is configured to read historical reasoning track information from the task bus to ensure execution of a reasoning task corresponding to the user interaction event; the historical reasoning track information includes historical intent reasoning track information, the historical reasoning track information is reasoning track information obtained after any agent reasons a historical user intent, and the historical reasoning track information is generated by the agent; The target agent is further configured to reason the user intent according to the historical reasoning track information, obtain a target user intent, and acquire current intent reasoning track information corresponding to the target user intent.

10. The multi-modal canvas system of claim 9, wherein, Further comprising a content preprocessor and a generative model; The content preprocessor is further configured to receive the current intent reasoning track information, the user intent and the event content sent by the target agent, and process the current intent reasoning track information, the user intent and the event content to obtain preprocessed content information; The generative model is configured to receive the preprocessed content information, and process the preprocessed content information to generate the target content information; The generative model is further configured to send the target content information to the target agent, so that the target agent saves reasoning track information corresponding to the target content information in the task bus.

Images