Document generation method and device based on multi-modal large language model, equipment and storage medium

Abstract

The invention discloses a document generation method, device and equipment based on a multi-modal large language model and a storage medium, and relates to the technical field of document generation, the method comprises the following steps: generating a workflow directed acyclic graph based on a workflow and an operator in a to-be-processed task; obtaining node information of the workflow directed acyclic graph through a graph structure reading analysis tool; generating a workflow document template based on the user configuration parameter information through a preset document typesetting tool; and generating a workflow document corresponding to the to-be-processed task based on the node information and the workflow document template. According to the method, the workflow directed acyclic graph can be generated based on the workflow and the operator in the to-be-processed task, and the workflow document is generated based on the node information of the workflow directed acyclic graph and the workflow document template generated according to the user configuration parameter information; the problems that in the prior art, when a workflow document is generated, a predefined template usually needs to be used, generation of diversified workflows cannot be supported, and universality is poor are solved.

Description

Technical Field

[0001] The present application relates to the technical field of document generation, and in particular to a document generation method, apparatus, device and storage medium based on a multimodal large language model. Background Art

[0002] In the process of improving the level of information management in enterprises, workflow technology has received extensive attention. Workflow can be used to represent the organization and automated execution of a series of tasks to achieve specific business goals. It involves the coordination of multiple steps and processes, including data transmission, task allocation and status tracking, to ensure that tasks are completed efficiently according to the predetermined order and rules.

[0003] Currently, workflow platforms can be used to assist in generating workflows based on conversations. Specifically, users can interact with the system in natural language to describe their needs in the conversational interface of the workflow platform, and then the platform can automatically generate corresponding workflow documents based on user input. However, this method usually requires the use of predefined fixed templates to generate workflow documents, which cannot support the generation of diversified workflow documents, resulting in poor versatility. Summary of the invention

[0004] The main purpose of this application is to provide a document generation method, device, equipment and storage medium based on a multimodal large language model, aiming to solve the technical problem that workflow documents in the prior art usually need to use predefined templates when generating, cannot support the generation of diversified workflows, and have poor versatility.

[0005] To achieve the above objectives, the present application proposes a document generation method based on a multimodal large language model, the method comprising:

[0006] Generate a workflow directed acyclic graph based on the workflows and operators in the tasks to be processed;

[0007] Obtaining node information of the workflow directed acyclic graph through a graph structure reading and analysis tool;

[0008] Generate a workflow document template based on user configuration parameter information through a preset document typesetting tool;

[0009] A workflow document corresponding to the task to be processed is generated based on the node information and the workflow document template.

[0010] In one embodiment, the step of generating a workflow directed acyclic graph based on the workflow and operators in the task to be processed includes:

[0011] Abstract the workflow of the task to be processed into a directed acyclic graph structure, and abstract the operators into graph nodes;

[0012] A workflow directed acyclic graph is generated based on the directed acyclic graph structure and the graph nodes.

[0013] In one embodiment, the step of generating the workflow document corresponding to the task to be processed based on the node information and the workflow document template includes:

[0014] Traversing the nodes in the directed acyclic graph of the workflow by topological sorting to determine the document structure hierarchy;

[0015] Determining whether the document structure hierarchy is logical by using a large language model;

[0016] If so, a workflow document corresponding to the task to be processed is generated based on the document structure hierarchy, the node information and the workflow document template.

[0017] In one embodiment, the step of traversing the nodes in the workflow directed acyclic graph by topological sorting to determine the document structure hierarchy includes:

[0018] Traversing and numbering the workflow directed acyclic graph in a topological sorting manner, and determining leaf nodes in the workflow directed acyclic graph in sequence;

[0019] Determine the node numbers corresponding to all nodes in the workflow directed acyclic graph based on the leaf node;

[0020] The step of judging whether the document structure hierarchy is logical by using a large language model comprises:

[0021] Whether the document structure hierarchy is logical is determined by a large language model based on the node numbers and operator information corresponding to each node.

[0022] In one embodiment, the step of generating the workflow document corresponding to the task to be processed based on the document structure hierarchy, the node information and the workflow document template includes:

[0023] Determine the document format according to the workflow document template;

[0024] Traversing the workflow directed acyclic graph according to the node numbers to set the document format for the nodes in the workflow directed acyclic graph according to the document format;

[0025] When the setting is completed, the node information is written into the workflow document template to generate a workflow document corresponding to the task to be processed.

[0026] In one embodiment, after the step of generating the workflow document corresponding to the task to be processed based on the node information and the workflow document template, the method further includes:

[0027] Generate text prompt information based on the function description information and output result corresponding to the operator;

[0028] Generate text analysis results and user analysis suggestion text based on the text prompt information through a multimodal large language model;

[0029] The text analysis result and the user analysis suggestion text are input into the workflow document.

[0030] In one embodiment, after the step of generating the workflow document corresponding to the task to be processed based on the node information and the workflow document template, the method further includes:

[0031] When receiving a user feedback request, obtaining user feedback information according to the user feedback request;

[0032] Determining whether to regenerate the workflow document according to the user feedback information;

[0033] If yes, return to the step of generating the workflow document corresponding to the task to be processed based on the node information and the workflow document template.

[0034] In addition, to achieve the above purpose, the present application also proposes a document generation device based on a multimodal large language model, the device comprising:

[0035] An acyclic graph generation module, used to generate a directed acyclic graph of a workflow based on the workflow and operators in the task to be processed;

[0036] A node information acquisition module, used to acquire the node information of the workflow directed acyclic graph through a graph structure reading and analysis tool;

[0037] A document template generation module is used to generate a workflow document template based on user configuration parameter information through a preset document typesetting tool;

[0038] A document generation module is used to generate a workflow document corresponding to the task to be processed based on the node information and the workflow document template.

[0039] In addition, to achieve the above-mentioned objectives, the present application also proposes a document generation device based on a multimodal large language model, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the document generation method based on a multimodal large language model as described above.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the document generation method based on the multimodal large language model as described above are implemented.

[0041] The present application provides a document generation method based on a multimodal large language model, and the present application discloses generating a workflow directed acyclic graph based on the workflow and operators in the task to be processed; obtaining the node information of the workflow directed acyclic graph through a graph structure reading and analysis tool; generating a workflow document template based on user configuration parameter information through a preset document typesetting tool; generating a workflow document corresponding to the task to be processed based on the node information and the workflow document template; compared with the prior art that uses a workflow platform based on dialogue to assist in generating a workflow, it is necessary to use a predefined fixed template and cannot support the generation of diversified workflows. Since the present invention can generate a workflow directed acyclic graph based on the workflow and operators in the task to be processed, and generate a workflow document based on the node information of the workflow directed acyclic graph and the workflow document template generated according to the user configuration parameter information, the technical problem that the prior art usually needs to use a predefined template when generating a workflow document, cannot support the generation of diversified workflows, and has poor versatility is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flowchart diagram of the first embodiment of the document generation method based on a multimodal large language model of the present application;

[0045] Figure 2 A flowchart diagram of the second embodiment of the document generation method based on a multimodal large language model of the present application;

[0046] Figure 3 A flowchart diagram of the third embodiment of the document generation method based on a multimodal large language model of the present application;

[0047] Figure 4 This is a schematic diagram of the module structure of a document generation device based on a multimodal large language model according to an embodiment of the present application;

[0048] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the document generation method based on the multimodal large language model in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of the embodiment of the present application is: generate a workflow directed acyclic graph based on the workflow and operators in the task to be processed; obtain the node information of the workflow directed acyclic graph through a graph structure reading and analysis tool; generate a workflow document template based on user configuration parameter information through a preset document typesetting tool; generate a workflow document corresponding to the task to be processed based on the node information and the workflow document template.

[0053] Since the prior art uses a workflow platform to assist in generating a workflow based on a dialogue and requires the use of a predefined fixed template, it is unable to support the generation of diversified workflows and has poor versatility.

[0054] The present application provides a solution that can generate a workflow directed acyclic graph based on the workflows and operators in the tasks to be processed, and generate a workflow document based on the node information of the workflow directed acyclic graph and a workflow document template generated according to user configuration parameter information, thereby solving the technical problem in the prior art that workflow documents usually need to use predefined templates when generating, cannot support the generation of diversified workflows, and have poor versatility.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a document generation device based on a multimodal large language model, etc. The following takes a document generation device based on a multimodal large language model as an example (hereinafter referred to as the device) to illustrate this embodiment and the following embodiments.

[0056] Based on this, the embodiment of the present application provides a document generation method based on a multimodal large language model, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the document generation method based on a multimodal large language model of the present application.

[0057] In this embodiment, the document generation method based on the multimodal large language model includes steps S10 to S40:

[0058] Step S10: Generate a workflow directed acyclic graph based on the workflow and operators in the task to be processed.

[0059] It should be noted that a workflow can be the steps and processes for executing tasks to be processed, and specific business goals can be achieved through workflows; operators can be the basic units for performing specific tasks or operations in workflows. In practical applications, each operator is responsible for a specific step, such as data processing, conversion, decision-making, data filtering, training of machine learning models, etc. In a workflow system, operators can be the core components used to define and automate each step in the workflow.

[0060] It should be noted that the above workflow directed acyclic graph may be a data structure for representing the dependency relationship between tasks in a workflow. In this embodiment, the device may abstract the workflow and operators of the tasks to be processed to construct a workflow directed acyclic graph.

[0061] Specifically, the step S10 includes: abstracting the workflow in the task to be processed into a directed acyclic graph structure, and abstracting the operator into a graph node; and generating a workflow directed acyclic graph based on the directed acyclic graph structure and the graph nodes.

[0062] In this embodiment, the workflow consists of a series of tasks or steps arranged in a specific order, wherein the operator, as the basic execution unit of the workflow, can be responsible for specific data processing tasks, such as data cleaning, conversion, feature extraction or model training. Since a complete workflow consists of operators and input and output, the workflow can be abstracted as a directed acyclic graph, with each operator as a graph node, and the sequence and logical relationship can be understood as directed edges. After completing the process of checking the legality of the workflow, the system can maintain the adjacency table to store the information of the directed acyclic graph in preparation for the follow-up, and additionally record the initial node, that is, the initial node with an in-degree of 0.

[0063] Step S20: Obtain node information of the workflow directed acyclic graph through a graph structure reading and analysis tool.

[0064] It is understandable that the above-mentioned graph structure reading and analysis tool can be a tool for reading and analyzing graph information, such as Igraph. In this embodiment, the graph structure reading and analysis tool can capture key information in the workflow directed acyclic graph and output it in a structured format for further processing and analysis.

[0065] It should be noted that the above-mentioned node information can be the attribute information of the operator node in the workflow directed acyclic graph, for example, the operator's identification ID, name, input, output, type and function description, etc. This information can be uniformly saved in the operator library in a structured file format (such as JSON).

[0066] In this embodiment, after the workflow of the task to be processed runs successfully, the device can obtain the directed acyclic graph node information of the workflow through the graph structure reading and analysis tool Igraph, so as to subsequently organize the text hierarchy according to this logical structure.

[0067] Step S30: Generate a workflow document template based on user configuration parameter information through a preset document typesetting tool.

[0068] It should be noted that the above-mentioned preset document typesetting tool can be a tool for generating high-quality documents and mathematical formula typesetting, such as LaTeX, wherein the document generated by LaTeX is a document format written in LaTeX code, which is commonly used in academic papers, books and technical documents. Its core feature is the efficient processing of complex typesetting requirements, especially mathematical formulas and tables. LaTeX documents can be converted to PDF or other formats through a specific compiler to generate professional and high-quality output results.

[0069] It should be understood that the above-mentioned user configuration parameter information may be the parameter information configured by the user for the document in the front-end interactive interface, such as font and title content, font size, font selection, line spacing and other typesetting formats, etc. This embodiment does not impose any restrictions on this.

[0070] In actual applications, the device can use LaTeX tools to control font selection, table settings, title size, structural hierarchy, image settings and other aspects of the text generation process through code, greatly reducing the difficulty of document typesetting, making the generated documents standardized, beautiful, and concise, and able to support complex mathematical formulas and complex data structure displays. Specifically, the device can use LaTeX tools (such as PyLaTex) to create tex files, set up a user interaction interface on the front end, obtain user-configured parameter information such as font and title content, font size, font selection, line spacing and other typesetting format parameters manually set by the user, and configure these user-configured parameter information. Then, the device can create a custom template based on user-defined parameters combined with a template engine (such as JinJa2) to generate a workflow document template that meets user needs.

[0071] Step S40: Generate a workflow document corresponding to the task to be processed based on the node information and the workflow document template.

[0072] In this embodiment, after generating a workflow document template, the device can run the template to generate a document that meets user needs, and then after obtaining the node information of the workflow directed acyclic graph, the node information can be written into the document, and finally a workflow document corresponding to the task to be processed is generated.

[0073] The present embodiment provides a document generation method based on a multimodal large language model, which discloses generating a workflow directed acyclic graph based on the workflow and operators in the task to be processed; obtaining the node information of the workflow directed acyclic graph through a graph structure reading and analysis tool; generating a workflow document template based on user configuration parameter information through a preset document typesetting tool; generating a workflow document corresponding to the task to be processed based on the node information and the workflow document template; compared with the prior art that uses a workflow platform based on dialogue to assist in generating a workflow, which requires the use of a predefined fixed template and cannot support the generation of diversified workflows, the present embodiment can generate a workflow directed acyclic graph based on the workflow and operators in the task to be processed, and generate a workflow document based on the node information of the workflow directed acyclic graph and the workflow document template generated according to the user configuration parameter information, thereby solving the technical problem that the prior art usually requires the use of a predefined template when generating a workflow document, cannot support the generation of diversified workflows, and has poor versatility.

[0074] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , Figure 2 A flowchart diagram is provided for Example 2 of the document generation method based on a multimodal large language model of the present application.

[0075] In this embodiment, step S40 includes steps S401 to S403:

[0076] Step S401: traverse the nodes in the workflow directed acyclic graph by topological sorting to determine the document structure hierarchy.

[0077] It should be understood that the topological sorting method can be a method for sorting vertices of a directed acyclic graph, which can make vertex u appear before vertex v in the sorting for each directed edge (u, v). This sorting method is usually used to solve dependency problems such as task scheduling and course arrangement.

[0078] It can be understood that the above-mentioned document structure hierarchy can be the organizational structure and logical relationship between the various parts within the document, including elements such as titles, paragraphs, and chapters. These elements can be arranged in a certain order and logic to form a complete and orderly document structure.

[0079] In this embodiment, the device can use the obtained directed acyclic graph to organize the document structure, and traverse the directed acyclic graph in sequence to read the node information to fill the document content. Specifically, the device can use the topological sorting search to obtain the operator information and set the document structure hierarchy according to the logical structure of the directed acyclic graph nodes, and output each node information to the document in sequence. Different data types, such as images and tables, are processed differently.

[0080] Step S402: judging whether the document structure hierarchy is logical by using a large language model.

[0081] It should be understood that a large language model is a machine learning model based on a neural network, which is designed to process and generate natural language text. These models are trained with large amounts of text data to learn language patterns, grammatical rules, and contextual associations. In practical applications, large language models can be used for a variety of natural language processing tasks, such as translation, text generation, question answering, etc. Their core lies in understanding and generating language, simulating the complexity and diversity of human language.

[0082] Further, the step S401 includes: traversing and numbering the workflow directed acyclic graph by topological sorting, and determining leaf nodes in the workflow directed acyclic graph in sequence; and determining node numbers corresponding to all nodes in the workflow directed acyclic graph based on the leaf nodes.

[0083] It can be understood that the leaf node may be a node without child nodes in the workflow directed acyclic graph, that is, a node with an in-degree of 0 in the workflow directed acyclic graph.

[0084] In this embodiment, the workflow directed acyclic graph can be traversed by topological sorting, and all nodes with an in-degree of 0 (i.e., leaf nodes) in the workflow directed acyclic graph are selected in turn. These nodes are regarded as the same level in the document structure, and then corresponding numbers can be added to these nodes. For example, the nodes of the first level can be numbered 1, 2, 3, ..., and the nodes of the second level can be numbered 1.1, 1.2, 2.1, 2.2, ..., and so on, where nodes with the same number prefix have a common parent node. During the traversal process, the operator identification information and number can be recorded to determine the node numbers corresponding to all nodes in the workflow directed acyclic graph.

[0085] Accordingly, the step S402 includes: judging whether the document structure hierarchy is logical based on the node numbers and operator information corresponding to each node through a large language model.

[0086] It should be noted that in order to ensure that the document structure organized in accordance with this method is reasonable, this embodiment can set up a verification mechanism. Specifically, the device can input the node number and operator information obtained from the directed acyclic graph of the workflow into the large language model, and then the large language model can be used to determine whether the hierarchical division of these steps is logical. If it is unreasonable, corresponding modification suggestions can be given. In addition, this embodiment can also call the knowledge graph of the operator library to analyze the dependencies between operators to verify the document structure. Finally, the system determines the hierarchical structure of the document.

[0087] Step S403: If it is in compliance, a workflow document corresponding to the task to be processed is generated based on the document structure hierarchy, the node information and the workflow document template.

[0088] Specifically, the step S403 includes: determining the document format according to the workflow document template; traversing the workflow directed acyclic graph according to the node number to set the document format of the nodes in the workflow directed acyclic graph according to the document format; when the setting is completed, writing the node information into the workflow document template to generate a workflow document corresponding to the task to be processed.

[0089] It should be noted that if the large language model determines that the document structure hierarchy is logical, the operator nodes in the workflow directed acyclic graph can be traversed in order of numbering according to the node numbers corresponding to each node in the workflow directed acyclic graph. For each node, the document format can be set, where the document format of the node can be determined based on the pre-generated workflow document template. Then, the device can read the node information such as the operator's name, input, output, function, specific description information, and write this node information into the document. After the document is finally written, a PDF file can be generated. Among them, the input and output data can distinguish different data types such as tables and images, and write them into the document in a reasonable form to achieve document typesetting.

[0090] In this embodiment, the device can use a graph structure reading and analysis tool to obtain the data structure of the complete workflow, and then parse the directed acyclic graph and use it to organize the structure of the document to obtain the document structure hierarchy to determine the section and title hierarchy of the article, so that the document has a clear context. At the same time, the device can obtain the input and output content of each operator node in the database and the specific function text of the operator, and present it in each subsection of the text, and then give the corresponding analysis content. This function allows users to understand each workflow subnode, especially the intermediate results.

[0091] In this embodiment, it is disclosed that nodes in a workflow directed acyclic graph are traversed by topological sorting to determine the document structure hierarchy; whether the document structure hierarchy is logical is determined by a large language model; if it is, a workflow document corresponding to the task to be processed is generated based on the document structure hierarchy, node information and workflow document template. Since this embodiment can generate a workflow document based on the document structure hierarchy, node information and workflow document template when the document structure hierarchy is logical, the generated workflow document has a clear context, which is convenient for users to understand the process and logic within the workflow, thereby improving the user experience.

[0092] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 3 , Figure 3 A flowchart diagram is provided for Example 3 of the document generation method based on a multimodal large language model of the present application.

[0093] In this embodiment, after step S40, the method further includes steps S50 to S70:

[0094] Step S50: Generate text prompt information based on the function description information and output result corresponding to the operator.

[0095] It should be understood that the above functional description information may be information used to describe the function of the operator; and the above output result is the output result of the node.

[0096] It can be understood that the above text prompt information can be input text or instructions provided to the large model, and the prompt information can guide the large model to generate a specific type of response.

[0097] Step S60: Generate text analysis results and user analysis suggestion text based on the text prompt information through a multimodal large language model.

[0098] It should be noted that the above text analysis results may be the results obtained after the large model analyzes the output results of the operator, for example, the processing of the original data by the operator, the results obtained, the characteristics of the new results, etc. This embodiment does not limit this.

[0099] It should be noted that the above-mentioned user analysis suggestion text can be a suggestion text for informing the user how to analyze the output results, for example, which parts of the table image data need to be focused on, the introduction of professional knowledge, etc. This embodiment does not limit this.

[0100] It should be understood that multimodal large language models can be language models that can process and generate multiple types of data (such as text, images, audio, etc.). They not only understand text, but can also be combined with other data types for comprehensive analysis and generation. For example, they can describe the scene in a picture, or convert text into corresponding images. This ability makes multimodal large language models widely used in many fields, such as natural language processing, computer vision, speech recognition, etc.

[0101] In this embodiment, for each node in the workflow directed acyclic graph, the function description information and output results of the operator can be obtained, and the text prompt information of the large model can be generated based on the function description information and output results of the operator. Then, the large model is used to generate analysis text and suggestions to obtain text analysis results and user analysis suggestion text.

[0102] Step S70: inputting the text analysis result and the user analysis suggestion text into the workflow document.

[0103] In practical applications, the multimodal large language model can support the generation of high-quality analysis reports. First, it can generate analytical text for the running results of workflow nodes, and can understand various data formats such as tables and images. In addition, it can also generate high-quality analysis text and support users to regenerate it to meet higher user needs. Since the large language model has rich professional knowledge, it can conduct targeted analysis of professional problems in specific fields, and at the same time realize academic text writing to meet the needs of work reports.

[0104] Furthermore, after step S40, the method further includes: upon receiving a user feedback request, obtaining user feedback information according to the user feedback request; determining whether to regenerate the workflow document according to the user feedback information; and if so, returning to the step of generating the workflow document corresponding to the task to be processed based on the node information and the workflow document template.

[0105] It should be understood that the above-mentioned user feedback request can be a request to instruct the device to regenerate the workflow document. In this embodiment, if the user is not satisfied with the generated workflow document, the user feedback request can be sent to the device. After receiving the user feedback request, the device can parse the user feedback request to obtain user feedback information, and then determine whether the user needs to regenerate the workflow document based on the user feedback information. If necessary, it can return to the step of generating the workflow document based on the node information and the workflow document template, and call the large model to regenerate the analysis results and the suggested text until a report that satisfies the user is generated. At the same time, this embodiment can provide a user dialogue interaction opinion function, so that the user can feedback modification opinions to the large model through natural language to help the model generate higher quality results.

[0106] It should be noted that in the document generation method based on the multimodal large language model provided by this solution, users can drag the workflow or run a predefined workflow template, and then select "Generate Report" to preview the workflow report document online or download it locally. In addition, this solution supports interactive functions such as regenerating documents or user-defined fonts, giving users great autonomy, while providing detailed process analysis and operator introductions, as well as related knowledge, so that users can fully understand the operation process of the workflow, greatly reducing the user's usage threshold. At the same time, the document can also provide analysis suggestions to guide users who lack professional knowledge.

[0107] In this embodiment, it is disclosed that text prompt information is generated based on the function description information and output results corresponding to the operator; text analysis results and user analysis suggestion text are generated based on the text prompt information through a multimodal large language model; the text analysis results and user analysis suggestion text are input into a workflow document. In this embodiment, the text analysis results and user analysis suggestion text corresponding to the workflow document can be generated through a multimodal large language model. Since the large language model has rich professional knowledge, it can conduct targeted analysis of professional problems in specific fields, and at the same time realize academic text writing to meet the needs of work reports.

[0108] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the document generation method based on a multimodal large language model of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0109] This application also provides a document generation device based on a multimodal large language model, please refer to Figure 4 , the document generation device based on the multimodal large language model includes:

[0110] An acyclic graph generation module 10, used to generate a workflow directed acyclic graph based on the workflow and operators in the task to be processed;

[0111] A node information acquisition module 20, used to acquire the node information of the workflow directed acyclic graph through a graph structure reading and analysis tool;

[0112] The document template generation module 30 is used to generate a workflow document template based on user configuration parameter information by using a preset document typesetting tool;

[0113] The document generation module 40 is used to generate a workflow document corresponding to the task to be processed based on the node information and the workflow document template.

[0114] The document generation device based on a multimodal large language model provided by the present application adopts the document generation method based on a multimodal large language model in the above-mentioned embodiment, which can solve the technical problem that workflow documents in the prior art usually need to use predefined templates when generating, cannot support the generation of diversified workflows, and have poor versatility. Compared with the prior art, the beneficial effects of the document generation device based on a multimodal large language model provided by the present application are the same as the beneficial effects of the document generation method based on a multimodal large language model provided by the above-mentioned embodiment, and the other technical features of the document generation device based on a multimodal large language model are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0115] The present application provides a document generation device based on a multimodal large language model, and the document generation device based on a multimodal large language model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the document generation method based on the multimodal large language model in the above-mentioned embodiment 1.

[0116] Reference below Figure 5 , which shows a schematic diagram of the structure of a document generation device based on a multimodal large language model suitable for implementing an embodiment of the present application. The document generation device based on a multimodal large language model in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The document generation device based on the multimodal large language model shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0117] like Figure 5As shown, the document generation device based on the multimodal large language model may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the document generation device based on the multimodal large language model are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the document generation device based on the multimodal large language model to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a document generation device based on a multimodal large language model with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0118] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0119] The document generation device based on a multimodal large language model provided by the present application adopts the document generation method based on a multimodal large language model in the above-mentioned embodiment, which can solve the technical problem of workflow document generation. Compared with the prior art, the beneficial effects of the document generation device based on a multimodal large language model provided by the present application are the same as the beneficial effects of the document generation method based on a multimodal large language model provided by the above-mentioned embodiment, and the other technical features in the document generation device based on a multimodal large language model are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0120] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0121] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0122] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the document generation method based on the multimodal large language model in the above-mentioned embodiment.

[0123] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0124] The computer-readable storage medium may be included in a document generation device based on a multimodal large language model; or may exist independently without being assembled into a document generation device based on a multimodal large language model.

[0125] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a document generation device based on a multimodal large language model, the document generation device based on the multimodal large language model: generates a workflow directed acyclic graph based on the workflow and operators in the task to be processed; obtains the node information of the workflow directed acyclic graph through a graph structure reading and analysis tool; generates a workflow document template based on user configuration parameter information through a preset document typesetting tool; and generates a workflow document corresponding to the task to be processed based on the node information and the workflow document template.

[0126] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0129] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned document generation method based on a multimodal large language model, and can solve the technical problem that workflow documents in the prior art usually need to use predefined templates when generating, cannot support the generation of diversified workflows, and have poor versatility. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the document generation method based on a multimodal large language model provided in the above-mentioned embodiment, and will not be repeated here.

[0130] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A document generation method based on a multimodal large language model, characterized in that: The method includes: Generate a workflow directed acyclic graph based on the workflows and operators in the tasks to be processed; Obtaining node information of the workflow directed acyclic graph through a graph structure reading and analysis tool; Generate a workflow document template based on user configuration parameter information through a preset document typesetting tool; A workflow document corresponding to the task to be processed is generated based on the node information and the workflow document template.

2. The method according to claim 1, characterized in that The step of generating a workflow directed acyclic graph based on the workflow and operators in the task to be processed includes: Abstract the workflow of the task to be processed into a directed acyclic graph structure, and abstract the operators into graph nodes; A workflow directed acyclic graph is generated based on the directed acyclic graph structure and the graph nodes.

3. The method according to claim 1, characterized in that The step of generating the workflow document corresponding to the task to be processed based on the node information and the workflow document template includes: Traversing the nodes in the directed acyclic graph of the workflow by topological sorting to determine the document structure hierarchy; Determining whether the document structure hierarchy is logical by using a large language model; If so, a workflow document corresponding to the task to be processed is generated based on the document structure hierarchy, the node information and the workflow document template.

4. The method according to claim 3, characterized in that The step of traversing the nodes in the workflow directed acyclic graph by topological sorting to determine the document structure hierarchy includes: Traversing and numbering the workflow directed acyclic graph in a topological sorting manner, and determining leaf nodes in the workflow directed acyclic graph in sequence; Determine the node numbers corresponding to all nodes in the workflow directed acyclic graph based on the leaf node; The step of judging whether the document structure hierarchy is logical by using a large language model comprises: Whether the document structure hierarchy is logical is determined by a large language model based on the node numbers and operator information corresponding to each node.

5. The method according to claim 4, characterized in that The step of generating the workflow document corresponding to the task to be processed based on the document structure hierarchy, the node information and the workflow document template includes: Determine the document format according to the workflow document template; Traversing the workflow directed acyclic graph according to the node numbers to set the document format for the nodes in the workflow directed acyclic graph according to the document format; When the setting is completed, the node information is written into the workflow document template to generate a workflow document corresponding to the task to be processed.

6. The method according to any one of claims 1 to 5, characterized in that After the step of generating the workflow document corresponding to the task to be processed based on the node information and the workflow document template, the method further includes: Generate text prompt information based on the function description information and output result corresponding to the operator; Generate text analysis results and user analysis suggestion text based on the text prompt information through a multimodal large language model; The text analysis result and the user analysis suggestion text are input into the workflow document.

7. The method according to any one of claims 1 to 5, characterized in that After the step of generating the workflow document corresponding to the task to be processed based on the node information and the workflow document template, the method further includes: When receiving a user feedback request, obtaining user feedback information according to the user feedback request; Determining whether to regenerate the workflow document according to the user feedback information; If yes, return to the step of generating the workflow document corresponding to the task to be processed based on the node information and the workflow document template.

8. A document generation device based on a multimodal large language model, characterized in that: The device comprises: An acyclic graph generation module, used to generate a directed acyclic graph of a workflow based on the workflow and operators in the task to be processed; A node information acquisition module, used to acquire the node information of the workflow directed acyclic graph through a graph structure reading and analysis tool; A document template generation module is used to generate a workflow document template based on user configuration parameter information through a preset document typesetting tool; A document generation module is used to generate a workflow document corresponding to the task to be processed based on the node information and the workflow document template.

9. A document generation device based on a multimodal large language model, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the document generation method based on a multimodal large language model as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the document generation method based on a multimodal large language model as described in any one of claims 1 to 7 are implemented.

Images