Composite content generation method
By building a knowledge base-driven language model for content generation, and combining manual operation information, the hallucination problems and human-computer collaboration problems in the content generation of large language models are solved, achieving high-quality content generation and effective human-computer interaction.
Patent Information
- Application Number
- CN202411989230.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-09
AI Technical Summary
When using large language models to generate content, the prior art faces hallucinations and the problem of difficulty in effectively perceiving manual operations, resulting in low quality of content generation and difficulty in working together with humans.
By building a knowledge base with graph data as the basic structure, the language model is driven to generate content, and manual operation information is obtained through software tool sampling, and query prompts are generated by combining knowledge base information to achieve the coordination of human-computer interaction and content generation.
It effectively improves the accuracy and quality of content generation, reduces the dependence on artificial technical capabilities, realizes the organic combination of language models and artificial, and improves the efficiency and effect of human-computer interaction.
Smart Images
Figure CN119962485A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information technology, and more particularly to a composite content generating method. Background Art
[0002] Content generation based on large language models or multimodal large language models has gradually become an important technical means to achieve process automation in many industries, especially by replacing traditional robotic process automation with intelligent agents. However, the hallucination problem of machine learning models themselves often becomes the main obstacle to ensuring content quality. Existing technologies include combining with static tools or using databases for assistance. In the field of human-computer interaction / collaboration, it is particularly important to improve the accuracy of model-generated content. At the same time, the model needs to perceive information about manual operations and use this as the basis for the next step of work.
[0003] In areas that require digital intelligence, such as building smart cities and constructing digital spacetime, AI content generation has become an increasingly important auxiliary decision-making and processing capability. In particular, intelligent agent technology based on large language models has increasingly become a mainstream technical means of content production or task planning. The current situation also places high demands on the output quality of the model, especially in work areas with frequent human-computer interaction. Relying solely on models to solve content generation faces the following two problems: 1. Content quality, how to avoid the illusion problem of large language models; 2. Perception of manual operations, how to effectively collaborate with humans.
[0004] Therefore, new solutions need to be proposed to solve this problem. Summary of the invention
[0005] To solve the above problems, this invention proposes content generation and human-computer interaction of language models driven by knowledge bases. This technology can flexibly adapt to different types of language models and software tools. By building a knowledge base as the core driver, the language model and artificial interaction are organically combined on the basis of ensuring the quality of content output.
[0006] The above technical objectives of the present invention are achieved through the following technical solutions: A composite content generation method comprises the following steps: S1. Given an application domain, construct a knowledge base with graph data as the basic structure, wherein the knowledge base includes work scripts, unit data and their associations; S2. fine-tuning the language model through instructions so that it can convert questions in the application field into accurate queries to the knowledge base, wherein the language model is a pre-trained large language model or a pre-trained multimodal large language model; S3. Build a software interface that interacts with the knowledge base and language model, and convert the knowledge base query results into instructions that can directly control the application software; S4. Build an application software status sampling mechanism to capture and collect manual operation information in real time, and combine the collected operation information with the knowledge base content to assist the language model in generating the next query prompt; S5. Realize the collaborative operation between human, model and knowledge base through interactive system, including: Users trigger the language model to generate knowledge base queries through text questions; The knowledge base query results are parsed and then the software operation is performed; The system continuously samples manual operations and provides feedback to optimize subsequent interactions.
[0007] The present invention is further configured as follows: the construction of the knowledge base includes: constructing a knowledge base for a given application field; realizing a program code parsing interface between the knowledge base and application software, and converting knowledge base query results into operating instructions executable by the application software.
[0008] The present invention is further configured as follows: the knowledge base uses graph data as a basic structure, and represents the relationship between various types of knowledge, scripts and data through a knowledge graph. The specific structure includes: A1. Work script node: stores the execution script of a specific action; A2. Unit data node: stores configuration data for specific knowledge or actions; A3. Association edge: represents the logical association between scripts, between unit data, and between scripts and data.
[0009] The present invention is further configured as follows: the knowledge base has an adaptive expansion mechanism, including: continuously collecting new work scripts and unit data; automatically analyzing and establishing an association between new content and existing knowledge; and continuously expanding and optimizing the knowledge system in an iterative manner.
[0010] The present invention is further configured as follows: the training language model is trained by fine-tuning instructions to align the pre-trained large language model or the pre-trained multimodal large language model to the content generation capability that cooperates with the application knowledge base.
[0011] The present invention is further configured as follows: the workflow of the application software state sampling mechanism includes: B1. Receive prompt questions input by the user; B2. Generate knowledge base query instructions from the language model; B3. Call the query engine to retrieve the knowledge base content; B4. Capture the user's software operation status in real time; B5. Feedback the operation status information to the language model for optimizing subsequent queries.
[0012] In summary, the present invention has the following beneficial effects: the present invention proposes a content generation system that combines a language model with a knowledge base, which can effectively generate content based on human-computer interaction. Users can combine with the content generation system by asking text questions or operating auxiliary design tools, and jointly operate application software for interactive design and development. The technology can flexibly adapt to different types of language models and software tools, and by building a knowledge base as the core drive, the language model and manual work are organically combined on the basis of ensuring the quality of content output. This intelligent technology greatly reduces the dependence on manual technical capabilities. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 It is a flowchart of the human-computer interaction system framework in the present invention; Figure 2 It is a schematic diagram of the process of the knowledge base graph data in the present invention; Figure 3 It is a schematic diagram of the process of realizing human-computer interaction in the present invention; Figure 4 A flowchart of the content generation and human-computer interaction in the present invention. DETAILED DESCRIPTION
[0014] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention. Example 1
[0015] The present invention proposes a composite content generation method, which constructs a knowledge base as the core engine to drive a large language model to generate content, and uses software tools to sample and obtain manual operation information, combines manual actions with knowledge base information to provide content generation prompt information for the large language model, and jointly operates application software, thereby realizing a complete set of content generation and human-computer interaction technology framework based on verifiable knowledge.
[0016] This invention consists of 4 parts: 1. Knowledge base driven content generation and human-computer interaction system framework; 2. Knowledge base engine construction method; 3. Language model integration; 4. Human-computer interaction realization.
[0017] First, the core technical framework of this invention is a content generation and human-computer interaction system driven by a knowledge base. The core of the system is the knowledge base, which drives the language model to generate content and interact with humans (see Appendix Figure 1 ). It is specifically manifested in the following 4 main steps: S1: Given an application domain, build a corresponding knowledge base; S2: Train a language model to generate queries to the knowledge base by asking questions about the application domain; S3: Build a software interface that interacts with the knowledge base and language model so that the knowledge base query results can be directly used to control the application software; S4: Build an application software status sampling mechanism to collect manual operation information. The collected information is combined with knowledge base information to assist the language model in generating query prompts.
[0018] Here, human-computer interaction is achieved by people and models + knowledge bases jointly operating application software.
[0019] Based on the application framework constructed in the above four steps, the process of content generation and human-computer interaction (see Appendix Figure 4 ) as shown: Users can enter prompt questions, and the language model will generate content related to the knowledge base query and call the query engine to obtain the knowledge base content. At the same time, users can directly operate the application software or make further operations based on the system operation. The sampling tool captures the application software operation status and feeds it back to the language model so that it can effectively perceive and align the current operation status when generating the next knowledge base query. The entire process manually drives the language model to operate the application software or directly operates the application software by people. The model operation needs to be carried out through the knowledge base, integrating the existing information in the knowledge base on the basis of saving manpower to achieve efficient human-computer collaboration.
[0020] Secondly, the construction of the knowledge base engine in the present invention includes two parts: 1. Building a knowledge base for a given application field; 2. Implementing the knowledge base and application software operation interface. The knowledge base is based on graph data (see Appendix Figure 2 ), using knowledge graphs to represent the relationships between various types of knowledge or scripts and data, making it easier to generate queries. The knowledge base for a given application domain includes the following information about the domain: A1: Work script: an execution script for a specific action, such as an operation script for drawing a specific element on a drawing tool; A2: Unit data: configuration data for specific knowledge or actions, such as the size and attributes of specific primitives in drawing tools, or application configuration in program development tools; A3: The relationship between scripts, between unit data, and between scripts and data: Based on domain knowledge, the content of 1 and 2 is associated; it serves as a connection between graph nodes.
[0021] The above contents A1 and A2 correspond to nodes in the knowledge graph, and A3 corresponds to edges in the knowledge graph. The construction of the knowledge base is carried out in an iterative manner in actual applications, that is, with the continuous development of the system, new work scripts, unit data and associations are continuously integrated into the knowledge base to expand the knowledge system. The interface between the knowledge base and the application software is reflected in the nodes or subgraphs in the knowledge graph as query results, which are executed or presented in the software tool. The implementation of this interface varies depending on the specific application software to be connected.
[0022] At the same time, in the technical framework of this invention, the language model is responsible for connecting to artificial prompt questions to generate knowledge base queries, and it also needs to integrate the sampling information returned by the application software to facilitate the generation of more accurate status queries. The key to the language model capability is to generate accurate knowledge base queries. To this end, a pre-trained large language model or a pre-trained multimodal large language model is selected, and instruction fine-tuning training is implemented by constructing question and answer corpora in specific application fields, and aligning it to the content generation capabilities that match the application knowledge base. In this invention, the language model is only required to have basic common sense capabilities and can support instruction fine-tuning training, so it can flexibly adapt to various types of large language models.
[0023] Finally, human-computer interaction is reflected in the conversion of language models into knowledge base queries through prompting questions, and then the query results are parsed to generate operation scripts or unit data that can be used by the application software. Another way is to manually operate the application software, and then the sampling tool captures the operation information and feeds back to the language model so that it can perceive the current state, so that the information can be effectively aligned the next time the knowledge base is queried. Human-computer interaction is not only used for collaboration, but the information from manual operations can be used to enrich the knowledge base and assist in the iterative expansion of the knowledge base. Therefore, the technical framework of this invention has the ability of self-learning with continuous optimization (see Appendix). Figure 3 ). Example 2
[0024] A composite content generation method is applied to SketchUp drawing software, and the workflow of knowledge base content conversion software tool operation includes the following steps: C1. Get data units by querying the knowledge base; C2. Determine the type of data unit (based on the annotation information of the data unit); C3. If it is a working script, call the script converter to convert the script into a Sketchup drawing script; C4. If it is graphic element information, call the drawing script generation tool to convert the graphic element information into a Sketchup drawing script.
[0025] Specifically, when the user inputs specific drawing instructions or questions, the language model will parse these instructions or questions and generate corresponding knowledge base query requests. After receiving the query request, the knowledge base query engine retrieves the relevant work scripts and graphic element information in the knowledge base. According to the search results, the system will call the corresponding software tool interface to convert the query results into scripts that can be understood and executed by the SketchUp drawing software. After receiving the converted script, the SketchUp drawing software performs the drawing operation to complete the user's drawing instructions. During the user's operation of the SketchUp drawing software, the application software state sampling mechanism will capture the user's operation status and feed this information back to the language model. The language model combines the user's operation status and knowledge base information to generate prompt information for effective perception and alignment of the current operation status during the next knowledge base query. If the user needs further interaction or has new questions, the system will repeat the above operations and continue to provide assistance and feedback. If there are no problems, it ends.
[0026] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A composite content generation method, characterized in that: The following steps are involved: S1. Given an application domain, construct a knowledge base with graph data as the basic structure, wherein the knowledge base includes work scripts, unit data and their associations; S2. fine-tuning the language model through instructions so that it can convert questions in the application field into accurate queries to the knowledge base, wherein the language model is a pre-trained large language model or a pre-trained multimodal large language model; S3. Build a software interface that interacts with the knowledge base and language model, and convert the knowledge base query results into instructions that can directly control the application software; S4. Build an application software status sampling mechanism to capture and collect manual operation information in real time, and combine the collected operation information with the knowledge base content to assist the language model in generating the next query prompt; S5. Realize the collaborative operation between human, model and knowledge base through interactive system, including: Users trigger the language model to generate knowledge base queries through text questions; The knowledge base query results are parsed and then the software operation is performed; The system continuously samples manual operations and provides feedback to optimize subsequent interactions.
2. A composite content generation method according to claim 1, characterized in that: The construction of the knowledge base includes: constructing a knowledge base in a given application field; realizing a program code parsing interface between the knowledge base and the application software, and converting the knowledge base query results into operating instructions executable by the application software.
3. A composite content generation method according to claim 2, characterized in that: The knowledge base uses graph data as its basic structure, and uses knowledge graphs to represent the relationships between various types of knowledge, scripts, and data. The specific structure includes: A1. Work script node: stores the execution script of a specific action; A2. Unit data node: stores configuration data for specific knowledge or actions; A3. Association edge: represents the logical association between scripts, between unit data, and between scripts and data.
4. A composite content generation method according to claim 2, characterized in that: The knowledge base has an adaptive expansion mechanism, including: continuously collecting new work scripts and unit data; automatically analyzing and establishing the association between new content and existing knowledge; and continuously expanding and optimizing the knowledge system through iteration.
5. A composite content generation method according to claim 1, characterized in that: The training language model is trained by fine-tuning instructions to align the pre-trained large language model or the pre-trained multimodal large language model to the content generation capability that matches the application knowledge base.
6. A composite content generation method according to claim 1, characterized in that: The workflow of the application software status sampling mechanism includes: B1. Receive prompt questions input by the user; B2. Generate knowledge base query instructions from the language model; B3. Call the query engine to retrieve the knowledge base content; B4. Capture the user's software operation status in real time; B5. Feedback the operation status information to the language model for optimizing subsequent queries.