Multi-point distributed interactive task system

Abstract

The invention relates to a multipoint distributed interaction system. The system comprises a resource management module, an agent module, a script generation module, a task module and an equipment control module which are connected in sequence, the task module, the equipment control module and the virtual human equipment are in communication connection, and the equipment control module and the field equipment are in communication connection. According to the method, the plot scripts and the interaction tasks can be automatically generated according to the keywords, tourists are supported to interact at multiple interaction points at the same time, and meanwhile, on-site interaction equipment is uniformly controlled according to behaviors of users. A gRPC communication protocol is adopted between the system and on-site interaction nodes and interaction equipment, so that data synchronization and real-time interaction among the nodes are ensured. According to the invention, seamless interaction experience is provided for tourists through multi-point interaction capability and a high-performance communication mechanism.

Description

Technical Field

[0001] The invention discloses a scenic spot multi-point distributed interactive task system, belonging to the field of tourism and virtual human interaction intersection. Background Art

[0002] Virtual human interaction belongs to the field of artificial intelligence. It is a kind of interaction between users and virtual characters generated by computers. Virtual characters can imitate human appearance, movements, voices and behaviors, and provide users with a natural interactive experience. Conventional multi-point distributed interactive experience is usually completed by dedicated staff or tour guides using manpower, so the actual effect is affected by the status of the staff. Compared with traditional manual interaction, virtual human interaction can better complete interactive tasks due to the characteristics of virtual humans such as high intelligence, strong stability and strong communication. Summary of the invention

[0003] The present invention is to realize a convenient multi-point distributed interactive task system in a scenic area, solve the problems of labor costs and environmental factors in a real environment through virtual human interaction technology, and simplify the creation process through automatic scripts generated by artificial intelligence. Artificial intelligence automatically generates plot scripts and interactive tasks based on keywords input by users, and sends them to the associated virtual human device. After receiving the relevant plot information, the virtual human device provides corresponding feedback based on the interactive behavior given by the user, and sends this information to other virtual human devices to ensure data synchronization. Finally, a complete distributed interactive task in the scenic area is completed.

[0004] The multi-point distributed interactive task system of a scenic spot described in the present invention comprises a resource management module, an intelligent body module, a scenario generation module, a task module, and a device control module connected in sequence; the task module and the device control module establish a communication connection with the virtual human device, and the device control module establishes a communication connection with the on-site device;

[0005] The script generation module is used to receive keywords input by the user, generate plot scripts and interactive tasks, and send data to the resource management module and the task module.

[0006] The intelligent agent module is used to simulate a virtual human according to the intelligent agent settings set by the user, and provide accurate and fast interactive feedback based on the behavior given by the user.

[0007] The resource management module is responsible for managing various resources in the system, including virtual human models, audio files, scene materials, etc., and ensuring efficient use and rapid scheduling of resources according to the plot and interactive task content.

[0008] The task module is responsible for receiving the interactive tasks generated by the script generation module, assigning them to the corresponding virtual human devices, receiving the information returned by the interaction points and synchronizing them to other virtual human devices.

[0009] The device control module is responsible for receiving the control signal returned by the virtual human device and sending it to the corresponding on-site device to implement the call of on-site resources.

[0010] Furthermore, the script generation module adopts artificial intelligence technology to generate relatively reasonable plot script data for users.

[0011] Furthermore, the intelligent agent module uses natural language processing technology, computer graphics technology and artificial intelligence technology to provide accurate and rapid feedback on user behavior.

[0012] Furthermore, the resource management module also has resource import and export functions, which is used for users to quickly organize resources.

[0013] Furthermore, after receiving the plot script and interactive task data, the task module sends the data to the virtual human device through gRPC communication and ensures data consistency.

[0014] Furthermore, the device control module is connected to the virtual human device through gRPC communication, and after receiving the feedback data, sends the data to the corresponding field device through gRPC communication.

[0015] Furthermore, the data sent by the task module includes the following contents: (1) task type, (2) trigger type, (3) field matching logic, and (4) field equipment used.

[0016] Furthermore, the data sent by the device control module includes the following: (1) device number, (2) specific operation, and (3) required resources.

[0017] Furthermore, the trigger types in the task module launch data include voice interaction and trigger recognition, and the field matching logic includes semantic similarity, text equality, keywords and matching any text.

[0018] The advantages of the present invention are that the gRPC communication protocol is used between the system and the on-site interactive nodes and interactive devices to ensure data synchronization and real-time interaction between the nodes. The innovation lies in its multi-point interactive capability and high-performance communication mechanism, which provides visitors with a seamless interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0020] Figure 1 A schematic diagram of a multi-point distributed interactive task system provided for this application.

[0021] Figure 2 A schematic diagram of the workflow of a multi-point distributed interactive task system provided for this application.

[0022] Figure 3 A schematic diagram of a script generation module provided for this application.

[0023] Figure 4 A schematic diagram of a knowledge graph provided for this application.

[0024] Figure 5 A schematic diagram of an example of an intelligent agent prompt word provided for this application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0027] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0028] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0029] Embodiment 1:

[0030] like Figure 1 As shown, a multi-point distributed interactive system provided by the present invention is used in conjunction with an interactive device. The system includes a resource management module, an intelligent body module, a script generation module, a task module, and a device control module connected in sequence; the task module and the device control module establish a communication connection with the virtual human device, and the device control module establishes a communication connection with the on-site device;

[0031] The script generation module receives keywords input by the user, generates corresponding scripts and interactive tasks, and sends the data to the resource management module and the task module.

[0032] The intelligent agent module is used to simulate a virtual person according to the intelligent agent settings set by the user, and receive the user's behavior, including sound and action, and generate corresponding reasonable feedback using methods such as natural language processing, computer graphics and artificial intelligence.

[0033] The resource management module is used to manage various resources imported by users, including virtual human models, audio files, scene materials, etc., and quickly dispatch corresponding resources according to task requirements.

[0034] The task module is used to receive the plot script and interactive tasks generated by the script generation module, and send the data to the virtual human device on site.

[0035] The device control module is used to receive the control signal returned by the virtual human device and send it to the corresponding on-site device to implement the call of on-site resources.

[0036] The script generation module adopts a transformer-based model (including but not limited to GPT-4), uses artificial intelligence technology to generate coherent and creative plot script data based on keywords entered by users, and optimizes the plot script through fine-tuning technology to fit the theme and style of the scenic spot.

[0037] The intelligent agent module combines natural language processing technology, computer graphics technology and artificial intelligence technology, adopts deep learning algorithm to provide the intelligent agent with the ability to understand and generate natural language, and uses OpenGL and DirectX technology to create a highly realistic 3D virtual character model.

[0038] The resource management module adopts an efficient database management system, such as MySQL or MongoDB, to manage various resources imported by users, including virtual human models, audio files, scene materials, etc., and has resource import and export functions, and quickly organizes resources through an automated tag system.

[0039] After receiving the plot script and interactive task data, the task module sends the data to the virtual human device through gRPC communication technology to ensure data consistency, and introduces a priority-based task scheduling algorithm to ensure reasonable task allocation.

[0040] The device control module is connected to the virtual human device through gRPC communication. After receiving the feedback data, it sends the data to the corresponding on-site device through gRPC communication to realize intelligent calling of on-site resources.

[0041] The task module sends data with the following contents: (1) task type; (2) trigger type; (3) field matching logic; (4) field equipment used, wherein the trigger type includes voice interaction and trigger recognition, and the field matching logic includes semantic similarity, text equality, keywords and matching any text. Semantic similarity refers to matching based on the similarity between the text and the voice input by the user, text equality refers to that the text and the voice input by the user are completely equal, keywords refers to the presence of the required keywords in the voice input by the user, and matching any text means that any voice input by the user can pass.

[0042] The data sent by the device control module includes the following: (1) device number; (2) specific operation; (3) required resources, and the Internet of Things (IoT) technology is used to connect with the on-site equipment through a wireless network to achieve efficient and reliable communication between devices.

[0043] like Figure 2 The figure is a flow chart of a multi-point distributed interactive task system for a scenic spot provided by an embodiment of the present application, which is described in detail as follows:

[0044] In step one, the user uploads the required resource files, which should at least include knowledge base files, multimedia files and device information.

[0045] The knowledge base file should be constructed in the form of knowledge graph, such as Figure 4 As shown. Multimedia files include audio and video files, image files, special effect files and other specific multimedia data used on on-site devices. Device information should include device name, model, routing address and corresponding function API interface.

[0046] The API interface in the device information can be stored in the following data structure: {base_url: string; path_template: string; method: string; content_type; authorization_type: string}, where base_url represents the base URL address, path_template represents the path template, method represents the request method, content_type represents the content type, and authorization_type represents the authorization type. For example:

[0047] {base_url:192.168.1.55; path_template:api / rpc / ${scene_id}; method:PUT; content_type:json; authorization_type:Bearer;}, as an example, the specific parameters and variable names can be replaced by other reasonable variable names.

[0048] Among them, a knowledge graph should include the following parts: Node information is used to represent objects or concepts in the real world, such as explosions and burning objects. Attribute information is used to represent the characteristics or descriptions of each node, such as high temperature and sound. Relationship information is used to represent the relationship or association type between different nodes, such as belongs to or is located in. Semantic label information is used to add information labels to nodes or relationships, enhance the semantic understanding ability of the graph, and facilitate subsequent natural language processing tasks. Inference rules are used to derive rules for new knowledge based on existing knowledge, adding reasoning capabilities to the knowledge graph.

[0049] All uploaded resources will be stored in the resource management module. Before uploading, the natural language processing library will be used to analyze the name and content of the resources, and one or more tags will be attached to each resource. For example, a video file of fireworks will have two tags, video and fireworks, to facilitate users to find resource files with the same tag later.

[0050] In step 2, the user sets the relevant information of the agent, including the agent prompt words, the associated knowledge base, etc.

[0051] When the user sets the agent prompt word, the prompt word should have the following Figure 5The format shown in the figure contains two parts: agent description and constraints. The agent description is used to provide task guidance and context to the agent. Constraints can improve the accuracy and relevance of AI answers. At the same time, setting format constraints can constrain the scope of topics and make the generated content meet specific structural requirements. For example, set an agent to generate the main goal. The prompt words can be {You are a plot design master. Please design the main goal of the story based on the theme, story background, player characters, non-player characters, and plot outline entered by the user. Design requirements: You only need to design the main goal, and no other content needs to be designed. Output format: 1. Output your results in JSON format.}, as an example, the specific content can be replaced by any suitable statement.

[0052] In step three, the user inputs a script introduction, which should at least include script information such as script style, script name and script description. The script generation module generates corresponding plot scripts and interactive task data based on the information input by the user.

[0053] Among them, the script information such as script style, script name and script description is used to describe the theme of the script. The information should be reasonable and relevant to ensure the coherence of the generated plot script. The specific format example is as follows: {"theme":"Looking for "Zheng He's Navigation Map","genre":"Suspense Reasoning","description":"Zheng He sailed to the West seven times and communicated with Western countries. He once drew "Zheng He's Navigation Map" to record all the routes of the voyage. Yanshan Garden is located in Taicang, Suzhou. It is a famous garden in Jiangnan and is known as the first garden in the southeast. It was originally the private garden of Wang Shizhen, a Shangshu writer in the Ming Dynasty. Please use the Ming Dynasty as the background of the era and "Yanshan Garden" and the scenes in the garden as the story locations."}.

[0054] The script generation module uses GPT-4 with a fine-tuning interface. GPT-4 can be replaced by any large language model that allows fine-tuning, including but not limited to artificial intelligence large language models. Figure 3 As shown in the figure, when using the large language model (LLM), the knowledge base and prompt words in step one and step two are used as fine-tuning parameters, and adapter tuning is used as the fine-tuning method. For a complete plot script, including multiple downstream tasks such as plot synopsis, character setting, and scene generation, if the same large language model is used on different downstream tasks, the results may be different. At the same time, fine-tuning a large model for all parameters of each downstream task will take a lot of computing resources and time. Therefore, the adapter fine-tuning method is used to use the specific knowledge base as the knowledge of the downstream task to quickly adapt to each downstream task, thereby ensuring that the large language model has a good generation effect for downstream tasks such as plot synopsis and character setting.

[0055] The specific process is as follows: before the large language model is generated, the prompt words and knowledge base are used as fine-tuning parameters, and parameter training is performed at the adapter layer. The trained parameters are used to fine-tune the large language model, so as to adapt to the current task while ensuring general performance. After that, the script information input by the user is generated through the fine-tuned large language model, and the generated plot script and interactive task data are sent to the task module.

[0056] In step 4, the task module sends the plot script and interactive task data information to the on-site virtual human equipment and equipment control module through grpc communication.

[0057] The interactive task information uses a priority-based task scheduling algorithm to ensure the reasonable progress of the task. When distributing interactive tasks, according to their importance to the plot I, the position of the task in the plot L, the time required to complete T, and different weights abc, after calculation aI+bL+cT=S, the respective priorities S can be obtained. They are sorted from low to high. The lower the priority, the more important it is, and it is completed first. As the script progresses, the position and importance of the task in the plot will change, resulting in changes in the priority of the task. After each task is completed, it will be re-sorted according to the new priority, so as to ensure that different tasks have different priorities at different stages of the plot, so as to ensure smooth task distribution and completion.

[0058] In step five, the device control module establishes a grpc two-way communication connection with the on-site equipment and the virtual human equipment and keeps the data consistent.

[0059] After the virtual human device receives information such as the role settings in the plot script, it uses computer graphics and AI technology to design the image of the digital virtual human. It can also use three-dimensional model character design software to design and model the image of the digital virtual human in advance, and use virtual reality technology to make software to render the digital virtual human in real time and simulate clothing, hair, etc.

[0060] The virtual human device also has a voice interaction function, allowing users to talk with virtual humans. The specific implementation is to first pre-process the user's voice information, including collection, enhancement, noise reduction, recognition and correction, and give the language model of the deep learning algorithm (OpenAI, etc.) to perform feature analysis and extraction on the voice information, and convert it into corresponding text information, and then understand the text through the language model, and generate text for interaction, and finally regenerate the generated interactive text into voice information.

[0061] In the present invention, the gRPC communication protocol is used between the system and the on-site interactive nodes and interactive devices to ensure data synchronization and real-time interaction between the nodes. The innovation lies in its multi-point interactive capability and high-performance communication mechanism, which provides visitors with a seamless interactive experience.

[0062] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-point distributed interactive system, used in conjunction with an interactive device, characterized in that: The system includes a resource management module, an intelligent body module, a scenario generation module, a task module, and a device control module which are connected in sequence; the task module and the device control module establish communication connections with the virtual human device, and the device control module establishes communication connections with the on-site device; The script generation module is used to receive keywords input by the user, generate corresponding scripts and interactive tasks, and send data to the resource management module and the task module; the keywords include script style, script name and script description; The agent module is used to simulate a virtual person according to the agent settings set by the user, receive the user's behavior, including sound and action, and generate corresponding reasonable feedback using natural language processing and / or computer graphics and artificial intelligence; The resource management module is used to manage various resources imported by users, including virtual human models, audio files, and / or scene materials, and to dispatch corresponding resources according to task requirements; The task module is used to receive the plot script and interactive tasks generated by the script generation module and send them to the virtual human device on site; The device control module is used to receive the feedback signal returned by the virtual human device and send it to the corresponding on-site device to realize the call of on-site resources.

2. The multi-point distributed interactive system according to claim 1, characterized in that: The script generation module adopts a transformer-based large language model, generates coherent plot script data according to keywords input by users through artificial intelligence, and optimizes the plot script data through fine-tuning to fit the theme and style of the scenic spot.

3. The multi-point distributed interactive system according to claim 2, characterized in that: Before generating plot script data, the large language model uses the prompt words and knowledge base set by the user as fine-tuning parameters, performs parameter training at the adapter layer, and fine-tunes the large language model with the trained parameters. After that, the keywords input by the user are used to generate a script through the fine-tuned large language model, and the generated plot script and interactive task data are sent to the task module.

4. The multi-point distributed interactive system according to claim 1, characterized in that: The intelligent agent module adopts a deep learning algorithm to provide the intelligent agent with the ability to understand and generate natural language and create a 3D virtual character model.

5. The multi-point distributed interactive system according to claim 1, characterized in that: The resource management module adopts a database management system to manage various resources imported by users, including virtual human models, audio files, and / or scene materials, and has resource import and export functions, and organizes resources through a tag system.

6. The multi-point distributed interactive system according to claim 1, characterized in that: After receiving the plot script and interactive task data, the task module sends the data to the virtual human device and distributes the tasks through a priority-based task scheduling algorithm.

7. The multi-point distributed interactive system according to claim 6, characterized in that: The priority-based task scheduling algorithm includes: when distributing interactive tasks, according to the importance I of the interactive task to the plot, the position L of the interactive task in the plot, the time required for completion T, and different weights a, b, c, aI+bL+cT=S is calculated to obtain the respective priorities S, and they are sorted in order from low to high. The lower the priority, the more important it is, and it is completed first; as the script progresses, the position and importance of the task in the plot change, resulting in changes in the priority of the task. After each task is completed, it is re-sorted according to the new priority, thereby ensuring that different tasks have different priorities at different stages of the plot, so as to ensure smooth task distribution and completion.

8. The multi-point distributed interactive system according to claim 1, characterized in that: The data sent by the task module includes the following contents: (1) task type, (2) trigger type, (3) field matching logic, and (4) on-site equipment used, wherein the trigger type includes voice interaction and trigger recognition, and the field matching logic includes semantic similarity, text equality, keywords, and matching any text.

9. The multi-point distributed interactive system according to any one of claims 1 to 8, characterized in that: The data sent by the device control module includes the following: (1) device number, (2) specific operation, and (3) required resources. The device control module is connected to the on-site device via a wireless network.

10. The multi-point distributed interactive system according to any one of claims 1 to 8, characterized in that: The virtual human device has a voice interaction function, allowing the user to communicate with the virtual human; first, the user's voice information is preprocessed, including collection, enhancement, noise reduction, recognition and correction, and the voice information is feature analyzed and extracted based on the language model of the deep learning algorithm, and converted into corresponding text information, and then the text is understood through the language model, and text for interaction is generated, and finally the generated interactive text is regenerated into voice information.

Images