Robot exhibition hall multi-machine linkage collaborative system and method based on private deployment

Abstract

The present invention discloses a robot exhibition hall multi-machine linkage collaborative system and method based on private deployment, the system includes a terminal and an edge end; the terminal includes a robot and a secondary terminal device; the robot is used to collect information data of the current exhibition hall scene in real time; the information data includes user voice; the edge end is used to: convert the user voice into text; based on the text, return a reply text; semantically decompose the reply text, generate robot task scheduling instructions, and send them to the robot through a local area network, so that the robot decomposes the instructions and distributes them to the secondary terminal device for operation; its beneficial effect is: various pre-processed data collected in the exhibition hall exhibition do not need to be uploaded to the cloud for calculation and distribution, but are directly calculated and processed at the edge end, thereby reducing the requirement for network bandwidth, and improving the timeliness of the robot response, while also avoiding the risk of data leakage during data uploading to the cloud.

Description

Technical Field

[0001] The present invention relates to the technical field of exhibition hall services, and in particular to a robot exhibition hall multi-machine linkage collaborative system and method based on private deployment. Background Art

[0002] The exhibition hall explanation humanoid robot is mainly used in exhibition hall scenes, directly facing customers, introducing and demonstrating exhibition hall products. During the service process of the robot, it is necessary to link the existing multi-terminal equipment in the exhibition hall to work together to achieve better display and explanation effects. For example: the exhibition hall explanation robot needs to explain and answer the exhibition hall content according to the questions asked by the current exhibition hall users, determine whether to turn on the speaker equipment according to the number of users, user location and other information, and display the corresponding content on the large screen in the exhibition hall according to the progress of the explanation. In addition, the explanation robot itself needs to have multiple capabilities such as welcoming guests, explaining, guiding, taking photos, and seeing off.

[0003] At present, most of the pre-processed data collected need to be uploaded to the cloud for calculation and processing. Since there are many devices in the exhibition hall, the network bandwidth requirements are high, which affects the timeliness of the robot's response and thus affects the user experience. Summary of the invention

[0004] In view of the technical defects mentioned in the background technology, the purpose of the embodiments of the present invention is to provide a robot exhibition hall multi-machine linkage collaborative system and method based on private deployment.

[0005] To achieve the above-mentioned purpose, in a first aspect, an embodiment of the present invention provides a robot exhibition hall multi-machine linkage collaborative system based on private deployment, the system comprising a terminal and an edge terminal; wherein the terminal comprises a robot and a secondary terminal device;

[0006] The robot is used to collect information data of the current exhibition hall scene in real time; wherein the information data includes user voice;

[0007] The edge end is used for:

[0008] Convert the user's speech into text;

[0009] Based on the text, return a reply text;

[0010] The reply text is semantically decomposed to generate robot task scheduling instructions, which are sent to the robot via a local area network, so that the robot decomposes the instructions and distributes them to the secondary terminal device for operation.

[0011] As a preferred implementation of the present application, the edge end is also used for:

[0012] Convert the reply text into a voice file and send it to the robot via a local area network;

[0013] The robot is also used to:

[0014] The voice file is received, and a reply voice is broadcasted through a microphone.

[0015] As a preferred implementation of the present application, the edge end includes a privately deployed edge end server, and the edge end server includes an ASR local module, an LLM local module, a TTS local module, a task scheduling calculation module, a knowledge base and a hot word library;

[0016] Convert the user speech into text through the ASR local module and the preset hot word library;

[0017] Sending the text to the LLM local module and knowledge base, and returning the reply text;

[0018] Convert the reply text into the voice file through the TTS local module;

[0019] The task scheduling calculation module performs task scheduling based on the disassembly result to obtain a task sequence; wherein the task sequence includes the task scheduling instruction.

[0020] As a preferred implementation of the present application, the robot includes a humanoid robot, and is linked with the secondary terminal device through a communication protocol; wherein the secondary terminal device includes a large screen, a tablet, a microphone and a speaker.

[0021] As a preferred implementation of the present application, the LLM local module trains the input text data through deep learning technology; thereby enabling the module to understand the user's input and generate corresponding replies.

[0022] In a second aspect, an embodiment of the present invention further provides a robot exhibition hall multi-machine linkage collaboration method based on private deployment, which is applied to the robot exhibition hall multi-machine linkage collaboration system based on private deployment described in the first aspect, wherein the system includes a terminal and an edge terminal, wherein the terminal includes a robot and a secondary terminal device; the method includes:

[0023] Collect information data of the current exhibition hall scene in real time; wherein the information data includes user voice;

[0024] Convert the user's speech into text;

[0025] Based on the text, return a reply text;

[0026] The reply text is semantically decomposed to generate robot task scheduling instructions, which are sent to the robot via a local area network, so that the robot decomposes the instructions and distributes them to the secondary terminal device for operation.

[0027] The technical solution provided by the embodiment of the present invention utilizes the edge infrastructure adopted by private deployment, reasonably utilizes the computing power of the robot body, performs data preprocessing on the terminal side, and simplifies the data communication and data storage in the overall process;

[0028] At the same time, through the deployed edge server, the pre-processed data collected in the exhibition hall does not need to be uploaded to the cloud for calculation and distribution, but is directly calculated and processed at the edge, thereby reducing the requirements for network bandwidth and improving the timeliness of the robot's response. At the same time, it can also avoid the risk of data leakage during the process of uploading data to the cloud;

[0029] And generate robot task scheduling instructions, then disassemble the instructions through the robot and distribute them to the secondary terminal device, control the coordinated operation of various equipment in the exhibition hall, and realize the anthropomorphic exhibition hall explanation experience for users;

[0030] And through the linkage of robots and multiple terminal devices, we can further combine the actual effects of exhibition hall operation to further enhance the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0032] Figure 1 It is a principle block diagram of a robot exhibition hall multi-machine linkage collaborative system based on private deployment provided by an embodiment of the present invention;

[0033] Figure 2 It is a flow chart of a multi-machine linkage and collaboration method in a robot exhibition hall based on private deployment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so as to describe the embodiments of the present application described herein.

[0036] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0037] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. Furthermore, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any appropriate combinations and / or subcombinations.

[0038] It should be noted that, unless otherwise specified, the technical terms in this embodiment have the common meanings understood in the relevant technical field.

[0039] A large language model (LLM) refers to a deep learning model trained with a large amount of text data. It can generate natural language text or understand the meaning of language text. A large language model can handle a variety of natural language tasks, such as text classification, question and answer, and dialogue.

[0040] ASR, Automatic Speech Recognition, automatic speech recognition;

[0041] TTS, Text To Speech, means from text to speech.

[0042] Please refer to Figure 1 , an embodiment of the present invention provides a robot exhibition hall multi-machine linkage collaborative system based on private deployment, the system includes a terminal and an edge terminal; wherein the terminal includes a robot and a secondary terminal device;

[0043] The robot is used to collect information data of the current exhibition hall scene in real time; wherein the information data includes user voice;

[0044] The edge end is used for:

[0045] Convert the user's speech into text;

[0046] Based on the text, return a reply text;

[0047] The reply text is semantically decomposed to generate robot task scheduling instructions, which are sent to the robot via a local area network, so that the robot decomposes the instructions and distributes them to the secondary terminal device for operation.

[0048] In this embodiment, when a user enters the exhibition hall, the robot automatically detects that someone is approaching and wakes up the system to greet them, while uploading the information to the edge end;

[0049] The robots are mainly humanoid robots, and there are multiple of them; and they are linked with the secondary terminal devices through a communication protocol; wherein the secondary terminal devices include large screens, tablets, microphones and speakers, etc., which are not elaborated here.

[0050] It should be noted that this embodiment is only described by taking voice data as an example, and is not intended to limit it; when applied, it may also include corresponding video data, current environment data, etc.;

[0051] The robot integrates multiple technologies such as environmental perception, artificial intelligence, image recognition, intelligent navigation and path planning, big data analysis, etc., and interacts with various types of terminals in the exhibition hall to create an intelligent, full-process service scene for exhibition customers.

[0052] In this embodiment, the edge end is also used for:

[0053] Convert the reply text into a voice file and send it to the robot via a local area network;

[0054] The robot is also used to:

[0055] The voice file is received, and a reply voice is broadcasted through a microphone.

[0056] When applied, the edge end includes a privately deployed edge end server, and the edge end server includes an ASR local module, an LLM local module, a TTS local module, a task scheduling calculation module, a knowledge base and a hot word library;

[0057] Convert the user speech into text through the ASR local module and the preset hot word library;

[0058] Sending the text to the LLM local module and knowledge base, and returning the reply text;

[0059] Convert the reply text into the voice file through the TTS local module;

[0060] The task scheduling calculation module performs task scheduling based on the disassembly result to obtain a task sequence; wherein the task sequence includes the task scheduling instruction.

[0061] In this embodiment, the LLM local module trains the input text data through deep learning technology; through the LLM large language model deployed in the LLM local module, the module can understand the user's input and generate corresponding replies;

[0062] The LLM local module converts the reply text generated by the system into speech, so that the user can hear smooth and natural speech output;

[0063] Then, through semantic understanding and knowledge base, the user's intention is inferred, thereby obtaining the task information to be performed in the scenario;

[0064] The task scheduling computing module is deployed with task scheduling programs for business scenarios, which are used to implement various specific business applications and perform task scheduling, such as welcoming guests in the exhibition hall, guiding, explaining business, walking on the route, taking group photos and selfies, etc.

[0065] Furthermore, during implementation, after receiving the instruction, the robot disassembles the instruction and distributes it to the secondary terminal device.

[0066] The disassembly is processed according to the coordinated operation logic of the required multiple devices, including voice control and execution), action linkage, and expression display (such as nodding, shaking head, blinking, moving lips, making faces, etc.).

[0067] The above solution uses the edge infrastructure adopted by private deployment, rationally utilizes the computing power of the robot body, pre-processes data on the terminal side, and simplifies the data communication and data storage in the overall process;

[0068] At the same time, through the deployed edge server, the pre-processed data collected in the exhibition hall does not need to be uploaded to the cloud for calculation and distribution, but is directly calculated and processed at the edge, thereby reducing the requirements for network bandwidth and improving the timeliness of the robot's response. At the same time, it can also avoid the risk of data leakage during the process of uploading data to the cloud;

[0069] And generate robot task scheduling instructions, then disassemble the instructions through the robot and distribute them to the secondary terminal device, control the coordinated operation of various equipment in the exhibition hall, and realize the anthropomorphic exhibition hall explanation experience for users;

[0070] And through the linkage of robots and multiple terminal devices, we can further combine the actual effects of exhibition hall operation to further enhance the user experience.

[0071] Reference Figure 2Based on the same inventive concept, an embodiment of the present invention further provides a robot exhibition hall multi-machine linkage collaboration method based on private deployment, which is applied to the robot exhibition hall multi-machine linkage collaboration system based on private deployment described above, the system includes a terminal and an edge terminal, the terminal includes a robot and a secondary terminal device; the method includes:

[0072] S101, collecting information data of the current exhibition hall scene in real time; wherein the information data includes user voice;

[0073] S102, converting the user's voice into text;

[0074] S103, returning a reply text based on the text;

[0075] S104, semantically decomposing the reply text, generating robot task scheduling instructions, and sending the instructions to the robot via the local area network, so that the robot decomposes the instructions and distributes them to the secondary terminal device for operation.

[0076] In this embodiment, the edge end includes a privately deployed edge end server, and the edge end server includes an ASR local module, an LLM local module, a TTS local module, a task scheduling calculation module, a knowledge base, and a hot word library;

[0077] Convert the user speech into text through the ASR local module and the preset hot word library;

[0078] Sending the text to the LLM local module and knowledge base, and returning the reply text;

[0079] Convert the reply text into the voice file through the TTS local module;

[0080] The task scheduling calculation module performs task scheduling based on the disassembly result to obtain a task sequence; wherein the task sequence includes the task scheduling instruction.

[0081] The method further comprises:

[0082] Convert the reply text into a voice file and send it to the robot via a local area network;

[0083] The robot receives the voice file and broadcasts a reply voice through a microphone;

[0084] The input text data is trained through deep learning technology, so that the LLM local module can understand the user's input and generate corresponding replies.

[0085] It should be noted that for a more specific description of the workflow of the method embodiment, please refer to the aforementioned system embodiment part, which will not be repeated here.

[0086] The whole solution makes reasonable use of the computing power of the robot body, pre-processes the data on the terminal side, and simplifies the data communication and data storage in the overall process;

[0087] At the same time, through the deployed edge server, the pre-processed data collected in the exhibition hall does not need to be uploaded to the cloud for calculation and distribution, but is directly calculated and processed at the edge, thereby generating faster network service response, meeting the real-time scene requirements of the exhibition hall, reducing the requirements for network bandwidth, and improving the timeliness of the robot response. At the same time, it can also avoid the risk of data leakage during the process of uploading data to the cloud;

[0088] And generate robot task scheduling instructions, then disassemble the instructions through the robot and distribute them to the secondary terminal device, control the coordinated operation of various equipment in the exhibition hall, and realize the anthropomorphic exhibition hall explanation experience for users;

[0089] And through the linkage of robots and multiple terminal devices, we can further combine the actual effects of exhibition hall operation to further enhance the user experience.

[0090] In several embodiments provided in the present application, it should be understood that the disclosed system and method can also be implemented in other ways. The system embodiment described above is only schematic, for example, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0091] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. When in use, each module only collects and stores user information with the full authorization of the user and in compliance with relevant laws and regulations, and protects the security and privacy of user data, and unauthorized access is strictly prohibited; data processing will be carried out within the scope prescribed by law and shall not exceed the purpose and scope authorized by the user; at the same time, users enjoy relevant rights such as access, correction, deletion, restriction of processing, and rejection of their personal data; and strictly abide by applicable laws and regulations and conduct compliance reviews.

[0092] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0093] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A robot exhibition hall multi-machine linkage collaborative system based on private deployment, characterized in that: The system includes a terminal and an edge terminal; wherein the terminal includes a robot and a secondary terminal device; The robot is used to collect information data of the current exhibition hall scene in real time; wherein the information data includes user voice; The edge end is used for: Convert the user's speech into text; Based on the text, returning a reply text; The reply text is semantically decomposed to generate robot task scheduling instructions, which are sent to the robot via a local area network, so that the robot decomposes the instructions and distributes them to the secondary terminal device for operation.

2. The robot exhibition hall multi-machine linkage collaborative system based on private deployment as claimed in claim 1, characterized in that: The edge end is also used for: Convert the reply text into a voice file and send it to the robot via a local area network; The robot is also used to: The voice file is received, and a reply voice is broadcasted through a microphone.

3. The robot exhibition hall multi-machine linkage collaborative system based on private deployment as claimed in claim 2, characterized in that: The edge end includes a privately deployed edge end server, and the edge end server includes an ASR local module, an LLM local module, a TTS local module, a task scheduling calculation module, a knowledge base and a hot word library; Convert the user speech into text through the ASR local module and the preset hot word library; Sending the text to the LLM local module and knowledge base, and returning the reply text; Convert the reply text into the voice file through the TTS local module; The task scheduling calculation module performs task scheduling based on the disassembly result to obtain a task sequence; wherein the task sequence includes the task scheduling instruction.

4. The robot exhibition hall multi-machine linkage collaborative system based on private deployment as claimed in claim 1, characterized in that: The robot includes a humanoid robot, and is linked with the secondary terminal device through a communication protocol; wherein the secondary terminal device includes a large screen, a tablet, a microphone and a speaker.

5. The robot exhibition hall multi-machine linkage collaborative system based on private deployment as claimed in claim 3, characterized in that: The LLM local module trains the input text data through deep learning technology, so that the module can understand the user's input and generate corresponding replies.

6. A multi-machine linkage collaborative method for a robot exhibition hall based on private deployment, characterized in that: The robot exhibition hall multi-machine linkage collaborative system based on private deployment as described in claim 1, the system includes a terminal and an edge terminal, the terminal includes a robot and a secondary terminal device; the method includes: Collect information data of the current exhibition hall scene in real time; wherein the information data includes user voice; Convert the user's speech into text; Based on the text, returning a reply text; The reply text is semantically decomposed to generate robot task scheduling instructions, which are sent to the robot via a local area network, so that the robot decomposes the instructions and distributes them to the secondary terminal device for operation.

7. The method for multi-machine linkage coordination in a robot exhibition hall based on private deployment as claimed in claim 6, characterized in that: The method further comprises: Convert the reply text into a voice file and send it to the robot via a local area network; The robot receives the voice file and broadcasts a reply voice through a microphone.

8. The method for multi-machine linkage coordination in a robot exhibition hall based on private deployment as claimed in claim 7, characterized in that: The edge end includes a privately deployed edge end server, and the edge end server includes an ASR local module, an LLM local module, a TTS local module, a task scheduling calculation module, a knowledge base and a hot word library; Convert the user speech into text through the ASR local module and the preset hot word library; Sending the text to the LLM local module and knowledge base, and returning the reply text; Convert the reply text into the voice file through the TTS local module; The task scheduling calculation module performs task scheduling based on the disassembly result to obtain a task sequence; wherein the task sequence includes the task scheduling instruction.

9. The method for multi-machine linkage coordination in a robot exhibition hall based on private deployment as claimed in claim 8, characterized in that: The method further comprises: The input text data is trained through deep learning technology, so that the LLM local module can understand the user's input and generate corresponding replies.

Images