Holographic walking digital robot based on large model technology and control method thereof

Abstract

The invention discloses a holographic walking digital robot based on a large model technology and a control method thereof, the robot can carry out holographic digital human image display and is provided with a chassis capable of walking, the overall implementation cost is low, the robot can move according to guidance space information, the interaction is vivid and visual, and the robot is convenient to use. According to the holographic walking digital robot based on the large model technology and the control method of the holographic walking digital robot, man-machine interaction and smooth interaction are achieved in the mode that a local large model and a cloud large model are combined, and therefore the holographic walking digital robot based on the large model technology and the control method of the holographic walking digital robot have the advantages of being low in cost, capable of conducting interaction based on space, flexible and natural in display image and smooth in interaction mode.

Description

Technical Field

[0001] The present invention relates to the field of robots and artificial intelligence technology, and in particular to a low-cost holographic walking digital robot based on large model technology that can interact based on space, has a flexible and natural display image, and has a smooth interactive mode, and a control method thereof. Background Art

[0002] With the development of digital human technology and large model technology, a number of virtual human images that combine large models and digital human technology have emerged, which can communicate with users in natural language; further, digital humans are projected through holographic technology, making them more three-dimensional;

[0003] At the same time, in the field of physical robots, physical robots with reception functions can complete user communication, and can also provide route guidance, cruising, etc.

[0004] However, both of these interaction methods have certain defects. Pure digital humans just stand in place on the screen and cannot interact in combination with spatial information; physical robots have problems such as high cost, inflexible image updates, and lack of vivid display.

[0005] Therefore, it is necessary to propose an improvement to overcome the defects of the prior art. Summary of the invention

[0006] The purpose of the present invention is to solve the problems in the prior art and to provide a holographic walking digital robot based on large model technology and a control method thereof which is low-cost, can interact based on space, has a flexible and natural display image, and has a smooth interactive mode.

[0007] The technical solution of the present invention is:

[0008] A holographic walking digital robot based on large model technology comprises a transparent box, a holographic projection device, a camera module, a central control host, a voice interaction module and a chassis; the holographic projection device is fixed in the transparent box and electrically connected to the central control host; the transparent box and the chassis are connected through a rotating shaft; the camera module is arranged on the chassis and electrically connected to the central control host; the central control host is arranged in the chassis; a roller is arranged at the bottom of the chassis close to the ground, and a motor is arranged in the chassis to drive the roller; the transparent box cooperates with the holographic projection device, A holographic display of a digital human is realized; the camera module is used to collect on-site video data; the central control host is used to perform large model calculations locally and control the robot and the holographic projection device to display the image of a digital human, and the central control host includes a network module and a battery module, which communicates with the cloud large model through the network module, and the battery module supplies power to the entire robot. The voice interaction module includes a microphone and a speaker, which is electrically connected to the central control host and is used for voice interaction with the user; the chassis is used to store the central control host, the voice interaction module and drive the robot to move.

[0009] As a preferred technical solution, the transparent box is made of acrylic material.

[0010] As a preferred technical solution, the holographic projection device includes three projection modules, and a complete digital human is displayed by splicing the images of the three projection modules; each of the projection modules includes four fan blades, and the fan blades can rotate at high speed. Each fan blade is provided with multiple LED lamp beads, and the three-dimensional display of the image is realized by the high-speed rotation of the lamp beads.

[0011] As a preferred technical solution, four rollers are provided.

[0012] As a preferred technical solution, the camera module is a wide-angle micro camera.

[0013] As a preferred technical solution, a transparent power supply film is provided in the transparent box, and the transparent power supply film electrically connects the holographic projection device with the central control host.

[0014] As a preferred technical solution, the transparent box is provided with a projection fixing device corresponding to the holographic projection device, and the holographic projection device is fixed to the transparent box via the projection fixing device.

[0015] As a preferred technical solution, the chassis is provided with a tray corresponding to the central control host, and the central control host is arranged in the tray.

[0016] The present invention also provides a control method for a holographic walking digital robot based on a large model technology, comprising the following steps: S1, training a local large model to enable it to have localized problem-solving capabilities, and deploying the local large model on the central control host; S2, real-time acquisition of the robot's working environment through the camera module and the voice interaction module, and when the user approaches, identifying the user's usage needs, if the user interacts with the robot, the robot enters an interactive mode, displays a holographic digital human image and performs voice interaction; S3, receiving the user's voice content, performing voice reply and corresponding digital human image display: generating corresponding question text information from the user's voice content, the local large model processes the question text information and answers the question to form local reply text information; if the local large model cannot answer the question text information, communicate with the cloud large model through the network module , the cloud-based large model processes the question text information and the answer, and after forming a cloud-based reply text information, transmits it to the central control host; the central control host is deployed with a text-to-speech conversion module to convert the local reply text information or the cloud-based reply text information into a voice stream; the central control host is deployed with a digital human synthesis module to generate a digital human image in combination with the voice stream content; the holographic projection device displays the digital human image; S4. During the interaction process, if it is found that the user requires the robot to move or it is judged that it needs to move based on the interaction content, the chassis moves the robot through the roller; during the interaction process, according to the user orientation collected by the camera module, the transparent box is rotated through the rotation axis to ensure that the digital human faces the user; S5. When it is recognized that the user is moving away or the user actively terminates the conversation, the interaction is terminated and the robot enters the standby state.

[0017] As a preferred technical solution, in step S3, when the digital human image is formed, streaming media is used for video synthesis; when the holographic projection device includes multiple projection modules, the digital human synthesis module divides the digital human image video and outputs it to the corresponding projection module to splice the digital human image when displaying it.

[0018] The present invention discloses a holographic walking digital robot based on large model technology and a control method thereof. The robot can display a holographic digital human image and has a walking chassis. The overall implementation cost is low, the robot can move according to the guidance space information, the interaction image is intuitive, and human-computer interaction is realized by combining a local large model with a cloud large model. Therefore, the present invention discloses a holographic walking digital robot based on large model technology and a control method thereof, which has the advantages of low cost, space-based interaction, flexible and natural display image, and smooth interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1This is a schematic diagram of the front structure of a specific implementation of a holographic walking digital robot based on large model technology of the present invention;

[0020] Figure 2 This is a schematic diagram of the back structure of a specific implementation of a holographic walking digital robot based on large model technology of the present invention;

[0021] Figure 3 This is a flowchart of a control method for a holographic walking digital robot based on large model technology of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0024] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0025] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0026] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a product or system. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0027] like Figure 1 and Figure 2 The figure shows a specific implementation of a holographic walking digital robot based on large model technology of the present invention. Figure 1 and Figure 2 As shown, a holographic walking digital robot based on large model technology in this embodiment includes a transparent box 1, a holographic projection device 2, a camera module 5, a central control host 6, a voice interaction module and a chassis 4. The holographic projection device 2 is fixed in the transparent box 1 and electrically connected to the central control host 6; the transparent box 1 is connected to the chassis 4 through a rotating shaft 3; the camera module 5 is arranged on the chassis 4 and electrically connected to the central control host 6; the central control host 6 is arranged in the chassis 4; a roller 7 is arranged at the bottom of the chassis 4 close to the ground, and a motor is arranged in the chassis 4 to drive the roller 7; the transparent box 1 cooperates with the holographic projection device 2 to realize the holographic display of the digital human; the camera module 5 is used to collect on-site video data; the central control host 6 is used to perform large model calculations locally and control the robot and control the holographic projection device 2 to display the image of the digital human, and the central control host 6 includes a network module and a battery module, communicates with the cloud large model through the network module, and the battery module supplies power to the robot as a whole, and the voice interaction module includes a microphone and a speaker, which is electrically connected to the central control host 6 for voice interaction with the user; the chassis 4 is used to store the central control host, the voice interaction module and drive the robot to move.

[0028] In this embodiment, in order to reduce production costs and to ensure durability, the transparent box 1 is made of acrylic material, and the transparent box 1 made of acrylic material is used to wrap and fix the holographic projection device 2.

[0029] In this embodiment, in order to ensure the display effect of the holographic image of the digital human, the holographic projection device 2 includes three projection modules, and the complete digital human is displayed by splicing the images of the three projection modules; each of the projection modules includes four fan blades, and the fan blades can rotate at high speed. Each fan blade is provided with multiple LED lamp beads. Through the high-speed rotation of the lamp beads, the POV (persistence of vision) of the human eye is used to realize the three-dimensional display of the image.

[0030] In this embodiment, the rotating shaft 3 connects the transparent box 1 and the chassis 4 , and the transparent box 1 can be controlled to rotate through the chassis 4 .

[0031] In this embodiment, in order to ensure the smooth movement of the chassis, four rollers 7 are provided and driven by a motor to achieve movement in a specified direction.

[0032] In this embodiment, in order to ensure the integrity of video acquisition, the camera module 5 uses a wide-angle micro camera to achieve the acquisition and analysis of on-site video data.

[0033] In this implementation, the central control host 6 is the brain of the entire robot system. It is interconnected with the large cloud model through the wifi or 4G, 5G and other mobile networks of the network module. It has a large local computing power to meet business needs and integrates battery function to supply power to the rollers, rotating shafts, and projection devices.

[0034] like Figure 2 As shown, in this embodiment, a transparent power supply film 8 is provided in the transparent box 1 , and the transparent power supply film 8 electrically connects the holographic projection device 2 with the central control host 6 to realize power supply and image display of the holographic projection device 2 .

[0035] like Figure 2 As shown, in this embodiment, the transparent box 1 is provided with a projection fixing device 9 corresponding to the holographic projection device 2 , and the projection fixing device 9 fixes the holographic projection device 2 and the transparent box 1 .

[0036] like Figure 2 As shown, in this embodiment, the chassis 4 is provided with a tray 10 corresponding to the central control host 6 , and the central control host 6 is arranged in the tray 10 and accommodated and fixed in the chassis 4 .

[0037] A holographic walking digital robot based on large model technology in this embodiment drives the entire digital robot forward through a chassis, realizes the movement of the digital person in the physical space, and can interact with information in combination with the physical space; a combination of horizontal movement of four rolling wheels and rotational movement of a rotating axis is adopted to ensure that the front of the digital person can always face the audience during the movement; a combination of a local large model and a cloud large model is used at the same time, which can take into account both timeliness and model capabilities. Routine questions and answers are quickly responded to through the local model, and questions that are out of range can be answered again by calling the remote large model; power supply to three projection devices is achieved through a transparent conductive film, and the final image is more beautiful than the traditional vertical column power supply method; a camera is added to complete intelligent judgment of the relative position of the audience and the on-site explanation situation, and the explanation is more natural.

[0038] like Figure 3 As shown, this embodiment also discloses a control method of a holographic walking digital robot based on a large model technology, comprising the following steps:

[0039] S1. Train the local big model to enable it to have local problem-solving capabilities, and deploy the local big model on the central control host. First, complete the training of the local big model, so that the robot has the professional capabilities required for the reception scene.

[0040] After the local large model is trained and deployed, the robot is ready to receive guests and wait on site:

[0041] S2. The robot collects the working environment of the robot in real time through the camera module and the voice interaction module. When the user approaches, the robot identifies the user's use needs. If the user interacts with the robot, the robot enters the interactive mode, displays the holographic digital human image and performs voice interaction. When in standby mode, the specific standby content can be pre-set (similar to the screen saver); when the user approaches, the robot actively starts to greet and enter the working state, or enters the working state when the user asks a question, and the user and the robot begin to interact.

[0042] S3, receiving user voice content, making voice reply and displaying the corresponding digital human image: generating corresponding question text information from the user voice content, the local big model processes the question text information and answers it, and forms local reply text information; if the local big model cannot answer the question text information, the network module communicates with the cloud big model, and the cloud big model processes the question text information and answers it, and after forming cloud reply text information, it is transmitted to the central control host; the central control host is equipped with a text-to-speech conversion module to convert the local reply text information or the cloud reply text information into a voice stream; the central control host is equipped with a digital human synthesis module to generate a digital human image in combination with the voice stream content; the holographic projection device displays the digital human image;

[0043] S4. During the interaction process, if the user requires the robot to move or the robot needs to move according to the interaction content, the chassis moves the robot through the rollers; during the interaction process, according to the user orientation captured by the camera module, the transparent box is rotated through the rotation axis to ensure that the digital human faces the user;

[0044] S5. When it is recognized that the user is away or the user actively terminates the conversation, the interaction is terminated and the device enters the standby state.

[0045] In this embodiment, when the digital human image is displayed in step S3, streaming media is used for video synthesis; the holographic projection device of the robot in this embodiment includes three projection modules, and the digital human synthesis module divides the digital human image video and outputs it to the corresponding projection module to splice the digital human image when displaying.

[0046] The present invention provides a holographic walking digital robot based on large model technology and a control method thereof, which can be applied to a venue explanation robot and a front desk reception robot. When used as a venue explanation robot, the large model is trained with venue knowledge so that the holographic digital robot can explain the contents displayed in the venue. When used as a front desk reception robot, the large model is trained with enterprise-related knowledge so that the holographic digital robot can receive visitors and, if necessary, can be connected to the telephone system to implement telephone calls to internal personnel of the enterprise. In actual application, the large model can be trained in a targeted manner according to specific needs to improve the application scenarios and intelligence level of the holographic digital robot.

[0047] The present invention discloses a holographic walking digital robot based on large model technology and a control method thereof. The robot can display a holographic digital human image and has a walking chassis. The overall implementation cost is low, the robot can move according to the guidance space information, the interaction image is intuitive, and human-computer interaction is realized by combining a local large model with a cloud large model. Therefore, the present invention discloses a holographic walking digital robot based on large model technology and a control method thereof, which has the advantages of low cost, space-based interaction, flexible and natural display image, and smooth interaction.

[0048] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of implementation of the present invention. That is, all equivalent changes and modifications made according to the content of the patent application scope of the present invention should belong to the technical scope of the present invention.

Claims

1. A holographic walking digital robot based on large model technology, characterized by: It includes a transparent box, a holographic projection device, a camera module, a central control host, a voice interaction module and a chassis; the holographic projection device is fixed in the transparent box and electrically connected to the central control host; the transparent box is connected to the chassis through a rotating shaft; the camera module is arranged on the chassis and electrically connected to the central control host; the central control host is arranged in the chassis; a roller is arranged at the bottom of the chassis close to the ground, and a motor is arranged in the chassis to drive the roller; the transparent box cooperates with the holographic projection device to realize the holographic display of the digital human The camera module is used to collect on-site video data; the central control host is used to perform large model calculations locally and control the robot and the holographic projection device to display a digital human image, and the central control host includes a network module and a battery module, which communicates with the cloud large model through the network module, and the battery module supplies power to the entire robot. The voice interaction module includes a microphone and a speaker, which is electrically connected to the central control host and is used for voice interaction with the user; the chassis is used to store the central control host, the voice interaction module and drive the robot to move.

2. The holographic walking digital robot based on large model technology according to claim 1 is characterized in that: The transparent box is made of acrylic material.

3. The holographic walking digital robot based on large model technology according to claim 1 is characterized in that: The holographic projection device includes three projection modules, and a complete digital human is displayed by splicing the images of the three projection modules; each of the projection modules includes four fan blades, and the fan blades can rotate at high speed. Each fan blade is provided with multiple LED lamp beads, and the three-dimensional display of the image is realized through the high-speed rotation of the lamp beads.

4. The holographic walking digital robot based on large model technology according to claim 1 is characterized in that: The number of rollers is four.

5. The holographic walking digital robot based on large model technology according to claim 1 is characterized in that: The camera module is a wide-angle micro camera.

6. The holographic walking digital robot based on large model technology according to claim 1 is characterized in that: A transparent power supply film is arranged in the transparent box, and the transparent power supply film electrically connects the holographic projection device with the central control host.

7. The holographic walking digital robot based on large model technology according to claim 1 is characterized by: The transparent box is provided with a projection fixing device corresponding to the holographic projection device, and the holographic projection device is fixed to the transparent box through the projection fixing device.

8. The holographic walking digital robot based on large model technology according to claim 1 is characterized by: The chassis is provided with a tray corresponding to the central control host, and the central control host is arranged in the tray.

9. A control method for a holographic walking digital robot based on large model technology according to any one of claims 1 to 8, characterized in that: The following steps are involved: S1. Train the local large model to enable it to have local problem-solving capabilities, and deploy the local large model on the central control host; S2, using the camera module and the voice interaction module to collect the robot's working environment in real time, and when the user approaches, identify the user's usage needs. If the user interacts with the robot, the robot enters the interactive mode, displays the holographic digital human image, and performs voice interaction; S3, receiving user voice content, making voice reply and displaying the corresponding digital human image: generating corresponding question text information from the user voice content, the local big model processes the question text information and answers it, and forms local reply text information; if the local big model cannot answer the question text information, the network module communicates with the cloud big model, and the cloud big model processes the question text information and answers it, and after forming cloud reply text information, it is transmitted to the central control host; the central control host is equipped with a text-to-speech conversion module to convert the local reply text information or the cloud reply text information into a voice stream; the central control host is equipped with a digital human synthesis module to generate a digital human image in combination with the voice stream content; the holographic projection device displays the digital human image; S4. During the interaction process, if the user requires the robot to move or the robot needs to move according to the interaction content, the chassis moves the robot through the rollers; during the interaction process, according to the user orientation captured by the camera module, the transparent box is rotated through the rotation axis to ensure that the digital human faces the user; S5. When it is recognized that the user is away or the user actively terminates the conversation, the interaction is terminated and the device enters the standby state.

10. The control method of a holographic walking digital robot based on large model technology according to claim 9 is characterized in that: In step S3, when the digital human image is formed, the video is synthesized by streaming media; when the holographic projection device includes multiple projection modules, the digital human synthesis module divides the digital human image video and outputs it to the corresponding projection module to splice the digital human image when displaying.

Images