Holographic projection method, holographic projection system and storage medium

Abstract

The invention relates to a holographic projection method, a holographic projection system and a storage medium. The holographic projection method comprises the following steps: in response to voice interaction information received by voice interaction equipment, generating voice reply information of the voice interaction information; a control instruction is generated and sent to a display terminal, the display terminal generates visual reply information, the visual reply information comprises limb actions and expressions of a digital virtual image, and the digital virtual image is displayed through a holographic image; and in response to receiving feedback information sent by the display terminal, playing the voice reply information, the feedback information being used for representing that the visual reply information has been generated and displayed. Through the method and the device, the cost of synchronously feeding back the visual information and the voice information based on the interaction information sent by the user is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of holographic projection, and in particular to a holographic projection method, a holographic projection system and a storage medium. Background Art

[0002] In the related technology, speakers have been used as voice assistants in daily life to help control smart homes and perform some simple voice interactions. When the voice interaction function of the speaker can only support basic interactions, and only voice communication is performed without controlling the speaker to perform functional operations, the speaker can only give mechanical replies and cannot give users a sense of companionship. In addition, since the speaker can only communicate in voice, it cannot give users visual feedback and a sense of immersion in the process of image interaction. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a holographic projection method, a holographic projection system and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a holographic projection method is provided, comprising: in response to a voice interaction device receiving voice interaction information, generating voice reply information for the voice interaction information; generating a control instruction, and sending the control instruction to a display terminal, wherein the display terminal generates visual reply information, wherein the visual reply information includes body movements and expressions of a digital virtual image, and the digital virtual image is displayed through a holographic image; in response to receiving feedback information sent by the display terminal, playing the voice reply information, wherein the feedback information is used to indicate that the visual reply information has been generated and displayed.

[0005] In one embodiment, the generation of voice response information of the voice interaction information includes: based on a GPT model, extracting semantic information of the voice interaction information, and generating voice response information including the semantic information, wherein the semantic information includes text response information and semantic emotion information; the GPT model is pre-trained based on a pre-trained model.

[0006] In one implementation, the generating of the control instruction includes: based on the UDP protocol between the voice interaction device and the display terminal, converting the text reply information into the control instruction.

[0007] In one implementation, the method further includes: in response to receiving an active interaction instruction issued by the display terminal, playing preset active interaction language information.

[0008] According to a second aspect of an embodiment of the present disclosure, a holographic projection method is provided, comprising: in response to a display terminal receiving a control instruction sent by a voice interaction device, generating visual reply information based on the control instruction, the visual reply information including body movements and expressions of a digital virtual image, and the digital virtual image is displayed through a holographic image; sending feedback information to the voice interaction device, and playing the visual reply information.

[0009] In one implementation, the digital avatar is installed in the display terminal in the form of an application package.

[0010] In one embodiment, the visual reply information includes the lip shape of the digital virtual image, and the method further includes: in response to the voice interaction device playing the voice reply information, synchronously acquiring audio information contained in the voice reply information; and adjusting the lip shape of the digital virtual image in real time based on the audio information.

[0011] In one embodiment, the display terminal is connected to a projection frame, the digital virtual image is projected by the display terminal onto the projection frame, and the projection frame includes a visual detection device. The method also includes: in response to a user facing the projection frame, generating an active interaction instruction, and sending the active interaction instruction to the voice interaction device, the active interaction instruction is used to control the display terminal and the voice playback device to synchronize active interaction visual information and active interaction voice information; in response to the active interaction instruction being sent to the voice interaction device, playing the pre-set active interaction visual information.

[0012] According to a third aspect of an embodiment of the present disclosure, a voice interaction device is provided, comprising: a generation unit, for generating voice reply information of the voice interaction information in response to the voice interaction device receiving the voice interaction information; a first processing unit, for generating a control instruction, and sending the control instruction to a display terminal, the display terminal generating visual reply information, the visual reply information including body movements and expressions of a digital virtual image, the digital virtual image being displayed through a holographic image; a first playback unit, for playing the voice reply information in response to receiving feedback information sent by the display terminal, the feedback information being used to indicate that the visual reply information has been generated and displayed.

[0013] In one embodiment, the generation unit generates voice response information of the voice interaction information in the following manner: based on a GPT model, the semantic information of the voice interaction information is extracted, and voice response information including the semantic information is generated, wherein the semantic information includes text response information and semantic emotional information; the GPT model is pre-trained based on a pre-trained model.

[0014] In one implementation, the first processing unit generates the control instruction in the following manner: based on the UDP protocol between the voice interaction device and the display terminal, the text reply information is converted into the control instruction.

[0015] In one implementation, the first processing unit is further configured to: in response to receiving an active interaction instruction issued by the display terminal, play preset active interaction language information.

[0016] According to the fourth aspect of an embodiment of the present disclosure, a display terminal is provided, comprising: a second processing unit, for generating visual reply information based on the control instruction received by the display terminal in response to a control instruction sent by a voice interaction device, wherein the visual reply information includes body movements and expressions of a digital virtual image, and the digital virtual image is displayed through a holographic image; and a second playback unit, for sending feedback information to the voice interaction device and playing the visual reply information.

[0017] In one implementation, the digital avatar is installed in the display terminal in the form of an application package.

[0018] In one embodiment, the visual reply information includes the lip shape of the digital virtual image, and the second processing unit is further used to: in response to the voice interaction device playing the voice reply information, synchronously obtain the audio information contained in the voice reply information; and adjust the lip shape of the digital virtual image in real time based on the audio information.

[0019] In one embodiment, the display terminal is connected to a projection frame, the digital virtual image is projected by the display terminal onto the projection frame, the projection frame includes a visual detection device, and the second processing unit is also used to: in response to a user facing the projection frame, generate an active interaction instruction, and send the active interaction instruction to the voice interaction device, the active interaction instruction is used to control the display terminal and the voice playback device to synchronize active interaction visual information and active interaction voice information; in response to the active interaction instruction being sent to the voice interaction device, play the pre-set active interaction visual information.

[0020] According to a fifth aspect of an embodiment of the present disclosure, a holographic projection system is provided, comprising the voice interaction device as described in the first aspect or any one embodiment of the first aspect and the display terminal as described in the second aspect or any one embodiment of the second aspect.

[0021] According to a sixth aspect of an embodiment of the present disclosure, a voice interaction device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the holographic projection method described in the first aspect or any one embodiment of the first aspect.

[0022] According to a seventh aspect of an embodiment of the present disclosure, a display terminal is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the holographic projection method described in the second aspect or any one of the implementations of the second aspect.

[0023] According to an eighth aspect of an embodiment of the present disclosure, a storage medium is provided, wherein instructions are stored in the storage medium. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the holographic projection method described in the first aspect or any one of the implementations of the first aspect.

[0024] According to a ninth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the holographic projection method described in the second aspect or any one of the embodiments of the second aspect.

[0025] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: intelligent interaction based on visual information and auditory information is realized based on a voice interaction device and a display terminal. Through the present disclosure, the cost of synchronously feeding back visual information and voice information based on interactive information sent by a user is reduced.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0028] Figure 1 The figure is a flow chart of a holographic projection method according to an exemplary embodiment.

[0029] Figure 2 The present invention is a flow chart showing a method for generating voice reply information according to an exemplary embodiment.

[0030] Figure 3 The invention is a flowchart of a method for generating a control instruction according to an exemplary embodiment.

[0031] Figure 4 The present invention is a flow chart showing a method for generating an active reply voice message according to an exemplary embodiment.

[0032] Figure 5 The present invention is a flowchart of a method for performing holographic projection based on control instructions according to an exemplary embodiment.

[0033] Figure 6 is a flow chart of a holographic projection method according to yet another exemplary embodiment.

[0034] Figure 7 is a flow chart of a holographic projection method according to another exemplary embodiment

[0035] Figure 8 It is a block diagram of a holographic projection method according to an exemplary embodiment of the present disclosure.

[0036] Fig. 9 It is a schematic diagram of a holographic projection method according to an exemplary embodiment of the present disclosure.

[0037] Fig.10 The present invention is a block diagram of a voice interaction device according to an exemplary embodiment.

[0038] Fig.11 It is a block diagram of a display terminal according to an exemplary embodiment.

[0039] Fig.12 It is a block diagram of a device for holographic projection according to an exemplary embodiment. DETAILED DESCRIPTION

[0040] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0041] The intelligent interaction method provided by the embodiments of the present disclosure is applied to scenarios in which a voice playback device and a projection device perform intelligent audio-visual language interaction with a user.

[0042] At present, smart speakers, as voice assistants in daily life, can be used to help control smart homes and even interact with users through voice. However, since the speakers can only communicate through voice, they cannot give users visual feedback such as body movements and facial expressions, resulting in a lack of immersion during the interaction. In order to improve the user experience, some manufacturers have developed the category of holographic projection speakers, adding devices for holographic projection on the basis of the inherent functions of smart speakers. When users interact with the speakers through voice, they are given voice feedback and visual feedback based on holographic projection at the same time. The current holographic projection speakers all regard the holographic projection device and the speaker as a whole, and the holographic projection device has high performance requirements, which leads to the need to improve the overall configuration of the speaker, thereby increasing the cost of the holographic projection speaker. However, for smart speakers, there is no improvement in the inherent functions, only the addition of the holographic projection function, which leads to excessively high costs and lack of cost performance.

[0043] Furthermore, the current maturity of voice interaction of speakers is insufficient. In related technologies, smart speakers interact with users based on built-in AI. The voice interaction function of speakers can only support basic interactions, perform functional operations, and make simple language replies. When the speaker is not controlled to perform functional operations and only voice communication is performed, the speaker can only make mechanical replies, reply to some common replies based on the preset language library, and can only make fixed replies when facing special questions. The degree of intelligence is insufficient. In addition, the built-in AI of smart speakers cannot imitate human emotions, achieve anthropomorphic effects, and cannot give users a sense of companionship.

[0044] In summary, the holographic projection speakers in the related art have the problems of high cost and low intelligence.

[0045] In view of this, the present disclosure proposes a holographic projection method based on a voice interaction device and a display terminal. Without changing the inherent structure of the voice interaction device, it is combined with the inherent display terminal to provide visual feedback and auditory feedback at the same time, thereby achieving the technical effect of a holographic projection speaker, and enabling the voice interaction device to access the GPT model for replying to interactive information, and interact with the user based on the GPT model, thereby improving the intelligence level of the voice interaction device.

[0046] Figure 1 is a flow chart of a holographic method according to an exemplary embodiment. Figure 1 As shown, the method includes steps S101 to S103.

[0047] In step S101, in response to the voice interaction device receiving the voice interaction information, voice reply information of the voice interaction information is generated, and a control instruction is generated.

[0048] In step S102, the control instruction is sent to the display terminal, and the display terminal generates visual reply information.

[0049] Among them, the visual response information includes the body movements and expressions of the digital virtual image, and the digital virtual image is displayed through a holographic image.

[0050] In step S103, in response to receiving the feedback information sent by the display terminal, a voice reply message is played.

[0051] Among them, the feedback information is used to represent the generated and displayed visual response information.

[0052] In the disclosed embodiment, the voice interaction device is connected to the network side and is wirelessly connected to the display terminal through a local area network. After the voice interaction device receives the voice interaction information sent by the user, it sends the voice interaction information to the network side, generates voice reply information for replying to the voice interaction information in response to the information fed back by the network side, and generates a control instruction for controlling the display terminal to perform visual reply information based on the information fed back by the network side. After the display terminal generates visual reply information based on the control instruction, it sends feedback information to the voice interaction device and displays the visual reply information. After the voice interaction device receives the feedback information, it plays the voice reply information. It can be understood that the delay in information transmission between the voice interaction device and the display terminal can be ignored, so it can be considered that the voice interaction device and the display terminal simultaneously feed back voice reply information and visual reply information to the user.

[0053] In the disclosed embodiment, the inherent display terminal (such as a tablet, mobile phone, etc.) is wirelessly connected to the voice interaction device, and visual response information is generated and displayed based on the control command issued by the voice interaction device. This avoids the cost increase caused by the integration of voice interaction devices and projection display devices, allowing users to experience the holographic projection speaker at a low cost.

[0054] In the embodiments of the present disclosure, the voice interaction device mainly plays the role of receiving information, transferring information and playing information. The voice reply information is not generated by the voice interaction device itself, but is generated by the language model pre-trained on the network side based on the voice interaction information after the voice interaction device sends the voice interaction information to the network side. The following embodiments of the present disclosure illustrate the method of generating voice reply information.

[0055] Figure 2 is a flow chart of a method for generating voice reply information according to an exemplary embodiment. Figure 2 As shown, the method includes step S201 to step S202.

[0056] In step S201, semantic information of speech interaction information is extracted based on the GPT model.

[0057] In step S202, speech reply information including semantic information is generated.

[0058] Among them, semantic information includes text reply information and semantic sentiment information.

[0059] In the disclosed embodiment, voice interaction information is acquired and analyzed based on a pre-trained GPT model, thereby generating voice response information containing text response information and semantic emotional information. It is understandable that based on the different training methods, the generated GPT model will differ in language style and simulated personality traits. The present disclosure can pre-train GPT models with different language styles and personality traits. Before using the intelligent interaction function of the language interaction device, the user can choose the GPT model of their interest, or actively switch to another GPT model after selecting a GPT model.

[0060] In the disclosed embodiment, the GPT model used to generate voice reply information is a large language model. The large language model trains a specific character based on specific character materials and changes the parameters of the activation layer of the neural network. It uses a deep learning model based on an artificial neural network (ANN) to generate text replies through a large amount of data analysis and training, and the large language model can be expanded with the expansion of the data set and the continuous optimization of the model. Based on the training and continuous optimization of the large language model, the large language model can generate appropriate answers based on the input language content. In summary, the large language model can realize personalized text generation solutions based on rich language data and powerful computing power, and through continuous iteration and improvement, it can continuously improve the accuracy and precision of its personalized text generation, thereby saving the interaction information between the robot and the user, so as to provide more localized, interesting and distinctive answers in subsequent conversations.

[0061] In summary, the present disclosure uses a large language model as a natural language processing technology to process and feedback the voice interaction information sent by the user, which has the following advantages: improving the language expression ability of the voice interaction device when making a voice reply; only fine-tuning is required on a relatively small data set to obtain better performance, solving the problem that the natural language processing model requires a large-scale data set; the large language model can not only generate natural language, but also recognize semantic information such as emotions in the text, that is, the large language model can be used for tasks such as sentiment analysis and text classification.

[0062] In the embodiments of the present disclosure, while the voice reply information is played through the voice interaction device, the terminal display device will also synchronously display the visual reply information, so that the user can obtain the audio-visual experience at the same time, and enhance the user's immersion. The visual reply information displayed by the terminal display device of the present disclosure is consistent with the voice reply information played by the voice interaction device, that is, the visual reply information is generated based on the GPT model to generate the voice interaction information. The following embodiments of the present disclosure illustrate the instructions for generating a control display terminal to display visual reply information. Figure 3 is a flow chart of a method for generating a control instruction according to an exemplary embodiment. Figure 3As shown, the method includes step S301 to step S302.

[0063] In step S301, semantic information of speech interaction information is extracted based on the GPT model.

[0064] In step S302, the semantic information is converted into control instructions based on the UDP protocol between the voice interaction device and the display terminal.

[0065] In the embodiment of the present disclosure, the voice interaction device and the display terminal are connected based on UDP broadcast communication, and the communication protocol of UDP broadcast communication specifies the control instructions of the voice interaction device to display visual interaction information on the display terminal. In the present disclosure, the visual interaction information displayed by the display terminal includes the body movements and facial expressions of the holographic digital virtual character. The communication protocol includes a control protocol for the virtual character. After the voice interaction information is generated, the semantic information contained in the semantic interaction information can be extracted, and the control instructions for controlling the digital virtual character based on the text reply information contained in the semantic information and the generation of semantic emotional information are performed, so that the digital virtual character performs the corresponding body movements and facial expressions. It is understandable that the user can pre-install multiple digital virtual characters in the display terminal, and the user can control the display terminal to switch digital virtual characters through voice instructions. It is further understandable that the holographic display of the digital virtual character will cause the power consumption of the display terminal to increase. When no instructions are received within a certain period of time, the display of the digital virtual character is turned off, so that it is in a dormant state, and the user can issue a voice instruction to wake up the digital virtual character.

[0066] In the disclosed embodiment, the voice interaction information generated based on the GPT model generates instructions for controlling the virtual character, so that the visual response information displayed by the terminal display device and the voice response information played by the voice interaction device are consistent, thereby improving the user's interaction experience.

[0067] In the embodiments of the present disclosure, in order to further enhance the user's interactive experience, the voice interaction device can be enabled to actively issue an interactive instruction. The following embodiments of the present disclosure illustrate the process of the voice interaction device actively issuing an interactive instruction.

[0068] Figure 4 is a flow chart showing a method for generating an active reply voice message according to an exemplary embodiment. Figure 4 As shown, the method includes step S401 to step S402.

[0069] In step S401, in response to receiving an active interaction instruction sent by a display terminal.

[0070] In step S402, the preset active interactive language information is played.

[0071] In the disclosed embodiment, the display terminal has the function of sensing the user. When sensing that the user performs a specific action, it can generate an active interaction instruction and send it to the voice interaction device, so that the voice interaction device synchronously sends active interaction language information and active interaction visual information.

[0072] In an exemplary embodiment of the present disclosure, the active language interaction information of the voice interaction device can be customized, and the specific content played and the voice played in different time periods can be customized. For example, the voice sent in the morning is "Good morning!", and the voice sent in the evening is "Welcome home".

[0073] In the embodiment of the present disclosure, corresponding to the voice interaction device, after receiving the control command sent by the voice interaction device, the display terminal generates visual response information consistent with the voice interaction device, and simultaneously displays the visual response information while playing the voice response information. The following embodiment of the present disclosure describes a method for a display terminal to generate and display visual response information based on a control command.

[0074] It should be noted that the above method can be executed by a speaker such as a smart speaker, but is not limited to this.

[0075] Figure 5 is a flow chart of a method for performing holographic projection based on control instructions according to an exemplary embodiment. Figure 5 As shown, the method includes step S501 to step S502.

[0076] In step S501, in response to the display terminal receiving a control instruction sent by the voice interaction device, visual reply information is generated based on the control instruction.

[0077] Among them, the visual response information includes the body movements and expressions of the digital virtual image, and the digital virtual image is displayed through a holographic image.

[0078] In step S502, feedback information is sent to the voice interaction device and visual reply information is played.

[0079] In the disclosed embodiment, after receiving the control command, the display terminal generates a command to control the holographic projection of the digital virtual character to make corresponding actions and facial expressions, and after generating the command, sends feedback information representing the generated command to the voice interaction device, prompting the voice interaction device to play the voice reply information, and synchronously displays the visual reply information, so that the holographic projection of the digital virtual character makes actions and expressions that conform to the context of the voice reply information. By synchronously feeding back audio-visual content to the user, the user's sense of immersion during interaction is enhanced.

[0080] In the embodiment of the present disclosure, the digital virtual image for displaying visual reply information by the user is not unique, and the user can select one based on his / her own preferences. The following embodiment of the present disclosure describes the installation form of the digital virtual image in the display terminal.

[0081] In one implementation of the disclosed embodiment, the digital virtual image is installed in the display terminal in the form of an application package.

[0082] In the disclosed embodiment, the digital virtual image is an installation package (apk) pre-made by the manufacturer of the display terminal or a third-party manufacturer, and the user selects and downloads it in the application store or theme store. It is understandable that the user can download the installation packages of multiple digital virtual images at the same time and install them in the display terminal, select the interactive object for intelligent interaction based on personal needs, or actively switch the digital virtual image during the interaction process.

[0083] In the disclosed embodiment, a three-dimensional image and multiple actions of a digital virtual character are established based on dedicated software (such as Unity), and relevant action control logic is developed to customize corresponding body movements, emotional movements, etc. according to the character model. And the control protocol set by UDP broadcast communication is used to complete the control of the corresponding actions of the character. For example: the actions of the character when angry or happy, and the corresponding lip shape, eye expression and body movement of the character when playing voice reply information, so as to make the interaction of the virtual character more intelligent and anthropomorphic. After completing the development logic, the installation package can be exported, and the display device can run the digital virtual image after downloading and installing the package, and control the digital virtual image to make corresponding actions, expressions and lip shapes based on the control instructions.

[0084] It is understandable that the three-dimensional modeling, action design and operation logic setting of digital virtual characters are relatively complex, and professionals are required to use software to obtain a complete digital virtual image. Users can only choose a digital virtual image for interaction from a variety of finished digital virtual images, and cannot participate in the customization of the digital virtual image during the process of creating the digital virtual image. It is further understandable that users can speak out on relevant platforms and make suggestions on the design of digital virtual images, so that professionals can design based on user opinions, thereby indirectly participating in the design of digital virtual images.

[0085] In the embodiments of the present disclosure, the body movements and facial expressions of the digital virtual image do not need to correspond in real time with the playback process of the language reply information. Instead, the digital virtual image only needs to make corresponding movements and expressions based on the semantic information conveyed in the entire language reply information. However, the digital virtual image needs to correspond to the language reply information in real time during the playback process of the language reply information. The following embodiments of the present disclosure illustrate the method for adjusting the lip shape of the virtual image during the playback of the voice reply information.

[0086] Figure 6is a flow chart of a holographic projection method according to another exemplary embodiment. Figure 6 As shown, the method includes step S601 to step S602.

[0087] In step S601, in response to the voice interaction device playing the voice reply information, the audio information included in the voice reply information is synchronously acquired.

[0088] In step S602, the lip shape of the digital avatar is adjusted in real time based on the audio information.

[0089] In the disclosed embodiment, the audio information of the voice reply information is synchronously obtained during the playback of the voice reply information, and the opening and closing of the digital virtual image's mouth shape corresponds to the voice reply information word by word, so that the opening and closing degree of the virtual image's mouth shape corresponds to the volume of the voice reply information playback sound. By synchronously adjusting the digital virtual image's mouth shape during the voice reply information process, the digital virtual image's body movements and facial expressions are kept consistent with the voice reply information, and the digital virtual image's mouth shape is also synchronized with the playback process of the voice reply information in real time. In this way, the digital virtual image is completely synchronized with the voice reply information, enhancing the immersion of the user during interaction.

[0090] In the embodiments of the present disclosure, the voice interaction device and the display terminal can not only respond to the interaction initiated by the user, but also actively initiate interaction with the user under certain conditions. The following embodiments of the present disclosure illustrate the method for the voice interaction device and the display terminal to actively initiate interaction.

[0091] Figure 7 is a flow chart of a holographic projection method according to another exemplary embodiment. Figure 7 As shown, the method includes step S701 to step S702.

[0092] In step S701, in response to the user facing the projection frame, an active interaction instruction is generated and sent to the voice interaction device.

[0093] In step S702, in response to the active interaction instruction being sent to the voice interaction device, pre-set active interaction visual information is played.

[0094] It should be noted that the above method can be executed by a display terminal such as a computer, a personal notebook, a tablet computer, etc., but is not limited thereto.

[0095] In the disclosed embodiment, the display terminal is fixed on a projection frame for displaying a digital virtual image, and the image displayed by the display terminal is projected on the projection screen of the projection frame, so as to realize a holographic display of the digital virtual image. Among them, a visual sensing device (monitoring camera) is provided on the projection frame, and the orientation of the visual sensing device is the front orientation of the holographic image of the digital virtual image. When the display monitors that the user is facing the projection frame (i.e., the digital virtual image), an instruction for active interaction can be issued to instruct the voice interaction device and the display terminal to synchronously send out interaction information, that is, to instruct the voice interaction device to send out voice interaction information (such as "Good morning", "Would you like some music") and at the same time instruct the display terminal to send out visual interaction information (such as waving, jumping, etc.)

[0096] It is understandable that, if the user needs to protect personal privacy, they can protect their privacy by changing the permissions of the program running the digital virtual image and turning off the visual sensing device on the projection frame.

[0097] In the disclosed embodiment, when the display terminal detects that the user is facing the digital virtual image, it actively issues instructions to instruct the display terminal and the voice interaction terminal to actively interact, thereby improving the level of intelligence and enhancing the user's experience of using the voice interaction device.

[0098] In an exemplary embodiment of the present disclosure, intelligent interaction based on voice and digital virtual images is accomplished based on a router, a smart speaker, a tablet terminal and a projection framework. Figure 8 The block diagram of the holographic projection method shown and Fig. 9The schematic diagram of the holographic projection method shown in the figure, the present disclosure adopts the following method to complete the intelligent interaction based on visual language and auditory language: establish a user data protocol broadcast communication local area network through a router, wirelessly connect the flat speaker to the flat terminal, and connect the large language model on the network side through the flat terminal. After the smart speaker receives the interactive information sent by the user, the interactive information is sent to the large language model on the network side. The large language model parses the interactive information and generates voice reply information, and sends the voice reply information to the smart speaker. The smart speaker converts the voice reply information into a control instruction based on the voice reply information and the communication protocol and sends the control instruction to the flat terminal. The flat terminal is pre-set with a digital virtual image that can be holographically projected. The flat terminal makes the digital virtual image make corresponding body movements and facial expressions while playing the voice reply information based on the control instruction. The flat terminal also obtains the audio information of the voice reply information while playing the voice reply information, and the lip shape of the digital virtual image is adjusted based on the audio information. The smart speaker and the flat terminal synchronously play the voice reply information and control the digital virtual image to perform the corresponding action, lip shape and expression, which improves the user experience and immersion during interaction. In addition, the tablet terminal can also perform detection based on the visual detection device on the projection frame. When it is detected that the user is facing the digital virtual image, the tablet terminal and the smart speaker will simultaneously send active interaction information, thereby improving the intelligence of the interactive device. In summary, the present disclosure achieves the same effect as the holographic projection speaker by splitting the holographic projection speaker and combining the existing voice interaction device and display terminal, thereby reducing the cost of synchronous visual and auditory interaction. And by connecting to the large language model on the network side and obtaining reply information, the intelligence of the language interaction device is improved, making the language interaction device more humanized, giving the user a sense of companionship, and enhancing the user's sense of immersion during interaction.

[0099] Based on the same concept, the embodiment of the present disclosure also provides a voice interaction device 100 and a display terminal 200.

[0100] It is understandable that the voice interaction device 100 and the display terminal 200 provided in the embodiment of the present disclosure include hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0101] Fig.101 is a block diagram of a voice interaction device 100 according to an exemplary embodiment. Fig.10 The device includes a generating unit 101, a first processing unit 102 and a first playing unit 103.

[0102] The generating unit 101 is used for generating voice reply information of the voice interaction information in response to the voice interaction device receiving the voice interaction information.

[0103] The first processing unit 102 is used to generate a control instruction and send the control instruction to the display terminal, and the display terminal generates visual response information. The visual response information includes the body movements and expressions of the digital virtual image. The digital virtual image is displayed through a holographic image.

[0104] The first playing unit 103 is used to play the voice reply information in response to receiving the feedback information sent by the display terminal, where the feedback information is used to indicate that the visual reply information has been generated and displayed.

[0105] In one embodiment, the generation unit 101 generates voice response information of the voice interaction information in the following manner: based on the GPT model, the semantic information of the voice interaction information is extracted, and the voice response information including the semantic information is generated, and the semantic information includes text response information and semantic emotion information. The GPT model is pre-trained based on a pre-trained model.

[0106] In one implementation, the first processing unit 102 generates the control instruction in the following manner: based on the UDP protocol between the voice interaction device and the display terminal, the text reply information is converted into the control instruction.

[0107] In one implementation, the first processing unit 102 is further configured to: in response to receiving an active interaction instruction sent by the display terminal, play preset active interaction language information.

[0108] Fig.11 2 is a block diagram of a display terminal 200 according to an exemplary embodiment. Fig.11 The terminal includes a second processing unit 201 and a second playing unit 202.

[0109] The second processing unit 201 is configured to generate visual response information based on the control instruction received by the display terminal from the voice interaction device, wherein the visual response information includes body movements and expressions of the digital virtual image, and the digital virtual image is displayed through a holographic image.

[0110] The second playing unit 202 is used to send feedback information to the voice interaction device and play visual reply information.

[0111] In one implementation, the digital avatar is installed in the display terminal in the form of an application package.

[0112] In one embodiment, the visual response information includes the lip shape of the digital avatar, and the second processing unit 201 is further configured to: in response to the voice response information being played by the voice interaction device, synchronously obtain the audio information included in the voice response information, and adjust the lip shape of the digital avatar in real time based on the audio information.

[0113] In one embodiment, the display terminal is connected to a projection frame, the digital virtual image is projected by the display terminal on the projection frame, the projection frame includes a visual detection device, and the second processing unit 201 is further used to: in response to the user facing the projection frame, generate an active interaction instruction, and send the active interaction instruction to the voice interaction device, the active interaction instruction is used to control the display terminal and the voice playback device to synchronize active interaction visual information and active interaction voice information. In response to the active interaction instruction being sent to the voice interaction device, the preset active interaction visual information is played.

[0114] Regarding the devices and terminals in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and will not be elaborated here.

[0115] Fig.12 3 is a block diagram of a device 300 for holographic projection according to an exemplary embodiment. For example, the device 300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0116] Reference Fig.12 , the device 300 may include one or more of the following components: a processing component 302 , a memory 304 , a power component 306 , a multimedia component 308 , an audio component 310 , an input / output (I / O) interface 312 , a sensor component 314 , and a communication component 316 .

[0117] The processing component 302 generally controls the overall operation of the device 300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 302 may include one or more modules to facilitate the interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate the interaction between the multimedia component 308 and the processing component 302.

[0118] The memory 304 is configured to store various types of data to support operations on the device 300. Examples of such data include instructions for any application or method operating on the device 300, contact data, phone book data, messages, pictures, videos, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0119] The power component 306 provides power to the various components of the device 300. The power component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 300.

[0120] The multimedia component 308 includes a screen that provides an output interface between the device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front camera and / or a rear camera. When the device 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0121] The audio component 310 is configured to output and / or input audio signals. For example, the audio component 310 includes a microphone (MIC), and when the device 300 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for outputting audio signals.

[0122] I / O interface 312 provides an interface between processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0123] The sensor assembly 314 includes one or more sensors for providing various aspects of status assessment for the device 300. For example, the sensor assembly 314 can detect the open / closed state of the device 300, the relative positioning of components, such as the display and keypad of the device 300, the sensor assembly 314 can also detect the position change of the device 300 or a component of the device 300, the presence or absence of user contact with the device 300, the orientation or acceleration / deceleration of the device 300, and the temperature change of the device 300. The sensor assembly 314 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 314 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0124] The communication component 316 is configured to facilitate wired or wireless communication between the device 300 and other devices. The device 300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0125] In an exemplary embodiment, the apparatus 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0126] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, and the instructions can be executed by the processor 320 of the device 300 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0127] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0128] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0129] It will be further understood that the terms “center”, “longitudinal”, “lateral”, “front”, “back”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.

[0130] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0131] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0132] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0133] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A holographic projection method, characterized in that: include: In response to the voice interaction device receiving the voice interaction information, generating voice reply information of the voice interaction information; Generate a control instruction, and send the control instruction to a display terminal, and the display terminal generates visual response information, the visual response information includes body movements and expressions of the digital virtual image, and the digital virtual image is displayed through a holographic image; In response to receiving feedback information sent by the display terminal, the voice reply information is played, and the feedback information is used to indicate that the visual reply information has been generated and displayed.

2. The method according to claim 1, characterized in that The generating of the voice reply information of the voice interaction information includes: Based on the GPT model, extract the semantic information of the voice interaction information, and generate voice reply information including the semantic information, wherein the semantic information includes text reply information and semantic emotion information; The GPT model is pre-trained based on a pre-trained model.

3. The method according to claim 1 or 2, characterized in that: The generating control instruction comprises: Based on the UDP protocol between the voice interaction device and the display terminal, the text reply information is converted into the control instruction.

4. The method according to claim 1, characterized in that The method further comprises: In response to receiving the active interaction instruction sent by the display terminal, pre-set active interaction language information is played.

5. A holographic projection method, characterized in that: include: In response to the display terminal receiving a control instruction sent by the voice interaction device, generating visual reply information based on the control instruction, the visual reply information including body movements and expressions of the digital virtual image, and the digital virtual image is displayed through a holographic image; Send feedback information to the voice interaction device and play the visual reply information.

6. The method according to claim 5, characterized in that The digital avatar is installed in the display terminal in the form of an application package.

7. The method according to claim 5, characterized in that The visual response information includes the lip shape of the digital avatar, and the method further includes: In response to the voice interaction device playing the voice reply information, synchronously acquiring audio information included in the voice reply information; Based on the audio information, the lip shape of the digital avatar is adjusted in real time.

8. The method according to claim 5, characterized in that The display terminal is connected to a projection frame, the digital virtual image is projected by the display terminal onto the projection frame, and the projection frame includes a visual detection device. The method further comprises: In response to the user facing the projection frame, an active interaction instruction is generated, and the active interaction instruction is sent to the voice interaction device, wherein the active interaction instruction is used to control the display terminal and the voice playback device to synchronize active interaction visual information and active interaction voice information; In response to the active interaction instruction being sent to the voice interaction device, the pre-set active interaction visual information is played.

9. A voice interaction device, characterized in that: include: A generating unit, configured to generate voice reply information of the voice interaction information in response to the voice interaction device receiving the voice interaction information; A first processing unit is used to generate a control instruction and send the control instruction to a display terminal, and the display terminal generates visual response information, wherein the visual response information includes body movements and expressions of the digital virtual image, and the digital virtual image is displayed through a holographic image; The first playback unit is used to play the voice reply information in response to receiving feedback information sent by the display terminal, where the feedback information is used to indicate that the visual reply information has been generated and displayed.

10. The device according to claim 9, characterized in that The generating unit generates the voice reply information of the voice interaction information in the following manner: Based on the GPT model, extract the semantic information of the voice interaction information, and generate voice reply information including the semantic information, wherein the semantic information includes text reply information and semantic emotion information; The GPT model is pre-trained based on a pre-trained model.

11. The device according to claim 9 or 10, characterized in that The first processing unit generates a control instruction in the following manner: Based on the UDP protocol between the voice interaction device and the display terminal, the text reply information is converted into the control instruction.

12. The device according to claim 9, characterized in that The first processing unit is further configured to: In response to receiving the active interaction instruction sent by the display terminal, pre-set active interaction language information is played.

13. A display terminal, characterized in that: include: A second processing unit is configured to generate visual reply information based on the control instruction received by the display terminal, wherein the visual reply information includes body movements and expressions of the digital virtual image, and the digital virtual image is displayed through a holographic image; The second playing unit is used to send feedback information to the voice interaction device and play the visual reply information.

14. The terminal according to claim 13, characterized in that: The digital avatar is installed in the display terminal in the form of an application package.

15. The terminal according to claim 13, characterized in that: The visual reply information includes the mouth shape of the digital avatar, and the second processing unit is further used for: In response to the voice interaction device playing the voice reply information, synchronously acquiring audio information included in the voice reply information; Based on the audio information, the lip shape of the digital avatar is adjusted in real time.

16. The terminal according to claim 13, characterized in that: The display terminal is connected to a projection frame, the digital virtual image is projected by the display terminal onto the projection frame, and the projection frame includes a visual detection device. The processing unit is also used for: In response to the user facing the projection frame, an active interaction instruction is generated, and the active interaction instruction is sent to the voice interaction device, wherein the active interaction instruction is used to control the display terminal and the voice playback device to synchronize active interaction visual information and active interaction voice information; In response to the active interaction instruction being sent to the voice interaction device, the pre-set active interaction visual information is played.

17. A holographic projection system, characterized in that: It comprises the voice interaction device according to any one of claims 1 to 4 and the display terminal according to any one of claims 5 to 8.

18. A voice interaction device, characterized in that: include: processor: a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the holographic projection method according to any one of claims 1 to 4.

19. A display terminal, characterized in that: include: processor: a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the holographic projection method according to any one of claims 5 to 8.

20. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor, the processor is enabled to execute the holographic projection method according to any one of claims 1 to 4.

21. A storage medium, characterized in that: Instructions are stored in the storage medium. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the holographic projection method according to any one of claims 5 to 8.

Images