Exhibition hall multi-screen shuttling digital human following interaction method

Abstract

The invention discloses an exhibition hall multi-screen shuttling digital human following interaction method, which is characterized in that an exhibition hall is provided with a plurality of exhibition stands, each exhibition stand is provided with a working machine, each working machine is connected with a server to form an exhibition hall system, and each working machine is connected with a microphone, a camera, a screen and a sound box; the multi-screen shuttling digital human following interaction method comprises the following steps: locking corresponding visitors by an exhibition hall system based on mouth shapes of the visitors saying preset keywords, and collecting voiceprint features of the corresponding visitors as voice recognition reference features; the exhibition hall system receives voice input of visitors based on voice recognition reference features, generates answer content through a general large model in combination with an exhibition hall field knowledge base and historical contexts, and transmits the content to the digital human front-end interaction system to drive a digital human to interact with a user on a screen; and the exhibition hall system obtains the moving state of the visitor of the user based on the voice information and / or action information of the visitor, and drives the digital person to move along with the visitor on the corresponding screen.

Description

Technical Field

[0001] The present application belongs to the field of computer image processing technology, and in particular relates to a digital human following interaction method for multiple screens in an exhibition hall. Background Art

[0002] At present, digital humans can integrate domain knowledge bases and vertical industry big models to achieve intelligent multimodal interaction capabilities with highly simulated images. They are often used in exhibition hall reception scenarios and can answer visitors' questions independently based on the exhibition content. They can also combine translation functions to provide more convenient services and communication methods for audiences from different countries and regions.

[0003] However, the current digital human exhibition hall reception is often carried out through the all-in-one machine of the holographic illusion warehouse or through a single work machine connected to the screen for fixed-point service. It is impossible to provide follow-up services to visitors. Digital humans can only be deployed in each exhibition content to provide explanations. This makes the visitor experience fragmented and the sense of immersion is destroyed. Especially in the scenario of receiving important leaders and customers, digital humans need to provide top-level personalized follow-up services to maximize the role of exhibition hall reception. Summary of the invention

[0004] The purpose of this application is to overcome the problem of insufficient immersion in the existing digital human exhibition hall reception, and to propose a digital human follow-up interaction method for multiple screens in the exhibition hall.

[0005] The purpose of this application is achieved through the following technical solutions:

[0006] A digital human follow-up interactive method for multiple screens in an exhibition hall, wherein the exhibition hall is provided with a plurality of exhibition stands, each exhibition stand is respectively equipped with a working machine, and each working machine is connected to a server to form an exhibition hall system, wherein the working machine is connected with a microphone, a camera, a screen and a speaker;

[0007] The screen is used to display the digital human, the microphone and the speaker are used for the visitor's digital human to perform voice interaction, and the camera is used to identify the speaker;

[0008] The multi-screen shuttle digital human following interaction method comprises:

[0009] The exhibition hall system locks the corresponding visitor based on the lip shape of the visitor when he / she says the preset keywords, and collects the voiceprint features of the corresponding visitor as reference features for voice recognition;

[0010] The exhibition hall system receives the visitor's voice input based on the sound recognition reference features, generates answer content through the universal large model combined with the exhibition hall domain knowledge base and historical context, and transmits the content to the digital human front-end interaction system to drive the digital human to interact with the user on the screen;

[0011] The exhibition hall system obtains the user's visitor's movement status based on the visitor's voice information and / or action information, and drives the digital person to move with the visitor on the corresponding screen.

[0012] According to a preferred embodiment, the working machine is deployed with a digital human interaction system, and the digital human interaction system includes a three-dimensional digital human and a digital human animation sequence library.

[0013] According to a preferred embodiment, a digital human backend system is deployed on the server.

[0014] The backend system includes an AI model library and a showroom domain knowledge base to provide AI capabilities. The AI ​​model library includes an audio-to-text model, a lip reading recognition model, a voiceprint recognition model, a person recognition model, and a general large model.

[0015] The exhibition hall domain knowledge base stores information such as exhibition hall exhibition content and explanation content; all digital human interaction systems use AI functions by calling the server digital human backend system.

[0016] According to a preferred embodiment, when a visitor visits, he interacts with the digital human through a microphone, and the audio input by the user is displayed on the screen through subtitles.

[0017] According to a preferred embodiment, the audio-to-text conversion process can be implemented by Whisper, Wav2vec, MASR or FunASR model.

[0018] According to a preferred implementation, each working machine in the exhibition hall system is numbered in the order of visiting, including: working machine 1, working machine 2, ..., working machine n.

[0019] According to a preferred embodiment, each working machine in the exhibition hall system is registered using a triplet, including the current working machine number, the left working machine number and the right working machine number.

[0020] According to a preferred embodiment, the process of the digital human accompanying the visitor's movement on the corresponding screen includes: accompanying the visitor to leave the current booth on the screen corresponding to the current booth, and accompanying the visitor to the corresponding booth area on the screen corresponding to the next booth to be visited.

[0021] According to a preferred embodiment, the digital human has a preset action sequence animation, including at least two types of departure animation and arrival animation;

[0022] The leaving animation is triggered when the user says a keyword containing the meaning of leaving or leaves the current camera range and enters the range of another camera. The leaving animation is that the digital human turns from the front to the side. If the next camera is to the left of the current camera, the digital human turns left, otherwise it turns right and walks to the edge of the screen until it disappears completely.

[0023] The arrival animation is triggered after the departure animation is completed and the current camera has captured the user. The arrival animation is that the digital human walks sideways from the edge of the screen to the center of the screen and turns to the front. If the previous camera is on the right side of the current camera, it appears from the right edge of the screen, otherwise it appears from the left edge of the screen.

[0024] According to a preferred embodiment, when the shooting ranges of the cameras overlap, the visitor will be photographed by the two booth cameras A and B at the same time, and the visitor will be dynamically identified using the character recognition model. A , K B , calculate the area of ​​the character recognition box S A , S B , when visitors interact, record the audio input time T A 、T B , and the peak volume D A , D B , if S A >S B 、T A <T B , D A >D B If two of the three conditions are met, the visitor is considered to be at booth A, otherwise at booth B.

[0025] The aforementioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0026] Beneficial effects of this application:

[0027] The setting of the application method can enable the digital human to display an interactive effect of shuttling between different screens to follow the visitors. Compared with the fixed-point service form commonly used by digital humans at present, it provides a more immersive and personalized experience effect and innovates in the service method. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the layout drawing of the booth you are applying for. DETAILED DESCRIPTION

[0029] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0030] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0031] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the application is usually placed when in use. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0032] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0033] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0034] In addition, the present application would like to point out that, in the present application, unless the specific structure, connection relationship, positional relationship, power source relationship, etc. are specifically written out, the structure, connection relationship, positional relationship, power source relationship, etc. involved in the present application are all known by those skilled in the art on the basis of the prior art without creative work.

[0035] Example

[0036] The present application discloses a digital human following interactive method for multiple screens in an exhibition hall. The exhibition hall is provided with a plurality of exhibition stands, each of which is respectively equipped with a working machine, and each working machine is connected to a server to form an exhibition hall system. The working machine is connected with a microphone, a camera, a screen and a speaker. Figure 1 shown.

[0037] The screen is used to display the digital human, the microphone and the speaker are used for the visitor's digital human to perform voice interaction, and the camera is used to identify the speaker.

[0038] Preferably, the working machine is deployed with a digital human interaction system, and the digital human interaction system includes a three-dimensional digital human and a digital human animation sequence library.

[0039] A digital human backend system is deployed on the server, and the backend system includes an AI model library and an exhibition hall domain knowledge base to provide AI capabilities. The AI ​​model library includes an audio-to-text model, a lip reading recognition model, a voiceprint recognition model, a character recognition model, and a general large model. The exhibition hall domain knowledge base stores information such as exhibition hall exhibition content and explanation content; all digital human interaction systems use AI functions by calling the server digital human backend system.

[0040] Preferably, when a visitor visits, he or she interacts with the digital human through a microphone, and the audio input by the user is displayed on the screen through subtitles. This allows the visitor to confirm the correctness of the input audio and determine whether it needs to be repeated. Furthermore, the audio-to-text conversion process can be implemented through Whisper, Wav2vec, MASR or FunASR models.

[0041] Preferably, each working machine in the exhibition hall system is numbered in the order of visiting, including: working machine 1, working machine 2, ..., working machine n.

[0042] Furthermore, considering that the exhibition hall may have a clockwise or counterclockwise layout, each workstation in the exhibition hall system is registered using a triplet, including the current workstation number, the left workstation number, and the right workstation number, which helps to clarify the location of each workstation.

[0043] The digital human following interaction method of the multi-screen shuttle of the present application includes:

[0044] The exhibition hall system locks the corresponding visitor based on the lip shape of the visitor when he / she says the preset keywords, and collects the voiceprint features of the corresponding visitor as the reference features for voice recognition. The exhibition hall system receives the visitor's voice input based on the reference features for voice recognition, and generates the answer content through the universal large model combined with the exhibition hall domain knowledge base and historical context, and transmits the content to the digital human front-end interactive system to drive the digital human to interact with the user on the screen; the exhibition hall system obtains the user's visitor's movement status based on the visitor's voice information and / or action information, and drives the digital human to move with the visitor on the corresponding screen.

[0045] Specifically, in order to avoid confusion of the target of following caused by company personnel accompanying, this application pre-sets keywords, and company personnel or signboards can guide visitors to say the keywords. The exhibition hall system identifies all people present through the character recognition model, and then identifies the lip shape through the lip reading recognition model, compares the keyword lip shape to locate the visitor, and finally records the visitor's voiceprint through the voiceprint recognition model. When there is audio input, the voiceprint record is compared, and only the audio that matches the visitor's voiceprint can be input into the digital human system.

[0046] Furthermore, the person recognition model can be implemented through the YOLOv9 model. The lip reading recognition model can be implemented through the MuAViC model. The voiceprint recognition model can be implemented through the ECAPA-TDNN, x-vector and other models.

[0047] Preferably, to ensure the continuity of cross-booth interactions, when a visitor is identified, the exhibition hall system will create a user space for him / her, record the historical context of the interaction, and input it into the universal macro model as a prompt word.

[0048] Preferably, the process of the digital human accompanying the visitor's movement on the corresponding screen includes: accompanying the visitor to leave the current booth on the screen corresponding to the current booth, and accompanying the visitor to arrive at the corresponding booth area on the screen corresponding to the next booth to be visited.

[0049] Furthermore, the digital human has a preset action sequence animation, including at least two types of departure animation and arrival animation;

[0050] The leaving animation is triggered when the user says a keyword containing the meaning of leaving or leaves the current camera range and enters the range of another camera. The leaving animation is that the digital human turns from the front to the side. If the next camera is to the left of the current camera, the digital human turns left, otherwise it turns right and walks to the edge of the screen until it disappears completely.

[0051] The arrival animation is triggered after the departure animation is completed and the current camera has captured the user. The arrival animation is that the digital human walks sideways from the edge of the screen to the center of the screen and turns to the front. If the previous camera is on the right side of the current camera, it appears from the right edge of the screen, otherwise it appears from the left edge of the screen.

[0052] Preferably, when the shooting ranges of the cameras overlap, the visitor will be photographed by the two booth cameras A and B at the same time, and the visitor will be dynamically identified using the person recognition model. A , K B , calculate the area of ​​the character recognition box S A , S B , when visitors interact, record the audio input time T A 、T B , and the peak volume D A , D B , if S A >S B 、T A <T B , D A >D B If two of the three conditions are met, the visitor is considered to be at booth A, otherwise at booth B.

[0053] The setting of the application method can enable the digital human to display an interactive effect of shuttling between different screens to follow the visitors. Compared with the fixed-point service form commonly used by digital humans at present, it provides a more immersive and personalized experience effect and innovates in the service method.

[0054] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A digital human following interactive method for multiple screens in an exhibition hall, characterized in that: The exhibition hall is provided with a number of exhibition stands, each of which is equipped with a working machine, and each working machine is connected to a server to form an exhibition hall system, and the working machine is connected to a microphone, a camera, a screen and a speaker; The screen is used to display the digital human, the microphone and the speaker are used for the visitor's digital human to perform voice interaction, and the camera is used to identify the speaker; The multi-screen shuttle digital human following interaction method comprises: The exhibition hall system locks the corresponding visitor based on the lip shape of the visitor when he / she says the preset keywords, and collects the voiceprint features of the corresponding visitor as reference features for voice recognition; The exhibition hall system receives the visitor's voice input based on the sound recognition reference features, generates answer content through the universal large model combined with the exhibition hall domain knowledge base and historical context, and transmits the content to the digital human front-end interaction system to drive the digital human to interact with the user on the screen; The exhibition hall system obtains the user's visitor's movement status based on the visitor's voice information and / or action information, and drives the digital person to move with the visitor on the corresponding screen.

2. The digital human following interaction method according to claim 1, characterized in that: The working machine is deployed with a digital human interaction system, which includes a three-dimensional digital human and a digital human animation sequence library.

3. The digital human following interaction method as claimed in claim 2, characterized in that: A digital human backend system is deployed on the server. The backend system includes an AI model library and a showroom domain knowledge base to provide AI capabilities. The AI ​​model library includes an audio-to-text model, a lip reading recognition model, a voiceprint recognition model, a person recognition model, and a general large model. The exhibition hall domain knowledge base stores information such as exhibition hall exhibition content and explanation content; all digital human interaction systems use AI functions by calling the server digital human backend system.

4. The digital human following interaction method as claimed in claim 3, characterized in that: When visitors visit, they interact with the digital human through a microphone, and the audio input by the user is displayed on the screen through subtitles.

5. The digital human following interaction method as claimed in claim 4, characterized in that: The audio-to-text conversion process can be implemented through Whisper, Wav2vec, MASR or FunASR models.

6. The digital human following interaction method according to claim 1, characterized in that: Each working machine in the exhibition hall system is numbered according to the order of visiting, including: working machine 1, working machine 2, ..., working machine n.

7. The digital human following interaction method according to claim 6, characterized in that: Each work machine in the exhibition hall system is registered using a triplet, including the current work machine number, the left work machine number, and the right work machine number.

8. The digital human following interaction method according to claim 1, characterized in that: The process of digital people accompanying visitors on the corresponding screen includes: Accompany the visitor to leave the current booth on the screen corresponding to the current booth, and accompany the visitor to the corresponding booth area on the screen corresponding to the next booth.

9. The digital human following interaction method as claimed in claim 8, characterized in that: The digital human has a preset action sequence animation, including at least two types of departure animation and arrival animation; The leaving animation is triggered when the user says a keyword containing the meaning of leaving or leaves the current camera range and enters the range of another camera. The leaving animation is that the digital human turns from the front to the side. If the next camera is to the left of the current camera, the digital human turns left, otherwise it turns right and walks to the edge of the screen until it disappears completely. The arrival animation is triggered after the departure animation is completed and the current camera has captured the user. The arrival animation is that the digital human walks sideways from the edge of the screen to the center of the screen and turns to the front. If the previous camera is on the right side of the current camera, it appears from the right edge of the screen, otherwise it appears from the left edge of the screen.

10. The digital human following interaction method according to claim 7, characterized in that: When the shooting ranges of the cameras overlap, the visitor will be captured by the two booth cameras A and B at the same time. The person recognition model is used to dynamically identify the visitor. There are two person recognition frames K A , K B , calculate the area of ​​the character recognition box S A , S B , when visitors interact, record the audio input time T A , T B , and the peak volume D A , D B , if S A >S B , T A <T B , D A >D B If two of the three conditions are met, the visitor is considered to be at booth A, otherwise at booth B.

Images