Elevator taking assistant system based on generative and cached digital human

Abstract

The invention relates to the technical field of intelligent elevators, and discloses an elevator taking assistant system based on a generative and cached digital human, which comprises a camera, an elevator taking assistant screen, a voice module ASR, an elevator taking assistant subsystem and an elevator controller, and is characterized in that the elevator taking assistant subsystem is respectively connected with the camera, the elevator taking assistant screen, the voice module ASR and the elevator controller; when a user enters an elevator waiting hall, a camera monitors the moving direction of the user, an elevator taking assistant screen prompts the user to call an elevator through voice and obtain elevator information, an elevator taking assistant subsystem sends an elevator calling instruction to an elevator controller, and the elevator taking assistant screen displays a corresponding digital human image video. And the elevator taking assistant subsystem identifies the moving direction of the user and broadcasts a voice prompt that the elevator arrives. According to the invention, information visualization interaction is carried out between the image of the digital person and the elevator user, elevator calling and elevator information query functions are provided for the user, visual interaction elevator calling experience is brought, and the availability and economical efficiency of a digital person elevator taking assistant are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent elevators, and in particular to an elevator assistant system based on a generative and cached digital human. Background Art

[0002] Generative digital human technology is a technology that uses computer graphics, artificial intelligence, machine learning and other technologies to create virtual characters with human appearance, expressions, movements, and language communication capabilities. Digital humans have a wide range of applications, ranging from entertainment, education, training to customer service and many other fields.

[0003] In most of the current digital human application scenarios, most of them are one-to-many digital human applications, such as live selling, online education, etc. In this scenario, most of them are to obtain video streams from the digital human platform, and then distribute the video in the live broadcast room, generally using one-way inflow. In the elevator assistant system, there will be a digital human image in each elevator lobby, and when the assistant interacts with passengers, it will involve the need to obtain multiple videos from the digital human platform, such as elevator assistant 1 is interacting with passenger 1, elevator assistant 2 is interacting with passenger 2, and elevator assistant n is interacting with passenger n. Multi-channel interaction is different from ordinary live broadcast, and multiple video streams need to be obtained from the digital platform. However, the current cost of multi-channel generative digital humans is relatively high, and the annual package of a single video stream is more than 20,000 yuan, and the elevator assistant cannot afford such expensive video stream costs. Therefore, a digital human elevator assistant system based on generative and caching is proposed to solve the availability and economy of elevator assistants. Summary of the invention

[0004] The present invention provides an elevator assistant system based on generative and cached digital humans, which is displayed in the elevator lobby, and uses the image of a digital human to interact with elevator users in a visualized manner, providing users with elevator calling and elevator information query functions, and bringing a visual interactive elevator calling experience, thereby solving the usability and economy of the digital human elevator assistant.

[0005] The present invention provides an elevator assistant system based on generative and cached digital human, which is suitable for building vertical elevator occasions, including a camera, an elevator assistant screen, a voice module ASR, an elevator assistant subsystem and an elevator controller, wherein the elevator assistant subsystem is respectively connected to the camera, the elevator assistant screen, the voice module ASR and the elevator controller;

[0006] When a user enters the elevator lobby, the camera is used to monitor the user's movements, the elevator assistant screen is used to prompt the user to call the elevator and obtain elevator information through voice commands, the voice module ASR recognizes the user's voice and converts it into text, the elevator assistant subsystem sends an elevator calling command to the elevator controller, and the elevator controller returns the elevator calling result so that the elevator assistant screen displays the corresponding digital human image video. When the elevator arrives, the elevator assistant subsystem recognizes the user's movements and broadcasts a voice prompt of the elevator's arrival.

[0007] Furthermore, it also includes a server, a digital human platform and an elevator maintenance subsystem, wherein the server and the digital human platform are connected to the elevator assistant subsystem via the Internet, and the elevator maintenance subsystem is connected to the elevator controller;

[0008] The voice module ASR recognizes the user's elevator maintenance voice instructions and converts them into text. The elevator assistant subsystem recognizes the user's voice semantics and requests the elevator maintenance information from the server, and queries the digital human video database through the maintenance information returned by the server and pulls and plays the maintenance information content video, or requests the server to generate a new video stream through the maintenance information returned by the server. The server requests the digital human platform to generate a video and sends the video stream to the elevator assistant subsystem. The elevator assistant subsystem displays the video stream and explains the elevator maintenance plan to the user in the form of a digital human.

[0009] Furthermore, the camera is a network digital camera installed in the elevator lobby, and the camera is used to monitor user movements, including a person stay detection algorithm and a person mouth shape change detection algorithm;

[0010] The elevator assistant screen is used to display the digital human image and provide information to the user. It is installed in the elevator lobby, runs a browser, and establishes a network connection with the elevator assistant subsystem through a web page;

[0011] The voice module ASR is used to convert the user's voice into text information;

[0012] The digital human platform is used to generate a digital human video and select a digital human image based on the submitted information;

[0013] The elevator maintenance subsystem is used to save the elevator maintenance plan and the maintenance plan execution status.

[0014] Furthermore, the elevator assistant subsystem includes an information processing module, a voice recognition module, an elevator calling module, a first digital human video storage module, a first digital human video database, a digital human push module, a server digital human video request module, an elevator and other information request module, and a first location identification module;

[0015] The information processing module is used to process input information and push the information input by the user to the elevator assistant screen;

[0016] The speech recognition module is used to understand the input information speech and analyze the user input information intention through the large model tool;

[0017] The elevator calling module is used to communicate with the elevator controller, send an elevator calling instruction to the elevator controller, and receive an elevator calling result returned by the elevator controller;

[0018] The first digital human video saving module is used to save the digital human video stream sent by the server;

[0019] The first digital human video database is used to retrieve the corresponding digital human video according to the video text information content, and store the corresponding relationship between the video text information content and the video position;

[0020] The digital human push module is used to push the digital human video stream to the elevator assistant screen;

[0021] The server digital human video request module is used to search the local digital human video database, and if the required digital human video does not exist, request the digital human video from the server, and after receiving the video returned by the server, save the video and corresponding information to the first digital human video database, and save the video to the local file system;

[0022] The elevator and other information request module is used to identify the user's intention to obtain information, determine whether the information is local information, if it is the current local elevator information, obtain the information from the elevator controller and generate a corresponding video, if it is remote server information, request the server for information and generate a corresponding video, and after obtaining the information video, push it to the elevator assistant screen for display and playback;

[0023] The first location identification module is used to store the elevator assistant number UUID A , carry the elevator assistant number and the current floor when communicating A information.

[0024] Furthermore, the elevator controller includes an elevator status acquisition module, an elevator command control module, a network communication module, and a second position identification module;

[0025] The elevator status acquisition module is used to collect data on the current floor, running speed, running direction, load, air quality, light brightness, and elevator energy consumption of the elevator. The data is used to display the elevator status of the elevator assistant subsystem;

[0026] The elevator command control module is used to send the elevator call command to the elevator and return the elevator call result to the elevator assistant subsystem;

[0027] The network communication module is used for the elevator status acquisition module to send the current elevator status to the elevator assistant subsystem through the network;

[0028] The second location identification module is used to store the elevator UUID Elv , carry UUID when combining data Elv .

[0029] Furthermore, the server includes a second digital human video storage module, a second digital human video database, a digital human platform digital human video request module, and an information acquisition module;

[0030] The second digital human video saving module is used to save the digital human video returned by the digital human platform;

[0031] The second digital human video database is used to retrieve the corresponding digital human video according to the video text information content, and store the corresponding relationship between the video text information content and the video position;

[0032] The digital human platform digital human video request module is used to search the second digital human video database. If the required digital human video does not exist, it is necessary to submit generation information to the digital human platform and request the generation of the digital human video. After receiving the video stream returned by the digital human platform, the video and corresponding information are saved to the second digital human video database, the video is saved to the server file system, and the video is pushed to the requested elevator assistant subsystem;

[0033] The information acquisition module is used to receive requests sent by the elevator assistant subsystem, including: if the weather needs to be obtained, a request is sent to the weather API and the weather request result is returned; to obtain elevator maintenance information, the elevator UUID is used to obtain the elevator maintenance information. Elv Number, request the elevator maintenance plan from the elevator maintenance subsystem, and return the elevator maintenance plan information.

[0034] Furthermore, a stop mark frame is set in front of the elevator assistant screen, and the camera monitors the user's movement. When the user stays in the stop mark frame for more than a preset detection time Time Pre When the elevator assistant subsystem stops, the camera sends a signal to the elevator assistant subsystem, and the elevator assistant subsystem starts playing the welcome message and video;

[0035] The voice module ASR collects user input voice information, converts the voice information into text, and outputs it to the elevator assistant subsystem. The elevator assistant subsystem integrates a large model. The large model presets the current voice recognition content prompt words and recognizes the voice information related to the current scene, eliminating invalid input.

[0036] Furthermore, the elevator assistant subsystem presets an intention management module, presets intention content and execution plan

[0037] [Intention1,Intention2…Intention n ], the intent contains multiple execution units, then

[0038] Intention1:[UnitKeyword,UnitRequestElv,UnitDigtalHumanShow,UnitEnd], where there can be multiple intent execution units, and the intent execution content is arranged through the intent execution units. The UnitKeyword unit contains the recognized keywords, and the intent to be executed is identified through the keywords;

[0039] Furthermore, the elevator assistant subsystem has stored the elevator assistant number UUID A 、Current floor Floor A When executing the elevator call command, the elevator call command packet [UUID A , Floor A ,FloorDest] is sent to the elevator controller, and the elevator controller A ,FloorDest sends a call to the elevator. After getting the result of the elevator dispatch, it uses UUID A Find the elevator assistant that needs to be replied, and send the dispatch result to the elevator assistant subsystem. The elevator assistant subsystem confirms whether the elevator is dispatched to the left or right elevator or displays the elevator result by number based on the preset elevator correspondence.

[0040] Furthermore, the elevator assistant subsystem and the server both include a digital human video database, in which the relationship between video and text content [{VideoUrl1, Text1}, {VideoUrl2, Text2}…{VideoUrln, Textn}] is stored. When video display is required, the database is searched through Text. When the Text is found to exist, the video address is obtained using VideoUrl. When the elevator assistant subsystem searches, the VideoUrl is pushed to the elevator assistant screen. When the elevator assistant subsystem cannot retrieve the corresponding relationship, the server is searched. When the server has the video, the server returns the video corresponding to the VideoUrl to the elevator assistant subsystem. The elevator assistant subsystem saves {VideoUrlnew, Textnew} to the database and pushes VideoUrl to the elevator assistant screen.

[0041] When the server cannot retrieve the corresponding relationship of Text, the server requests the digital human platform to generate the video corresponding to the Text, and requires the digital human to mark the start and end of the video stream with [VideoStart, VideoEnd]. When the server receives the stream starting with VideoStart, it starts the video stream saving process. When it receives the VideoEnd mark, it ends the video stream interception function, saves the saved VideoUrl and Text to the database in the data format of {VideoUrl, Text}, and returns the video corresponding to VideoUrl to the elevator assistant subsystem. The elevator assistant subsystem saves {VideoUrlnew, Textnew} to the database and pushes VideoUrl to the elevator assistant screen.

[0042] The beneficial effects of the present invention are:

[0043] The present invention is applied in vertical elevator scenarios of buildings, and includes a camera, an elevator assistant screen, a voice module ASR, an elevator assistant subsystem, an elevator controller, a server, a digital human platform and an elevator maintenance subsystem. When a user enters the elevator lobby, the camera monitors the user's movements, the elevator assistant screen recognizes the user's voice, and sends an elevator call instruction to the elevator controller, so that the elevator assistant screen displays a corresponding digital human image video to prompt the user to take the elevator; the voice module ASR recognizes the user's elevator maintenance voice instruction, the elevator assistant subsystem recognizes the user's voice semantics and requests elevator maintenance information from the server, the elevator assistant subsystem displays the maintenance information content video, and explains the elevator maintenance plan to the user in the form of a digital human image; the digital human image is used to interact with the elevator user in a visualized manner, providing the user with elevator call and elevator information query functions, and bringing a visual interactive elevator call experience, thereby solving the usability and economy of the digital human elevator assistant. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The figure is a schematic diagram of an elevator lobby according to an embodiment of the present invention.

[0045] Figure 2 It is a structural schematic diagram of the elevator assistant system based on the generative and cached digital human of the present invention.

[0046] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0048] The present invention provides a digital human elevator assistant system based on generative and caching, which is applied in vertical elevator situations in buildings to bring users a visual interactive elevator calling experience and solve the usability and economy of the digital human elevator assistant.

[0049] like Figure 1-2 As shown, the present invention provides an elevator assistant system based on generative and cached digital humans, including a camera, an elevator assistant screen, a voice module ASR, an elevator assistant subsystem and an elevator controller, wherein the elevator assistant subsystem is respectively connected to the camera, the elevator assistant screen, the voice module ASR and the elevator controller.

[0050] When a user enters the elevator lobby, the camera is used to monitor the user's movements, the elevator assistant screen is used to prompt the user to call the elevator and obtain elevator information through voice commands, the voice module ASR recognizes the user's voice and converts it into text, the elevator assistant subsystem sends an elevator calling command to the elevator controller, and the elevator controller returns the elevator calling result so that the elevator assistant screen displays the corresponding digital human image video. When the elevator arrives, the elevator assistant subsystem recognizes the user's movements and broadcasts a voice prompt of the elevator's arrival.

[0051] In one embodiment, the present invention further includes a server, a digital human platform and an elevator maintenance subsystem, wherein the server and the digital human platform are connected to the elevator assistant subsystem via the Internet, and the elevator maintenance subsystem is connected to the elevator controller;

[0052] The voice module ASR recognizes the user's elevator maintenance voice instructions and converts them into text. The elevator assistant subsystem recognizes the user's voice semantics and requests the elevator maintenance information from the server, and queries the digital human video database through the maintenance information returned by the server and pulls and plays the maintenance information content video, or requests the server to generate a new video stream through the maintenance information returned by the server. The server requests the digital human platform to generate a video and sends the video stream to the elevator assistant subsystem. The elevator assistant subsystem displays the video stream and explains the elevator maintenance plan to the user in the form of a digital human.

[0053] The workflow of the system of the present invention is:

[0054] S1. The user enters the elevator lobby. The elevator assistant subsystem in the elevator lobby monitors the user's movements through a camera. When the user arrives in front of the elevator assistant screen, the elevator assistant digital image actively broadcasts a welcome message, prompting the user to call the elevator and obtain elevator information by voice, such as: "Welcome to the elevator. You can use voice to call the elevator and obtain elevator information."

[0055] S2. The user says "I want to go to the third floor" by voice. The elevator assistant subsystem recognizes the user's voice and displays the user's input content on the screen. The elevator assistant recognizes the semantics of the user's voice, understands that the user intends to call the elevator, sends the elevator call command to the elevator control system, and the elevator control system returns the elevator call result. The digital human image displays the digital human image video according to the elevator call result. If you take the left elevator, you use the left hand to guide. If you take the right elevator, you use the right hand to guide. If there are multiple elevators, you use the right hand to guide, and the elevator number is displayed.

[0056] S3. The user uses voice to ask "When will the elevator be maintained?" The elevator assistant subsystem recognizes the user's voice and displays the user's input content on the screen. The elevator assistant recognizes the user's voice semantics and understands the user's intention to obtain maintenance information. It requests the elevator maintenance information from the server. After the server returns the maintenance information, the elevator assistant subsystem queries the digital human image video library. If the maintenance information content video content already exists, it directly pulls the video content and broadcasts it. If the video does not exist, the elevator assistant subsystem requests the server to generate a new video stream. The server requests the digital human platform to generate a video. The digital human platform generates a video stream. The server obtains the streaming video stream and sends the video stream to the elevator assistant subsystem. The video stream is saved to the digital human image video library during the video stream download process. The elevator assistant subsystem displays the video stream and explains the elevator maintenance plan to the user using the digital human image.

[0057] S4. After the user arrives at the elevator, the elevator assistant subsystem identifies the user's movements and announces "Your elevator has arrived, please go and take it."

[0058] In one embodiment, the system of the present invention is specifically configured as follows:

[0059] 1. Camera

[0060] The camera is a network digital camera installed in the elevator lobby. The camera is used to monitor user movements, such as whether the user stays in front of the elevator assistant screen, whether the user is talking, etc. The camera contains algorithms such as person stay detection and person mouth shape change detection.

[0061] 2. Elevator Assistant Screen

[0062] The elevator assistant screen is used to display a digital human image and provide information to users. It is installed in the elevator lobby, runs a browser, and establishes a network connection with the elevator assistant subsystem through a web page.

[0063] 3. Voice module ASR

[0064] The voice module ASR is used to convert the user's voice into text information.

[0065] 4. Elevator Assistant Subsystem

[0066] The elevator assistant subsystem includes an information processing module, a voice recognition module, an elevator calling module, a first digital human video storage module, a first digital human video database, a digital human push module, a server digital human video request module, an elevator and other information request module, and a first location identification module.

[0067] 1) The information processing module is used to process input information and push the information input by the user to the elevator assistant screen;

[0068] 2) The speech recognition module is used to understand the input information voice and analyze the user's input information intention through a large model tool; for example, the content including "I want to go upstairs", "I want to go to the third floor", "I call the third floor" is the elevator calling intention; the content including "I want to know the weather", "I want to know when the maintenance will be done", "I want to obtain the elevator operation status" is the information acquisition intention.

[0069] 3) The elevator calling module is used to communicate with the elevator controller, send an elevator calling instruction to the elevator controller, and receive the elevator calling result returned by the elevator controller.

[0070] 4) The first digital human video saving module is used to save the digital human video stream sent by the server.

[0071] 5) The first digital human video database is used to retrieve the corresponding digital human video through the video text information content, and save the corresponding relationship between the video text information content and the video position.

[0072] 6) The digital human push module is used to push the digital human video stream to the elevator assistant screen.

[0073] 7) The server digital human video request module is used to search the local digital human video database. If the required digital human video does not exist, it is necessary to request the digital human video from the server. After receiving the video returned by the server, the video and corresponding information are saved to the first digital human video database, and the video is saved to the local file system.

[0074] 8) The elevator and other information request module is used to identify the user's intention to obtain information and determine whether the information is local information. If it is the current local elevator information, it obtains the information from the elevator controller and generates a corresponding video. If it is remote server information, it requests the server for information and generates a corresponding video. After obtaining the information video, it is pushed to the elevator assistant screen for display and playback.

[0075] 9) The first location identification module is used to store the elevator assistant number UUID A , carry the elevator assistant number and the current floor when communicating A information.

[0076] 5. Elevator controller

[0077] The elevator controller includes an elevator status acquisition module, an elevator command control module, a network communication module, and a second position identification module.

[0078] 1) The elevator status acquisition module is used to collect data on the current floor, running speed, running direction, load, air quality, light brightness, and elevator energy consumption of the elevator. The data is used to display the elevator status of the elevator assistant subsystem.

[0079] 2) The elevator command control module is used to send the elevator call command to the elevator and return the elevator call result to the elevator assistant subsystem.

[0080] 3) The network communication module is used by the elevator status acquisition module to send the current elevator status to the elevator assistant subsystem through the network.

[0081] 4) The second location identification module is used to store the elevator UUID Elv , carry UUID when combining data Elv .

[0082] 6. Server

[0083] The server includes a second digital human video storage module, a second digital human video database, a digital human platform digital human video request module, and an information acquisition module.

[0084] 1) The second digital human video saving module is used to save the digital human video returned by the digital human platform.

[0085] 2) The second digital human video database is used to retrieve the corresponding digital human video through the video text information content, and save the corresponding relationship between the video text information content and the video position.

[0086] 3) The digital human video request module of the digital human platform is used to search the second digital human video database. If the required digital human video does not exist, it is necessary to submit generation information to the digital human platform and request to generate a digital human video. After receiving the video stream returned by the digital human platform, the video and corresponding information are saved to the second digital human video database, the video is saved to the server file system, and the video is pushed to the requested elevator assistant subsystem.

[0087] 4) The information acquisition module is used to receive requests sent by the elevator assistant subsystem. If you need to obtain the weather, send a request to the weather API and return the weather request result. If you need to obtain elevator maintenance information, use the elevator UUID Elv Number, request the elevator maintenance plan from the elevator maintenance subsystem, and return the elevator maintenance plan information.

[0088] 7. Digital Human Platform

[0089] The digital human platform is a digital human video generation platform provided by a third-party commercial company, which can generate digital human videos based on the submitted information and can select the digital human image.

[0090] 8. Elevator maintenance subsystem

[0091] The elevator maintenance subsystem is used to save the elevator maintenance plan and the maintenance plan execution status.

[0092] In one embodiment, the detection of the user's intention to call the elevator is implemented as follows:

[0093] like Figure 1 As shown in the figure, a stop mark box is added to the area in front of the elevator assistant screen, and camera 1 is used to monitor the user's movement. When the user stays in front of the stop mark box for more than the preset detection time Time Pre When the elevator assistant stops, the camera sends a signal to the elevator assistant subsystem, and the elevator assistant subsystem starts playing the welcome message and video.

[0094] In one embodiment, the user voice recognition method is:

[0095] A speech recognition ASR module is installed in front of the elevator assistant screen. The speech recognition ASR module collects the user's input voice information. The ASR module converts the voice information into text and outputs it to the elevator assistant subsystem.

[0096] The elevator assistant subsystem integrates a large model, and the large model becomes a preset prompt for the current voice recognition content, such as "You are a professional voice recognition personnel. The current scenario is elevator calling and elevator information acquisition. Please filter out the following information that meets the current conditions." The large model recognizes voice information related to the current scenario and eliminates invalid input.

[0097] In one embodiment, user intent is identified and performed:

[0098] In the elevator assistant subsystem, preset the intention management system, preset the intention content and execution plan

[0099] [Intention1,Intention2…Intention n ], the intent contains multiple execution units, then

[0100] Intention1: [UnitKeyword, UnitRequestElv, UnitDigtalHumanShow, UnitEnd], where there can be multiple intent execution units, and the intent execution content is arranged through the intent execution units. The UnitKeyword unit contains recognized keywords, such as [call elevator, go upstairs, go downstairs, go to {n} floors], etc., and the keywords are used to identify which intent should be executed.

[0101] For example, for the intent recognition of "I want to go to the third floor", the UnitKeyword is "go to {third} floor", which matches Intention1. After the UnitKeyword is executed, the floor to be reached is extracted as 3. The request command is sent to the elevator controller through UnitRequestElv, and the elevator call result is received. The elevator call result video is displayed through UnitDigtalHumanShow, and the intent process is completed through UnitEnd.

[0102] In one embodiment, the elevator calling process:

[0103] The elevator assistant subsystem has stored the elevator assistant number UUID A 、Current floor Floor A When the elevator call command is executed, the elevator call command packet [UUID A , Floor A ,Floor Dest ] is sent to the elevator controller, and the elevator controller A ,Floor Dest Send an elevator call command to the elevator, and after obtaining the elevator dispatch result, use UUID A Find the elevator assistant that needs to be replied, and send the dispatch result to the elevator assistant subsystem. The elevator assistant subsystem confirms whether the elevator is dispatched to the left or right elevator or displays the elevator result by number based on the preset elevator correspondence.

[0104] In one embodiment, the video stream obtains:

[0105] There is a digital human video information database in both the elevator assistant subsystem and the server, which stores the relationship between video and text content [{VideoUrl1, Text1}, {VideoUrl2, Text2}…{VideoUrl n ,Text n}], when video display is required, the database is searched through Text. When the Text is found to exist, the video address is obtained using VideoUrl. When the elevator assistant subsystem searches, the elevator assistant pushes VideoUrl to the elevator assistant screen. If the elevator assistant subsystem cannot retrieve the corresponding relationship, the server is searched. If the server has the video, the server returns the video corresponding to VideoUrl to the elevator assistant subsystem. The elevator assistant subsystem {VideoUrl new ,Text new}Save it to the database and push the VideoUrl to the elevator assistant screen.

[0106] When the server cannot retrieve the corresponding relationship of the text, the server requests the digital human platform to generate the video corresponding to the text and requires the digital human to generate the video corresponding to the text in [Video Start , Video End ] Mark the start and end of the video stream. When the server receives a video Start When the video stream is started, the video stream saving process is started. End When the mark is set, the video stream interception function is terminated, the saved VideoUrl and Text are saved to the database in the data format of {VideoUrl, Text}, and the video corresponding to the VideoUrl is returned to the elevator assistant subsystem. The elevator assistant subsystem returns {VideoUrl new ,Text new}Save it to the database and push the VideoUrl to the elevator assistant screen.

[0107] In one embodiment, the video plays:

[0108] The elevator assistant screen runs a browser, establishes a webSocket connection with the elevator assistant subsystem through the web page, and plays the video with the Video component of the web page. When the elevator assistant subsystem needs to update the currently playing video, it sends VideoUrl to the web page through webSocket, and the web page creates a new Video component and points the src to the VideoUrl sent in the webSocket, and switches the current Video component to the newly created Video component to complete the video playback.

[0109] If there is no new video at present, the elevator assistant subsystem will ask the web page to play the waiting transition video.

[0110] In one embodiment, the screen text information is updated:

[0111] Similar to VideoUrl update, screen text information is sent through webScoket, and the web page updates the current display information according to the instructions of webScoket.

[0112] The present invention is applied in vertical elevator scenarios of buildings, and includes a camera, an elevator assistant screen, a voice module ASR, an elevator assistant subsystem, an elevator controller, a server, a digital human platform and an elevator maintenance subsystem. When a user enters the elevator lobby, the camera monitors the user's movements, the elevator assistant screen recognizes the user's voice, and sends an elevator call instruction to the elevator controller, so that the elevator assistant screen displays a corresponding digital human image video to prompt the user to take the elevator; the voice module ASR recognizes the user's elevator maintenance voice instruction, the elevator assistant subsystem recognizes the user's voice semantics and requests elevator maintenance information from the server, the elevator assistant subsystem displays the maintenance information content video, and explains the elevator maintenance plan to the user in the form of a digital human image; the digital human image is used to interact with the elevator user in a visualized manner, providing the user with elevator call and elevator information query functions, and bringing a visual interactive elevator call experience, thereby solving the usability and economy of the digital human elevator assistant.

[0113] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0114] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A digital human elevator assistant system based on generative and cached methods, applied to vertical elevators in buildings, characterized in that: It includes a camera, an elevator assistant screen, a voice module ASR, an elevator assistant subsystem and an elevator controller, wherein the elevator assistant subsystem is respectively connected to the camera, the elevator assistant screen, the voice module ASR and the elevator controller; When a user enters the elevator lobby, the camera is used to monitor the user's movements, the elevator assistant screen is used to prompt the user to call the elevator and obtain elevator information through voice commands, the voice module ASR recognizes the user's voice and converts it into text, the elevator assistant subsystem sends an elevator calling command to the elevator controller, and the elevator controller returns the elevator calling result so that the elevator assistant screen displays the corresponding digital human image video. When the elevator arrives, the elevator assistant subsystem recognizes the user's movements and broadcasts a voice prompt of the elevator's arrival.

2. The elevator assistant system based on generative and cached digital human according to claim 1 is characterized in that: It also includes a server, a digital human platform and an elevator maintenance subsystem, wherein the server and the digital human platform are connected to the elevator assistant subsystem via the Internet, and the elevator maintenance subsystem is connected to the elevator controller; The voice module ASR recognizes the user's elevator maintenance voice instructions and converts them into text. The elevator assistant subsystem recognizes the user's voice semantics and requests the elevator maintenance information from the server, and queries the digital human video database through the maintenance information returned by the server and pulls and plays the maintenance information content video, or requests the server to generate a new video stream through the maintenance information returned by the server. The server requests the digital human platform to generate a video and sends the video stream to the elevator assistant subsystem. The elevator assistant subsystem displays the video stream and explains the elevator maintenance plan to the user in the form of a digital human.

3. The elevator assistant system based on generative and cached digital human according to claim 2 is characterized in that: The camera is a network digital camera installed in the elevator lobby. The camera is used to monitor user movements, including a person stay detection algorithm and a person mouth shape change detection algorithm; The elevator assistant screen is used to display the digital human image and provide information to the user. It is installed in the elevator lobby, runs a browser, and establishes a network connection with the elevator assistant subsystem through a web page; The voice module ASR is used to convert the user's voice into text information; The digital human platform is used to generate a digital human video and select a digital human image based on the submitted information; The elevator maintenance subsystem is used to save the elevator maintenance plan and the maintenance plan execution status.

4. The elevator assistant system based on generative and cached digital human according to claim 2 is characterized in that: The elevator assistant subsystem includes an information processing module, a voice recognition module, an elevator calling module, a first digital human video storage module, a first digital human video database, a digital human push module, a server digital human video request module, an elevator and other information request module, and a first location identification module; The information processing module is used to process input information and push the information input by the user to the elevator assistant screen; The speech recognition module is used to understand the input information speech and analyze the user input information intention through the large model tool; The elevator calling module is used to communicate with the elevator controller, send an elevator calling instruction to the elevator controller, and receive an elevator calling result returned by the elevator controller; The first digital human video saving module is used to save the digital human video stream sent by the server; The first digital human video database is used to retrieve the corresponding digital human video according to the video text information content, and store the corresponding relationship between the video text information content and the video position; The digital human push module is used to push the digital human video stream to the elevator assistant screen; The server digital human video request module is used to search the local digital human video database, and if the required digital human video does not exist, request the digital human video from the server, and after receiving the video returned by the server, save the video and corresponding information to the first digital human video database, and save the video to the local file system; The elevator and other information request module is used to identify the user's intention to obtain information, determine whether the information is local information, if it is the current local elevator information, obtain the information from the elevator controller and generate a corresponding video, if it is remote server information, request the server for information and generate a corresponding video, and after obtaining the information video, push it to the elevator assistant screen for display and playback; The first location identification module is used to store the elevator assistant number UUID A , carry the elevator assistant number and the current floor when communicating A information.

5. The elevator assistant system based on generative and cached digital human according to claim 2 is characterized in that: The elevator controller includes an elevator status acquisition module, an elevator command control module, a network communication module, and a second position identification module; The elevator status acquisition module is used to collect data on the current floor, running speed, running direction, load, air quality, light brightness, and elevator energy consumption of the elevator. The data is used to display the elevator status of the elevator assistant subsystem; The elevator command control module is used to send the elevator call command to the elevator and return the elevator call result to the elevator assistant subsystem; The network communication module is used for the elevator status acquisition module to send the current elevator status to the elevator assistant subsystem through the network; The second location identification module is used to store the elevator UUID Elv , carry UUID when combining data Elv .

6. The elevator assistant system based on generative and cached digital human according to claim 2 is characterized in that: The server includes a second digital human video storage module, a second digital human video database, a digital human platform digital human video request module, and an information acquisition module; The second digital human video saving module is used to save the digital human video returned by the digital human platform; The second digital human video database is used to retrieve the corresponding digital human video according to the video text information content, and store the corresponding relationship between the video text information content and the video position; The digital human platform digital human video request module is used to search the second digital human video database. If the required digital human video does not exist, it is necessary to submit generation information to the digital human platform and request the generation of the digital human video. After receiving the video stream returned by the digital human platform, the video and corresponding information are saved to the second digital human video database, the video is saved to the server file system, and the video is pushed to the requested elevator assistant subsystem; The information acquisition module is used to receive requests sent by the elevator assistant subsystem, including: if the weather needs to be obtained, a request is sent to the weather API and the weather request result is returned; Get elevator maintenance information through elevator UUID Elv Number, request the elevator maintenance plan from the elevator maintenance subsystem, and return the elevator maintenance plan information.

7. The elevator assistant system based on generative and cached digital human according to claim 2 is characterized in that: A stay mark frame is set in front of the elevator assistant screen, and the camera monitors the user's movement. When the user stays in the stay mark frame for more than a preset detection time Time Pre When the elevator assistant subsystem stops, the camera sends a signal to the elevator assistant subsystem, and the elevator assistant subsystem starts playing the welcome message and video; The voice module ASR collects user input voice information, converts the voice information into text, and outputs it to the elevator assistant subsystem. The elevator assistant subsystem integrates a large model. The large model presets the current voice recognition content prompt words and recognizes the voice information related to the current scene, eliminating invalid input.

8. The elevator assistant system based on generative and cached digital human according to claim 7 is characterized in that: The elevator assistant subsystem presets an intention management module, presets intention content and execution plan [Intention1, Intention2…Intention n ], the intent contains multiple execution units, then Intention1:[UnitKeyword,UnitRequestElv,UnitDigtalHumanShow,UnitEnd], where there can be multiple intention execution units, and the intention execution content is arranged through the intention execution units. The UnitKeyword unit contains recognized keywords, and the intention to be executed is identified through the keywords.

9. The elevator assistant system based on generative and cached digital human according to claim 8 is characterized in that: The elevator assistant subsystem has stored the elevator assistant number UUID A 、Current floor Floor A When executing the elevator call command, the elevator call command packet [UUID A , Floor A ,FloorDest] is sent to the elevator controller, and the elevator controller A ,FloorDest sends a call to the elevator. After getting the result of the elevator dispatch, it uses UUID A Find the elevator assistant that needs to be replied, and send the dispatch result to the elevator assistant subsystem. The elevator assistant subsystem confirms whether the elevator is dispatched to the left or right elevator or displays the elevator result by number based on the preset elevator correspondence.

10. The elevator assistant system based on generative and cached digital human according to claim 2 is characterized in that: The elevator assistant subsystem and the server both include a digital human video database, in which the relationship between video and text content [{VideoUrl1, Text1}, {VideoUrl2, Text2}…{VideoUrln, Textn}] is stored. When video display is required, the database is searched through Text. When the Text is found to exist, the video address is obtained using VideoUrl. When the elevator assistant subsystem searches, VideoUrl is pushed to the elevator assistant screen. When the elevator assistant subsystem cannot retrieve the corresponding relationship, the server is searched. When the server exists the video, the server returns the video corresponding to VideoUrl to the elevator assistant subsystem. The elevator assistant subsystem saves {VideoUrlnew, Textnew} to the database and pushes VideoUrl to the elevator assistant screen. When the server cannot retrieve the corresponding relationship of Text, the server requests the digital human platform to generate the video corresponding to the Text, and requires the digital human to mark the start and end of the video stream with [VideoStart, VideoEnd]. When the server receives the stream starting with VideoStart, it starts the video stream saving process. When it receives the VideoEnd mark, it ends the video stream interception function, saves the saved VideoUrl and Text to the database in the data format of {VideoUrl, Text}, and returns the video corresponding to VideoUrl to the elevator assistant subsystem. The elevator assistant subsystem saves {VideoUrlnew, Textnew} to the database and pushes VideoUrl to the elevator assistant screen.

Images