Robot visitor welcome system and method based on elevator data

Abstract

The invention relates to a robot visitor welcome system and method based on elevator data, and the system comprises a video monitoring system which is used for obtaining a video stream in an elevator collected by a camera in real time, and carrying out the analysis and processing of the video stream to obtain the face data of elevator passengers; the visitor system is used for acquiring face data of the elevator taking personnel through the video monitoring system and identifying identity information of the elevator taking personnel according to the face data; and the welcome robot is used for obtaining identity information of the elevator taking personnel when the target floor of the elevator taking personnel is judged to be the floor where the welcome robot is located, and a corresponding welcome mode is adopted for the elevator taking personnel according to the obtained identity information. According to the guest greeting robot, the guest greeting robot greets the guest by combining the video monitoring system and the guest system, the guest greeting robot is not limited by hardware conditions of the guest greeting robot, and the guest greeting experience of the guest can be improved by adopting the corresponding guest greeting mode for the identity information of the guest.

Description

Technical Field

[0001] The present application relates to the technical field of welcoming robots, and in particular to a welcoming robot visitor welcoming system and method based on elevator data. Background Art

[0002] The reception service robots currently used in building scenarios basically adopt passive and universal reception solutions.

[0003] Passive reception mainly means that visitors need to actively come to the reception robot and stay a certain distance away before the reception robot can recognize the biological characteristics of people based on its own hardware equipment (RGB camera, infrared sensor device) and start the reception service. This method is limited by the hardware conditions of the reception robot and has certain requirements for the distance and angle of personnel recognition. It is very easy to have problems such as missed recognition, misrecognition, and long recognition time, which will cause a very poor reception experience for visitors.

[0004] The generalized reception is mainly reflected in the fact that the reception robot cannot identify the identity and intention of the visitor, so it can only use the process plan of the generalized reception service based on the identity of the stranger. Ultimately, the visitor and the reception robot can only make corresponding service actions after multiple rounds of interaction through the interface + voice. This service method and service efficiency of the reception robot have greatly reduced the visitor's visiting experience and the functional value of the reception robot's own reception service. Summary of the invention

[0005] In view of the above problems, the present application provides a robot visitor welcoming system and method based on elevator data to solve the problem that the welcoming robots in existing building scenarios use a passive welcoming method, which is limited by the hardware conditions of the welcoming robots and the universal welcoming method has low service efficiency, affecting the visitor's welcoming experience.

[0006] To achieve the above purpose, the inventor provides a robot visitor welcoming system based on elevator data, comprising:

[0007] A video monitoring system, which is connected to a camera in the elevator and is used to obtain the video stream in the elevator collected by the camera in real time, and to perform monitoring, display and analysis to obtain facial data of passengers;

[0008] A visitor system, the visitor system is connected to the video surveillance system, and the visitor system is used to obtain facial data of elevator passengers through the video surveillance system, and identify the identity information of the elevator passengers based on the facial data;

[0009] A welcoming robot is connected to the elevator and the visitor system. The welcoming robot is used to obtain the elevator status data in real time. When there are passengers in the elevator, when it is determined that the passenger's target floor is the floor where the welcoming robot is located, the visitor system is notified to obtain the passenger's identity information. When the passenger reaches the current floor, the corresponding welcoming mode is adopted for the passenger according to the obtained identity information.

[0010] In some embodiments, the welcoming robot is also used to:

[0011] When the current elevator passenger is identified as an employee through identity information, when the elevator arrives at the current floor and the elevator door is open, the time to send the door opening command to the company's access control is calculated based on the distance from the elevator door to the employee's corresponding company's access control. When the time is reached, the door opening command is sent to the company's access control.

[0012] In some embodiments, the welcoming robot is also used to:

[0013] When the current elevator passenger is identified as a stranger through identity information, when the elevator arrives at the current floor and the elevator door is open, the elevator passenger is asked for the required service and the universal welcoming process is started.

[0014] In some embodiments, the welcoming robot is also used to:

[0015] When the current elevator rider is identified as a scheduled visitor through identity information, the elevator will broadcast a welcome message based on the visitor information when it arrives at the current floor and the elevator door is open, obtain the location information of the person being visited from the visitor system, and ask the scheduled visitor whether he or she needs guidance service. When it is confirmed that the scheduled visitor needs guidance service, the scheduled visitor will be guided based on the location information of the person being visited.

[0016] In some embodiments, the guest system is further configured to:

[0017] When the elevator passenger is identified as a scheduled visitor through identity information, a notification message is sent to the person being visited.

[0018] In some embodiments, the welcoming robot is also used to:

[0019] The image data collected by the camera in the elevator is processed by a convolutional neural network to obtain the first feature;

[0020] For the visitor information provided by the visitor system, the second feature is extracted through natural language processing and image processing technology;

[0021] The first feature and the second feature are combined to obtain a comprehensive feature;

[0022] Based on the fused features, a deep learning model is used to generate a welcoming strategy.

[0023] In some embodiments, the welcoming robot is also used to provide welcoming service to passengers with the highest service priority when identity information of multiple passengers in the elevator is detected. The service priority is: the service priority of scheduled visitors is higher than the service priority of strangers, and the service priority of strangers is higher than the service priority of employees.

[0024] Another technical solution is also provided, a robot visitor welcoming method based on elevator data, which is applied to the robot visitor welcoming system based on elevator data, and includes the following steps:

[0025] The welcoming robot obtains elevator status data in real time;

[0026] When the welcoming robot determines through the elevator status data that a passenger is using the elevator, and the passenger's target floor is the floor where the welcoming robot is located, the welcoming robot notifies the visitor system to periodically obtain valid facial data detected in the elevator from the video surveillance system;

[0027] The visitor system identifies the face data identified by the video surveillance system, obtains the identity information of the elevator passengers, and synchronizes it to the welcoming robot;

[0028] When the elevator reaches the floor where the welcoming robot is located and starts, the welcoming robot adopts the corresponding welcoming mode according to the identity information of the elevator occupants.

[0029] In some embodiments, the “when the elevator reaches the floor where the welcoming robot is located and starts, the welcoming robot adopts the corresponding welcoming mode according to the identity information of the elevator passenger” comprises the following steps:

[0030] When the current elevator passenger is identified as an employee through identity information, when the elevator arrives at the current floor and the elevator door is open, the time to send the door opening command to the company's access control is calculated based on the distance from the elevator door to the employee's company's access control. When the time is reached, the door opening command is sent to the company's access control.

[0031] When the current passenger is identified as a stranger through identity information, the elevator will actively move to the elevator door. When the elevator arrives at the current floor and the door is open, the elevator will ask the passenger for the required service and start the universal welcoming process;

[0032] When the current elevator passenger is identified as a scheduled visitor through identity information, the system will actively move to the elevator door. When the elevator reaches the current floor and the elevator door is open, a welcome message will be broadcast based on the visitor information, and the location information of the person being visited will be obtained from the visitor system. The scheduled visitor will be asked whether a guiding service is required. When it is confirmed that the scheduled visitor needs the guiding service, the scheduled visitor will be guided based on the location information of the person being visited.

[0033] In some embodiments, the “welcoming robot adopts a corresponding welcoming mode according to the identity information of the elevator passenger” specifically includes the following steps:

[0034] The image data collected by the camera in the elevator is processed by a convolutional neural network to obtain the first feature;

[0035] For the visitor information provided by the visitor system, the second feature is extracted through natural language processing and image processing technology;

[0036] The first feature and the second feature are combined to obtain a comprehensive feature;

[0037] Based on the fused features, a deep learning model is used to generate a welcoming strategy.

[0038] Different from the prior art, the above technical solution has the ability to obtain elevator status data in real time. When there are passengers in the elevator, the welcoming robot can determine the user's target floor. When it is detected that the passenger's target floor is the floor where the welcoming robot is located, the welcoming robot will notify the visitor system to periodically obtain valid facial data detected in the elevator from the video surveillance system. After the visitor system obtains valid facial data, it identifies the passenger based on the facial data, obtains the passenger's identity information, and sends the identity information to the welcoming robot. When the passenger reaches the floor where the welcoming robot is located, the welcoming robot can use the corresponding welcoming mode to welcome the passenger based on the passenger's identity. The welcoming robot welcomes visitors by combining the video surveillance system and the visitor system, and is no longer limited by the hardware conditions of the welcoming robot itself. By using the corresponding welcoming mode for the visitor's identity information, the visitor's welcoming experience can be improved.

[0039] The above-mentioned records related to the invention content are only an overview of the technical solution of the present application. In order to enable ordinary technicians in the field to more clearly understand the technical solution of the present application, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purpose and other purposes, features and advantages of the present application easier to understand, the following is an explanation in combination with the specific implementation mode and drawings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present application and other related contents, and shall not be considered as limitations of the present application.

[0041] In the drawings of the specification:

[0042] Figure 1 A schematic diagram of a structure of a robot visitor welcoming system based on elevator data according to a specific implementation method;

[0043] Figure 2 Another structural schematic diagram of the robot visitor welcoming system based on elevator data described in the specific implementation method;

[0044] Figure 3 Another structural schematic diagram of the robot visitor welcoming system based on elevator data described in the specific implementation method;

[0045] Figure 4 A schematic flow chart of a robot visitor welcoming method based on elevator data according to a specific implementation method;

[0046] Figure 5 A schematic diagram of a process flow for generating the welcoming strategy described in the specific implementation method.

[0047] The reference numerals in the above drawings are described as follows:

[0048] 110. Video surveillance system,

[0049] 120. Visitor system;

[0050] 130. Welcoming robot. DETAILED DESCRIPTION

[0051] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0052] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0053] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0054] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.

[0055] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0056] Without further limitations, in this application, the words "include", "comprises", "has" or other similar expressions used in the sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0057] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.

[0058] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0059] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0060] See also Figure 1-2 , this embodiment provides a robot visitor welcoming system based on elevator data, including:

[0061] A video monitoring system 110, which is connected to a camera in the elevator. The video monitoring system 110 is used to obtain the video stream in the elevator collected by the camera in real time, and to perform monitoring display and analysis to obtain facial data of passengers;

[0062] A visitor system 120, wherein the visitor system 120 is connected to the video surveillance system 110, and the visitor system 120 is used to obtain facial data of elevator passengers through the video surveillance system 110, and identify the identity information of the elevator passengers based on the facial data;

[0063] The welcoming robot 130 is connected to the elevator and the visitor system 120. The welcoming robot 130 is used to obtain the elevator status data in real time. When there are passengers in the elevator, when it is determined that the passenger's target floor is the floor where the welcoming robot 130 is located, the visitor system 120 is notified to obtain the passenger's identity information. When the passenger reaches the current floor, the corresponding welcoming mode is adopted for the passenger according to the obtained identity information.

[0064] The welcoming robot 130 has the ability to obtain elevator status data in real time. When there are passengers in the elevator, the welcoming robot 130 can determine the user's target floor. When it is detected that the passenger's target floor is the floor where the welcoming robot 130 is located, the welcoming robot 130 will notify the visitor system 120 to periodically obtain valid facial data detected in the elevator from the video surveillance system 110. After the visitor system 120 obtains valid facial data, it identifies the passenger according to the facial data, obtains the passenger's identity information, and sends the identity information to the welcoming robot 130. When the passenger reaches the floor where the welcoming robot 130 is located, the welcoming robot 130 can use the corresponding welcoming mode to welcome the passenger according to the passenger's identity. The welcoming robot 130 welcomes the visitor by combining the video surveillance system 110 and the visitor system 120, and is no longer limited by the hardware conditions of the welcoming robot 130 itself. By using the corresponding welcoming mode for the visitor's identity information, the visitor's welcoming experience can be improved.

[0065] In some embodiments, the welcoming robot 130 is also used to:

[0066] When the current elevator passenger is identified as an employee through identity information, when the elevator arrives at the current floor and the elevator door is open, the time to send the door opening command to the company's access control is calculated based on the distance from the elevator door to the employee's corresponding company's access control. When the time is reached, the door opening command is sent to the company's access control.

[0067] When the welcoming robot 130 recognizes that the identity information of the elevator passenger is an employee, then when the elevator arrives at the floor where the welcoming robot 130 is located and the elevator door is in the open state, the time for sending the door opening command can be estimated based on the distance from the elevator door to the company access control. When the calculated time is reached, the door opening command is sent to the company access control corresponding to the employee, so that when the employee reaches the company access control, the service purpose of actively opening the door for the employee is achieved.

[0068] In some embodiments, the welcoming robot 130 is also used to:

[0069] When the current elevator passenger is identified as a stranger through identity information, the system will actively move to the elevator door. When the elevator arrives at the current floor and the door is open, the system will ask the elevator passenger for the required service and start the universal welcoming process.

[0070] When the identity information of the elevator passenger is recognized as a stranger, the welcoming robot 130 will actively move to the elevator door. When the elevator arrives at the current floor and the elevator door is open, it will actively ask the stranger whether he needs service and simultaneously start the universal welcoming process.

[0071] In some embodiments, the welcoming robot 130 is also used to:

[0072] When the current elevator passenger is identified as a scheduled visitor through identity information, when the elevator arrives at the current floor and the elevator door is open, a welcome message is broadcast according to the visitor information, and the location information of the person being visited is obtained from the visitor system 120. The scheduled visitor is asked whether a guidance service is required. When it is confirmed that the scheduled visitor requires a guidance service, the scheduled visitor is guided according to the location information of the person being visited.

[0073] When it is identified that the person taking the elevator is a scheduled visitor, the welcoming robot 130 will actively move to the elevator entrance. When the elevator reaches the current floor and the elevator door is open, the welcoming words will be accurately broadcast according to the visitor information. At the same time, based on the location information of the person being visited obtained from the visitor system 120, the welcoming robot will actively ask the scheduled visitor whether he needs to be provided with services such as guidance. When the scheduled visitor needs guidance services, the scheduled visitor will be guided according to the location information of the person being visited.

[0074] In some embodiments, the guest system 120 is further configured to:

[0075] When the elevator passenger is identified as a scheduled visitor through identity information, a notification message is sent to the person being visited.

[0076] At the same time, when the visitor system 120 detects that the elevator passenger has made an appointment for the visitor, a notification message is sent to the mobile phone of the person being visited. The notification message can be a text message sent to the mobile phone of the person being visited or a voice message sent by calling the mobile phone of the person being visited, so that the person being visited knows that his / her appointment visitor has arrived, so that the person being visited can calmly prepare to receive the upcoming visitor.

[0077] In some embodiments, the welcoming robot 130 is also used to:

[0078] The image data collected by the camera in the elevator is processed by a convolutional neural network to obtain the first feature;

[0079] For the visitor information provided by the visitor system 120, extract the second feature by natural language processing and image processing technology;

[0080] The first feature and the second feature are combined to obtain a comprehensive feature;

[0081] Based on the fused features, a deep learning model is used to generate a welcoming strategy.

[0082] In order to improve the recognition accuracy and service accuracy of the welcoming robot 130 and provide personalized welcoming services to visitors, the image data collected by the camera in the elevator is integrated with the information in the visitor system 120. Specifically, the welcoming strategy is generated as follows:

[0083] 1. Data processing of camera books in elevators;

[0084] For the image data X collected by the camera in the elevator t , processed by convolutional neural network (CNN), extracting the first feature F t :

[0085] F t =CNN(X t ).

[0086] Among them, the feature extraction process includes several convolutional layers, pooling layers and fully connected layers. The specific formula is as follows:

[0087] F t =FC(ReLU(Pool(Conv(X t )))). Among them, Conv represents the convolution operation, Pool represents the pooling operation, ReLU represents the activation function, and the fully connected layer (FC) is used to generate the final feature representation.

[0088] 2. Visitor system 120 information processing;

[0089] For the visitor information V provided by the visitor system 120 ttht The second feature F is extracted by using natural language processing (NLP) and image processing technology. v ;

[0090] F v =thP(V ttht )+Ctt(V ithgt ), where NLP is used to process text information in visitor information, and CNN is used to process visitor photos in visitor information.

[0091] 3. Data fusion;

[0092] The first feature F extracted by the camera in the elevator t and the second feature F extracted by the visitor system 120 v Fusion to generate comprehensive features F fusion :

[0093] F fusion =αF t +βF v ; where α and β are fusion weights, satisfying α+β=1. The fusion method can be weighted averaging, splicing, etc.

[0094] 4. Generate welcoming strategy.

[0095] Based on the comprehensive feature F fusion , using the deep learning model to generate the welcoming strategy S:

[0096] S=Dtt(F fusion ), where DNN represents a deep neural network, which generates a welcoming strategy including robot actions and language output through several hidden layers and output layers.

[0097] By integrating the camera data in the elevator and the information of the visitor system 120, the recognition accuracy and service accuracy of the welcoming robot 130 are improved. At the same time, the system can process the image data and visitor information in the elevator in real time to achieve rapid response and accurate service. The welcoming robot 130 generates intelligent welcoming and guiding strategies based on the integrated data to improve the user experience.

[0098] See also Figure 3 In some embodiments, the robot and the elevator use the Lora protocol for communication. The welcoming robot 130 obtains information about the elevator door, floor, etc. from the elevator every 500ms. The video surveillance system 110 and the camera in the elevator use the RTSP protocol for communication, and obtains the video stream in the elevator in real time for monitoring, display and analysis. The welcoming robot 130 and the visitor system 120 use the MQTT protocol for communication to achieve the interaction of visitor information and related instructions. The visitor system 120 and the video surveillance system 110 use the HTTP protocol for communication to obtain the face data in the elevator for identification and analysis by the visitor system 120. The visitor system 120 and the visited person use telephone or text messages to notify the visitor of visitor information.

[0099] In some embodiments, the welcoming robot 130 is also used to provide welcoming services to passengers with the highest service priority when identity information of multiple passengers is detected in the elevator. The service priority is: the service priority of scheduled visitors is higher than the service priority of strangers, and the service priority of strangers is higher than the service priority of employees.

[0100] When multiple passengers in the elevator reach the floor where the robot is located, the passenger with the highest service priority will be greeted according to the service priority level.

[0101] By providing a robot visitor welcoming system based on elevator data, the welcoming robot 130 has the ability to obtain elevator status data in real time. Therefore, when a user uses the elevator, the welcoming robot 130 can determine the user's target floor. When it is detected that the user's target floor is the same as the floor where the welcoming robot 130 is located, the welcoming robot 130 will notify the visitor system 120 to obtain the valid face data detected in the elevator from the video surveillance system 110 at a regular time. After the visitor system 120 obtains the face data in the elevator, it will compare it with the face of the reservation information database and the employee information database to identify whether the identity of the elevator passenger is a company employee, a visitor or a stranger. The visitor system 120 will then synchronize the information of the elevator passenger to the welcoming robot 130, so the welcoming robot 130 can adopt different welcoming modes according to the identity of the elevator passenger to achieve the purpose of active welcoming and precise service. If the visitor system 120 detects that the elevator passenger is a scheduled visitor, it will also notify the person being visited so that the person being visited can calmly prepare to receive the upcoming visitor.

[0102] The building system basically includes elevator video and video surveillance system 110, so the visitor system 120 and the video surveillance system 110 only need to be connected by simple protocol. It will not increase the hardware cost, but can greatly improve the initiative of the robot to welcome guests and the ability of precise service.

[0103] See also Figure 4 In another embodiment, a robot visitor welcoming method based on elevator data is applied to the robot visitor welcoming system based on elevator data in the above embodiment, comprising the following steps:

[0104] Step S410: The welcoming robot obtains elevator status data in real time;

[0105] Step S420: When the welcoming robot determines through the elevator status data that a passenger is using the elevator, and the passenger's target floor is the floor where the welcoming robot is located, the welcoming robot notifies the visitor system to periodically obtain valid facial data detected in the elevator from the video surveillance system;

[0106] Step S430: The visitor system identifies the face data identified by the video surveillance system, obtains the identity information of the elevator passenger, and synchronizes it to the welcoming robot;

[0107] Step S440: When the elevator reaches the floor where the welcoming robot is located and starts, the welcoming robot adopts a corresponding welcoming mode according to the identity information of the elevator occupant.

[0108] The welcoming robot has the ability to obtain elevator status data in real time. When there are passengers in the elevator, the welcoming robot can determine the user's target floor. When it detects that the passenger's target floor is the floor where the welcoming robot is located, the welcoming robot will notify the visitor system to periodically obtain valid facial data detected in the elevator from the video surveillance system. After the visitor system obtains valid facial data, it will identify the passenger based on the facial data, obtain the passenger's identity information, and send the identity information to the welcoming robot. When the passenger reaches the floor where the welcoming robot is located, the welcoming robot can use the corresponding welcoming mode to welcome the passenger based on the passenger's identity. The welcoming robot welcomes visitors by combining the video surveillance system and the visitor system. It is no longer limited by the hardware conditions of the welcoming robot itself. By using the corresponding welcoming mode for the visitor's identity information, the visitor's welcoming experience can be improved.

[0109] In some embodiments, the “when the elevator reaches the floor where the welcoming robot is located and starts, the welcoming robot adopts the corresponding welcoming mode according to the identity information of the elevator passenger” comprises the following steps:

[0110] When the current elevator passenger is identified as an employee through identity information, when the elevator arrives at the current floor and the elevator door is open, the time to send the door opening command to the company's access control is calculated based on the distance from the elevator door to the employee's company's access control. When the time is reached, the door opening command is sent to the company's access control.

[0111] When the current passenger is identified as a stranger through identity information, when the elevator arrives at the current floor and the elevator door is open, the passenger is asked for the required service and the universal welcoming process is started;

[0112] When the current elevator rider is identified as a scheduled visitor through identity information, the elevator will broadcast a welcome message based on the visitor information when it arrives at the current floor and the elevator door is open, obtain the location information of the person being visited from the visitor system, and ask the scheduled visitor whether he or she needs guidance service. When it is confirmed that the scheduled visitor needs guidance service, the scheduled visitor will be guided based on the location information of the person being visited.

[0113] The welcoming robot can adopt the corresponding welcoming mode according to the different identity information of the elevator passengers. When the welcoming robot recognizes that the identity information of the elevator passenger is an employee, when the elevator arrives at the floor where the welcoming robot is located and the elevator door is open, the time to send the door opening command can be estimated based on the distance from the elevator door to the company access control. When the calculated time is reached, the door opening command is sent to the company access control corresponding to the employee, so that the employee can reach the company access control to achieve the service purpose of actively opening the door for the employee. When the identity information of the elevator passenger is recognized as a stranger, the welcoming robot will actively move to the elevator door. When the elevator arrives at the current floor and the elevator door is open, it will actively ask the stranger whether he needs service and start the universal welcoming process simultaneously. When it is identified that the person taking the elevator is a scheduled visitor, the welcoming robot will actively move to the elevator entrance. When the elevator reaches the current floor and the elevator door is open, the welcoming words will be accurately broadcast according to the visitor information. At the same time, based on the location information of the person being visited obtained from the visitor system, the robot will actively ask the scheduled visitor whether he needs to be provided with services such as guidance. When the scheduled visitor needs guidance services, the robot will guide the scheduled visitor according to the location information of the person being visited.

[0114] See also Figure 5 In some embodiments, the “welcoming robot adopts a corresponding welcoming mode according to the identity information of the elevator passenger” specifically includes the following steps:

[0115] Step S510: Processing the image data collected by the camera in the elevator through a convolutional neural network to obtain a first feature;

[0116] Step S520: extracting a second feature from the visitor information provided by the visitor system through natural language processing and image processing technology;

[0117] Step S530: Fusing the first feature and the second feature to obtain a comprehensive feature;

[0118] Step S540: Generate a welcoming strategy using a deep learning model based on the fusion features.

[0119] In order to improve the recognition accuracy and service accuracy of the welcoming robot and provide personalized welcoming services to visitors, the image data collected by the camera in the elevator is integrated with the information in the visitor system. Specifically, the welcoming strategy is generated as follows:

[0120] 1. Data processing of camera books in elevators;

[0121] For the image data X collected by the camera in the elevator t , processed by convolutional neural network (CNN), extracting the first feature F t :

[0122] Ft =CNN(X t ).

[0123] Among them, the feature extraction process includes several convolutional layers, pooling layers and fully connected layers. The specific formula is as follows:

[0124] F t =FC(ReLU(Pool(Conv(X t )))). Among them, Conv represents the convolution operation, Pool represents the pooling operation, ReLU represents the activation function, and the fully connected layer (FC) is used to generate the final feature representation.

[0125] 2. Visitor system information processing;

[0126] For the visitor information provided by the visitor system ttht The second feature F is extracted by using natural language processing (NLP) and image processing technology. v ;

[0127] F v =thP(V ttht )+Ctt(V ithgt ), where NLP is used to process text information in visitor information, and CNN is used to process visitor photos in visitor information.

[0128] 3. Data fusion;

[0129] The first feature F extracted by the camera in the elevator t The second feature F extracted by the visitor system v Fusion to generate comprehensive features F fusion :

[0130] F fusion =αF t +βF v ; where α and β are fusion weights, satisfying α+β=1. The fusion method can be weighted averaging, splicing, etc.

[0131] 4. Generate welcoming strategy.

[0132] Based on the comprehensive feature F fusion , using the deep learning model to generate the welcoming strategy S:

[0133] S=Dtt(F fusion ), where DNN represents a deep neural network, which generates a welcoming strategy including robot actions and language output through several hidden layers and output layers.

[0134] By integrating the camera data in the elevator and the visitor system information, the recognition accuracy and service accuracy of the welcoming robot are improved. At the same time, the system can process the image data and visitor information in the elevator in real time to achieve rapid response and accurate service. The welcoming robot generates intelligent welcoming and guidance strategies based on the integrated data to improve the user experience.

[0135] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concept of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. A robot visitor welcoming system based on elevator data, characterized in that: include: A video monitoring system, which is connected to a camera in the elevator and is used to obtain the video stream in the elevator collected by the camera in real time, and to perform monitoring, display and analysis to obtain facial data of passengers; A visitor system, the visitor system is connected to the video surveillance system, and the visitor system is used to obtain facial data of elevator passengers through the video surveillance system, and identify the identity information of the elevator passengers based on the facial data; A welcoming robot is connected to the elevator and the visitor system. The welcoming robot is used to obtain the elevator status data in real time. When there are passengers in the elevator, when it is determined that the passenger's target floor is the floor where the welcoming robot is located, the visitor system is notified to obtain the passenger's identity information. When the passenger reaches the current floor, the corresponding welcoming mode is adopted for the passenger according to the obtained identity information.

2. The robot visitor welcoming system based on elevator data according to claim 1 is characterized in that: The welcoming robot is also used for: When the current elevator passenger is identified as an employee through identity information, when the elevator arrives at the current floor and the elevator door is open, the time to send the door opening command to the company's access control is calculated based on the distance from the elevator door to the employee's corresponding company's access control. When the time is reached, the door opening command is sent to the company's access control.

3. The elevator data-based welcoming robot visitor welcoming system according to claim 1, characterized in that: The welcoming robot is also used for: When the current elevator passenger is identified as a stranger through identity information, the system will actively move to the elevator door. When the elevator arrives at the current floor and the door is open, the system will ask the elevator passenger for the required service and start the universal welcoming process.

4. The robot visitor welcoming system based on elevator data according to claim 1, characterized in that: The welcoming robot is also used for: When the current elevator passenger is identified as a scheduled visitor through identity information, the system will actively move to the elevator door. When the elevator reaches the current floor and the elevator door is open, a welcome message will be broadcast based on the visitor information, and the location information of the person being visited will be obtained from the visitor system. The scheduled visitor will be asked whether a guiding service is required. When it is confirmed that the scheduled visitor needs the guiding service, the scheduled visitor will be guided based on the location information of the person being visited.

5. The robot visitor welcoming system based on elevator data according to claim 1, characterized in that: The guest system is also used to: When the elevator passenger is identified as a scheduled visitor through identity information, a notification message is sent to the person being visited.

6. The robot visitor welcoming system based on elevator data according to claim 1, characterized in that: The welcoming robot is also used for: The image data collected by the camera in the elevator is processed by a convolutional neural network to obtain the first feature; For the visitor information provided by the visitor system, the second feature is extracted through natural language processing and image processing technology; The first feature and the second feature are combined to obtain a comprehensive feature; Based on the fused features, a deep learning model is used to generate a welcoming strategy.

7. The robot visitor welcoming system based on elevator data according to claim 1, characterized in that: The welcoming robot is also used to provide welcoming service to the passengers with the highest service priority when it detects the identity information of multiple passengers in the elevator. The service priority is: the service priority of scheduled visitors is higher than that of strangers, and the service priority of strangers is higher than that of employees.

8. A robot visitor welcoming method based on elevator data, the method being applied to the robot visitor welcoming system based on elevator data as claimed in any one of claims 1 to 7, characterized in that: The following steps are involved: The welcoming robot obtains elevator status data in real time; When the welcoming robot determines through the elevator status data that a passenger is using the elevator, and the passenger's target floor is the floor where the welcoming robot is located, the welcoming robot notifies the visitor system to periodically obtain valid facial data detected in the elevator from the video surveillance system; The visitor system identifies the face data identified by the video surveillance system, obtains the identity information of the elevator passengers, and synchronizes it to the welcoming robot; When the elevator reaches the floor where the welcoming robot is located and starts, the welcoming robot adopts the corresponding welcoming mode according to the identity information of the elevator occupants.

9. The robot visitor welcoming method based on elevator data according to claim 8, characterized in that: The specific steps of "when the elevator reaches the floor where the welcoming robot is located and starts, the welcoming robot adopts the corresponding welcoming mode according to the identity information of the elevator passengers" are as follows: When the current elevator passenger is identified as an employee through identity information, when the elevator arrives at the current floor and the elevator door is open, the time to send the door opening command to the company's access control is calculated based on the distance from the elevator door to the employee's company's access control. When the time is reached, the door opening command is sent to the company's access control. When the current passenger is identified as a stranger through identity information, the elevator will actively move to the elevator door. When the elevator arrives at the current floor and the door is open, the elevator will ask the passenger for the required service and start the universal welcoming process; When the current elevator passenger is identified as a scheduled visitor through identity information, the system will actively move to the elevator door. When the elevator reaches the current floor and the elevator door is open, a welcome message will be broadcast based on the visitor information, and the location information of the person being visited will be obtained from the visitor system. The scheduled visitor will be asked whether a guiding service is required. When it is confirmed that the scheduled visitor needs the guiding service, the scheduled visitor will be guided based on the location information of the person being visited.

10. The robot visitor welcoming method based on elevator data according to claim 8, characterized in that: The "welcoming robot adopts a corresponding welcoming mode according to the identity information of the elevator passengers" specifically includes the following steps: The image data collected by the camera in the elevator is processed by a convolutional neural network to obtain the first feature; For the visitor information provided by the visitor system, the second feature is extracted through natural language processing and image processing technology; The first feature and the second feature are combined to obtain a comprehensive feature; Based on the fused features, a deep learning model is used to generate a welcoming strategy.

Images