Multi-dimensional man-machine interaction elevator mirror system

Abstract

The multi-dimensional man-machine interaction elevator mirror system comprises elevator mirrors (1), WiFi transmitting modules (2) and WiFi receiving modules (3), the elevator mirrors (1) are arranged on the outer sides of elevator doors of all floors and the inner side wall of an elevator car, and the WiFi transmitting modules (2) and the WiFi receiving modules (3) are arranged at the top and the bottom of the elevator car correspondingly; the elevator mirror (1) comprises a glass body (4), a touch display screen (5), a camera (6) and a controller (7) are arranged in the glass body (4), and the controller (7) is connected with the touch display screen (5) and the camera (6); the elevator mirror serves as a carrier to be used for information obtaining and displaying, the crowding degree in the elevator car is accurately known in a picture recognition counting mode, the positions of passengers are positioned in a non-inductive mode according to WiFi sensing so as to adjust the instruction input area of the touch display screen, and the elevator mirror is suitable for various crowds and good in practicability.

Description

Technical Field

[0001] The present invention relates to the field of automatic control, and particularly to an elevator mirror system with multi-dimensional human-computer interaction. Background Art

[0002] An elevator refers to a permanent transportation device that serves several specific floors in a building and whose car moves on at least two columns of rigid tracks and is widely used in most areas. Currently, the interior of an elevator usually has a control panel set at a fixed position for contact pressing or card induction to control the arrival floor, which cannot adapt to users of different heights and those with limited mobility, such as children or wheelchair users. Adding an auxiliary control panel will occupy part of the interior space of the car and still cannot achieve full coverage. At the same time, the existing elevator only measures the number of people in the car by weight and cannot accurately reflect the degree of crowding inside. It may be that the per capita mass is light, resulting in the car not being overweight but the interior space is still relatively crowded. At this time, stopping at non-target floors cannot continue to carry passengers, thus leading to low transportation efficiency. Summary of the Invention

[0003] The purpose of the present invention is to provide an elevator mirror system with multi-dimensional human-computer interaction. The present invention uses the elevator mirror as a carrier for information acquisition and display, accurately knows the degree of crowding inside the car through the method of picture recognition and counting, and adjusts the command input area of the touch display screen according to the position of passengers sensed by WiFi in a non-intrusive manner, which is applicable to various groups of people and has good practicability.

[0004] The technical solution provided by the present invention is as follows: An elevator mirror system with multi-dimensional human-computer interaction includes an elevator mirror, a WiFi transmitting module, and a WiFi receiving module. The elevator mirror is arranged on the outer side of each floor elevator door and the inner side wall of the elevator car. The WiFi transmitting module and the WiFi receiving module are respectively arranged on the top and bottom of the elevator car. The elevator mirror includes a glass body, on which there are a touch display screen, a camera, and a controller. The controller is connected to the touch display screen and the camera, and is wirelessly connected to the WiFi transmitting module and the WiFi receiving module. The touch display screen is used to receive the controller's command for information display and receive commands and output them to the controller. The camera is used to obtain the pictures inside and outside the elevator car, and the controller receives the pictures for recognition and counting to obtain the number of passengers and the number of passengers waiting to board. The WiFi transmitting module transmits WiFi signals, and the WiFi receiving module receives the WiFi signals after passing through the elevator car and transmits them to the controller. The controller analyzes the channel state information in the WiFi signals through WiFi sensing to obtain the gestures and gesture positions of the passengers in the car. The controller of the elevator mirror in the car adjusts the command input area of the touch display screen according to the gestures and gesture positions.

[0005] In the elevator mirror system with the above-mentioned multi-dimensional human-machine interaction, the controller of the elevator mirror inside the elevator car communicates wirelessly with the controllers of the elevator mirrors outside the elevator doors on each floor via a WiFi transmission module.

[0006] In the aforementioned elevator mirror system with multi-dimensional human-machine interaction, the process of the controller for identification and counting is carried out according to the following steps:

[0007] Step A1: Receive the picture inside the elevator car when the car door is closed.

[0008] Step A2: Perform noise reduction, contrast enhancement, and color space conversion processing on the picture.

[0009] Step A3: Perform face recognition on the dynamic and static features in the picture through a target detection model and generate detection frames.

[0010] Step A4: Remove redundant detection frames through non-maximum suppression and retain the detection frames that reach the confidence threshold.

[0011] Step A5: Count the number of detection frames in Step A4 to obtain the counting result.

[0012] In the aforementioned elevator mirror system with multi-dimensional human-machine interaction, there are at least two elevator mirrors inside the elevator car, which are respectively arranged on two adjacent inner sidewalls of the elevator car. The average value of the counting results of the controllers of all the elevator mirrors inside the elevator car is taken as the number of passengers.

[0013] In the aforementioned elevator mirror system with multi-dimensional human-machine interaction, the process of the controller for WiFi perception is carried out according to the following steps:

[0014] Step B1: Extract the channel state information including amplitude and phase data from the WiFi signal through protocol parsing.

[0015] Step B2: Perform filtering and wavelet denoising processing on the channel state information.

[0016] Step B3: Calculate the amplitude difference and phase difference in the channel state information processed in Step B2, generate a time-frequency diagram through short-time Fourier transform, and capture the micro-Doppler effect generated by gestures.

[0017] In the aforementioned elevator mirror system with multi-dimensional human-machine interaction, the process of the controller for gesture recognition is carried out according to the following steps:

[0018] Step B3.1.1: Input the time-frequency diagram into a convolutional neural network to extract the corresponding spatial features.

[0019] Step B3.1.2: Input the spatial features and time-frequency diagrams into the Transformer network, capture the temporal dependencies of the spatial features through the self-attention mechanism, and thus obtain the continuous frame changes corresponding to the gesture actions.

[0020] Step B3.1.3: Map and classify the continuous frame changes in Step B3.1.2 through a Softmax classifier, and output the corresponding gesture categories.

[0021] In the aforementioned elevator mirror system for multi-dimensional human-computer interaction, the process of the controller for gesture positioning is carried out according to the following steps:

[0022] Step B3.2.1: Offline collect the phase difference and amplitude difference signal features formed by gestures at different positions, and construct a fingerprint database associated with the position coordinates and signal features.

[0023] Step B3.2.2: Obtain the phase difference and amplitude difference of the current gesture.

[0024] Step B3.2.3: Match the signal features in the fingerprint database through similarity to obtain multiple signal features with a similarity higher than the threshold.

[0025] Step B3.2.4: Use weighted average to calculate the position coordinates of multiple signal features, and the result is the position coordinate of the current gesture.

[0026] In the aforementioned elevator mirror system for multi-dimensional human-computer interaction, a microphone and a speaker connected to the controller are provided outside the glass.

[0027] Compared with the prior art, the present invention obtains the pictures inside and outside the elevator car through a camera, and the controller receives the pictures for recognition and counting to obtain the number of passengers and the number of waiting passengers, accurately reflecting the degree of congestion inside the car, and assisting the elevator to judge whether to stop by combining the number of waiting passengers outside the floor; the WiFi signal is transmitted through the WiFi transmitting module, the WiFi receiving module receives the WiFi signal after passing through the elevator car and transmits it to the controller, and the controller analyzes the channel state information in the WiFi signal through WiFi sensing to obtain the gestures and gesture positions of the passengers in the car. The controller of the elevator mirror in the car adjusts the command input area of the touch display screen according to the gestures and gesture positions, and quickly and accurately adjusts the display area of the touch display screen through a contactless sensing method, thereby being applicable to various people. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic structural diagram of the present invention;

[0029] Figure 2 is a schematic structural diagram of the back side inside the elevator car of the present invention;

[0030] Figure 3 Schematic diagram of module connection of the present invention

[0031] The labels in the attached drawings are: 1, elevator mirror; 2, WiFi transmitting module; 3, WiFi receiving module; 4, vitreous body; 5, touch display screen; 6, camera; 7, controller; 8, microphone; 9, speaker Specific implementation mode

[0032] The present invention will be further described below in conjunction with the embodiments and the attached drawings, but it shall not be used as the basis for limiting the present invention

[0033] Embodiment: An elevator mirror system for multi-dimensional human-computer interaction, as shown in the attached Figure 1 and attached Figure 2 figures, including an elevator mirror 1, a WiFi transmitting module 2 and a WiFi receiving module 3. The WiFi transmitting module uses three MIMO (multiple input multiple output) dual-band routers, is configured with more than three antennas, supports 2.4 / 5G bands, the WiFi receiving module uses a USB wireless network card, supports 802.11n / ac protocol, has at least four receiving ends, and is distributed in a grid pattern. The elevator mirror 1 is fixed on the outer sides of the elevator doors on each floor and the three side walls inside the elevator car. The WiFi transmitting module 2 and the WiFi receiving module 3 are respectively arranged on the top and bottom of the elevator car; the elevator mirror 1 includes an ITO vitreous body 4, a touch display screen 5, a camera 6 and a controller 7 are integrated on the vitreous body 4, a microphone 8 and a speaker 9 are fixed outside the glass body, as shown in the attached Figure 3As shown in the figure, the controller 7 uses Rockchip RK3399, which is connected to the touch display screen 5 and the camera 6. The controller 7 is wirelessly connected to the WiFi transmitting module 2 and the WiFi receiving module 3. The touch display screen of the elevator mirror in the car is used to display information such as weather and news, and the touch display screen of the outer elevator door displays elevator status information such as the number of people in the elevator and the running direction; the touch display screen 5 is used to receive instructions from the controller 7 for information display and receive instructions and output them to the controller 7; the camera 6 is used to obtain the picture inside the elevator car and the picture outside the elevator. The controller 7 receives the picture for recognition and counting to obtain the number of passengers and the number of waiting passengers. When the number threshold is exceeded, information is transmitted to the elevator main unit through the WiFi transmitting module to control the elevator not to stop at non-target floors; the WiFi transmitting module 2 transmits WiFi signals, and the WiFi receiving module 3 receives the WiFi signals after passing through the elevator car and transmits them to the controller 7. The controller 7 analyzes the channel state information in the WiFi signals through WiFi sensing to obtain the gestures and gesture positions of the passengers in the car. The controller 7 of the elevator mirror 1 in the car adjusts the instruction input area of the touch display screen 5 according to the gestures and gesture positions, and obtains the instructions expressed by the passengers through the gestures, including instructions such as calling out / retracting the console, zooming in, and zooming out. According to the gesture position, corresponding display instructions are executed on the corresponding horizontal plane; voice information is received through the microphone. After receiving the voice information, the controller performs voice recognition through the Transformer voice recognition model and performs corresponding preset control according to the voice recognition result; the controller 7 of the elevator mirror 1 in the elevator car communicates wirelessly with the controller 7 of the elevator mirror 1 outside the elevator doors on each floor through the WiFi transmitting module 2, realizes the information transmission of the number of people inside and the docking position, and at the same time connects to communication devices such as mobile phones through the local area network to meet various Internet of Things functions.

[0034] The process of the controller 7 for recognition and counting is carried out according to the following steps:

[0035] Step A1: Receive the picture inside the elevator car when the car door is closed, and the internal personnel remain relatively static;

[0036] Step A2: Perform noise reduction, contrast enhancement, and color space conversion processing on the picture to improve the picture clarity and highlight the details of the face target at the same time;

[0037] Step A3: Perform face recognition on the dynamic and static features in the picture through the YOLO object detection model and generate detection frames.

[0038] Step A4: Remove redundant detection frames through non-maximum suppression, and retain the detection frames that reach the 0.5 confidence threshold;

[0039] Step A5: Count the number of detection frames in Step A4 to obtain the counting result.

[0040] There are three elevator mirrors 1 inside the elevator car, which are respectively fixed on the inner side walls of the elevator car. The average value of the counting results of the controllers 7 of all the elevator mirrors 1 inside the elevator car is taken as the number of passengers to ensure the integrity of target detection coverage. At the same time, mean calculation is used to reduce the interference of target overlap.

[0041] The process of the controller 7 performing WiFi sensing is carried out according to the following steps:

[0042] Step B1: Extract the channel state information including amplitude and phase data in the WiFi signal by parsing the 802.11ac protocol.

[0043] Step B2: Perform filtering and wavelet denoising processing on the channel state information to eliminate high-frequency noise and suppress environmental interference at the same time;

[0044] Step B3: Calculate the amplitude difference and phase difference in the channel state information processed in Step B2, and generate a 10ms×1Hz time-frequency diagram through short-time Fourier transform to capture the micro-Doppler effect generated by gestures.

[0045] The process of the controller 7 performing gesture recognition is carried out according to the following steps:

[0046] Step B3.1.1: Input the time-frequency diagram into the convolutional neural network to extract the corresponding spatial features;

[0047] Step B3.1.2: Input the spatial features and the time-frequency diagram into the Transformer network, and capture the temporal dependence of the spatial features through the self-attention mechanism, so as to obtain the continuous frame changes corresponding to the gesture actions;

[0048] Step B3.1.3: Map and recognize the classification of the continuous frame changes in Step B3.1.2 through the Softmax classifier, and output the corresponding gesture categories. The Softmax classifier includes gestures such as waving, opening the palm, and summoning.

[0049] The process of the controller 7 performing gesture positioning is carried out according to the following steps:

[0050] Step B3.2.1: Offline collect the phase difference and amplitude difference signal features formed by gestures at different positions, and construct a fingerprint library that associates the position coordinates with the signal features;

[0051] Step B3.2.2: Obtain the phase difference and amplitude difference of the current gesture;

[0052] Step B3.2.3: Match the similarity with the signal features in the fingerprint library to obtain multiple signal features with a similarity higher than the threshold.

[0053] Step B3.2.4: Calculate the position coordinates of multiple signal features using weighted average, and the result is the position coordinates of the current gesture.

[0054] In summary, the present invention integrates camera vision recognition, WiFi signal sensing and voice input, supports multiple interaction modes such as touch, gesture and voice, takes into account the operation needs of users with different heights and mobility, and solves the problems of fixed and insufficient coverage of traditional elevator control panels. Through multi-camera video processing and object detection models, combined with the non-maximum suppression algorithm, high-precision counting of the number of passengers in the car is achieved; based on the analysis of the channel state information of WiFi signals (including the capture of the micro-Doppler effect), combined with convolutional neural networks and Transformer networks for gesture recognition and position localization, the command input area of the touch display screen is dynamically adjusted to improve interaction convenience; the elevator mirrors inside the car and outside the elevator doors on each floor are wirelessly communicated through WiFi modules to synchronize information such as the number of people in the car, congestion status and running direction in real time, assisting the elevator to intelligently judge whether to stop and optimizing the transportation efficiency; at the same time, it supports networking with terminal devices such as mobile phones to expand the functional scenarios. Through signal processing technologies such as noise reduction and wavelet denoising, as well as multi-elevator mirror position deployment and mean calculation of counting results, interference such as occlusion and overlap is reduced to ensure detection accuracy in complex environments; the offline constructed WiFi signal fingerprint library combined with the weighted average positioning algorithm improves the robustness of gesture position recognition; an intelligent human-computer interaction system is constructed with the elevator mirror as the carrier, which has significant advantages in improving user experience, optimizing transportation efficiency and enhancing system adaptability, and is applicable to multiple scenarios such as residential buildings, commercial buildings and public facilities.

Claims

1. An elevator mirror system for multi-dimensional human-computer interaction, comprising an elevator mirror (1), a WiFi transmitting module (2) and a WiFi receiving module (3). The elevator mirror (1) is arranged on the outer side of each floor elevator door and the inner side wall of the elevator car. The WiFi transmitting module (2) and the WiFi receiving module (3) are respectively arranged at the top and bottom of the elevator car. It is characterized in that: The elevator mirror (1) includes a vitreous body (4), on which a touch display screen (5), a camera (6) and a controller (7) are provided. The controller (7) is connected to the touch display screen (5) and the camera (6), and the controller (7) is wirelessly connected to a WiFi transmitting module (2) and a WiFi receiving module (3); the touch display screen (5) is used to receive instructions from the controller (7) for information display and receive instructions and output them to the controller (7); the camera (6) is used to obtain the picture inside the elevator car and the picture outside the elevator. The controller (7) receives the pictures for recognition and counting to obtain the number of passengers and the number of passengers waiting to board; the WiFi transmitting module (2) transmits a WiFi signal, and the WiFi receiving module (3) receives the WiFi signal after passing through the elevator car and transmits it to the controller (7). The controller (7) analyzes the channel state information in the WiFi signal through WiFi sensing to obtain the gestures and gesture positions of the passengers in the car. The controller (7) of the elevator mirror (1) inside the car adjusts the instruction input area of the touch display screen (5) according to the gestures and gesture positions.

2. The elevator mirror system for multi-dimensional human-computer interaction according to claim 1, wherein: The controller (7) of the elevator mirror (1) inside the elevator car communicates wirelessly with the controller (7) of the elevator mirror (1) outside the elevator door on each floor through the WiFi transmitting module (2).

3. The elevator mirror system for multi-dimensional human-computer interaction according to claim 1, characterized in that: The process of the controller (7) for recognition and counting is carried out according to the following steps: Step A1: Receive the picture inside the elevator car when the car door is closed. Step A2: Perform noise reduction, contrast enhancement and color space conversion processing on the picture. Step A3: Perform face recognition on the dynamic and static features in the picture through a target detection model and generate detection frames. Step A4: Remove redundant detection frames through non-maximum suppression and retain the detection frames that reach the confidence threshold. Step A5: Count the number of detection frames in Step A4 to obtain the counting result.

4. The elevator mirror system for multi-dimensional human-computer interaction according to claim 3, characterized in that: There are at least two elevator mirrors (1) inside the elevator car, and they are respectively arranged on two adjacent inner side walls of the elevator car. The average value of the counting results of the controllers (7) of all the elevator mirrors (1) inside the elevator car is taken as the number of passengers.

5. The elevator mirror system for multi-dimensional human-computer interaction according to claim 1, wherein: The process of the controller (7) for WiFi sensing is carried out according to the following steps: Step B1: Extract the channel state information including amplitude and phase data in the WiFi signal through parsing the protocol. Step B2: Perform filtering and wavelet denoising processing on the channel state information. Step B3: Calculate the amplitude difference and phase difference in the channel state information processed in Step B2, generate a time-frequency diagram through short-time Fourier transform, and capture the micro-Doppler effect generated by the gesture.

6. The elevator mirror system for multi-dimensional human-computer interaction according to claim 5, wherein: The process of the controller (7) for gesture recognition is carried out according to the following steps: Step B3.1.1: Input the time-frequency diagram into a convolutional neural network to extract the corresponding spatial features. Step B3.1.2: Input the spatial features and the time-frequency diagram into a Transformer network, and capture the temporal dependence of the spatial features through the self-attention mechanism, so as to obtain the continuous frame change corresponding to the gesture action. Step B3.1.3: Map and identify the classification of the continuous frame changes in Step B3.1.2 through a Softmax classifier, and output the corresponding gesture categories.

7. The elevator mirror system for multi-dimensional human-computer interaction according to claim 5, characterized in that: The process of gesture positioning by the controller (7) is carried out according to the following steps: Step B3.2.1: Offline collect the phase difference and amplitude difference signal features formed by gestures at different positions, and construct a fingerprint database associated with the position coordinates and signal features; Step B3.2.2: Obtain the phase difference and amplitude difference of the current gesture; Step B3.2.3: Match the signal features in the fingerprint database through similarity to obtain multiple signal features with a similarity higher than the threshold. Step B3.2.4: Use weighted average to calculate the position coordinates of multiple signal features, and the result is the position coordinate of the current gesture.

8. The elevator mirror system for multi-dimensional human-computer interaction according to claim 1, characterized in that: A microphone (8) and a speaker (9) connected to the controller (7) are provided outside the vitreous body (4).

Images