Intelligent elevator taking assisting system based on machine vision

Abstract

The invention discloses an intelligent elevator taking assisting system based on machine vision, which is characterized by comprising an industrial camera, a weak infrared ray emission instrument, a weak infrared image acquisition instrument and a development board integration module, the industrial camera is fixed above the elevator car; the weak infrared ray emitter is fixed on the left side of the elevator car; the weak infrared image acquisition instrument is fixed on the right side of the elevator car; the development board integration module is fixed in a gap between a residential elevator landing door and a car door and comprises NVIDIA edge detection equipment, a power supply lithium battery, a voltage conversion device and a relay device. The NVIDIA edge detection device is connected with the industrial camera, the weak infrared ray emitter and the weak infrared image collector. Through the face recognition and posture recognition technology, the identity and the sight position of an owner can be automatically detected, and non-contact elevator taking is achieved. Compared with traditional card swiping or key operation, the elevator taking convenience is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of elevator safety protection devices and machine vision technology, and in particular to an intelligent auxiliary elevator riding system based on machine vision. Background Art

[0002] With the continuous development and progress of science and technology, the elevator industry, as a special equipment in the manufacturing industry, is the first to bear the brunt and is closely connected with people's daily life. As of 2024, the total number of special equipment in the country will reach 19.5525 million units, including 9.6446 million elevators. The popularity of community elevators has developed significantly in China in recent years. Especially in the context of accelerated urbanization, more and more residential buildings in communities are equipped with elevators. However, despite the increase in the coverage rate of elevators, many communities still have many problems in terms of safety, convenience and intelligence. The elevator management in some communities is relatively loose, lacking an effective access control and identity authentication system, which makes it easy for any unauthorized person to enter and use the elevator, adding potential safety hazards to the owners; many traditional elevators still use key operating systems and lack intelligent technology support. When the owner's hands are full of items, such as carrying heavy objects, pushing a stroller or carrying other items, it becomes quite inconvenient to press the elevator button. In addition, the elevator's response speed and the recognition mechanism of the destination floor are also relatively backward. Therefore, there is an urgent need for a convenient and intelligent auxiliary elevator system and method to enhance the safety management of the community, provide owners (especially disabled passengers) with a better elevator experience and improve the quality of life of the owners.

[0003] The invention patent "Passenger Assistance System for Elevators" with application publication number CN109748163A receives auxiliary service requests through the elevator system controller and transports the elevator car to the floor indicated by the service request. Compared with traditional community elevators, it has improved the convenience of elevators to a certain extent, but it still cannot effectively solve the problems of arbitrary entry and exit and use of community elevators, and the privacy and security of owners cannot be well guaranteed. In addition, the transmission method of auxiliary service requests is easily interfered by the external environment, which has certain limitations.

[0004] The invention patent "Elevator Passenger Interface with Special Auxiliary Features" with application publication number CN103974888B designs a touch interface for elevator passengers with special auxiliary features. The touch screen is configured to allow passengers to indicate their service request for the elevator by touching the screen, and to determine the destination floor number requested by the passenger in combination with the auxiliary button and subsequent operation of the controller. Compared with traditional residential elevators, it also improves the convenience of the elevator to a certain extent, but it also cannot solve the problem that the passenger's hands are occupied and it is inconvenient to operate the elevator auxiliary interface, and there is a certain inconvenience.

[0005] The invention patent "Elevator Riding Assistance Device" with application publication number CN115515886A and the invention patent "Step Lifting Assistance Device and Step Lifting Assistance Method for Passenger Conveyor" with application publication number CN112239131B are both designed for escalators in large supermarkets such as shopping malls, and are not suitable for use in community elevators.

[0006] Therefore, how to better improve the safety of community owners taking elevators, better meet the community owners' elevator experience, and improve the convenience and intelligence of community elevators are key issues that need to be urgently solved at this stage. Summary of the invention

[0007] The purpose of the present invention is to provide an intelligent assisted elevator system based on machine vision to solve the problems existing in the above-mentioned prior art. This technology can be widely used in community elevators in daily life. By performing face recognition on people entering the elevator, it is determined whether they are owners in the community. If they are not owners, they need to swipe their cards to take the elevator; if they are owners, the system uses the Ultralytics v8 pose human posture recognition network to detect the key points of various parts of the owner's body, infer the owner's height, control the weak infrared rays and the owner's binocular sight level, and the infrared image acquisition instrument captures the image of the reflected light in real time, and combines image processing technology to estimate the convergence point of the owner's sight on the elevator floor button. The system transmits the obtained convergence point information to the elevator control system and automatically controls the elevator to go to the corresponding floor. The system and method are efficient, accurate and applicable to a variety of scenarios, especially in terms of privacy protection and community security. This can not only effectively improve the safety of community owners, but also reduce human intervention and improve the user's elevator experience.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] An intelligent auxiliary elevator riding system based on machine vision, characterized in that it includes: an industrial camera, a weak infrared ray emitter, a weak infrared image acquisition device and a development board integrated module;

[0010] The industrial camera is fixed above the elevator car;

[0011] The weak infrared ray transmitter is fixed on the left side of the elevator car;

[0012] The weak infrared image collector is fixed on the right side of the elevator car;

[0013] The development board integrated module is fixed in the gap between the community elevator floor door and the car door, and includes: NVIDIA edge detection equipment, power supply lithium battery, voltage conversion device and relay device; the NVIDIA edge detection equipment is respectively connected to the industrial camera, the weak infrared ray emitter and the weak infrared image acquisition device.

[0014] Preferably, the industrial camera takes a photo of the person entering the elevator car, and performs face recognition on the photo through the recognition algorithm deployed on the NVIDIA edge detection device to determine whether the person is a community owner; the recognition algorithm adopts an improved YOLOv8 network model, and proposes a loss function with adaptive scaling weights based on the characteristics of the face data set, and adjusts the loss function through dynamic adjustment coefficients; a priori frame update method is proposed to adjust the width and height of the detection frame according to the symmetry and proportion characteristics of the face; and an attention mechanism is designed to weightedly fuse the outputs of channel attention and spatial attention through an adaptive fusion mechanism to generate the final attention feature.

[0015] Preferably, the weak infrared ray emitter is connected to the NVIDIA edge detection device via WIFI, and the direct position of the weak infrared light is automatically adjusted according to the height of the owner determined by the posture recognition algorithm, so that the direct position of the light is aligned with the owner's eyes.

[0016] Preferably, the weak infrared image collector is directly connected to the NVIDIA edge detection device, and the collected reflected infrared light image is transmitted to the NVIDIA edge detection device. After the machine vision image processing algorithm, the angles between multiple groups of infrared light reflected by the owner's eye lens and retina are obtained.

[0017] Preferably, the direction of the owner's line of sight is determined by the rotation angle of the owner's eyeball, and the floor button the owner is looking at is determined based on the relative position between the weak infrared ray emitter and the elevator floor button panel; the information of the floor button the owner is looking at is obtained and transmitted to the community elevator control system, and the control system automatically adjusts the elevator to make the elevator go to the floor the owner wants to go to.

[0018] Preferably, the voltages required by the weak infrared ray emitter, the weak infrared image collector and the NVIDIA edge detection device are different; by respectively fixing the relay device and the voltage conversion device at the left and right ends of the power supply lithium battery, the weak infrared ray emitter, the weak infrared image collector and the NVIDIA edge detection device are provided with the power they need respectively.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] (1) Through face recognition and gesture recognition technology, the system can automatically detect the owner's identity and line of sight, enabling contactless elevator riding. Compared with traditional card swiping or button operation, this greatly improves the convenience of elevator riding, especially when people are carrying items and it is inconvenient to operate.

[0021] (2) The Ultralytics v8 pose network has high accuracy in detecting human posture and combines visual infrared technology for line of sight tracking, making the system far more accurate than traditional elevator control systems, ensuring that the elevator can reach the designated floor quickly and accurately, effectively reducing waiting time and operational errors.

[0022] (3) The system automatically selects the elevator floor through gesture recognition and gaze tracking technology, avoiding the process of owners manually pressing the elevator floor button, effectively preventing the owner's floor information from being leaked, protecting the owner's personal privacy, and greatly improving the safety of the community and buildings.

[0023] (4) Through machine vision technology, the system can automatically determine the owner's identity and control the elevator, reducing the need for manual operation. Compared with the traditional elevator operation process, this system is more intelligent and automated, improving the user experience.

[0024] (5) This system is not only suitable for ordinary residential areas, but can also be used in other places with high requirements for security and privacy, and has a wider range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 A schematic diagram of the system structure of an embodiment of the present invention;

[0027] Figure 2 A structural diagram of a relay device and a voltage conversion device according to an embodiment of the present invention;

[0028] Figure 3 It is a diagram of the actual detection process of an embodiment of the present invention;

[0029] Figure 4 Schematic diagram of the system workflow of an embodiment of the present invention.

[0030] Explanation of the accompanying drawings: 1. Elevator car; 2. Elevator car door panel; 3. Weak infrared ray emitter; 4. Industrial camera; 5. Elevator floor button panel; 6. Weak infrared image collector; 7. Development board integrated module; 8. Relay device; 9. Voltage conversion device. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] like Figure 1 As shown, it is a schematic diagram of the system structure of this embodiment, including: an industrial camera 4, a weak infrared ray transmitter 3, an infrared image collector 6 and a development board integrated module 7; the industrial camera 4 is fixed above the elevator car 1; the weak infrared ray transmitter 3 is fixed on the left side of the elevator car 1; the infrared image collector 6 is fixed on the right side of the elevator car 1; the development board integrated module 7 is fixed in the gap between the community elevator floor door and the car door, including: NVIDIA edge detection equipment, power supply lithium battery (24V), voltage conversion device 9 and relay device 8; the NVIDIA edge detection equipment is respectively connected to the industrial camera 4, the weak infrared ray transmitter 3 and the infrared image collector 6.

[0034] The development board integrated module 7 is fixed in the gap between the community elevator floor door and the car door by different screws. The NVIDIA edge detection device is fixed in the development board integrated module 7 by screws, and is connected to the industrial camera 4, the power supply lithium battery (24V), the relay device 8 and the voltage conversion device 9, and mainly runs the relevant detection algorithm. Since the voltages required by the weak infrared ray emitter 3, the weak infrared image acquisition device 6 and the NVIDIA edge detection device are different, such as Figure 2 The relay device 8 and the voltage conversion device 9 shown are respectively fixed to the left and right ends of the power supply lithium battery (24V), so as to provide each partial module with the power it needs.

[0035] like Figure 1As shown, the industrial camera 4 is fixed above the elevator car 1 by a fixing device and connected to the NVIDIA edge detection device. The improved YOLOv8 target detection algorithm deployed on the NVIDIA edge detection device is used to perform face recognition on the person entering the elevator car 2 to determine whether he is a community owner. In order to improve the accuracy of community face detection, the original YOLOv8 network is improved according to the characteristics of the face data set, mainly focusing on the loss function and the prior frame, and adding the attention mechanism, as follows:

[0036] (1) Since face datasets may have imbalanced categories of different face features, which may lead to imbalanced positive and negative samples. In order to further enhance the adaptability of the network to different scales and feature complexities, a dynamic adjustment coefficient is introduced on the basis of the original loss function. Used to adaptively scale weights. The improved loss function L ad-loss As shown below:

[0037]

[0038] Among them, p t Represents the model's predicted probability for the correct category; γ represents the adjustment parameter, which is used to control the degree of weight reduction of the sample; Where k(x) represents the dynamic adjustment coefficient, which is automatically adjusted based on the complexity of the face image detected by the network.

[0039] (2) The prior frame is an important part of the network. In order to better adapt to the geometric features of the face, the face and symmetry and proportion features are used to perform geometric statistical analysis on the face data set and dynamically adjust the size and shape of the prior frame. The update formula is as follows:

[0040] W new =λ·W origin

[0041] H new =μ·H origin

[0042] Among them, W new and H new They represent the width and height of the updated prior box respectively, and λ and μ represent the scaling factors extracted according to the geometric features of the face dataset.

[0043] (3) Considering that the scales of facial features of different people vary greatly, especially in the specific scene at the elevator door, the size of faces at different distances varies significantly. Therefore, the designed attention mechanism is introduced into the detection head of the original YOLOv8 network to adaptively select the features of faces detected at different scales, thereby improving the detection accuracy of faces. The overall structure of the designed attention mechanism can be divided into a channel attention module, a spatial attention module, and an adaptive fusion mechanism. The channel attention module is used to assign different weights to different feature channels; the spatial attention module is used to allocate attention on the spatial dimension of the feature map and select important spatial areas. The adaptive fusion mechanism performs weighted fusion on the outputs of the above two to generate the final attention feature. The details are as follows:

[0044] 1) Channel Attention Module

[0045] The channel attention module is used to dynamically adjust the weight of each channel according to the global information and obtain the global information of the channel through global average pooling.

[0046]

[0047] Among them, X(i,j) represents the pixel value of the input feature map, H and W represent the height and width of the feature map (corresponding to the width and height of the prior box). The global information is processed through a fully connected network, and the generated weight f is activated using the ReLU function. ca Multiply it with the input feature X to get the weighted channel feature:

[0048]

[0049] X ca =f ca ·X

[0050] Among them, δ is the ReLU activation function, is the Sigmoid function, w i Expressed as weight.

[0051] 2) Spatial Attention Module

[0052] The spatial attention module is responsible for selecting important areas in the feature map. The channel features are fused through maximum pooling to obtain spatial information. The fused spatial features are convolved to generate the spatial attention weights f sa :

[0053]

[0054] In the formula, f spatial Represented as spatial information.

[0055] The generated spatial attention weight fsa With input feature X ca Multiply them together to get the weighted spatial feature X sa The specific formula is as follows:

[0056] X sa =f sa ·X ca .

[0057] 3) Adaptive fusion mechanism

[0058] In order to enhance the flexibility of the attention mechanism, an adaptive fusion weight is introduced based on channel attention and spatial attention, and the weights of the two attentions are dynamically allocated according to the complexity of the features. The weights can be adaptively adjusted according to the complexity of the features or the task requirements. Π is represented by the adaptive weight, and the fused feature X final It is expressed by the following formula:

[0059] X final =Π·X ca +(1-Π)·X sa

[0060] Adding the above three methods to the original YOLOv8 network structure can improve the detection accuracy of the detection network for face datasets to a certain extent.

[0061] The main workflow of the improved YOLOv8 target detection algorithm is as follows:

[0062] (1) Construct a facial image dataset of community owners, use Labelme software to annotate targets and key points, and divide the dataset into training and validation sets according to a certain ratio;

[0063] (2) The constructed data set is trained through the improved YOLOv8 target detection network to obtain a model for real-time recognition of the facial features and key points of community owners. Since the improved training model has a large number of parameters, the training model is lightweight;

[0064] (3) Deploy the lightweight model to NVIDIA edge detection devices to detect people entering the community elevator in real time;

[0065] If you are not the owner of this community, you need to swipe your card to use the elevator. If you are the owner of this community, the Ultralytics v8 pose network combined with the industrial camera 4 will further detect the key points of various parts of the owner's body to infer the owner's height and the height of the line of sight.

[0066] like Figure 3As shown, the weak infrared ray transmitter 3 is fixed on the left side of the elevator car 1, and is connected to the NVIDIA edge detection device through its own WIFI module. The direct position of the weak infrared light is automatically adjusted according to the height of the owner, so that the direct position of the light is basically flush with the position of the owner's eyes.

[0067] The weak infrared image acquisition device 6 is fixed to the right side of the elevator car through a fixing device and is directly connected to the NVIDIA edge detection device. Figure 3 As shown in the figure, the weak infrared ray transmitter illuminates the owner's eyes. Due to the special structure of the human eye, under the premise of keeping the relative position between the owner's head and the weak infrared ray transmitter unchanged, the infrared light reflected by the cornea will not change its angle, while the infrared light reflected by the retina will be at the same angle as the owner's line of sight. After the owner enters the elevator, he looks directly at the elevator floor button panel. At this time, the infrared light images reflected by the cornea and retina are collected in real time through the weak infrared image collector.

[0068] The infrared light image collected by the weak infrared image collector 6 is transmitted to the NVIDIA edge detection device in real time, and then processed by the image processing algorithm, such as Figure 3 As shown, the angles σ between multiple sets of infrared light reflected from the main lens and the retina are obtained.

[0069] The main process of machine vision image processing algorithm is as follows:

[0070] (1) Hough transform: used to detect the circular boundary of the pupil of the owner in the infrared light image collected by the weak infrared image acquisition device 6, and determine the center position of the pupil;

[0071] (2) Gaussian filtering: After determining the pupil center position, the Gaussian filtering algorithm is used to extract the position of the infrared reflection point in the image and track the change of the corneal reflection point R g ;

[0072] (3) Pupil-corneal reflection model: Based on the position of the pupil center and the position change between the corneal reflection points, the pupil-corneal reflection model is constructed to calculate the angle σ of eyeball rotation, thereby inferring the owner's line of sight. The specific calculation method is as follows:

[0073] The geometric point at the center of the pupil is P c =(x c ,y c ), the reflection point of weak infrared light on the cornea is R g =(x g ,y g ), the direction of the owner's gaze point after eye rotation is Pupil center P c and corneal reflex point R gThe relative position change ΔP between them is calculated by the following formula:

[0074] ΔP=Pc-Rg=(xc-xg,yc-yg)

[0075] According to the principles of geometric optics, the rotation angle σ of the eyeball can be calculated by the relative position change and the radius r of the eyeball. e The specific calculation formula is as follows:

[0076]

[0077] The angle σ (eyeball rotation angle) between the infrared light reflected from the eyelid and the retina can be further decomposed into the horizontal θ x and vertical θ y The component in the direction is given by the following formula:

[0078]

[0079] According to the rotation angle σ of the eyeball, the owner’s sight direction can be calculated. The owner’s initial sight direction is The calculation is performed using the following formula, and R(σ) is specifically as follows.

[0080]

[0081] R(σ) represents the rotation of the owner's eye in the horizontal and vertical directions.

[0082] Determine the direction of the owner's sight through multiple sets of angles σ And according to the relative position between the weak infrared ray transmitter 3 and the elevator floor button panel 5 (in Figure 3 The floor button that the owner is looking at is determined (indicated by a black dot in the figure). The obtained floor button information that the owner is looking at is transmitted to the community elevator control system, and the control system adjusts the elevator to go to the floor that the owner wants to go to. The overall workflow of the system in this embodiment is as follows Figure 4 shown.

[0083] In the description of the present invention, it should be understood that the fixed or positional relationship indicated in the relevant description is based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience of describing the present patent method, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0084] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An intelligent assisted elevator system based on machine vision, characterized in that: include: Industrial camera, weak infrared ray emitter, weak infrared image acquisition instrument and development board integrated module; The industrial camera is fixed above the elevator car; The weak infrared ray transmitter is fixed on the left side of the elevator car; The weak infrared image collector is fixed on the right side of the elevator car; The development board integrated module is fixed in the gap between the community elevator floor door and the car door, including: NVIDIA edge detection equipment, power supply lithium battery, voltage conversion device and relay device; The NVIDIA edge detection device is connected to the industrial camera, the weak infrared ray emitter and the weak infrared image collector respectively.

2. The intelligent assisted elevator riding system based on machine vision according to claim 1 is characterized in that: The industrial camera takes a photo of the person entering the elevator car, and performs face recognition on the photo through the recognition algorithm deployed on the NVIDIA edge detection device to determine whether the person is a community owner; the recognition algorithm adopts an improved YOLOv8 network model, and proposes a loss function with adaptive scaling weights based on the characteristics of the face data set, and adjusts the loss function through dynamic adjustment coefficients; a priori frame update method is proposed to adjust the width and height of the detection frame according to the symmetry and proportion characteristics of the face; and an attention mechanism is designed to perform weighted fusion on the outputs of channel attention and spatial attention through an adaptive fusion mechanism to generate a final attention feature.

3. The intelligent assisted elevator riding system based on machine vision according to claim 1 is characterized in that: The weak infrared ray emitter is connected to the NVIDIA edge detection device via WIFI, and the direct position of the weak infrared light is automatically adjusted based on the height of the owner determined by the posture recognition algorithm, so that the direct position of the light is aligned with the owner's eyes.

4. The intelligent assisted elevator riding system based on machine vision according to claim 1 is characterized in that: The weak infrared image acquisition instrument is directly connected to the NVIDIA edge detection device, and the collected reflected infrared light images are transmitted to the NVIDIA edge detection device. After the machine vision image processing algorithm, the angles between multiple groups of infrared light reflected by the owner's eye lens and retina are obtained.

5. The intelligent assisted elevator riding system based on machine vision according to claim 4 is characterized in that: The direction of the owner's sight is determined by the rotation angle of the owner's eyeball, and the floor button the owner is looking at is determined according to the relative position between the weak infrared ray transmitter and the elevator floor button panel; The information of the floor button that the owner is looking at is obtained and transmitted to the community elevator control system. The control system automatically adjusts the elevator to make the elevator go to the floor that the owner wants to go to.

6. The intelligent assisted elevator riding system based on machine vision according to claim 1 is characterized in that: The voltages required by the weak infrared ray emitter, the weak infrared image collector and the NVIDIA edge detection device are different; by respectively fixing the relay device and the voltage conversion device at the left and right ends of the power supply lithium battery, the weak infrared ray emitter, the weak infrared image collector and the NVIDIA edge detection device are provided with the power they need respectively.

Images