Elevator control method and control device

Abstract

The present invention relates to an elevator control method, comprising: step 1, obtaining a video stream in a car in real time through a surveillance camera in the car; step 2, extracting a video frame image according to the obtained video stream; step 3, inputting the extracted video frame image into a target detection model and a face detection model for recognition, and outputting a first recognition result and a second recognition result respectively; step 4, judging whether there are only pets / infants and young children but no adults in the first recognition result, then controlling the elevator door to remain open; otherwise, the elevator is not intervened, and judging whether there is a successfully matched face in the second recognition result, if so, triggering a floor button in the car corresponding to the matched face information; if not, controlling the elevator to execute a restricted mode; the present invention also relates to a control device for realizing the above control method. The present invention can realize contactless elevator riding and avoid the occurrence of pets / infants riding the elevator alone.

Description

Technical Field

[0001] The present invention relates to elevator control technology, and in particular to an elevator control method and a control device. Background Art

[0002] With the rapid development of the national economy and the continuous acceleration of urbanization, the number of elevators in my country is also increasing rapidly. Data from the China Elevator Industry Association show that the number of elevators in my country has increased from 1.37 million in 2009 to about 7.0975 million in 2019, with an annual compound growth rate of 16.13%. By the end of 2020, the number of elevators has reached 7.8 million. According to statistics, the number of elevators in the world in 2020 was 19.57 million, and China's elevators accounted for nearly 40% (39.86%) of the world's elevators in use. According to the purpose of the elevator, elevators can be divided into: passenger elevators, freight elevators, medical elevators, sightseeing elevators, vehicle elevators, etc. The elevators that are closely related to our daily life are mainly two types: cabin passenger elevators and freight elevators. The common characteristics of these two elevators include: fixed space size, large flow, high frequency of use, etc. The proportion of elevator use has increased year by year, and with it comes a series of problems such as elevator safety and hygiene.

[0003] In recent years, with the rise of artificial intelligence, more and more industries have actively embraced these technologies, aiming to apply advanced technologies to various traditional industries in order to bring a series of conveniences to people's daily lives. The traditional elevator usage plan is that passengers control the elevator by clicking the button of the floor they want to go to. This traditional click-type operation method is somewhat lagging and also has widespread hygiene problems. Therefore, how to realize the intelligent operation of the elevator and realize the contactless operation of the elevator has been placed in front of elevator practitioners.

[0004] The present invention is proposed in view of the above. Summary of the invention

[0005] An object of the present invention is to overcome the deficiencies of the prior art and provide an elevator control method to improve the safety of elevator use, realize contactless elevator riding, and improve the convenience of elevator use.

[0006] Another object of the present invention is to provide a control device for implementing the above elevator control method.

[0007] In order to achieve the first invention objective, the present invention adopts the following technical solution:

[0008] An elevator control method comprises the following steps:

[0009] Step 1: Obtain the video stream in the car in real time through the surveillance camera in the car;

[0010] Step 2: extracting video frame images according to the acquired video stream;

[0011] Step 3: Input the extracted video frame image into the target detection model and the face detection model for recognition, and output the recognition result;

[0012] Step 4: Control the elevator according to the recognition result;

[0013] In the step 3, the target detection model is at least configured to identify whether the input image contains pets, infants and adults, and output a first recognition result;

[0014] The method for the face detection model to perform recognition is:

[0015] Step a1: Use the face detection model provided by OpenCV to detect whether there is a face in the image, and after detecting the face, locate the face position in the image;

[0016] Step a2: extracting a face image according to the located face position, matching the extracted face with faces pre-stored in a database, and outputting a second recognition result;

[0017] The step 4 includes performing the following steps according to the first recognition result:

[0018] Step S1, determining whether there are pets / infants and adults in the first recognition result; if there are pets / infants but no adults, executing the first intervention instruction; if there are both pets / infants and adults, or there are adults but no pets / infants, not executing the first intervention instruction;

[0019] The first intervention instruction is: controlling the elevator door to remain in an open state and stop the elevator from moving up and down;

[0020] The step 4 further includes performing the following steps according to the second recognition result:

[0021] Step S2: determine whether there is a successfully matched face in the second recognition result. If yes, obtain the floor information corresponding to the face and trigger the floor button corresponding to the floor in the car; if no, control the elevator to execute the restricted mode;

[0022] The restriction mode is to suspend the use authority of the car floor buttons or limit the use floors of the elevator.

[0023] Furthermore, the target detection model distinguishes infants and adults by:

[0024] Step b1: Use the deep learning algorithm to identify the human body using the ssd-mobilenet network combined with the OpenCV library, and select the identified human body with a detection frame;

[0025] Step b2, calculating the height value of the detection frame, if the height value of the detection frame is less than or equal to the preset height value, then outputting the recognition result as an infant, if it is greater, then outputting the recognition result as an adult;

[0026] The selection range of the preset value is 0.5 meter to 1 meter.

[0027] Further, in step three,

[0028] The object detection model is also configured to identify whether an input image contains an electric vehicle;

[0029] The step 4 further includes performing the following steps according to the first recognition result:

[0030] Step S'1, determining whether there is an electric vehicle in the first recognition result, if yes, executing the second intervention instruction, if no, not executing the second intervention instruction;

[0031] The second intervention instruction is: controlling the elevator door to remain open and playing a voice alarm prompt in the car;

[0032] The priority of the second intervention instruction is higher than that of the first intervention instruction, and the elevator preferentially executes the instruction with the higher priority.

[0033] Further, in step three,

[0034] The object detection model is also configured to identify whether there is garbage in the input image;

[0035] The step 4 further includes performing the following steps according to the first recognition result:

[0036] Step S"1, determine whether there is garbage in the first recognition result, if yes, execute step S"2, if no, do not execute;

[0037] Step S"2, determining the retention time of the garbage, if it exceeds the preset time, executing the third intervention instruction, if it does not exceed the preset time, not executing the third intervention instruction;

[0038] The third intervention instruction is: sending a reminder message to the elevator management background or the mobile phone of the maintenance personnel;

[0039] The first intervention instruction has a higher priority than the third intervention instruction.

[0040] Further, the step three also includes inputting the extracted video frame image into a behavior recognition model for recognition, wherein the behavior recognition model is configured to recognize whether the input image contains dangerous behavior / action and output a third recognition result;

[0041] The step 4 further includes performing the following steps according to the third recognition result:

[0042] Step S'1, determining whether there is any dangerous behavior / action in the third recognition result, if yes, executing step S'2, otherwise not executing;

[0043] Step S'2, determining whether the elevator has abnormal vibration, if yes, executing the fourth intervention instruction, if no, not executing the fourth intervention instruction;

[0044] The fourth intervention instruction is: playing a voice alarm prompt in the car and sending a reminder message to the elevator management background or the maintenance personnel's mobile phone;

[0045] The fourth intervention instruction has the same priority as the third intervention instruction.

[0046] Furthermore, the behavior recognition model adopts a deep learning algorithm and uses a resnet50-convLSTM network for behavior recognition;

[0047] The behavior recognition model used for recognition is a trained model, and its training method is:

[0048] Step c1, obtaining Internet surveillance video and cabin surveillance video, and generating video clips;

[0049] Step c2: randomly cropping the video at a fixed time and marking them respectively;

[0050] Step c3, dividing the annotated files into training set, validation set and test set according to a preset ratio;

[0051] Step c4: Apply the resnet50-convLSTM network to train the training set, use the validation set to determine the hyperparameters, and use the test set to verify the performance of the generated model.

[0052] Furthermore, the dangerous behaviors / actions include fighting, jumping, running, falling and door-climbing behaviors / actions.

[0053] Furthermore, the target detection model in step 3 is a trained model, and its training method is:

[0054] Step d1, by obtaining the video in the car, exporting the RGB color image;

[0055] Step d2: For the exported images, select the data with the target and add them to the data set;

[0056] Step d3: Use labelImg software to label the target to be detected and obtain an XML labeling file;

[0057] Step d4: Use the image and the annotation file as training data and input them into the SSD-MobileNet network for training, and finally obtain a trained model file.

[0058] Furthermore, the face detection model in step 3 is a trained model, and its training method is:

[0059] Step e1, obtaining a clear single-frame color face image of the owner or tenant;

[0060] Step e2, performing data enhancement processing on the acquired image to expand a single image into multiple images;

[0061] Step e3: Use the LBPH algorithm to train the image obtained after data enhancement, and complete the training after the model converges to obtain the face detection model.

[0062] In order to achieve the second invention objective, the present invention adopts the following technical solution:

[0063] An elevator control device, used to execute the above control method, the elevator control device comprises:

[0064] A surveillance camera installed in the elevator car with an integrated microphone;

[0065] An image processing module, connected to the surveillance camera, for processing the acquired video;

[0066] AI intelligent development board, connected to the image processing module, for identifying the processed image;

[0067] The elevator control module is used to control the movement of the elevator door and the up and down movement of the elevator;

[0068] The gyroscope is used to obtain the acceleration of the elevator during movement and is connected to the elevator control module;

[0069] The external speaker module is used to play voice prompts and is connected to the elevator control module;

[0070] As well as a communication module and a power module; the AI ​​intelligent development board is connected to the elevator control module through the communication module, and the power module supplies power to other modules.

[0071] The technical solution of the present invention brings the following beneficial effects:

[0072] 1. The present invention can realize contact-free elevator riding and can prevent pets and infants from riding the elevator alone, thereby effectively reducing the occurrence of accidents, greatly providing convenience for passengers and property personnel, and improving the safety and convenience of elevator riding.

[0073] 2. The target detection model and face detection model of the present invention can perform detection and recognition simultaneously, which not only has high detection efficiency, but also the detection models are all deep learning models, with low misdiagnosis probability and fast recognition speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 : Flow chart of Embodiment 1 of the present invention;

[0075] Figure 2 : System diagram of the control device of the present invention;

[0076] Figure 3 : Flow chart of embodiment 2 of the present invention. DETAILED DESCRIPTION

[0077] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0078] Embodiment 1

[0079] like Figure 1 and Figure 2 As shown, an elevator control method comprises the following steps:

[0080] Step 1: Obtain the video stream in the car in real time through the surveillance camera in the car;

[0081] Step 2: extracting a video frame image according to the acquired video stream, wherein the extracted video frame image is an RGB image;

[0082] Step 3: Input the extracted video frame image into the target detection model and the face detection model for recognition, and output the recognition result;

[0083] Step 4: Control the elevator based on the recognition result.

[0084] Specifically, in step 3, the target detection model is configured to identify whether the input image contains a pet, an infant, or an adult, and output a first recognition result;

[0085] The target detection model is a trained model, and its training method is:

[0086] Step d1, by obtaining the video in the car, exporting the RGB color image;

[0087] Step d2: For the exported images, select the data with the target and add them to the data set;

[0088] Step d3: Use labelImg software to label the targets to be detected to obtain an xml labeling file; the targets include pets, infants, adults, electric vehicles and garbage;

[0089] Step d4: Use the image and the annotation file as training data and input them into the SSD-MobileNet network for training, and finally obtain a trained model file.

[0090] Since it is difficult for the target detection model to directly distinguish between infants and adults, the target recognition model in the present invention distinguishes between infants and adults by the following method:

[0091] Step b1: Use the deep learning algorithm to identify the human body using the ssd-mobilenet network combined with the OpenCV library, and select the identified human body with a detection frame;

[0092] Step b2, calculating the height value of the detection frame, if the height value of the detection frame is less than or equal to the preset height value, then outputting the recognition result as an infant, if it is greater, then outputting the recognition result as an adult;

[0093] The selection range of the preset value is 0.5 meter to 1 meter.

[0094] In step 3, the face detection model is configured to recognize the face in the input image and output a second recognition result. Specifically, the face detection model performs recognition in the following manner:

[0095] Step a1: Use the face detection model provided by OpenCV to detect whether there is a face in the image, and after detecting the face, locate the face position in the image;

[0096] Step a2: extract the face image according to the located face position, match the extracted face with the face pre-stored in the database, and output a second recognition result.

[0097] The face detection model is a trained model, and its training method is:

[0098] Step e1, obtaining a clear single-frame color face image of the owner or tenant;

[0099] Step e2, performing data enhancement processing on the acquired image to expand a single image into multiple images;

[0100] Step e3: Use the LBPH algorithm to train the image obtained after data enhancement, and complete the training after the model converges to obtain the face detection model.

[0101] The difference between this face detection model and other face card readers is that it can detect and match faces in the entire car at the same time, without requiring each passenger to scan their face for detection in sequence. This can greatly improve the efficiency of algorithm detection and enhance the passenger's elevator experience.

[0102] Specifically, the step 4 includes performing the following steps according to the first recognition result:

[0103] Step S1, determining whether there are pets / infants and adults in the first recognition result; if there are pets / infants but no adults, executing the first intervention instruction; if there are both pets / infants and adults, or there are adults but no pets / infants, not executing the first intervention instruction;

[0104] The first intervention instruction is: controlling the elevator door to remain in an open state and stop the elevator from moving up and down;

[0105] The step 4 further includes performing the following steps according to the second recognition result:

[0106] Step S2: determine whether there is a successfully matched face in the second recognition result. If yes, obtain the floor information corresponding to the face and trigger the floor button corresponding to the floor in the car; if no, control the elevator to execute the restricted mode;

[0107] The restriction mode is to suspend the use authority of the car floor buttons or limit the use floors of the elevator.

[0108] Preferably, in step S2, when the floor button in the car is triggered according to the face matching result, a voice prompt can be given first. After the button in the car is automatically triggered, the passenger can also cancel it manually and select other floor buttons.

[0109] Preferably, in step 3, the object detection model is further configured to identify whether the input image contains electric vehicles and garbage;

[0110] The step 4 further includes performing the following steps according to the first recognition result:

[0111] Step S'1, determine whether there is an electric vehicle in the first recognition result, if so, execute the second intervention instruction, if not, do not execute the second intervention instruction, the second intervention instruction is: control the elevator door to remain open, and play a voice alarm prompt in the car, the content of the voice alarm prompt can be: electric vehicles are prohibited from entering.

[0112] The priority of the second intervention instruction is higher than that of the first intervention instruction. The elevator executes the instruction with higher priority first. For example, when there is an electric vehicle in the first recognition result, the second intervention instruction is directly executed without considering other intervention instructions.

[0113] The step 4 further includes performing the following steps according to the first recognition result:

[0114] Step S"1, determine whether there is garbage in the first recognition result, if yes, execute step S"2, if no, do not execute;

[0115] Step S"2, determining the retention time of the garbage, if it exceeds the preset time, executing the third intervention instruction, if it does not exceed the preset time, not executing the third intervention instruction;

[0116] The third intervention instruction is: sending a reminder message to the elevator management background or the mobile phone of the maintenance personnel;

[0117] The priority of the first intervention instruction is higher than that of the third intervention instruction, that is, the elevator executes the second intervention instruction first. If there is no second intervention instruction, the first intervention instruction is executed first. If there is no first intervention instruction, the third intervention instruction is executed first. If there is no third intervention instruction, the elevator door operates normally without voice alarm prompt.

[0118] The intervention instructions in the above step S1, step S"1, and step S"2 may be generated simultaneously. When generated simultaneously, they will be executed according to the priority.

[0119] Embodiment 2

[0120] like Figure 3 As shown, the difference between this embodiment and the first embodiment is that:

[0121] In step three, the extracted video frame image is also input into a behavior recognition model for identification. The behavior recognition model is configured to identify whether the input image contains dangerous behaviors / actions and output a third identification result; the dangerous behaviors / actions include fighting, jumping, running, falling and door-climbing behaviors / actions.

[0122] The behavior recognition model adopts a deep learning algorithm and uses a resnet50-convLSTM network for behavior recognition.

[0123] The behavior recognition model used for recognition is a trained model, and its training method is:

[0124] Step c1, obtaining Internet surveillance video and cabin surveillance video, and generating video clips;

[0125] Step c2: randomly cropping the video at a fixed time and marking them respectively;

[0126] Step c3, dividing the annotated files into training set, validation set and test set according to a preset ratio;

[0127] Step c4: Apply the resnet50-convLSTM network to train the training set, use the validation set to determine the hyperparameters, and use the test set to verify the performance of the generated model.

[0128] The step 4 also includes the following steps:

[0129] Step S'1, determining whether there is any dangerous behavior / action in the third recognition result, if yes, executing step S'2, otherwise not executing;

[0130] Step S'2: determine whether the elevator has abnormal vibration, if yes, execute the fourth intervention instruction, if no, do not execute the fourth intervention instruction.

[0131] The fourth intervention instruction is: play a voice alarm prompt in the car, and send a reminder message to the elevator management background or the maintenance personnel's mobile phone; the content of the voice alarm prompt can be multiple, and correspond to the specific identified dangerous behaviors / actions, such as when a fight is identified, a voice prompt can be given: no fighting; when jumping is identified, a voice prompt can be given: no jumping; when running is identified, a voice prompt can be given: no running; when a fall is identified, a helpline number can be played by voice, and when door prying is identified, a voice prompt can be given: no door prying.

[0132] The fourth intervention instruction has the same priority as the third intervention instruction.

[0133] The method for judging whether the elevator has abnormal vibration is to judge the acceleration value of the elevator by reading the parameters of the gyroscope. When the acceleration value is abnormal, it is judged that the elevator has abnormal vibration.

[0134] In order to achieve the second invention objective, the present invention adopts the following technical solution:

[0135] An elevator control device, used to implement the elevator control method in the above-mentioned embodiment 1 or embodiment 2, the elevator control device comprising:

[0136] A surveillance camera installed in the elevator car with an integrated microphone;

[0137] An image processing module, connected to the surveillance camera, for processing the acquired video;

[0138] AI intelligent development board, connected to the image processing module, for identifying the processed image;

[0139] The elevator control module is used to control the movement of the elevator door and the up and down movement of the elevator;

[0140] The gyroscope is used to obtain the acceleration of the elevator during movement and is connected to the elevator control module;

[0141] The external speaker module is used to play voice prompts and is connected to the elevator control module;

[0142] As well as a communication module and a power module; the AI ​​intelligent development board is connected to the elevator control module through the communication module, and the power module supplies power to other modules.

[0143] The monitoring camera is used to obtain the video stream in the car in real time, and transmit the obtained video stream to the image processing module. The image processing module decodes the video, extracts the video frame image, packages it and sends it to the AI ​​intelligent development board. The AI ​​intelligent development board is used to run the target detection model, face detection model and behavior recognition model, and transmit the recognition result to the elevator control module. The elevator control module generates control instructions according to the recognition result and controls the operation of the elevator.

[0144] The above is an embodiment of the present invention. It should be pointed out that for ordinary technicians in this field, various modifications and improvements can be made without departing from the principle of the present invention, which should also be regarded as the protection scope of the present invention.

Claims

1. An elevator control method, The following steps are involved: Step 1: Obtain the video stream in the car in real time through the surveillance camera in the car; Step 2: extracting video frame images according to the acquired video stream; Step 3: Input the extracted video frame image into the target detection model and the face detection model for recognition, and output the recognition result; Step 4: Control the elevator according to the recognition result; It is characterized in that in the step three, the target detection model is at least configured to identify whether the input image contains pets, infants and adults, and output a first recognition result; The method for the face detection model to perform recognition is: Step a1: Use the face detection model provided by OpenCV to detect whether there is a face in the image, and after detecting the face, locate the face position in the image; Step a2: extracting a face image according to the located face position, matching the extracted face with faces pre-stored in a database, and outputting a second recognition result; The step 4 includes performing the following steps according to the first recognition result: Step S1, determining whether there are pets, infants and adults in the first recognition result; if there are pets or infants but no adults, the first intervention instruction is executed; if there are pets and / or infants and adults, or if there are adults but no pets and infants, the first intervention instruction is not executed; The first intervention instruction is: controlling the elevator door to remain in an open state and stop the elevator from moving up and down; The step 4 further includes performing the following steps according to the second recognition result: Step S2: determine whether there is a successfully matched face in the second recognition result. If yes, obtain the floor information corresponding to the face and trigger the floor button corresponding to the floor in the car; if no, control the elevator to execute the restricted mode; The restriction mode is to suspend the use of the car floor button or limit the use of the elevator floor; The step three also includes inputting the extracted video frame image into a behavior recognition model for recognition, wherein the behavior recognition model is configured to recognize whether the input image contains dangerous behavior / action and output a third recognition result; The step 4 further includes performing the following steps according to the third recognition result: Step S'1, determining whether there is any dangerous behavior / action in the third recognition result, if yes, executing step S'2, otherwise not executing; Step S'2, determining whether the elevator has abnormal vibration, if yes, executing the fourth intervention instruction, if no, not executing the fourth intervention instruction; The fourth intervention instruction is: playing a voice alarm prompt in the car, and sending a reminder message to the elevator management background or the maintenance personnel's mobile phone.

2. An elevator control method according to claim 1, It is characterized in that The object detection model distinguishes infants from adults by: Step b1: Use the deep learning algorithm to identify the human body using the ssd-mobilenet network combined with the OpenCV library, and select the identified human body with a detection frame; Step b2, calculating the height value of the detection frame, if the height value of the detection frame is less than or equal to the preset height value, then outputting the recognition result as an infant, if it is greater, then outputting the recognition result as an adult; The selection range of the preset height value is 0.5 meter to 1 meter.

3. An elevator control method according to claim 1, It is characterized in that In the step three, The object detection model is also configured to identify whether an input image contains an electric vehicle; The step 4 further includes performing the following steps according to the first recognition result: Step S'1, determining whether there is an electric vehicle in the first recognition result, if yes, executing the second intervention instruction, if no, not executing the second intervention instruction; The second intervention instruction is: controlling the elevator door to remain open and playing a voice alarm prompt in the car; The priority of the second intervention instruction is higher than that of the first intervention instruction, and the elevator preferentially executes the instruction with the higher priority.

4. An elevator control method according to claim 3, It is characterized in that In the step three, The object detection model is also configured to identify whether there is garbage in the input image; Step S"1, determine whether there is garbage in the first recognition result, if yes, execute step S"2, if no, do not execute; Step S"2, determining the retention time of the garbage, if it exceeds the preset time, executing the third intervention instruction, if it does not exceed the preset time, not executing the third intervention instruction; The third intervention instruction is: sending a reminder message to the elevator management background or the mobile phone of the maintenance personnel; The first intervention instruction has a higher priority than the third intervention instruction.

5. An elevator control method according to claim 4, It is characterized in that The fourth intervention instruction has the same priority as the third intervention instruction.

6. An elevator control method according to claim 1, It is characterized in that The behavior recognition model adopts a deep learning algorithm and uses a resnet50-convLSTM network for behavior recognition; The behavior recognition model used for recognition is a trained model, and its training method is: Step c1, obtaining Internet surveillance video and cabin surveillance video, and generating video clips; Step c2: randomly cropping the video at a fixed time and marking them respectively; Step c3, dividing the annotated files into training set, validation set and test set according to a preset ratio; Step c4: Apply the resnet50-convLSTM network to train the training set, use the validation set to determine the hyperparameters, and use the test set to verify the performance of the generated model.

7. An elevator control method according to claim 6, It is characterized in that The dangerous behaviors / actions include fighting, jumping, running, falling and door-climbing behaviors and actions.

8. An elevator control method according to claim 4, It is characterized in that The target detection model in step 3 is a trained model, and its training method is: Step d1, by obtaining the video in the car, exporting the RGB color image; Step d2: For the exported images, select the data with the target and add them to the data set; Step d3: Use labelImg software to label the target to be detected and obtain an XML labeling file; Step d4: Use the image and the annotation file as training data and input them into the SSD-MobileNet network for training, and finally obtain a trained model file.

9. An elevator control method according to claim 1, It is characterized in that The face detection model in step 3 is a trained model, and its training method is: Step e1, obtaining a clear single-frame color face image of the owner or tenant; Step e2, performing data enhancement processing on the acquired image to expand a single image into multiple images; Step e3: Use the LBPH algorithm to train the image obtained after data enhancement, and complete the training after the model converges to obtain the face detection model.

10. An elevator control device, It is characterized in that For executing the control method according to any one of claims 1 to 9, the elevator control device comprises: A surveillance camera installed in the elevator car with an integrated microphone; An image processing module, connected to the surveillance camera, for processing the acquired video; AI intelligent development board, connected to the image processing module, for identifying the processed image; The elevator control module is used to control the movement of the elevator door and the up and down movement of the elevator; The gyroscope is used to obtain the acceleration of the elevator during movement and is connected to the elevator control module; The external speaker module is used to play voice prompts and is connected to the elevator control module; As well as a communication module and a power module; the AI ​​intelligent development board is connected to the elevator control module through the communication module, and the power module supplies power to other modules.

Images