Elevator light curtain control method, light curtain system and elevator based on machine vision

Abstract

The present invention discloses an elevator light curtain control method, a light curtain system and an elevator based on machine vision, and relates to the technical field of elevator light curtain control. It includes: setting a monitoring area to capture video images and obtaining multiple image data sets through image processing; performing object detection, area detection, anomaly detection and parallel operation analysis based on the image data set, generating corresponding control signals and controlling the operation and scheduling of the elevator and the elevator light curtain; generating control decisions through safety self-check and establishing a light curtain control optimization model, and optimizing the operation mode and control means of the elevator light curtain. The present invention uses machine vision technology to monitor the elevator light curtain in real time, and generates corresponding control signals through object detection, area detection, abnormality detection of the elevator light curtain and parallel operation analysis of the elevator to regulate the elevator and the elevator light curtain, thereby improving the accuracy of the elevator light curtain in identifying objects and the reliability of the elevator control, thereby effectively improving the safety and stability of the elevator.

Description

Technical Field

[0001] The present invention belongs to the technical field of elevator light curtain control, and in particular relates to an elevator light curtain control method based on machine vision, a light curtain system and an elevator. Background Art

[0002] The elevator light curtain is a safety device installed at the elevator door, usually composed of infrared sensors or lasers. Its main function is to detect whether there are people or objects at the elevator door to avoid door pinching or other safety accidents. Although the elevator light curtain plays an important role in the elevator control system, it still faces some problems and challenges in actual use, such as: false alarms caused by improper light curtain sensitivity settings or external light interference, long light curtain control time due to slow response speed after detecting people or objects, elevator door opening errors due to incomplete sensor detection or obstructions, causing people pinching, and security vulnerabilities caused by equipment failure / aging / untimely maintenance, etc.

[0003] Machine vision is a technology that uses computers and special equipment to process and interpret images or videos. Integrating machine vision into the elevator light curtain control system can use cameras and image processing technology to monitor the status of the light curtain at the elevator door, thereby achieving control of the elevator door switch. This avoids safety hazards caused by obstructions to a certain extent, and also improves the automation level, safety and reliability of the elevator.

[0004] However, there are still many problems with the elevator light curtain control system based on machine vision. For example, the various complex environmental conditions at the elevator door (such as light at different heights, various background noises / radiation from other electronic equipment, reflections at the door, etc.) may affect the recognition accuracy and stability of machine vision; the dynamic changes of the elevator door cause the light curtain system to be unable to quickly and accurately distinguish between obstructed objects and normal passers-by; how to avoid mutual interference when multiple elevators are used in parallel; and real-time and safety issues caused by untimely response. Summary of the invention

[0005] The object of the present invention is to provide an elevator light curtain control method, a light curtain system and an elevator based on machine vision, which can be achieved by the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides an elevator light curtain control method based on machine vision, comprising the following steps:

[0007] S1, install a visual sensor at the elevator door and set up a monitoring area of ​​the visual light curtain to capture the video image of the monitoring area in real time;

[0008] S2, performing image processing on the video image and obtaining a first image data set, a second image data set and a third image data set according to image features;

[0009] S3, performing object detection and region detection based on the first image data set, and generating a first control signal and a second control signal respectively according to the detection results;

[0010] S4, adjusting the sensitivity of the elevator light curtain and controlling the opening and closing of the elevator door according to the first control signal; adjusting the range of the monitoring area according to the second control signal;

[0011] S5, performing abnormality detection of the elevator light curtain based on the second image data set, and generating an abnormality signal according to the detection result;

[0012] S6, performing abnormality processing and fault analysis according to the abnormal signal;

[0013] S7, performing elevator parallel operation analysis based on the third image data set, and generating a third control signal according to the analysis result;

[0014] S8, coordinate control and dispatch the elevator operation according to the third control signal;

[0015] S9, obtain the operation data of the elevator and the elevator light curtain and perform safety self-inspection, and perform light curtain protection according to the self-inspection results;

[0016] S10, establishing a light curtain control optimization model based on the video image and using it to analyze the real-time state of the elevator light curtain, generating a control decision according to the analysis result and using it to optimize the operation mode and control means of the elevator light curtain;

[0017] Among them, the features in the first image data set include elevator doors, elevator waiting areas, elevator cars and objects; the features in the second image data set include elevator doors, elevator cars, elevator waiting areas and elevator floors; the features in the third image data set include objects, elevator cars, elevator waiting areas and elevator floors.

[0018] Preferably, object detection is performed in step S3, specifically including the following contents:

[0019] S31, using a fast multi-target detection algorithm to quickly detect the first image data set to obtain a target object;

[0020] S32, tracking the target object using a target tracking algorithm to obtain the position of the target object and the motion trajectory of the target object;

[0021] S33, judging the moving intention of the target object according to the position of the target object and the moving trajectory of the target object;

[0022] S34, obtaining the running state of the elevator, and judging whether the target object has a safety risk according to the moving intention, and generating a first control signal according to the judgment result;

[0023] The fast multi-target detection algorithm in step S31 specifically includes the following contents:

[0024] S311, selecting a feature map from the first image data set and adjusting the pixel size of the feature map;

[0025] S312, inputting the feature map into a network model, wherein the network model is used to divide the input feature map into N×N grids, and if the center point of a target in the feature map falls into a grid, the grid is responsible for detecting the target;

[0026] S313: Each grid predicts M corresponding bounding boxes. Each bounding box consists of 6 prediction values, including the center coordinates of the bounding box (x, y), the width and height of the bounding box (w, h), the category confidence X of the target in the bounding box, and the error between the bounding box and the anchor box. ;

[0027] S314, during testing, the category information predicted by each grid is multiplied by the category confidence to obtain the confidence score that the target of the bounding box belongs to a certain category. The calculation formula is expressed as:

[0028] ;

[0029] represents the conditional probability, which is the probability of belonging to the i-th category under the condition that the bounding box exists; Indicates that the bounding box belongs to the i-th category under the condition that there is an object;

[0030] Wherein, the network model is a VOLO network structure; the category confidence X is expressed as:

[0031] ;

[0032] Indicates whether there is a target in the grid. If the center point of the target bounding box does not fall into the grid, it is 0, otherwise it is 1; Represents the intersection-over-union ratio of the predicted bounding box and the true bounding box;

[0033] The anchor frame is a frame of fixed size and aspect ratio generated at each pixel of the feature map;

[0034] error The processing formula includes:

[0035] ; ;

[0036] ; ;

[0037] Represents the error between the central horizontal coordinate of the jth anchor box and the predicted value of the bounding box; Represents the error between the central ordinate of the jth anchor box and the predicted value of the bounding box; Represents the error between the width of the jth anchor box and the predicted value of the bounding box; Represents the error between the height of the jth anchor box and the predicted value of the bounding box; represents the jth anchor box; represents the i-th bounding box.

[0038] Preferably, the target tracking algorithm in step S32 specifically includes the following contents:

[0039] S321, inputting the feature map into the target detector in the fast multi-target detection algorithm and obtaining the bounding box of the first frame and its category confidence;

[0040] S322, setting high and low category confidence thresholds respectively according to the detection results, and dividing the bounding box into a high-score bounding box and a low-score bounding box based on the thresholds;

[0041] S323, if the score of the bounding box is higher than the threshold of the high-scoring bounding box, the bounding box is matched with the target trajectory box predicted by the Kalman filter through the Hungarian algorithm for the first time. If the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set. If the match is not successful, the unmatched trajectory is temporarily stored to wait for the second match;

[0042] S324, if the score of the bounding box is lower than the threshold of the high-scoring bounding box and higher than the threshold of the low-scoring bounding box, match it with the temporarily stored unmatched trajectory. If the match is successful, update its trajectory to the current frame trajectory set. If the match is still unsuccessful, delete the low-scoring bounding box, and determine whether the unmatched trajectory meets the deletion condition. According to the determination result, delete it or merge it into the current frame trajectory set and wait for the next frame to match.

[0043] S325, if the bounding box of the last predicted target still fails to match, it is identified as a newly appeared bounding box, and a new track and a new target are assigned to it.

[0044] Preferably, determining the moving intention of the target object in step S33 specifically includes the following contents:

[0045] S331, respectively obtaining the center coordinates of the target object position in the Nth frame image and the N-1th frame image, calculating the center coordinate difference between two adjacent frames of image and obtaining the displacement of the target object;

[0046] S332, calculating the moving speed of the target object in two adjacent frames of images;

[0047] S333, obtaining a moving direction of the target object according to the moving trajectory of the target object;

[0048] S334, judging the moving intention of the target object according to the moving speed and moving direction of the target object, if the moving intention is that the target object is heading towards the elevator, executing step S34, otherwise executing step S335;

[0049] S335, delete the N-1th frame image, obtain the N+1th frame image and return to execute step S331.

[0050] Preferably, region detection is performed in step S3, specifically including the following contents:

[0051] S3A, acquiring the trajectory change of the elevator door, and the image information of the elevator waiting area and the elevator floor based on the first image data set;

[0052] S3B, setting a first pre-monitoring area according to the trajectory change of the elevator door;

[0053] S3C, setting a second pre-monitoring area according to the image information of the elevator waiting area and the elevator floor;

[0054] S3D, respectively calculating the area of ​​the first pre-monitoring area and the second pre-monitoring area, and adding the areas of the two areas as the range of the visual light curtain monitoring area;

[0055] S3E, generating a second control signal after determining the range of the monitoring area;

[0056] Among them, the elevator door includes the elevator floor door and the elevator car door;

[0057] Step S3B specifically includes the following contents:

[0058] S3B1, when the opening and closing speeds of the elevator floor door and the elevator car door are the same, the length of the area when the elevator door is not closed is set as the length of the first pre-monitoring area, and the length of the area when the elevator door is fully opened is set as the width of the first pre-monitoring area, and the maximum area of ​​the first pre-monitoring area is calculated;

[0059] S3B2, when the opening and closing speeds of the elevator floor door and the elevator car door are inconsistent, the maximum area of ​​the first pre-monitoring area is calculated with the faster opening and closing speed as the reference target, specifically:

[0060] When the opening and closing speed of the elevator car door is greater than the opening and closing speed of the elevator floor door, the maximum area of ​​the first pre-monitoring area is: the length of the elevator car door track change × the width of the first pre-monitoring area and the length of the elevator car door track change × the width of the elevator floor door track change and the sum of the intermediate gap area;

[0061] When the opening and closing speed of the elevator floor door is greater than the opening and closing speed of the elevator car door, the maximum area of ​​the first pre-monitoring area is: the length of the elevator floor door trajectory change × the width of the first pre-monitoring area and the length of the elevator floor door trajectory change × the width of the elevator car door trajectory change and the sum of the intermediate gap area.

[0062] Preferably, abnormality detection of the elevator light curtain is performed in step S5, which specifically includes the following contents:

[0063] S51, detecting the door opening and closing state and the running state of the elevator to obtain a first abnormality detection result;

[0064] S52, detecting the components and circuit interfaces of the elevator light curtain to obtain a second abnormality detection result;

[0065] S53, performing abnormal behavior detection on the elevator waiting area and the elevator floor to obtain a third abnormal result;

[0066] S54, combining the first abnormality detection result, the second abnormality detection result and the third abnormality detection result together as a total abnormality detection result output and generating an abnormality signal;

[0067] The detection of the door opening and closing state in step S51 further includes:

[0068] S511, acquiring image features of an elevator door and motion characteristics of an elevator car from the second image data set;

[0069] S512, analyzing the image features of the elevator door, if the elevator floor door and / or the elevator car door are in an open state when the elevator car moves up and down, it is determined that the elevator door has a closing failure; if the elevator floor door and / or the elevator car door are in a closed state for a long time when the elevator car stops at a floor, it is determined that the elevator door has a opening failure;

[0070] S513, analyzing the movement characteristics of the elevator car. If the bottom of the elevator car is not parallel to the floor when the elevator car stops at the floor / or there is a height difference, it is determined that there is a parking failure of the elevator car; if the running speed of the elevator car exceeds the set speed threshold when it moves up and down, it is determined that there is an operation failure of the elevator car.

[0071] Preferably, in step S7, an elevator parallel operation analysis is performed, which specifically includes the following contents:

[0072] S71, acquiring an image from the third image data set and converting it into a grayscale image, performing binarization processing and feature extraction to obtain a human feature sample;

[0073] S72, training a classifier based on the human feature samples and performing binary judgment on the head and non-head, and adding the judgment results to obtain the number of people inside and outside the elevator on the same floor;

[0074] S73, calculating the total number of people in the elevator cars, elevator waiting areas, and elevator floors of multiple elevators on the same floor and counting the total number of people on multiple floors, and analyzing the remaining capacity inside the multiple elevators;

[0075] S74, setting the elevator response time between two adjacent floors of the same elevator; if there are multiple elevators, setting multiple different elevator response times;

[0076] S75, calculating the stop time of the elevator at each floor according to the total number of people on the multiple floors and the remaining capacity inside the multiple elevators; calculating the time for the elevator to arrive at the response floor according to the floor position of the elevator stop and the set elevator response time;

[0077] S76, performing comprehensive calculation based on the elevator's parking time and the time it takes for the elevator to arrive at the response floor, generating an elevator coordination decision and outputting an elevator dispatching instruction;

[0078] S77, coordinating and scheduling the parallel operation of multiple elevators according to the elevator scheduling instruction.

[0079] Preferably, a safety self-check is performed in step S9, specifically including the following contents:

[0080] S91, detect the current state of the elevator according to the operation data of the elevator and determine whether there is a light curtain blocking signal in the current state, and generate a control command to stop the transmitting end of the elevator light curtain according to the determination result, specifically:

[0081] If the elevator is currently in the parked state and the door is closed and there is no call signal for the current elevator, determine whether there is a light curtain blocking signal in the door closed state. If not, generate and send a control command; otherwise, re-detect the current state of the elevator;

[0082] S92, the receiving end of the elevator light curtain outputs a light curtain status signal and determines whether it is a light curtain blocking signal. If the light curtain status signal is a light curtain blocking signal, the elevator light curtain is not faulty, otherwise the elevator light curtain is faulty;

[0083] S93, after determining that the elevator light curtain fails, obtaining the real-time distance between the elevator light curtain transmitting end and the elevator light curtain receiving end according to the operation data of the elevator light curtain;

[0084] S94, determining a distance threshold based on the distance between the receiving ends and the transmission angle of the transmitting end;

[0085] S95, compare the real-time distance with the distance threshold and determine the threshold interval in which the real-time distance is located, and then control the scanning mode of the elevator light curtain based on the threshold interval, specifically:

[0086] If the real-time distance is within the first threshold interval, a first scanning mode is adopted;

[0087] If the real-time distance is within the second threshold interval, a second scanning mode is adopted;

[0088] If the real-time distance is within the third threshold interval, a third scanning mode is adopted;

[0089] The distance threshold includes a first distance threshold and a second distance threshold; the first threshold interval is: lower than the first distance threshold; the second threshold interval is: between the first distance threshold and the second distance threshold; the third threshold interval is: higher than the second distance threshold;

[0090] The first scanning method is: scanning with a single-point light beam;

[0091] The second scanning method is: scanning with a combined light beam;

[0092] The third scanning method is: scanning by intermittently alternating single-point light beam and combined light beam.

[0093] In a second aspect, an embodiment of the present application provides a machine vision-based light curtain system, which is applied to the elevator light curtain control method as described above, including a visual light curtain monitoring module, a video image processing module, an elevator light curtain control module, an abnormality detection processing module, a parallel operation scheduling module, and a safety self-check optimization module;

[0094] The visual light curtain monitoring module installs a visual sensor at the elevator door and sets the monitoring area of ​​the visual light curtain to capture the video image of the monitoring area in real time;

[0095] The video image processing module performs image processing on the video image and obtains a first image data set, a second image data set and a third image data set according to image features;

[0096] The elevator light curtain control module performs object detection and area detection based on the first image data set, and generates a first control signal and a second control signal according to the detection results; adjusts the sensitivity of the elevator light curtain and controls the opening and closing of the elevator door according to the first control signal; and adjusts the range of the monitoring area according to the second control signal;

[0097] The abnormality detection processing module performs abnormality detection of the elevator light curtain based on the second image data set, generates an abnormality signal according to the detection result; performs abnormality processing and fault analysis according to the abnormal signal;

[0098] The parallel operation scheduling module performs elevator parallel operation analysis based on the third image data set, generates a third control signal according to the analysis result, and coordinates and schedules the elevator operation according to the third control signal;

[0099] The safety self-check optimization module obtains the operation data of the elevator and the elevator light curtain and performs safety self-check, and performs light curtain protection according to the self-check results; establishes a light curtain control optimization model based on the video image and uses it to analyze the real-time status of the elevator light curtain, generates a control decision according to the analysis result and uses it to optimize the operation mode and control means of the elevator light curtain;

[0100] Among them, the features in the first image data set include elevator doors, elevator waiting areas, elevator cars and objects; the features in the second image data set include elevator doors, elevator cars, elevator waiting areas and elevator floors; the features in the third image data set include objects, elevator cars, elevator waiting areas and elevator floors.

[0101] In a third aspect, an embodiment of the present application provides an elevator based on machine vision, wherein the elevator adopts the elevator light curtain control method as described above.

[0102] The beneficial effects of the present invention are:

[0103] (1) The present invention utilizes machine vision technology to monitor the elevator light curtain in real time, and divides the obtained video image into multiple image information sets to perform object detection, area detection, abnormal detection of the elevator light curtain, and analysis of elevator parallel operation, and generates corresponding control signals. Different control signals are then used to perform different operations and adjustments on the elevator and the elevator light curtain, thereby improving the accuracy of the elevator light curtain in identifying objects and the reliability of the elevator control, thereby effectively improving the safety and stability of the elevator; the present invention also improves the operating efficiency and safety performance of the elevator and the elevator light curtain through safety self-checking and establishing a light curtain control optimization model, thereby comprehensively improving the control performance of the elevator light curtain.

[0104] (2) The present invention first installs a visual sensor at the elevator door and sets a monitoring area of ​​the visual light curtain to capture the video image of the monitoring area in real time, and then processes the video image to obtain a first image data set, a second image data set and a third image data set respectively; performs object detection and area detection based on the first image data set, generates a first control signal and a second control signal respectively according to the detection results, and adjusts the sensitivity of the elevator light curtain and controls the opening and closing of the elevator door according to the first control signal, thereby avoiding the occurrence of the elevator door pinching incident; also adjusts the range of the monitoring area according to the second control signal, controls the visual monitoring system to re-divide the monitoring area; and then performs elevator light curtain detection based on the second image data set. The system detects abnormalities of the light curtain, generates abnormal signals according to the detection results, and performs abnormal processing and fault analysis according to the abnormal signals; then analyzes the parallel operation of the elevators through the third image data set, generates a third control signal according to the analysis results, and coordinates the operation of the elevator with the third control signal. Control and dispatch; finally, a safety self-check is performed by obtaining the operation data of the elevator and the elevator light curtain, the working state is switched according to the self-check results, and the light curtain protection is performed, and a light curtain control optimization model is established based on the video image and the real-time state of the elevator light curtain is analyzed, and control decisions are further generated and the operation and control of the elevator light curtain are optimized according to the generated decisions, thereby improving the reliability, real-timeness and safety of the elevator light curtain recognition.

[0105] (3) The present invention adopts a fast multi-target detection algorithm to perform object detection, and allows the prediction classification and prediction position to be performed simultaneously in a regression manner, which greatly reduces the time required for detection. At the same time, in order to solve the problem of multiple targets and objects of different sizes, this embodiment adopts the VOLO network structure to give the classification results and precise positions of the targets in a regression manner, so that the network can cope with objects of different sizes, increases the detection recall rate and is easy to train.

[0106] (4) The present invention avoids safety risks such as pinching of the target object (passenger) by judging the moving intention of the target object. On the basis of the above-mentioned target detection and target tracking, the accuracy of target detection and target motion judgment is achieved, thereby further improving the reliability of light curtain detection and the safety of passengers riding the elevator; at the same time, the range of the monitoring area is calculated by analyzing the trajectory changes of the elevator door, and dynamically adjusted, thereby ensuring the real-time, accuracy and reliability of the elevator light curtain detection.

[0107] (5) The present invention calculates the number of people on each floor, sets the response time of multiple elevators, and the floor buttons pressed by passengers inside and outside the elevators, comprehensively analyzes the operating conditions of multiple elevators, and comprehensively dispatches the parallel operation of multiple elevators based on the number of people in the elevators and the response time, thereby avoiding mutual interference when multiple elevators are used in parallel and real-time and safety problems caused by untimely response, thereby greatly improving the safety of elevators and the real-time performance of elevator operation.

[0108] (6) The present invention first determines whether the elevator light curtain is faulty / failed. If a fault occurs, the scanning mode is further switched according to the distance between the transmitting end and the receiving end, thereby ensuring that different modes can be used to ensure the operation of the elevator light curtain when the elevator light curtain fails or fails, thereby achieving stability and safety in the operation of the elevator light curtain. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] For better understanding and implementation, the technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0110] Figure 1 A flowchart of a method for controlling an elevator light curtain based on machine vision provided in an embodiment of the present application;

[0111] Figure 2 A schematic diagram of the structure of a machine vision-based light curtain system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0112] In order to further explain the technical means and effects taken by the present invention to achieve the predetermined invention purpose, exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of methods and systems consistent with some aspects of the present application as detailed in the attached claims.

[0113] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to any or all possible combinations of one or more associated listed items.

[0114] The specific implementation methods, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments. Example 1

[0115] See also Figure 1 The present application embodiment provides an elevator light curtain control method based on machine vision, comprising the following steps:

[0116] S1, install a visual sensor at the elevator door and set up a monitoring area of ​​the visual light curtain to capture the video image of the monitoring area in real time;

[0117] S2, performing image processing on the video image and obtaining a first image data set, a second image data set and a third image data set according to image features;

[0118] S3, performing object detection and region detection based on the first image data set, and generating a first control signal and a second control signal respectively according to the detection results;

[0119] S4, adjusting the sensitivity of the elevator light curtain and controlling the opening and closing of the elevator door according to the first control signal; adjusting the range of the monitoring area according to the second control signal;

[0120] S5, performing abnormality detection of the elevator light curtain based on the second image data set, and generating an abnormality signal according to the detection result;

[0121] S6, performing abnormality processing and fault analysis according to the abnormal signal;

[0122] S7, performing elevator parallel operation analysis based on the third image data set, and generating a third control signal according to the analysis result;

[0123] S8, coordinate control and dispatch the elevator operation according to the third control signal;

[0124] S9, obtain the operation data of the elevator and the elevator light curtain and perform safety self-inspection, and perform light curtain protection according to the self-inspection results;

[0125] S10, establishing a light curtain control optimization model based on the video image and using it to analyze the real-time state of the elevator light curtain, generating a control decision according to the analysis result and using it to optimize the operation mode and control means of the elevator light curtain;

[0126] Among them, the features in the first image data set include elevator doors, elevator waiting areas, elevator cars and objects; the features in the second image data set include elevator doors, elevator cars, elevator waiting areas and elevator floors; the features in the third image data set include objects, elevator cars, elevator waiting areas and elevator floors.

[0127] Specifically, the existing elevator light curtain may cause false alarms due to various complex environmental conditions (such as light, noise, reflection, etc.) and improper sensitivity settings. The light curtain system may not be able to quickly and accurately distinguish between obstructed objects and normal passengers due to dynamic changes in the elevator door. The recognition ability may be reduced due to elevator or light curtain failures, resulting in false detection and missed detection. When multiple elevators are used in parallel, they may cause mutual interference, which may cause loopholes in the detection of the elevator light curtain and thus make its reliability and safety unable to be guaranteed. Therefore, in order to solve the above problems, the present application provides a control method for an elevator light curtain, firstly installing a visual sensor at the elevator door and setting a monitoring area of ​​the visual light curtain to capture the video image of the monitoring area in real time, and then processing the video image to obtain a first image data set, a second image data set, and a third image data set respectively; performing object detection and area detection based on the first image data set, generating a first control signal and a second control signal respectively according to the detection results, and adjusting the sensitivity of the elevator light curtain and controlling the opening and closing of the elevator door according to the first control signal, thereby avoiding the occurrence of elevator door pinching incidents; and adjusting the range of the monitoring area according to the second control signal to control the visual monitoring system to re- The monitoring area is divided; based on the second image data set, an abnormality detection of the elevator light curtain is performed, an abnormality signal is generated according to the detection result, and abnormality processing and fault analysis are performed according to the abnormal signal; then, the parallel operation of the elevator is analyzed through the third image data set, and a third control signal is generated according to the analysis result, and the operation of the elevator is coordinated and controlled and dispatched by the third control signal; finally, a safety self-inspection is performed by obtaining the operation data of the elevator and the elevator light curtain, and the working state is switched and the light curtain protection is performed according to the self-inspection result, and a light curtain control optimization model is established based on the video image and the real-time state of the elevator light curtain is analyzed, and a control decision is further generated and the operation and control of the elevator light curtain are optimized according to the generated decision. The present application utilizes machine vision technology to monitor the elevator light curtain in real time, and divides the obtained video image into multiple image information sets to perform object detection, area detection, abnormal detection of the elevator light curtain, and analysis of elevator parallel operation, and generates corresponding control signals, and then uses different control signals to perform different operations and adjustments on the elevator and the elevator light curtain, thereby improving the accuracy of the elevator light curtain in identifying objects and the reliability of the elevator control, thereby effectively improving the safety and stability of the elevator; it also improves the operating efficiency and safety performance of the elevator and the elevator light curtain through safety self-inspection and the establishment of a light curtain control optimization model, thereby comprehensively improving the control performance of the elevator light curtain.

[0128] It should be noted that the first image data set is used to detect whether there is an object in the monitoring area and if so, what type of object it is and what properties it has (such as size, moving direction, living / non-living object, moving speed, etc.). It also provides data support for subsequent adjustment of the scope of the monitoring area through area detection; the second image data set is used to perform abnormal detection on the elevator light curtain, and find out the abnormal conditions of the elevator light curtain and the abnormal operations of the elevator caused by detection; and the third image data set is used to analyze the parallel operation of the elevator, so as to provide data support for subsequent coordinated control; the video image content contained in the above three image data sets of this embodiment will be different due to different functions, but the most critical thing is the different functions and effects of the three image data sets.

[0129] In an embodiment provided in the present application, image processing is performed in step S2, including the following contents:

[0130] Image preprocessing: remove noise from video images and smooth the images to improve the results of subsequent processing steps; convert video images from RGB color space to other color spaces (such as grayscale, HSV, etc.) to facilitate processing of specific tasks; resize video images;

[0131] Image enhancement and restoration: Improve the quality and visual effects of images, such as enhancing contrast and sharpening images; repair defects or damaged parts in images, such as removing occluded objects and filling missing areas; select mask smoothing algorithms for filtering operations, histogram equalization, and linear broadening of image contrast;

[0132] Image segmentation: By identifying and analyzing the features in the video image, it is divided into several regions; the features in each region are similar;

[0133] Image extraction and screening: extract features from the segmented regional images, classify and screen the regional images according to the features, and group them into three types of image data sets (first image data set, second image data set, and third image data set).

[0134] Specifically, this embodiment performs image preprocessing on video images to make the images smoother, and performs image enhancement and restoration to make the images more complete and of higher quality; divides images with similar features into multiple regions through image segmentation; and selects and groups several regional images into three types of image data sets according to actual needs through image extraction and screening. This embodiment provides data support for subsequent detection and analysis based on different image data sets through image processing.

[0135] In an embodiment provided in the present application, object detection is performed in step S3, which specifically includes the following contents:

[0136] S31, using a fast multi-target detection algorithm to quickly detect the first image data set to obtain a target object;

[0137] S32, tracking the target object using a target tracking algorithm to obtain the position of the target object and the motion trajectory of the target object;

[0138] S33, judging the moving intention of the target object according to the position of the target object and the moving trajectory of the target object;

[0139] S34, obtaining the running state of the elevator, and judging whether the target object has a safety risk according to the moving intention, and generating a first control signal according to the judgment result;

[0140] The fast multi-target detection algorithm in step S31 specifically includes the following contents:

[0141] S311, selecting a feature map from the first image data set and adjusting the pixel size of the feature map;

[0142] S312, inputting the feature map into a network model, wherein the network model is used to divide the input feature map into N×N grids, and if the center point of a target in the feature map falls into a grid, the grid is responsible for detecting the target;

[0143] S313: Each grid predicts M corresponding bounding boxes. Each bounding box consists of 6 prediction values, including the center coordinates (x, y) of the bounding box, the width and height (w, h) of the bounding box, the category confidence X of the target in the bounding box, and the error between the bounding box and the anchor box. ;

[0144] S314, during testing, the category information predicted by each grid is multiplied by the category confidence to obtain the confidence score that the target of the bounding box belongs to a certain category. The calculation formula is expressed as:

[0145] ;

[0146] represents the conditional probability, which is the probability of belonging to the i-th category under the condition that the bounding box exists; Indicates that the bounding box belongs to the i-th category under the condition that there is an object;

[0147] Wherein, the network model is a VOLO network structure; the category confidence X is expressed as:

[0148] ;

[0149] Indicates whether there is a target in the grid. If the center point of the target bounding box does not fall into the grid, it is 0, otherwise it is 1; Represents the intersection-over-union ratio of the predicted bounding box and the true bounding box;

[0150] The anchor frame is a frame of fixed size and aspect ratio generated at each pixel of the feature map;

[0151] error The processing formula includes:

[0152] ; ;

[0153] ; ;

[0154] Represents the error between the central horizontal coordinate of the jth anchor box and the predicted value of the bounding box; Represents the error between the central ordinate of the jth anchor box and the predicted value of the bounding box; Represents the error between the width of the jth anchor box and the predicted value of the bounding box; Represents the error between the height of the jth anchor box and the predicted value of the bounding box; represents the jth anchor box; represents the i-th bounding box.

[0155] Specifically, this embodiment adopts a fast multi-target detection algorithm to perform object detection, and uses regression to simultaneously predict classification and predicted position, which greatly reduces the time required for detection. At the same time, in order to solve the problem of multiple targets and objects of different sizes, this embodiment uses the VOLO network structure to give the classification results and precise positions of the targets in a regression manner, so that the network can cope with objects of different sizes, increases the recall rate of detection and is easy to train.

[0156] It should be noted that the VOLO network structure used in this embodiment is a network structure based on the YOLO (You Look Only Once) algorithm and using regression analysis. Compared with the traditional two-stage detection algorithm that first generates candidate frames and then classifies and regresses the candidate frames, the VOLO network structure does not need to generate a series of pre-selected frames that may contain the objects to be detected. It can determine the category of the target and obtain the position detection frame of the target in one step. Therefore, it has a faster detection speed and can meet the needs of real-time detection of elevator light curtains. The higher the value of indicates the higher the correlation between the predicted bounding box and the true bounding box, but when the two do not intersect, The value of is 0.

[0157] In an embodiment provided in the present application, the target tracking algorithm in step S32 specifically includes the following contents:

[0158] S321, inputting the feature map into the target detector in the fast multi-target detection algorithm and obtaining the bounding box of the first frame and its category confidence;

[0159] S322, setting high and low category confidence thresholds respectively according to the detection results, and dividing the bounding box into a high-score bounding box and a low-score bounding box based on the thresholds;

[0160] S323, if the score of the bounding box is higher than the threshold of the high-scoring bounding box, the bounding box is matched with the target trajectory box predicted by the Kalman filter (i.e., the predicted bounding box) for the first time through the Hungarian algorithm. If the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set. If the match is not successful, the unmatched trajectory is temporarily stored to wait for the second match;

[0161] S324, if the score of the bounding box is lower than the threshold of the high-scoring bounding box and higher than the threshold of the low-scoring bounding box, match it with the temporarily stored unmatched trajectory. If the match is successful, update its trajectory to the current frame trajectory set. If the match is still unsuccessful, delete the low-scoring bounding box, and determine whether the unmatched trajectory meets the deletion condition. According to the determination result, delete it or merge it into the current frame trajectory set and wait for the next frame to match.

[0162] S325, if the bounding box of the last predicted target still fails to match, it is identified as a newly appeared bounding box, and a new track and a new target are assigned to it.

[0163] Specifically, this embodiment obtains the position of the target object and the motion trajectory of the target object through the target tracking algorithm. Based on the YOLO detector, the similarity between the detection frame (boundary frame) and the tracking trajectory is used to remove the background from the low-scoring bounding frame while retaining the high-scoring bounding frame, and to mine the object detection frame that is discarded due to environmental factors such as occlusion or blur or illumination, thereby reducing the possibility of missed detection and false detection and improving the coherence of the trajectory. At the same time, the use of Kalman filtering can improve the accuracy of prediction.

[0164] In an embodiment provided in the present application, determining the moving intention of the target object in step S33 specifically includes the following contents:

[0165] S331, respectively obtaining the center coordinates of the target object position in the Nth frame image and the N-1th frame image, calculating the center coordinate difference between two adjacent frames of image and obtaining the displacement of the target object;

[0166] S332, calculating the moving speed of the target object in two adjacent frames of images;

[0167] S333, obtaining a moving direction of the target object according to the moving trajectory of the target object;

[0168] S334, judging the moving intention of the target object according to the moving speed and moving direction of the target object, if the moving intention is that the target object is heading towards the elevator, executing step S34, otherwise executing step S335;

[0169] S335, delete the N-1th frame image, obtain the N+1th frame image and return to execute step S331.

[0170] Specifically, the present application determines whether the target object is moving toward or away from the elevator by calculating the displacement, moving speed, and moving direction of the target object in two adjacent frames of images. If it is moving toward the elevator, the running state of the elevator is obtained in step S34, and the elevator is analyzed to determine whether it is parked at the corresponding floor or in an up and down moving state at this time, and whether there is a safety risk of accidental injury or being pinched when the target object moves toward the elevator. If there is a risk, a first control signal is generated, and the operation of the elevator is controlled by the first control signal; if the target object moves away from the elevator, the N-1 frame image is deleted, the N+1 frame image is obtained, and the step S331 is returned to start a new round of judgment process. This embodiment avoids safety risks such as pinching of the target object (passenger) by judging the moving intention of the target object. On the basis of the above-mentioned target detection and target tracking, the accuracy of target detection and target action judgment is achieved, thereby further improving the reliability of light curtain detection and the safety of passengers riding the elevator.

[0171] In an embodiment provided in the present application, region detection is performed in step S3, which specifically includes the following contents:

[0172] S3A, acquiring the trajectory change of the elevator door, and the image information of the elevator waiting area and the elevator floor based on the first image data set;

[0173] S3B, setting a first pre-monitoring area according to the trajectory change of the elevator door;

[0174] S3C, setting a second pre-monitoring area according to the image information of the elevator waiting area and the elevator floor;

[0175] S3D, respectively calculating the area of ​​the first pre-monitoring area and the second pre-monitoring area, and adding the areas of the two areas as the range of the visual light curtain monitoring area;

[0176] S3E, generating a second control signal after determining the range of the monitoring area;

[0177] Among them, the elevator door includes the elevator floor door and the elevator car door;

[0178] Step S3B specifically includes the following contents:

[0179] S3B1, when the opening and closing speeds of the elevator floor door and the elevator car door are the same, the length of the area when the elevator door is not closed is set as the length of the first pre-monitoring area, and the length of the area when the elevator door is fully opened is set as the width of the first pre-monitoring area, and the maximum area of ​​the first pre-monitoring area is calculated;

[0180] S3B2, when the opening and closing speeds of the elevator floor door and the elevator car door are inconsistent, the maximum area of ​​the first pre-monitoring area is calculated with the faster opening and closing speed as the reference target, specifically:

[0181] When the opening and closing speed of the elevator car door is greater than the opening and closing speed of the elevator floor door, the maximum area of ​​the first pre-monitoring area is: the length of the elevator car door track change × the width of the first pre-monitoring area and the length of the elevator car door track change × the width of the elevator floor door track change and the sum of the intermediate gap area;

[0182] When the opening and closing speed of the elevator floor door is greater than the opening and closing speed of the elevator car door, the maximum area of ​​the first pre-monitoring area is: the length of the elevator floor door trajectory change × the width of the first pre-monitoring area and the length of the elevator floor door trajectory change × the width of the elevator car door trajectory change and the sum of the intermediate gap area.

[0183] Specifically, this embodiment takes the trajectory change of the elevator door as the analysis target, calculates the area size of the first pre-monitoring area by analyzing the opening and closing speeds of the elevator floor door and the elevator car door, and then includes the elevator waiting area and the elevator floor in the monitoring range. By calculating the area of ​​these two parts, the area of ​​the first pre-monitoring area and the area of ​​the second pre-monitoring area are added to define the size of the entire monitoring area. It should be noted that since the trajectory change of the elevator door is in a real-time changing state, this embodiment generates a second control signal based on the result of the area detection, and continuously adjusts the range of the monitoring area in real time with the second control signal. This embodiment calculates the range of the monitoring area by analyzing the trajectory change of the elevator door, and dynamically adjusts it, thereby ensuring the real-time, accuracy and reliability of the elevator light curtain detection.

[0184] In an embodiment provided by the present application, abnormality detection of the elevator light curtain is performed in step S5, which specifically includes the following contents:

[0185] S51, detecting the door opening and closing state and the running state of the elevator to obtain a first abnormality detection result;

[0186] S52, detecting the components and circuit interfaces of the elevator light curtain to obtain a second abnormality detection result;

[0187] S53, performing abnormal behavior detection on the elevator waiting area and the elevator floor to obtain a third abnormal result;

[0188] S54, combining the first abnormality detection result, the second abnormality detection result and the third abnormality detection result together as a total abnormality detection result output and generating an abnormality signal;

[0189] The detection of the door opening and closing state in step S51 further includes:

[0190] S511, acquiring image features of the elevator door and motion characteristics of the elevator car from the second image data set; the motion characteristics include up and down movement, running speed, and stop floors, etc.;

[0191] S512, analyzing the image features of the elevator door, if the elevator floor door and / or the elevator car door are in an open state when the elevator car moves up and down, it is determined that the elevator door has a closing failure; if the elevator floor door and / or the elevator car door are in a closed state for a long time when the elevator car stops at a floor, it is determined that the elevator door has a opening failure;

[0192] S513, analyzing the movement characteristics of the elevator car. If the bottom of the elevator car is not parallel to the floor when the elevator car stops at the floor / or there is a height difference, it is determined that there is a parking failure of the elevator car; if the running speed of the elevator car exceeds the set speed threshold when it moves up and down, it is determined that there is an operation failure of the elevator car.

[0193] Specifically, in addition to detecting the door opening and closing status and the operating status of the elevator, this embodiment also detects the components and circuit interfaces of the elevator light curtain, specifically the sensors and detectors of the elevator light curtain, the power supply and data interfaces, and the circuits in the control system. By detecting the above contents, it is found and confirmed that there is a fault in the elevator light curtain and the fault location is accurately located, which is convenient for subsequent staff to repair and maintain the fault and ensure the normal operation of the elevator light curtain; the abnormal behavior of the elevator waiting area and the elevator floor is also detected, mainly including that there are people and objects still in the plane of the elevator door during the closing process of the elevator, or the elevator door is closed and opened repeatedly, or the elevator door fails to close due to a fault and stops at a position where the door is not completely closed, etc., and the abnormal personnel in the elevator waiting area and the abnormal parking of the elevator floor are confirmed by detection.

[0194] It should be noted that the abnormality handling and fault analysis described in step S6 of the present application are performed based on the abnormalities and faults detected and confirmed in the above step S5, and are alarm prompts and timely fault handling for the above faults, thereby achieving the effect of all-round monitoring of abnormalities and real-time handling. This embodiment does not specifically limit the means of abnormality handling, and appropriate maintenance means can be selected for fault handling according to actual needs.

[0195] In an embodiment provided in the present application, an elevator parallel operation analysis is performed in step S7, which specifically includes the following contents:

[0196] S71, acquiring an image from the third image data set and converting it into a grayscale image, performing binarization processing and feature extraction to obtain a human feature sample;

[0197] S72, training a classifier based on the human feature samples and performing binary judgment on the head and non-head, and adding the judgment results to obtain the number of people inside and outside the elevator on the same floor;

[0198] S73, calculating the total number of people in the elevator cars, elevator waiting areas, and elevator floors of multiple elevators on the same floor and counting the total number of people on multiple floors, and analyzing the remaining capacity inside the multiple elevators;

[0199] S74, setting the elevator response time between two adjacent floors of the same elevator; if there are multiple elevators, setting multiple different elevator response times;

[0200] S75, calculating the stop time of the elevator at each floor according to the total number of people on the multiple floors and the remaining capacity inside the multiple elevators; calculating the time for the elevator to arrive at the response floor according to the floor position of the elevator stop and the set elevator response time;

[0201] S76, performing comprehensive calculation based on the elevator's parking time and the time it takes for the elevator to arrive at the response floor, generating an elevator coordination decision and outputting an elevator dispatching instruction;

[0202] S77, coordinating and scheduling the parallel operation of multiple elevators according to the elevator scheduling instruction.

[0203] Specifically, this embodiment calculates the number of people on each floor, sets the response time of multiple elevators, and the floor buttons pressed by passengers in and outside the elevator (the floor the passengers want to reach, the floor where the passengers call the elevator), comprehensively analyzes the operation status of multiple elevators, and comprehensively schedules the parallel operation of multiple elevators according to the number of elevator passengers and the response time, thereby avoiding mutual interference when multiple elevators are used in parallel, and real-time and safety problems caused by untimely response, thereby greatly improving the safety of the elevators and the real-time performance of the elevator operation.

[0204] In an embodiment provided in the present application, a safety self-check is performed in step S9, specifically including the following contents:

[0205] S91, detect the current state of the elevator according to the operation data of the elevator and determine whether there is a light curtain blocking signal in the current state, and generate a control command to stop the transmitting end of the elevator light curtain according to the determination result, specifically:

[0206] If the elevator is currently in the parked state and the door is closed and there is no call signal for the current elevator, determine whether there is a light curtain blocking signal in the door closed state. If not, generate and send a control command; otherwise, re-detect the current state of the elevator;

[0207] S92, the receiving end of the elevator light curtain outputs a light curtain status signal and determines whether it is a light curtain blocking signal. If the light curtain status signal is a light curtain blocking signal, the elevator light curtain is not faulty, otherwise the elevator light curtain is faulty;

[0208] S93, after determining that the elevator light curtain fails, obtaining the real-time distance between the elevator light curtain transmitting end and the elevator light curtain receiving end according to the operation data of the elevator light curtain;

[0209] S94, determining a distance threshold based on the distance between the receiving ends and the transmission angle of the transmitting end;

[0210] S95, compare the real-time distance with the distance threshold and determine the threshold interval in which the real-time distance is located, and then control the scanning mode of the elevator light curtain based on the threshold interval, specifically:

[0211] If the real-time distance is within the first threshold interval, a first scanning mode is adopted;

[0212] If the real-time distance is within the second threshold interval, a second scanning mode is adopted;

[0213] If the real-time distance is within the third threshold interval, a third scanning mode is adopted;

[0214] The distance threshold includes a first distance threshold and a second distance threshold; the first threshold interval is: lower than the first distance threshold; the second threshold interval is: between the first distance threshold and the second distance threshold; the third threshold interval is: higher than the second distance threshold;

[0215] The first scanning method is: scanning with a single-point light beam;

[0216] The second scanning method is: scanning with a combined light beam;

[0217] The third scanning method is: scanning by intermittently alternating single-point light beam and combined light beam.

[0218] It should be noted that a single-point beam refers to a beam emitted by a transmitting end and a corresponding beam received by a receiving end; a combined beam refers to a beam emitted by a transmitting end and a corresponding plurality of receiving ends receiving the beams; and alternating scanning refers to the scanning of a single-point beam and a combined beam alternately within a set interval.

[0219] Specifically, this embodiment first determines whether the elevator light curtain is faulty / failed. If a fault occurs, the scanning mode is further switched according to the distance between the transmitting end and the receiving end, thereby ensuring that different methods can be used to ensure the operation of the elevator light curtain when the elevator light curtain fails or fails, thereby achieving stability and safety in the operation of the elevator light curtain. Example 2

[0220] See also Figure 2 , the embodiment of the present application provides a machine vision-based light curtain system, which is applied to the elevator light curtain control method as described above, including a visual light curtain monitoring module, a video image processing module, an elevator light curtain control module, an abnormality detection processing module, a parallel operation scheduling module and a safety self-check optimization module;

[0221] The visual light curtain monitoring module installs a visual sensor at the elevator door and sets the monitoring area of ​​the visual light curtain to capture the video image of the monitoring area in real time;

[0222] The video image processing module performs image processing on the video image and obtains a first image data set, a second image data set and a third image data set according to image features;

[0223] The elevator light curtain control module performs object detection and area detection based on the first image data set, and generates a first control signal and a second control signal according to the detection results; adjusts the sensitivity of the elevator light curtain and controls the opening and closing of the elevator door according to the first control signal; and adjusts the range of the monitoring area according to the second control signal;

[0224] The abnormality detection processing module performs abnormality detection of the elevator light curtain based on the second image data set, generates an abnormality signal according to the detection result; performs abnormality processing and fault analysis according to the abnormal signal;

[0225] The parallel operation scheduling module performs elevator parallel operation analysis based on the third image data set, generates a third control signal according to the analysis result, and coordinates and schedules the elevator operation according to the third control signal;

[0226] The safety self-check optimization module obtains the operation data of the elevator and the elevator light curtain and performs safety self-check, and performs light curtain protection according to the self-check results; establishes a light curtain control optimization model based on the video image and uses it to analyze the real-time status of the elevator light curtain, generates a control decision according to the analysis result and uses it to optimize the operation mode and control means of the elevator light curtain;

[0227] Among them, the features in the first image data set include elevator doors, elevator waiting areas, elevator cars and objects; the features in the second image data set include elevator doors, elevator cars, elevator waiting areas and elevator floors; the features in the third image data set include objects, elevator cars, elevator waiting areas and elevator floors. Example 3

[0228] An embodiment of the present application provides an elevator based on machine vision, wherein the elevator adopts the elevator light curtain control method as described above.

[0229] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0230] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0231] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0232] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A method for controlling an elevator light curtain based on machine vision, characterized in that: The steps include: S1, install a visual sensor at the elevator door and set up a monitoring area of ​​the visual light curtain to capture the video image of the monitoring area in real time; S2, performing image processing on the video image and obtaining a first image data set, a second image data set and a third image data set according to image features; S3, performing object detection and region detection based on the first image data set, and generating a first control signal and a second control signal respectively according to the detection results; S4, adjusting the sensitivity of the elevator light curtain and controlling the opening and closing of the elevator door according to the first control signal; adjusting the range of the monitoring area according to the second control signal; S5, performing abnormality detection of the elevator light curtain based on the second image data set, and generating an abnormality signal according to the detection result; S6, performing abnormality processing and fault analysis according to the abnormal signal; S7, performing elevator parallel operation analysis based on the third image data set, and generating a third control signal according to the analysis result; S8, coordinate control and dispatch the elevator operation according to the third control signal; S9, obtain the operation data of the elevator and the elevator light curtain and perform safety self-inspection, and perform light curtain protection according to the self-inspection results; S10, establishing a light curtain control optimization model based on the video image and using it to analyze the real-time state of the elevator light curtain, generating a control decision according to the analysis result and using it to optimize the operation mode and control means of the elevator light curtain; The features in the first image data set include elevator doors, elevator waiting areas, elevator cars, and objects; the features in the second image data set include elevator doors, elevator cars, elevator waiting areas, and elevator floors; the features in the third image data set include objects, elevator cars, elevator waiting areas, and elevator floors; In step S3, object detection is performed, which specifically includes the following contents: S31, using a fast multi-target detection algorithm to quickly detect the first image data set to obtain a target object; S32, using a target tracking algorithm to track the target object, and obtaining the position of the target object and the motion trajectory of the target object, specifically including the following contents: S321, inputting the feature map into the target detector in the fast multi-target detection algorithm and obtaining the bounding box of the first frame and its category confidence; S322, setting high and low category confidence thresholds respectively according to the detection results, and dividing the bounding box into a high-score bounding box and a low-score bounding box based on the detection results; S323, if the score of the bounding box is higher than the threshold of the high-scoring bounding box, the bounding box is matched with the target trajectory box predicted by the Kalman filter through the Hungarian algorithm for the first time. If the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set. If the match is not successful, the unmatched trajectory is temporarily stored to wait for the second match; S324, if the score of the bounding box is lower than the threshold of the high-scoring bounding box and higher than the threshold of the low-scoring bounding box, match it with the temporarily stored unmatched trajectory. If the match is successful, update its trajectory to the current frame trajectory set. If the match is still unsuccessful, delete the low-scoring bounding box, and determine whether the unmatched trajectory meets the deletion condition. According to the determination result, delete it or merge it into the current frame trajectory set and wait for the next frame to match. S325, if the bounding box of the last predicted target still fails to match, it is identified as a newly appeared bounding box, and a new trajectory and a new target are assigned to it; S33, judging the moving intention of the target object according to the position of the target object and the moving trajectory of the target object, specifically including the following contents: S331, respectively obtaining the center coordinates of the target object position in the Nth frame image and the N-1th frame image, calculating the center coordinate difference between two adjacent frames of image and obtaining the displacement of the target object; S332, calculating the moving speed of the target object in two adjacent frames of images; S333, obtaining a moving direction of the target object according to the moving trajectory of the target object; S334, judging the moving intention of the target object according to the moving speed and moving direction of the target object, if the moving intention is that the target object is heading towards the elevator, executing step S34, otherwise executing step S335; S335, delete the N-1th frame image, obtain the N+1th frame image and return to step S331; S34, obtaining the running status of the elevator, and judging whether the target object has a safety risk according to the moving intention, and generating a first control signal according to the judgment result.

2. The elevator light curtain control method based on machine vision according to claim 1, characterized in that: The fast multi-target detection algorithm in step S31 specifically includes the following contents: S311, selecting a feature map from the first image data set and adjusting the pixel size of the feature map; S312, inputting the feature map into a network model, wherein the network model is used to divide the input feature map into N×N grids, and if the center point of a target in the feature map falls into a grid, the grid is responsible for detecting the target; S313: Each grid predicts M corresponding bounding boxes. Each bounding box consists of 6 prediction values, including the center coordinates of the bounding box (x, y), the width and height of the bounding box (w, h), the category confidence X of the target in the bounding box, and the error between the bounding box and the anchor box. ; S314, during testing, the category information predicted by each grid is multiplied by the category confidence to obtain the confidence score that the target of the bounding box belongs to a certain category. The calculation formula is expressed as: ； represents the conditional probability, which is the probability of belonging to the i-th category under the condition that the bounding box exists; Indicates that the bounding box belongs to the i-th category under the condition that there is an object; Wherein, the network model is a VOLO network structure; the category confidence X is expressed as: ； Indicates whether there is a target in the grid. If the center point of the target bounding box does not fall into the grid, it is 0, otherwise it is 1; Represents the intersection-over-union ratio of the predicted bounding box and the true bounding box; The anchor frame is a frame of fixed size and aspect ratio generated at each pixel of the feature map; error The processing formula includes: ； ； ； ； Represents the error between the central horizontal coordinate of the jth anchor box and the predicted value of the bounding box; Represents the error between the central ordinate of the jth anchor box and the predicted value of the bounding box; Represents the error between the width of the jth anchor box and the predicted value of the bounding box; Represents the error between the height of the jth anchor box and the predicted value of the bounding box; represents the jth anchor box; represents the i-th bounding box.

3. The elevator light curtain control method based on machine vision according to claim 1, characterized in that: In step S3, region detection is performed, which specifically includes the following contents: S3A, acquiring the trajectory change of the elevator door, and the image information of the elevator waiting area and the elevator floor based on the first image data set; S3B, setting a first pre-monitoring area according to the trajectory change of the elevator door; S3C, setting a second pre-monitoring area according to the image information of the elevator waiting area and the elevator floor; S3D, respectively calculating the area of ​​the first pre-monitoring area and the second pre-monitoring area, and adding the areas of the two areas as the range of the visual light curtain monitoring area; S3E, generating a second control signal after determining the range of the monitoring area; Among them, the elevator door includes the elevator floor door and the elevator car door; Step S3B specifically includes the following contents: S3B1, when the opening and closing speeds of the elevator floor door and the elevator car door are the same, the length of the area when the elevator door is not closed is set as the length of the first pre-monitoring area, and the length of the area when the elevator door is fully opened is set as the width of the first pre-monitoring area, and the maximum area of ​​the first pre-monitoring area is calculated; S3B2, when the opening and closing speeds of the elevator floor door and the elevator car door are inconsistent, the maximum area of ​​the first pre-monitoring area is calculated with the faster opening and closing speed as the reference target, specifically: When the opening and closing speed of the elevator car door is greater than the opening and closing speed of the elevator floor door, the maximum area of ​​the first pre-monitoring area is: the length of the elevator car door track change × the width of the first pre-monitoring area and the length of the elevator car door track change × the width of the elevator floor door track change and the sum of the intermediate gap area; When the opening and closing speed of the elevator floor door is greater than the opening and closing speed of the elevator car door, the maximum area of ​​the first pre-monitoring area is: the length of the elevator floor door trajectory change × the width of the first pre-monitoring area and the length of the elevator floor door trajectory change × the width of the elevator car door trajectory change and the sum of the intermediate gap area.

4. The elevator light curtain control method based on machine vision according to claim 1, characterized in that: In step S5, abnormality detection of the elevator light curtain is performed, which specifically includes the following contents: S51, detecting the door opening and closing state and the running state of the elevator to obtain a first abnormality detection result; S52, detecting the components and circuit interfaces of the elevator light curtain to obtain a second abnormality detection result; S53, performing abnormal behavior detection on the elevator waiting area and the elevator floor to obtain a third abnormal result; S54, combining the first abnormality detection result, the second abnormality detection result and the third abnormality detection result together as a total abnormality detection result output and generating an abnormality signal; The detection of the door opening and closing state in step S51 further includes: S511, acquiring image features of an elevator door and motion characteristics of an elevator car from the second image data set; S512, analyzing the image features of the elevator door, if the elevator floor door and / or the elevator car door are in an open state when the elevator car moves up and down, it is determined that the elevator door has a closing failure; if the elevator floor door and / or the elevator car door are in a closed state for a long time when the elevator car stops at a floor, it is determined that the elevator door has a opening failure; S513, analyzing the movement characteristics of the elevator car. If the bottom of the elevator car is not parallel to the floor when the elevator car stops at the floor / or there is a height difference, it is determined that there is a parking failure of the elevator car; if the running speed of the elevator car exceeds the set speed threshold when it moves up and down, it is determined that there is an operation failure of the elevator car.

5. The elevator light curtain control method based on machine vision according to claim 1, characterized in that: In step S7, the elevator parallel operation analysis is performed, which specifically includes the following contents: S71, acquiring an image from the third image data set and converting it into a grayscale image, performing binarization processing and feature extraction to obtain a human feature sample; S72, training a classifier based on the human feature samples and performing binary judgment on the head and non-head, and adding the judgment results to obtain the number of people inside and outside the elevator on the same floor; S73, calculating the total number of people in the elevator cars, elevator waiting areas, and elevator floors of multiple elevators on the same floor and counting the total number of people on multiple floors, and analyzing the remaining capacity inside the multiple elevators; S74, setting the elevator response time between two adjacent floors of the same elevator; If there are multiple elevators, set different elevator response times; S75, calculating the stop time of the elevator on each floor according to the total number of people on the multiple floors and the remaining capacity in the multiple elevators; Calculate the time it takes for the elevator to arrive at the response floor based on the floor where the elevator stops and the set elevator response time; S76, performing comprehensive calculation based on the elevator's parking time and the time it takes for the elevator to arrive at the response floor, generating an elevator coordination decision and outputting an elevator dispatching instruction; S77, coordinating and scheduling the parallel operation of multiple elevators according to the elevator scheduling instruction.

6. The elevator light curtain control method based on machine vision according to claim 1, characterized in that: In step S9, a safety self-check is performed, which specifically includes the following contents: S91, detect the current state of the elevator according to the operation data of the elevator and determine whether there is a light curtain blocking signal in the current state, and generate a control command to stop the transmitting end of the elevator light curtain according to the determination result, specifically: If the elevator is currently in the parked state and the door is closed and there is no call signal for the current elevator, determine whether there is a light curtain blocking signal in the door closing state. If not, generate and send a control command; otherwise, re-detect the current state of the elevator; S92, the receiving end of the elevator light curtain outputs a light curtain state signal and determines whether it is a light curtain blocking signal. If the light curtain state signal is a light curtain blocking signal, the elevator light curtain is not faulty, otherwise the elevator light curtain is faulty; S93, after determining that the elevator light curtain fails, obtaining the real-time distance between the elevator light curtain transmitting end and the elevator light curtain receiving end according to the operation data of the elevator light curtain; S94, determining a distance threshold based on the distance between the receiving ends and the transmission angle of the transmitting end; S95, comparing the real-time distance with the distance threshold and determining the threshold interval in which the real-time distance is located, and then controlling the scanning mode of the elevator light curtain based on the threshold interval, specifically: If the real-time distance is within the first threshold interval, a first scanning mode is adopted; If the real-time distance is within the second threshold interval, a second scanning mode is adopted; If the real-time distance is within the third threshold interval, a third scanning mode is adopted; The distance threshold includes a first distance threshold and a second distance threshold; the first threshold interval is: lower than the first distance threshold; the second threshold interval is: between the first distance threshold and the second distance threshold; the third threshold interval is: higher than the second distance threshold; The first scanning method is: scanning with a single-point light beam; The second scanning method is: scanning with a combined light beam; The third scanning method is: scanning by intermittently alternating single-point light beam and combined light beam.

7. A light curtain system based on machine vision, applied to the elevator light curtain control method according to any one of claims 1 to 6, characterized in that: It includes visual light curtain monitoring module, video image processing module, elevator light curtain control module, abnormal detection processing module, parallel operation scheduling module and safety self-check optimization module; The visual light curtain monitoring module installs a visual sensor at the elevator door and sets the monitoring area of ​​the visual light curtain to capture the video image of the monitoring area in real time; The video image processing module performs image processing on the video image and obtains a first image data set, a second image data set and a third image data set according to image features; The elevator light curtain control module performs object detection and area detection based on the first image data set, and generates a first control signal and a second control signal according to the detection results; adjusts the sensitivity of the elevator light curtain and controls the opening and closing of the elevator door according to the first control signal; and adjusts the range of the monitoring area according to the second control signal; The abnormality detection processing module performs abnormality detection of the elevator light curtain based on the second image data set, and generates an abnormality signal according to the detection result; Performing abnormality processing and fault analysis according to the abnormal signal; The parallel operation scheduling module performs elevator parallel operation analysis based on the third image data set, generates a third control signal according to the analysis result, and coordinates and schedules the elevator operation according to the third control signal; The safety self-check optimization module obtains the operation data of the elevator and the elevator light curtain and performs safety self-check, and performs light curtain protection according to the self-check results; establishes a light curtain control optimization model based on the video image and uses it to analyze the real-time status of the elevator light curtain, generates a control decision according to the analysis result and uses it to optimize the operation mode and control means of the elevator light curtain; The features in the first image data set include elevator doors, elevator waiting areas, elevator cars, and objects; the features in the second image data set include elevator doors, elevator cars, elevator waiting areas, and elevator floors; The features in the third image data set include objects, elevator cars, elevator waiting areas, and elevator floors.

8. An elevator based on machine vision, characterized in that: The elevator adopts the elevator light curtain control method as described in any one of claims 1-6.

Images