Elevator car entrapment identification and warning method, system, and device based on image processing

Abstract

The application relates to the technical field of reading and identifying image data, in particular to an elevator car trapped person identification and early warning method, system and device based on image processing; in order to overcome the low accuracy of the existing car trapped person identification, the application first obtains an image in the car according to the comparison result of the elevator car door closing time and the closing time threshold; then, after the enhanced image in the car is subjected to boundary box regression and classification processing, a suitable detection frame is screened out through confidence, classification type and classification probability, and a car passenger detection image capable of reflecting the number of passengers and the position of passengers is obtained; finally, based on the position information, posture information and emotional information of the passengers in the car passenger detection image, it is judged whether the passengers are elevator repair workers and whether they are trapped in the elevator, so as to decide whether to send a trapped person early warning information; the application has the advantages of high identification efficiency and good identification accuracy when used in elevator car trapped person identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of reading and identifying image data, in particular to an elevator car occupant identification and early warning method, system and device based on image processing. BACKGROUND

[0002] When a person is trapped in an elevator car, it is crucial to quickly discover and rescue. Elevator car occupant identification based on images can achieve real-time monitoring, enabling timely detection of occupant trapping events and thus quick action for rescue. Compared to manual monitoring of elevator cars, elevator car occupant identification based on images can avoid the problems of human oversight or misjudgment, reduce human error, improve identification accuracy, and also enable automated monitoring, reducing the investment in human resources and costs.

[0003] However, due to image quality and character features within the image, existing car image identification technology has low accuracy in identifying passengers in the elevator car, and is also prone to identifying workers performing elevator maintenance as ordinary passengers, and issuing false elevator car occupant trapping early warning information, affecting the normal work of security personnel. SUMMARY

[0004] The purpose of the present application is to provide an elevator car occupant identification and early warning method, system and device based on image processing with good identification effect.

[0005] The technical solution of the present application is as follows:

[0006] An elevator car occupant identification and early warning method based on image processing, comprising the following operations:

[0007] S1, if the elevator car door closing time exceeds the closing time threshold, obtain the car interior image, and the car interior image is subjected to image enhancement processing to obtain a car enhanced image;

[0008] S2, the car enhanced image is subjected to boundary box regression and classification processing to obtain an initial car classification image containing multiple detection boxes in the same detection area, each detection box containing multiple classification types and corresponding classification probabilities; retain the detection box corresponding to the maximum confidence value in the same detection area in the initial car classification image to obtain an optimal car classification image; retain the detection box in the optimal car classification image whose classification type includes passengers and whose classification probability corresponding to the passenger classification type is greater than the probability threshold to obtain a car passenger detection image;

[0009] S3, based on the passenger position and posture classification result in each detection box in the car passenger detection image, determine whether the passenger is an elevator maintenance worker; if all passengers are elevator maintenance workers, do not send a car trapped person warning message; if all passengers are not elevator maintenance workers, execute S4; the operation of determining whether the passenger is an elevator maintenance worker is specifically: obtaining the pixel gradient value of each position point of the current detection box in the car passenger detection image, to obtain the current detection box pixel gradient distribution image; obtaining the position point in the current detection box pixel gradient distribution image, whose pixel gradient is greater than the standard pixel gradient and the straight line distance from the detection box is less than the distance threshold, to obtain the current passenger edge contour point; after curve fitting of all passenger edge contour points, curve closure processing is performed to obtain the current passenger posture contour map; the current passenger posture contour map is subjected to posture classification processing to obtain the posture classification result of the current passenger; the position coordinates of the bottom center of the current detection box are taken as the position of the current passenger; if the posture classification result of the current passenger is that the posture type is maintenance, and the corresponding posture type probability is not less than the posture probability threshold, and the position of the current passenger is in the maintenance position area, then the current passenger is an elevator maintenance worker; if the posture classification result of the current passenger is that the posture type is not maintenance, or the posture classification type is maintenance but the corresponding posture type probability is less than the posture probability threshold, then the current passenger is not an elevator maintenance worker;

[0010] S4, obtain the face area of each passenger on each detection box in the car passenger detection image as the region of interest of each passenger, and obtain the emotional information of the region of interest of each passenger; if the emotional information of the passenger includes panic, and the posture type of the passenger includes escape, and the proportion of the number of passengers in the escape area is greater than the proportion threshold, then the passenger is trapped in the elevator car, and a car trapped person warning message is sent.

[0011] The operation of classification processing in S2 is specifically: obtaining the similarity between the current detection box corresponding image and each standard real object image in the car boundary regression image obtained by boundary box regression processing of the car enhanced image, taking the label of the standard real object image with a similarity greater than a similarity threshold as the classification type of the current detection box, and the similarity corresponding to the classification type as the classification probability of the current detection box; the car boundary regression image, and the classification type and corresponding classification probability of all detection boxes form an initial car classification image.

[0012] The similarity obtaining method is: the detection frame corresponding image is subjected to segmentation processing to obtain a detection frame segmentation image containing a plurality of segmentation regions; based on the pixel standard deviation of each segmentation region in the detection frame segmentation image, the pixel standard deviation of each segmentation region corresponding to the initial region in the standard real object image, and the pixel covariance of each segmentation region corresponding to the initial region in the standard real object image, the region similarity value of each segmentation region and the corresponding initial region is obtained; the region similarity value of each segmentation region and the corresponding initial region is multiplied by the weight of the corresponding segmentation region, and then summed to obtain the similarity.

[0013] The operation of the bounding box regression processing further includes semantic feature extraction on the car bounding box regression graph to obtain a car bounding box regression semantic feature graph for performing the operation of classification processing; the operation of the semantic feature extraction processing is specifically: the car bounding box regression graph is fused after being subjected to graph convolution processing of different convolution scales to obtain the car bounding box regression semantic feature graph.

[0014] In the process of the graph convolution processing, the adjacency matrix corresponding to the output graph of the previous layer is added to the initial adjacency matrix to obtain an updated adjacency matrix as the adjacency matrix of the current layer input.

[0015] The operation of the image enhancement processing in S1 is specifically: the car interior image is subjected to Gaussian blur processing to obtain a car Gaussian image; the car Gaussian image and the car interior image are subjected to image subtraction processing to obtain a car secondary detail image; and the car secondary detail image and the car interior image are subjected to fusion processing to obtain a car enhanced image.

[0016] The operation of the posture classification processing in S3 can be: a neural network is trained by using the obtained historical car ordinary passenger posture data set and the historical car elevator repair worker posture data set to obtain a trained neural network; and the current passenger posture contour graph is subjected to the trained neural network processing to obtain a posture classification result of the current passenger containing a posture type and a posture type probability.

[0017] An elevator car trapped person identification and early warning system based on image processing, comprising:

[0018] The car enhanced image generation module is configured to, if the elevator car door closing time exceeds the closing time threshold, obtain a car interior image, and subject the car interior image to image enhancement processing to obtain a car enhanced image.

[0019] The car passenger detection map generation module is used for processing the car enhanced image through the bounding box regression and classification, obtaining an initial car classification map containing multiple detection boxes in the same detection area, and each detection box containing multiple classification types and corresponding classification probabilities.

[0020] The elevator maintenance worker judgment module is used for judging whether the passengers are elevator maintenance workers based on the position and posture classification result of each passenger in the car passenger detection map. If all passengers are elevator maintenance workers, no car trapping warning information is sent. If all passengers are not elevator maintenance workers, the car trapping identification and warning information sending module is executed. The operation of judging whether the passengers are elevator maintenance workers is specifically as follows: the pixel gradient value of each position point of the image corresponding to the current detection box in the car passenger detection map is obtained, and a current detection box pixel gradient distribution map is obtained. The position point with a pixel gradient greater than a standard pixel gradient and a distance from the detection box less than a distance threshold in the current detection box pixel gradient distribution map is obtained, and a current passenger edge contour point is obtained. After curve fitting of all passenger edge contour points, curve closure processing is performed to obtain a current passenger posture contour map. The posture classification result of the current passenger is obtained through posture classification processing of the current passenger posture contour map. The position coordinates of the center of the bottom of the current detection box are taken as the position of the current passenger. If the posture classification result of the current passenger includes a maintenance posture type and the corresponding posture type probability is not less than a posture probability threshold, and the position of the current passenger is in a maintenance position area, the current passenger is an elevator maintenance worker. If the posture classification result of the current passenger does not include a maintenance posture type, or the posture classification type is maintenance but the corresponding posture type probability is less than the posture probability threshold, the current passenger is not an elevator maintenance worker.

[0021] The car trapping identification and warning information sending module is used for obtaining the face area of each passenger on the car passenger detection map as the region of interest of each passenger, and obtaining the emotional information of the region of interest of each passenger. If the emotional information of the passenger includes panic, and the number of passengers whose posture type includes avoidance and whose position is in an avoidance area accounts for more than a proportion threshold, the passenger is trapped in the elevator car, and the car trapping warning information is sent.

[0022] An elevator car trapping identification and warning device based on image processing includes a processor and a memory, wherein the processor implements the above-mentioned elevator car trapping identification and warning method based on image processing when executing the computer program saved in the memory.

[0023] A computer readable storage medium for storing a computer program, wherein the computer program is executed by a processor to implement the image processing-based elevator car occupant identification and early warning method.

[0024] The present application has the advantages of:

[0025] The image processing-based elevator car occupant identification and early warning method provided by the present application first acquires a car interior image according to a comparison result of an elevator car door closing time and a closing time threshold value, and improves the quality of the car interior image through image enhancement processing; then, after the car enhanced image is subjected to bounding box regression and classification processing, appropriate detection boxes are screened out through confidence, classification type and classification probability, and a car passenger detection image that can reflect the number and position of passengers is obtained; finally, based on the position information and posture information of passengers in the car passenger detection image, it is determined whether the passengers are elevator maintenance workers; if the passengers in the elevator car are not elevator maintenance workers, it is determined whether the passengers are trapped based on the emotional information, posture information and passenger number proportion information of passengers in the danger area, so as to determine whether to send a car occupant early warning information; the method has high identification efficiency and good identification accuracy. DETAILED DESCRIPTION

[0026] The present embodiment provides an image processing-based elevator car occupant identification and early warning method, which includes the following operations:

[0027] S1, if the elevator car door closing time exceeds the closing time threshold value, a car interior image is acquired, and the car interior image is subjected to image enhancement processing to obtain a car enhanced image;

[0028] S2, the car enhanced image is subjected to bounding box regression and classification processing to obtain an initial car classification image containing multiple detection boxes in the same detection area, each detection box containing multiple classification types and corresponding classification probabilities;

[0029] The detection box corresponding to the maximum confidence value in the same detection area in the initial car classification image is retained to obtain an optimal car classification image;

[0030] The detection box in the optimal car classification image is retained, the classification type of which includes passengers and the classification probability corresponding to the classification type of which is greater than a probability threshold value, to obtain a car passenger detection image;

[0031] S3, based on the position and posture classification result of passengers in each detection box in the car passenger detection image, it is determined whether the passengers are elevator maintenance workers; if all the passengers are elevator maintenance workers, no car occupant early warning information is sent; if all the passengers are not elevator maintenance workers, S4 is executed;

[0032] The operation of determining whether the passenger is an elevator repair worker specifically includes: obtaining pixel gradient values of each position point of a current detection frame image in the car passenger detection image to obtain a current detection frame pixel gradient distribution image; obtaining position points in the current detection frame pixel gradient distribution image, which have a pixel gradient greater than a standard pixel gradient and a straight-line distance from the detection frame less than a distance threshold, to obtain current passenger edge contour points; performing curve fitting on all passenger edge contour points and performing curve closure processing to obtain a current passenger posture contour map; and performing posture classification processing on the current passenger posture contour map to obtain a posture classification result of the current passenger.

[0033] Taking a position coordinate of a center of a bottom of the current detection frame as a position of the current passenger.

[0034] If the posture classification result of the current passenger includes a maintenance posture type and a corresponding posture type probability is not less than a posture probability threshold value, and the position of the current passenger is in a maintenance position area, the current passenger is an elevator repair worker.

[0035] If the posture classification result of the current passenger does not include a maintenance posture type, or the posture classification result includes a maintenance posture type but the corresponding posture type probability is less than the posture probability threshold value, the current passenger is not an elevator repair worker.

[0036] S4, obtaining a face area of each passenger on each detection frame in the car passenger detection image as a region of interest of each passenger, and obtaining emotional information of the region of interest of each passenger; if the emotional information of the passenger includes panic, and the posture type of the passenger includes an escape posture type, and a proportion of the number of passengers in an escape area is greater than a proportion threshold value, the passenger is trapped in the elevator car, and a car trapped person warning information is sent.

[0037] S1, if an elevator car door closing time exceeds a closing time threshold value, obtaining a car interior image, and obtaining a car enhanced image through image enhancement processing of the car interior image.

[0038] According to a comparison result of the elevator car door closing time (elevator car running time) and the closing time threshold value, the car interior image is obtained, which is used for subsequent judgment of whether the elevator car has a trapped person event; if the elevator car door closing time exceeds the closing time threshold value, it represents that the elevator running time exceeds the running time threshold value, and the elevator may have a fault, so that the car interior image obtained after image enhancement processing is a car enhanced image with rich details, which is used for improving the accuracy of subsequent image recognition, thereby improving the accuracy of elevator car trapped person recognition.

[0039] The image enhancement processing operation specifically includes: performing Gaussian blur processing on the in-car image to obtain a car Gaussian image; performing image subtraction processing on the car Gaussian image and the in-car image to obtain a car secondary detail image; and performing fusion processing on the car secondary detail image and the in-car image to obtain a car enhanced image. Specifically, the in-car image is subjected to Gaussian blur processing based on a preset blur radius and a blur area to obtain the car Gaussian image; the car Gaussian image and the in-car image are subjected to pixel subtraction processing at corresponding positions to implement the image subtraction processing, thereby obtaining the car secondary detail image; and the car secondary detail image and the in-car image are subjected to channel superposition processing to implement fusion, thereby obtaining the car enhanced image.

[0040] When the elevator car is in a relatively dark environment, the in-car image quality is reduced, and the accuracy of subsequent image recognition is reduced. To solve this technical problem, the embodiment further includes, before the image enhancement processing operation, performing color temperature compensation processing on the in-car image to obtain an in-car corrected image, which is used to perform the image enhancement processing operation. The color temperature compensation operation specifically includes: based on the color temperature difference between the in-car image and a standard car image, and the red channel value, green channel value and blue channel value of each position point of the in-car image, obtaining the red channel color temperature increment value, green channel color temperature increment value and blue channel color value increment value of each position point of the in-car image relative to the standard car image; based on the red channel color temperature increment value, green channel color temperature increment value and blue channel color value increment value of each position point of the in-car image, obtaining the balance value of each position point of the in-car image; dividing the pixel value of each position point of the in-car image by the corresponding balance value to obtain the color temperature update value of each position point, thereby implementing the color temperature compensation processing and obtaining the in-car corrected image.

[0041] S2, the car enhanced image is subjected to bounding box regression and classification processing to obtain an initial car classification image in which each detection box contains multiple classification types and corresponding classification probabilities in the same detection region; a detection box corresponding to the maximum confidence value in the same detection region in the initial car classification image is retained to obtain an optimal car classification image; and a detection box in which the classification type includes passengers and the classification probability corresponding to the classification type of passengers is greater than a probability threshold in the optimal car classification image is retained to obtain a car passenger detection image.

[0042] After the bounding box regression and classification processing of the car enhanced image, appropriate detection boxes are screened out through the confidence, classification type and classification probability to obtain a car passenger detection image that can reflect the number of passengers (the number of detection boxes) and the position of passengers (the position of the detection box), thereby improving the accuracy of subsequent differentiation between elevator repair workers and ordinary passengers and improving the accuracy of elevator car entrapment identification.

[0043] First, the enhanced image of the car is processed by bounding box regression to obtain a car boundary regression map containing multiple detection boxes within the same detection region. Bounding box regression is an existing technique, and will not be described in detail here to save space.

[0044] Then, the images (images corresponding to the detection boxes) within each detection box region in the car boundary regression map are classified. Specifically, the classification process involves: obtaining the similarity between the image corresponding to the current detection box and each standard physical object image in the car physical object database within the car boundary regression map obtained from the enhanced car image through boundary box regression processing; assigning the label of the standard physical object image with a similarity greater than a similarity threshold as the classification type of the current detection box; and using the similarity corresponding to the classification type as the classification probability of the current detection box. The car boundary regression map, along with the classification types and corresponding classification probabilities of all detection boxes, forms the initial car classification map.

[0045] The similarity is obtained as follows: the image corresponding to the detection box is segmented to obtain a detection box segmentation map containing several segmented regions; based on the pixel standard deviation of each segmented region in the detection box segmentation map, the pixel standard deviation of the corresponding initial region in the standard physical image, and the pixel covariance of the corresponding initial region in the standard physical image, the region similarity value between each segmented region and the corresponding initial region is obtained; the region similarity value between each segmented region and the corresponding initial region is multiplied by the weight of the corresponding segmented region and then summed to obtain the similarity.

[0046] In addition, to improve the accuracy of classification, the bounding box regression process includes extracting semantic features from the car boundary regression map to obtain a car regression semantic feature map, which is then used to perform classification.

[0047] The semantic feature extraction process is as follows: the car boundary regression map is fused after graph convolution processing at different convolution scales to obtain the car regression semantic feature map.

[0048] Graph convolution processing can be performed using the following formula: H l =sigmoid(A l ·H l-1 ·W l ), H l H is the output of the l-th convolutional layer. l-1 H is the output of the (l-1)th convolutional layer. 0 W is the pixel matrix representing the segmentation of the car boundary regression map, and W is the pixel matrix formed by the average pixel values ​​of each block region in the segmented car boundary regression. l Let A be the weight of the l-th convolutional layer. l is the adjacency matrix input to the l-th convolutional layer.

[0049] Further, in order to improve the effect of the graph convolution processing, in the process of the graph convolution processing, the adjacency matrix corresponding to the output graph of the previous layer is added to the initial adjacency matrix to obtain an updated adjacency matrix as the adjacency matrix of the current layer input.

[0050] Next, the detection frame corresponding to the maximum confidence value in the same detection region in the initial car classification graph is retained to obtain an optimal car classification graph capable of improving the calculation efficiency. The confidence is the intersection-over-union of the detection frame and the real frame (preset anchor frame).

[0051] Finally, the detection frame in which the classification type includes passengers and the classification probability corresponding to the classification type of passengers is greater than the probability threshold in the optimal car classification graph is retained to obtain a car passenger detection graph.

[0052] S3, based on the position and posture type of the passenger in each detection frame in the car passenger detection graph, it is judged whether the passenger is an elevator repairman; if all passengers are elevator repairmen, no car trapping warning information is sent; if all passengers are not elevator repairmen, S4 is executed.

[0053] Based on the position information and posture information of the passenger in the detection frame, it is judged whether the passenger is an elevator repairman. If all are elevator repairmen (by default, elevator repairmen are not allowed to enter the elevator car after elevator car maintenance), it is proved that the elevator car does not open the elevator door within the specified time because of normal maintenance, and the car trapping warning information does not need to be sent. If all passengers are not elevator repairmen, the judgment of elevator trapping and slow elevator running speed in S4 needs to be made.

[0054] The operation of judging whether the passenger is an elevator repairman is specifically: obtaining the pixel gradient value (the pixel difference between the current position point and the previous position point) of each position point of the current detection frame in the car passenger detection graph to obtain a current detection frame pixel gradient distribution graph; obtaining the position point in the current detection frame pixel gradient distribution graph whose pixel gradient is greater than the standard pixel gradient and whose straight-line distance from the detection frame is less than the distance threshold to obtain the current passenger edge contour point; after curve fitting of all passenger edge contour points, curve closure processing is performed to obtain the current passenger posture contour graph; the current passenger posture contour graph is subjected to posture classification processing to obtain the posture classification result of the current passenger; at the same time, the position coordinates of the center of the bottom of the current detection frame are taken as the position of the current passenger; if the posture classification result of the current passenger is the maintenance type and the corresponding posture type probability is not less than the posture probability threshold, and the position of the current passenger is in the maintenance position region, the current passenger is an elevator repairman; if the posture classification result of the current passenger is not maintenance, or the posture classification result of the current passenger is maintenance but the corresponding posture type probability is less than the posture probability threshold, the current passenger is not an elevator repairman.

[0055] The operation of the posture classification processing can be: using the obtained historical car posture database (historical car ordinary passenger posture data set and historical car elevator maintenance worker posture data set) to train a neural network to obtain a trained neural network; the current passenger posture contour map is processed by the trained neural network to obtain the posture classification result of the current passenger containing the posture type and the posture type probability. In the historical car posture database, each picture has a corresponding posture type label, and the posture type includes maintenance, ordinary riding, and refuge, etc. Among them, in the maintenance posture type, the passenger's body does not contact the car, but the hand contacts the car (direct contact or indirect contact), and the head line of sight direction points to the hand; in the refuge posture type, the passenger's body contacts the car, the passenger's back is straight and pastes the car wall, the hands and feet are separated, the knees are bent and squat, and / or the hand holds the elevator car wall.

[0056] The operation of the posture classification processing can also be: obtaining the current passenger posture contour map, and the similarity between each historical car posture graph in the historical car posture database, taking the label of the historical car posture graph corresponding to the maximum similarity value as the posture type of the current passenger, and taking the maximum similarity value as the posture type probability of the current passenger. The posture type and the posture type probability of the current passenger form the posture classification result of the current passenger. The operation of calculating the similarity has been described in S2, and therefore will not be described in detail here to save space.

[0057] In S4, the face region of each passenger on each detection frame in the car passenger detection graph is obtained as the region of interest of each passenger, and the emotional information of the region of interest of each passenger is obtained. If the emotional information of the passenger includes panic, the posture type of the passenger includes refuge, and the proportion of the number of passengers in the refuge area is greater than the proportion threshold, the passenger is trapped in the elevator car, and a car trapped person warning information is sent.

[0058] To further improve the accuracy of elevator car trapped person identification and avoid misjudgment of trapped person identification caused by the elevator speed being reduced and the elevator door not being opened within the preset time, if all passengers are not elevator maintenance workers, the face region of each passenger on each detection frame in the car passenger detection graph is obtained as the region of interest of each passenger. The emotional information of the region of interest of each passenger is obtained. If the emotional information of all passengers includes panic, the posture type of the passenger includes refuge posture, and the proportion of the number of passengers in the refuge area is greater than the proportion threshold, the passenger is trapped in the elevator car, and a car trapped person warning information is sent.

[0059] The region of interest is the region corresponding to the height of the first distance threshold from the top of the detection frame in the detection frame. The refuge area is an area with a distance from the elevator car wall less than the first distance threshold. The proportion of the number of passengers in the refuge area is the ratio of the number of passengers in the refuge area to the total number of passengers.

[0060] The emotion information of the passenger's region of interest is obtained based on a historical trapped passenger facial region dataset and a neural network, and the process is similar to the above-mentioned operation of classifying the posture by using the neural network. Therefore, the operation is not described in detail. In the historical trapped passenger facial region dataset, each picture has a corresponding emotion information classification label, and the emotion information types include panic, joy, calmness, etc. In the panic emotion information, the passenger's forehead is wrinkled, the eyebrows are raised and gathered, the upper eyelid is raised, the lower eyelid is tightened, or the line of sight is directed to the elevator door.

[0061] The embodiment also provides an elevator car trapped person recognition and early warning system based on image processing, which comprises:

[0062] The car enhanced image generation module is configured to obtain an in-car image if the closing time of the elevator car door exceeds the closing time threshold, and the in-car image is subjected to image enhancement processing to obtain a car enhanced image.

[0063] The car passenger detection map generation module is configured to obtain an initial car classification map by subjecting the car enhanced image to bounding box regression and classification processing, so that the car enhanced image contains multiple detection boxes in the same detection region, each detection box contains multiple classification types and corresponding classification probabilities; obtain an optimal car classification map by retaining the detection box corresponding to the maximum confidence value in the same detection region in the initial car classification map; and obtain a car passenger detection map by retaining the detection box in the optimal car classification map, wherein the classification type of the detection box includes a passenger, and the classification probability corresponding to the classification type of the passenger is greater than a probability threshold.

[0064] The elevator repairman judgment module is configured to judge whether the passengers are elevator repairmen based on the position and posture classification result of each passenger in the car passenger detection image. If all the passengers are elevator repairmen, no car trapping warning information is sent. If all the passengers are not elevator repairmen, the car trapping identification and warning information sending module is executed. The operation of judging whether the passengers are elevator repairmen specifically comprises: obtaining the pixel gradient value of each position point of the current detection frame image in the car passenger detection image to obtain a current detection frame pixel gradient distribution image; obtaining the position point with a pixel gradient greater than a standard pixel gradient and a distance from the detection frame less than a distance threshold in the current detection frame pixel gradient distribution image to obtain a current passenger edge contour point; after curve fitting of all the passenger edge contour points, curve closure processing is performed to obtain a current passenger posture contour map; the current passenger posture contour map is subjected to posture classification processing to obtain a current passenger posture classification result; the position coordinate of the bottom center of the current detection frame is taken as the position of the current passenger; if the posture type in the current passenger posture classification result is repair, the corresponding posture type probability is not less than a posture probability threshold, and the position of the current passenger is in a repair position area, the current passenger is an elevator repairman; if the posture type in the current passenger posture classification result is not repair, or the posture classification type is repair but the corresponding posture type probability is less than the posture probability threshold, the current passenger is not an elevator repairman.

[0065] The car trapping identification and warning information sending module is configured to obtain the face area of each passenger on each detection frame in the car passenger detection image as the region of interest of each passenger, and obtain the emotional information of the region of interest of each passenger. If the emotional information of the passenger includes panic, the posture type of the passenger includes an escape posture, and the proportion of the number of passengers in the escape area is greater than a proportion threshold, the passenger is trapped in the elevator car, and the car trapping warning information is sent.

[0066] The embodiment also provides an image processing-based elevator car trapping identification and warning method.

[0067] The embodiment also provides a computer readable storage medium for storing a computer program, wherein the computer program is executed by a processor to implement the image processing-based elevator car trapping identification and warning method.

[0068] The embodiment provides an elevator car person-trapping identification and early warning method based on image processing. First, according to the comparison result of the elevator car door closing time and the closing time threshold, the image in the car is obtained, and the quality of the image in the car is improved through image enhancement processing. Then, after the car enhanced image is subjected to boundary box regression and classification processing, the appropriate detection frame is screened out through the confidence, the classification type and the classification probability, and the car passenger detection image capable of reflecting the passenger quantity and the passenger position is obtained. Finally, based on the position information and the posture information of the passengers in the car passenger detection image, it is judged whether the passengers are elevator repair workers. If the passengers in the elevator car are not elevator repair workers, whether the passengers are trapped is judged based on the emotional information, the posture information of the passengers and the passenger quantity proportion information in the refuge area, so as to decide whether to send the car person-trapping early warning information. The method has high identification efficiency and good identification accuracy.

Claims

1. A method for identifying and warning of people trapped in an elevator car based on image processing, characterized in that, This includes the following operations: S1. If the elevator car door closing time exceeds the closing time threshold, the image inside the car is acquired. The image inside the car is then processed by image enhancement to obtain an enhanced image. Before the image enhancement process, the image inside the car undergoes color temperature compensation to obtain a corrected image, which is then used to perform the image enhancement process. Specifically, based on the color temperature difference between the image inside the car and the standard car image, and the red, green, and blue channel values ​​at each location point in the image inside the car, the red channel color temperature increment, green channel color temperature increment, and blue channel color temperature increment at each location point in the image inside the car relative to the standard car image are obtained. Based on the color temperature increments of the red channel, green channel, and blue channel at each location point in the image inside the car, the balance value of each location point in the image inside the car is obtained. Divide the pixel value of each location point in the image inside the car by the corresponding balance value to obtain the color temperature update value of each location point, and obtain the corrected image inside the car. S2. The enhanced car image is processed by bounding box regression and classification to obtain an initial car classification image containing multiple detection boxes in the same detection area, and each detection box contains multiple classification types and corresponding classification probabilities. The classification process is as follows: The similarity between the current detection box image and each standard physical object image in the physical object database is obtained from the bounding box regression map of the enhanced car image. The labels of the standard physical objects with similarity greater than a similarity threshold are used as the classification type of the current detection box, and the similarity of the classification type is used as the classification probability of the current detection box. The physical object regression map, the classification types of all detection boxes, and their corresponding classification probabilities form the initial car classification map. The similarity is obtained as follows: The image corresponding to the detection box is segmented to obtain a detection box segmentation map containing several segmented regions. Based on the pixel standard deviation of each segmented region in the detection box segmentation map, the pixel standard deviation of the corresponding initial region in the standard physical object map, and the pixel covariance of the corresponding initial region in the standard physical object map, the region similarity value between each segmented region and its corresponding initial region is obtained. The region similarity value between each segmented region and its corresponding initial region is multiplied by the weight of the corresponding segmented region and then summed to obtain the similarity. The optimal car classification map is obtained by retaining the detection boxes corresponding to the maximum confidence value in the same detection area within the initial car classification map. The optimal car classification map retains the detection boxes that include passenger as a classification type and whose classification probability is greater than the probability threshold, thus obtaining the car passenger detection map. S3. Based on the passenger position and posture classification results in each detection box of the car passenger detection map, determine whether the passenger is an elevator maintenance worker; if all passengers are elevator maintenance workers, do not send a car entrapment warning message. If none of the passengers are elevator maintenance workers, proceed to step S4; The specific steps for determining whether a passenger is an elevator maintenance worker are as follows: First, obtain the pixel gradient value of each point in the image corresponding to the current detection box in the passenger detection map of the car, thus obtaining the pixel gradient distribution map of the current detection box. Second, obtain the points in the pixel gradient distribution map of the current detection box where the pixel gradient is greater than the standard pixel gradient and the straight-line distance from the detection box is less than the distance threshold, thus obtaining the current passenger's edge contour points. Third, perform curve fitting on all passenger edge contour points and then perform curve closure processing to obtain the current passenger's posture contour map. Finally, perform posture classification processing on the current passenger's posture contour map to obtain the posture classification result of the current passenger. The coordinates of the bottom center of the current detection frame are used as the current passenger's position; If the current passenger's posture classification result shows that the posture type is "maintenance" and the corresponding posture type probability is not less than the posture probability threshold, and the current passenger's position is within the maintenance location area, then the current passenger is an elevator maintenance worker. If the current passenger's posture classification result is not "maintenance", or if the posture classification type is "maintenance" but the corresponding posture type probability is less than the posture probability threshold, then the current passenger is not an elevator maintenance worker. S4. Obtain the facial region of the passenger in each detection box in the passenger detection map of the car, as the region of interest for each passenger, and obtain the emotional information of the region of interest for each passenger; the region of interest is the area corresponding to the top of the detection box and the height from the top of the detection box at a distance of a first distance threshold. If a passenger's emotional information includes panic, and the passenger's posture type includes a risk-avoidance posture, and the number of passengers located in a risk-avoidance area is greater than the percentage threshold, then a warning message for passengers trapped in the elevator car will be sent.

2. The elevator car entrapment identification and early warning method based on image processing according to claim 1, characterized in that, The bounding box regression process also includes extracting semantic features from the car boundary regression map to obtain a car regression semantic feature map, which is used to perform classification processing. The semantic feature extraction process is as follows: the car boundary regression map is fused after graph convolution processing at different convolution scales to obtain the car regression semantic feature map.

3. The elevator car entrapment identification and early warning method based on image processing according to claim 2, characterized in that, During graph convolution processing, the adjacency matrix corresponding to the output graph of the previous layer is added to the initial adjacency matrix to obtain the updated adjacency matrix, which is then used as the adjacency matrix input to the current layer.

4. The elevator car entrapment identification and early warning method based on image processing according to claim 1, characterized in that, The image enhancement processing in S1 is specifically as follows: The image inside the car is Gaussian blurred to obtain a Gaussian image of the car; the Gaussian image of the car and the image inside the car are subtracted to obtain a sub-detail image of the car; the sub-detail image of the car and the image inside the car are fused to obtain an enhanced image of the car.

5. An elevator car entrapment identification and early warning system based on image processing, characterized in that, include: The car enhancement image generation module starts timing after the elevator car door closes. If the elevator car door running time exceeds the running time threshold, it acquires an image inside the car. This image undergoes image enhancement processing to obtain an enhanced car image. Before the image enhancement processing, the image inside the car undergoes color temperature compensation processing to obtain a corrected image inside the car, which is then used to perform the image enhancement processing. Specifically, based on the color temperature difference between the image inside the car and the standard car image, and the red, green, and blue channel values ​​at each location point in the image inside the car, it obtains the red channel color temperature increment, green channel color temperature increment, and blue channel color temperature increment at each location point in the image inside the car relative to the standard car image. Based on the color temperature increments of the red channel, green channel, and blue channel at each location point in the image inside the car, the balance value of each location point in the image inside the car is obtained. Divide the pixel value of each location point in the image inside the car by the corresponding balance value to obtain the color temperature update value of each location point, and obtain the corrected image inside the car. The elevator passenger detection map generation module is used to process the enhanced elevator image through bounding box regression and classification to obtain an initial elevator classification map containing multiple detection boxes within the same detection area, with each detection box containing multiple classification types and corresponding classification probabilities. The classification process is as follows: Obtain the similarity between the current detection box image and each standard physical object image in the physical object database from the bounding box regression map obtained by bounding box regression processing of the enhanced car image. The labels of the standard physical objects with similarity greater than a similarity threshold are used as the classification type of the current detection box, and the similarity corresponding to the classification type is used as the classification probability of the current detection box. The car boundary regression map, along with the classification types and corresponding probabilities of all detection boxes, forms the initial car classification map. The similarity is obtained by segmenting the image corresponding to the detection box to obtain a detection box segmentation map containing several segmented regions. Based on the detection box segmentation map, the similarity of each segmented region... The pixel standard deviation, the pixel standard deviation of the corresponding initial region in the standard physical image for each segmented region, and the pixel covariance of the corresponding initial region in the standard physical image for each segmented region are used to obtain the region similarity value between each segmented region and its corresponding initial region. The region similarity value between each segmented region and its corresponding initial region is multiplied by the weight of the corresponding segmented region and then summed to obtain the similarity score. The detection boxes corresponding to the maximum built-in confidence of the same detection region in the initial car classification image are retained to obtain the optimal car classification image. The detection boxes in the optimal car classification image that include the category of passenger and whose category probability is greater than the probability threshold are retained to obtain the car passenger detection image. The elevator repairman determination module is used to determine whether a passenger is an elevator repairman based on the classification results of the position and posture of passengers within each detection frame in the passenger detection image of the car. If all passengers are elevator repairmen, no car entrapment warning information is sent; if none of the passengers are elevator repairmen, the car entrapment identification and warning information sending module is executed. The operation of determining whether a passenger is an elevator repairman specifically involves: obtaining the pixel gradient value of each position point in the image corresponding to the current detection frame in the passenger detection image of the car to obtain the pixel gradient distribution map of the current detection frame; obtaining the position points in the pixel gradient distribution map of the current detection frame where the pixel gradient is greater than the standard pixel gradient and the straight-line distance from the detection frame is less than the distance threshold, and thus determining the current passenger. Edge contour points; after curve fitting of all passenger edge contour points, curve closure processing is performed to obtain the current passenger posture contour map; the current passenger posture contour map is processed by posture classification to obtain the current passenger posture classification result; the position coordinates of the bottom center of the current detection box are taken as the current passenger position; if the posture type in the current passenger posture classification result is maintenance, and the corresponding posture type probability is not less than the posture probability threshold, and the current passenger position is within the maintenance position area, then the current passenger is an elevator maintenance worker; if the posture type in the current passenger posture classification result is not maintenance, or the posture type is maintenance but the corresponding posture type probability is less than the posture probability threshold, then the current passenger is not an elevator maintenance worker; The elevator car entrapment identification and early warning information sending module is used to acquire the facial region of the passenger in each detection box in the passenger detection map of the elevator car, as the region of interest for each passenger, and to acquire the emotional information of each passenger's region of interest; the region of interest is the area corresponding to the top of the detection box and the height at which the distance from the top of the detection box is a first distance threshold; if the passenger's emotional information includes panic, and the passenger's posture type includes risk avoidance, and the proportion of passengers in the risk avoidance area is greater than the proportion threshold, then the passenger is trapped in the elevator car, and an elevator car entrapment early warning information is sent.

6. An elevator car entrapment identification and early warning device based on image processing, characterized in that, It includes a processor and a memory, wherein the processor implements the image processing-based elevator car entrapment identification and early warning method as claimed in any one of claims 1-4 when executing a computer program stored in the memory.

7. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the image processing-based elevator car entrapment identification and early warning method as described in any one of claims 1-4.