Detection method of objects placed on the ground in elevators in multiple scenarios

Abstract

The present invention discloses a method for detecting objects placed on the ground in an elevator under multiple scenarios, including: data collection and labeling: collecting video data in the elevator, processing and labeling the video data accordingly, and training the labeled data; determining an operation mode: the operation mode includes an indoor mode and an outdoor mode. When it is the outdoor mode, an image enhancement algorithm is used to convert a low-light or high-light image into a normal-light image; an operation stage: determining whether the operation mode is an indoor mode or an outdoor mode, and then detecting objects placed on the ground in the elevator; the present invention is suitable for detecting objects left on the ground in an elevator car under different lighting conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of elevator equipment, and in particular to a method for detecting objects placed on the ground in an elevator in multiple scenarios. Background Art

[0002] Regarding the detection of objects left on the ground in elevators, from the perspective of adaptability to different lighting environments in elevators: traditional detection methods are suitable for elevator scenes under constant light sources, but not for elevators that can be illuminated by natural light, such as sightseeing elevators, transparent elevators in villas, etc., for the detection and early warning of objects left on the ground in elevator cabins, so its versatility in multiple scenarios is slightly insufficient; from the perspective of the classification and identification of objects left on the ground: traditional detection methods do not classify and identify objects left on the ground, so they cannot distinguish the value of objects on the floor of the elevator. When the information determined by the program as objects left on the ground is sent back to the property and other relevant personnel, the relevant personnel are required to The secondary determination of the item category reduces processing efficiency to a certain extent. From the perspective of the completeness of the detection process: during the continuous detection of the video, no detection of items left on the ground is performed when there are people in the elevator. This is because during the period when there are people in the elevator, items may fall and items left on the ground may be taken away by others, so the function is slightly insufficient. From the perspective of tracking the disappearance and whereabouts of items left on the ground: the traditional detection method itself does not detect items when there are people in the elevator, so there is naturally no behavior recognition function to identify the behavior of passengers picking up and handling items left on the ground, and thus to a certain extent, it cannot track the whereabouts of items left on the ground. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a method for detecting objects left on the ground in an elevator under multiple scenarios. The present invention is suitable for detecting objects left on the ground in an elevator car under different lighting conditions, and can classify and identify common garbage and objects left on the ground in the elevator car. When there are people in the elevator car, a certain degree of classification detection of objects left on the ground can be performed, and a certain degree of suspect determination and analysis capability is available if the objects left on the ground are taken away by others.

[0004] To achieve the above object, the technical solution adopted by the present invention is: a method for detecting objects placed on the ground in an elevator in multiple scenarios, comprising the following steps:

[0005] S100, data collection and labeling: Collect video data in the elevator, and process and label the video data accordingly to form a floor information dataset A, a ground object dataset B, a garbage object dataset C, a human body dataset D, a behavior recognition dataset E of people in the elevator on left-behind objects, and a pedestrian re-identification dataset F;

[0006] S200, training the labeled data: training the labeled data set to obtain a target detection model M1, a skeleton behavior recognition model M2, and a pedestrian re-identification model M3;

[0007] S300, determining an operation mode: the operation mode includes an indoor mode and an outdoor mode. When the operation mode is the outdoor mode, an image enhancement algorithm is used to convert a low-light or high-light image into a normal-light image;

[0008] S400, operation stage: determine whether the operation mode is indoor mode or outdoor mode according to step S300, and then detect the objects left on the ground in the elevator using the target detection model M1, skeleton behavior recognition model M2 and pedestrian re-identification model M3 of step S200 trained according to the floor information dataset A, ground object dataset B, garbage object dataset C, human body dataset D, elevator occupant behavior recognition dataset E and pedestrian re-identification dataset F of step S100.

[0009] As a preferred implementation, step S100 specifically includes the following steps:

[0010] S101, obtaining video data DT1 collected by the elevator camera within a recent specified time period;

[0011] S102, obtaining video data DT2 containing a floor information display screen from video data DT1, and classifying and labeling the video data DT2 according to the floor information of the display screen to form a floor information data set A;

[0012] S103, perform human detection on the video data DT1 to obtain video segment data DT3 without people and video segment data DT4 with people; classify and label the video data containing objects placed on the ground in the video segment data DT3 without people according to the categories of the objects placed on the ground, to form a dataset of objects placed on the ground B; classify and label the video data containing garbage in the obtained video segment data DT3 without people according to the categories of garbage, to form a dataset of garbage objects C;

[0013] S104, select and mark the ground area in the elevator in the recent day in the unmanned video segment data DT3, select the car rectangular frame ground area of ​​the best picture as a single car rectangular frame ground area data set, and record the corresponding coordinate data;

[0014] S105, marking the human body in the acquired video segment data DT4 data with people, to form a human body data set D;

[0015] S106. Obtain the video data DT5 containing user behavior from the video segment data DT4 containing people, use the AlphaPose human posture estimation algorithm to obtain the human joint information of the people in the human detection area containing user behavior, save the skeleton diagram, and define the behavior segmentation labels of people towards the objects left on the ground, ensure that each video has one label, and the video time length is less than two seconds, and number the video folders and establish indexes of behavior labels and video files, so as to obtain the behavior recognition data set E of people in the elevator towards the objects left behind.

[0016] As another preferred implementation, the step S100 further includes the following steps:

[0017] S107, if the operation mode is the outdoor mode, the video data containing the objects left on the ground in the acquired video segment data DT3 without people is used to obtain the range of illumination components that can normally identify the video frame of the objects left on the ground by using an image enhancement algorithm;

[0018] S108. Detect and track pedestrians in the elevator video data and capture pedestrian video frames, set the elevator user confidence threshold filtering operation and manual filtering operation to obtain a preliminary pedestrian re-identification data set, and then use the FastReID-based model and feature extraction components to extract pedestrian features. The extracted pedestrian features are clustered and frame archiving and manual screening are performed to obtain a pedestrian re-identification data set F with the corresponding ID in the corresponding folder.

[0019] As another preferred implementation, the step S200 specifically includes the following steps:

[0020] S201, using the YOLOv5 target detection algorithm to perform sample balanced model training on the floor information dataset A, the ground object dataset B, the garbage object dataset C, and the human body dataset D, to obtain a target detection model M1;

[0021] S202, using the ST-GCN skeleton behavior recognition algorithm to convert the behavior recognition data set E of people in the elevator to the objects left on the ground into a file suitable for training, and then perform model training to obtain a skeleton behavior recognition model M2;

[0022] S203: Use the FastReID-based person re-identification model to perform model training to obtain a person re-identification model M3.

[0023] As another preferred embodiment, in step S300, the image enhancement algorithm includes a visual communication frame enhancement module, which is used to process low-light and high-light video frames and convert them into normal-light frames. In the process of iterative illumination adaptive image enhancement, when the illumination components of the enhanced image are extracted and compared with the range of normal illumination components, if they are not in a reasonable range of normal illumination, a convergence rationality judgment and trend processing mechanism is embedded to promote the optimization of the image enhancement process.

[0024] As another preferred implementation, the image enhancement algorithm is used to convert the low-light or high-light image into a normal-light image as follows:

[0025] After acquiring the RGB frame, convert it into an HSI frame, and then use the multi-scale Gaussian filter function to compress the image dynamic range while calculating the image illumination component and recording it, and then determine whether the maximum number of iterations has been reached. If so, convert it into RGB frame data output, otherwise determine whether it is a normal illumination video frame. If it is a normal illumination video frame, convert it into an RGB frame for output;

[0026] Otherwise, it is determined whether it is the first time to determine whether this frame is a normal image. If not, the iteration flag position is 1 and the number of iterations is recorded. Otherwise, the iteration flag position is 0. Then it is determined whether it is a low-light image. If it is a low-light frame, the image illumination component flag position is 0, and the camera response model is used. Where f and h represent the brightness mapping function and exposure rate of the camera response model, respectively, and a, b and γ represent the camera parameters and adjustable parameters, respectively;

[0027] Under the condition that the low-light image F0 is underexposed, in order to make the virtual exposure image F1 better exposed, the optimal h value is determined and then the pixels with higher exposure in F0 are removed, and then the pixel point set V with underexposed pixels is constructed according to V={F0(x)|G(x)<ξ}, where G and ξ represent the illumination component of F0 and the expected normal illumination component interval respectively;

[0028] The brightness matrix is ​​constructed using the brightness of all pixels in V to determine the h value, based on the information entropy of the brightness matrix B of all pixels in V Among them, p i Represents the probability of each gray level i appearing in the brightness matrix, and then maximizes the image entropy to determine the optimal exposure rate value h and obtain the virtual exposure image F1;

[0029] Then, it is judged whether the iteration flag is 1. If it is not 1, fusion processing is performed. Otherwise, convergence rationality judgment and trend processing are performed: the illumination component after the last image enhancement is compared with the illumination component of the current image. If it is smaller than the illumination component after the last image enhancement, based on the positive correlation between the weight matrix and the illumination of the scene in the complex illumination image, the enhancement parameter of the adjustment weight matrix J is set to the calculated value:

[0030] Initial value of enhancement parameter * (100% + 5% * number of iterations)

[0031] Then F0 and F1 are fused:

[0032] (F2=J°F0+(1-J°F1)), where J and operator ° respectively represent the multiplication of the corresponding elements of the weight matrix and the matrix with uniform number of rows and columns;

[0033] Since the weight value of well-exposed pixels is higher than that of underexposed pixels, the weight matrix J is positively correlated with the illumination of the scene in the complex illumination image, so J = G τ , τ represents the parameter controlling the enhancement degree, and the fused image F2 is obtained. The returned image is used to extract the illumination component again using the multi-scale Gaussian filter function to determine whether the image is a normal illumination image. If not, it is iterated again, otherwise the image is converted into an RGB image output;

[0034] If it is a high-light image, the image grayscale value is inverted, the image illumination component flag is set to 1, and low-light image enhancement processing is performed. After processing, the grayscale value is inverted again to determine whether the image is a normal-light image. If not, it is iterated again, otherwise the image is converted into an RGB image output.

[0035] As another preferred implementation, the step S400 specifically includes the following steps:

[0036] S401, the operation mode setting takes effect: before the camera is turned on, the operation mode is set to indoor mode or outdoor mode, and the default is outdoor mode;

[0037] S402, automatic setting of parameters of the floor area of ​​the rectangular frame in the elevator: automatically setting the corresponding area according to the existing coordinate data of the floor area of ​​the rectangular frame of the elevator;

[0038] S403, frame update;

[0039] S404, determining the operation mode, if it is the outdoor mode, performing image enhancement, and then proceeding to the next step; if it is the indoor mode, no processing is performed, and directly proceeding to the next step;

[0040] S405, target detection and result recording in the rectangular floor area of ​​the elevator: the position and number of human bodies in the rectangular floor area of ​​the elevator can be used to identify the position, type and number of items on the floor, the position, type and number of garbage, and the floor information for target detection, and the relevant information is recorded;

[0041] S406, whether there is human body detection result: through the data record of the process detection of whether there are passengers in the elevator car area, it is determined that if there are passengers in the elevator car area, the performance flag position is 1, and it is determined whether the proportion of the human body detection frame part in the elevator floor area exceeds the specified threshold. If it exceeds, the flag is reset, and the process returns to step S403; otherwise, the process proceeds to the next step; if there is no person in the elevator, the performance flag position is 0, and the process proceeds to the next step;

[0042] S407, garbage identification: Based on the target detection result record in the rectangular ground area of ​​the elevator, if it is determined that there is garbage, the video frame and the garbage information are sent back to the background and the next step is entered; if not, the next step is directly entered;

[0043] S408, processing statistical sorting of relevant information of identifiable items placed on the ground and unidentifiable areas in two threads: one thread performs statistical sorting of relevant information of identifiable items placed on the ground; the second thread performs frame difference between the ground area except the detection frame of people, items placed on the ground, and garbage in the current video frame and the same ground area of ​​the stored empty ladder frame, and then checks whether there is a pixel area within the set threshold range by the absolute value of the difference between the absolute value of the pixel difference after the frame difference and the absolute value of the adjacent difference value. If there is, there is no unidentifiable object, and directly enters the comparison of the identifiable ground object area with the convex hull area and the garbage area. If there is no such information, the pixels not within the threshold range are displayed in the form of a convex hull in the frame, and the convex hull containing pixels less than the specified threshold is removed. The identifiable leftover item area is compared with the convex hull area and the garbage area, and the same area is deleted. The information of the unidentifiable area and the identifiable items on the ground are sorted again. It is determined whether the number of identifiable items on the ground and the unidentifiable area is greater than 0. If not, the flag is reset, and the process returns to step S403. Otherwise, the process proceeds to the next step.

[0044] S409, determining whether it is the first time to detect the identifiable ground object and the unidentifiable area information on the day, if so, reset the flag bit and return to step S403, otherwise, proceed to the next step;

[0045] S410, determining by a behavior recognition algorithm whether a person has held the identifiable ground object or the unidentifiable area in the previous frame, if so, the object or area is preliminarily determined to be a non-left-on-ground object, and proceeding to the next step; otherwise, the object or area is preliminarily determined to be a left-on-ground object, and proceeding to the next step;

[0046] S411, compare the identifiable items on the ground and the unidentifiable area of ​​the previous frame to see if the same area of ​​the current frame has disappeared. If not, use the pedestrian re-identification algorithm, behavior recognition algorithm and existing record information to determine whether the same person in this frame and the previous frame has the behavior of holding the identifiable items on the ground or the unidentifiable area. If not, send the video frame and the information of the lost items back to the property and other relevant departments, reset the flag bit, and return to step S403; if yes, it is a normal person in the elevator car holding his own items, reset the flag bit, and return to step S403; otherwise, go to the next step;

[0047] S412, detect whether someone has left the elevator in this frame. If no one has left the elevator, use the behavior recognition algorithm to determine whether someone has picked up or held the shelf items or areas. If yes, determine whether the shelf items or unrecognizable areas in the same area of ​​the previous frame are determined to be items left on the ground. If yes, send the video frame of the target person and the information about the lost items to the background, reset the flag, and return to step S403. If not, reset the flag, and return to step S403. On the contrary, if the recognizable shelf items or unrecognizable areas are blocked by a person, send the video frame and the information about the blocked shelf items to the background, reset the flag, and return to step S403. If someone has left the elevator, proceed to the next step.

[0048] S413, using the behavior recognition algorithm to determine whether someone has picked up the items on the shelf but has not touched the items and the unrecognizable area in the previous frame. If not, save and record the information of the person taking away the items on the shelf, reset the flag, and return to step S403; if there is such behavior, proceed to the next step;

[0049] S414, judging whether the placed item or the unrecognizable area in the same area of ​​the previous frame is determined as an item left on the ground according to the record, if not, framing the video frame of the suspect who took away the placed item and may have picked up the placed item but did not touch the item and the unrecognizable area, and saving the information that the items of others were taken away by someone other than the suspect, resetting the flag bit, and returning to step S403; otherwise, going to the next step;

[0050] S415. The video frame of the target person in the previous frame in which the target person picks up the placed items but does not touch the items and the unrecognizable area, and the information about the lost items are saved and recorded. The information about the person who may have taken the placed items is also saved and recorded. The flag is reset and the process returns to step S403.

[0051] The beneficial effects of the present invention are:

[0052] 1. The present invention utilizes a visual communication image enhancement algorithm to perform iterative adaptive image enhancement on low-light and high-light images, convert low-light and high-light images into normal-light images, and convert abnormal-light images into normal images with more information, higher clarity, and higher contrast, thereby increasing target detection and recognition rates.

[0053] 2. The present invention utilizes a self-trained target detection model to detect classifiable objects placed on the ground and unrecognizable areas, and adopts a joint detection method defined by pixel convex hull areas to perform detailed classification detection processing on the areas left on the ground, thereby being able to inform property and other relevant personnel of the detailed categories of objects left in the areas of the ground, enabling them to more effectively obtain more effective information.

[0054] 3. The present invention can detect items placed on the ground when there are people in the elevator car, without missing the frequency of events;

[0055] 4. The present invention adds the premise of detecting items placed on the ground when there are people in the elevator car, and uses a behavior recognition algorithm (the model is a self-trained behavior recognition model) to identify the behavior of people who take away items placed on the ground, thereby locating the whereabouts of the items placed on the ground to a certain extent.

[0056] 5. The present invention can classify and identify items placed on the ground in an elevator car under different lighting conditions through visual communication image enhancement algorithms, behavior recognition algorithms, target detection algorithms, pedestrian re-identification and other visual processing methods. It can also timely classify, detect and identify items placed on the ground when there are people but the elevator floor area is not fully occupied from the camera's perspective and when there is no one. On the premise that the items are lost on the ground to a certain extent, an analysis result is given on the whereabouts of the items, thereby playing a security role for the property of elevator users to a certain extent, and using the detected garbage category to inform the property and other departments to purify the environment inside the elevator. At the same time, this early warning method can be widely used in elevator cars under different lighting conditions to a certain extent, and is suitable for the interior of elevator cars in various environments (tourist attractions, family residences, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of the application scenario structure of an embodiment of the present invention;

[0058] Figure 2 is a workflow diagram of a visual communication frame enhancement module in an embodiment of the present invention;

[0059] Figure 3 It is a flowchart of the operation phase in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0061] Example

[0062] like Figure 1 As shown, this embodiment is based on Figure 1 The following is a specific description of the car application scenario shown in the figure:

[0063] 1. There is a rear camera in the car, which is located at the rear side of the car.

[0064] 2. The rear camera has a digital recognition function and can recognize the numbers displayed on the display screen inside the elevator. At the same time, the rear camera has functions such as human body detection, behavior recognition, garbage recognition, detection of objects placed on the ground, pedestrian re-identification, and image enhancement.

[0065] 3. At least one of the top and walls inside the elevator is made of transparent material or is the interior of a common elevator (none of the top, bottom, and surrounding walls inside the elevator is made of transparent material).

[0066] A method for detecting objects placed on the ground in an elevator under multiple scenarios is divided into a data collection and labeling stage, a training stage, an operation mode establishment and submodule stage, and an operation stage.

[0067] In the data collection and annotation stage, the main process is as follows:

[0068] Step 1, obtain the video data DT1 of the elevator car in the current application scenario (inside a residence, inside and outside a shopping mall, etc.) collected by the camera within a recent specified time period (the default is one month, and the user can specify the time period threshold during specific implementation);

[0069] Step 2, obtain the video data DT2 containing the floor information display screen from DT1 and classify and label the data according to the floor information of the display screen to form a floor information data set A;

[0070] Step 3, use the YOLOv5 target detection algorithm to perform human detection on the video data DT1, and obtain the video segment data DT3 without people, and the data with people is DT4; classify and annotate the video data containing objects on the ground in the obtained video segment data DT3 without people according to the category of objects on the ground, forming a dataset of objects on the ground B, and classify and annotate the video data containing garbage (garbage category is proposed according to the scene environment) in the obtained video segment data DT3 without people according to the category of garbage, forming a dataset of garbage objects C;

[0071] Step 4: Mark the ground area of ​​the elevator in the recent day in the unmanned video segment data DT3, select the car rectangular frame ground area of ​​the best picture as a single car rectangular frame ground area data set, and record the corresponding coordinate data;

[0072] Step 5: Mark the human body in the acquired video segment data DT4 to form a human body data set D;

[0073] Step 6, obtain the video data DT5 containing behaviors such as elevator users picking up items on the ground (the arms and torso are not in a normal standing posture, with bending knees, picking up with two or one hand, etc.), carrying items on the ground (the arms and torso are not in a normal standing posture, with bending knees, picking up with two or one hand, etc.), and holding items on the ground (with upright torso, arms slanted, etc.) in the video segment data DT4 with people, and use the AlphaPose human posture estimation algorithm to obtain the human joint information of the people in the human detection area containing the above behaviors, and save the skeleton diagram (save the sequence number and coordinates of each joint point in each frame, and the corresponding skeleton diagram of each frame), and define the behavior segmentation label of the person on the left-behind items, ensure that there is one label for each video, and the video time length is less than two seconds, and number the video folders and establish the index of the behavior label and the video file, so as to obtain the behavior recognition data set E of the person on the left-behind items in the elevator;

[0074] Step 7: If the application scenario is an elevator outside a shopping mall, the video data of the unoccupied video segment data DT3 containing the items on the ground is obtained using an image enhancement algorithm (see the submodule stage image enhancement module for details) to obtain the illumination component range R of the video frame that can normally identify the left-behind items;

[0075] Step 8. Use the YOLOv5 target detection algorithm and DeepSort multi-target tracking algorithm to detect and track pedestrians in the elevator scene video and capture pedestrian video frames. Set the confidence threshold filtering operation and manual filtering operation for elevator users to obtain a preliminary pedestrian re-identification data set (ID numbering for pedestrians and serial numbering for pedestrian images under this ID). Then use the FastReID-based model (the backbone network feature extraction model is ResNet50, and the feature integration part uses Attention pooling operation). Use the feature extraction component to extract pedestrian features, cluster the extracted pedestrian features, archive the frames, and manually screen them to obtain the pedestrian re-identification data set F of the corresponding ID in the corresponding folder (only data of the same ID in a folder).

[0076] During the training phase, the main process is as follows:

[0077] Step 1: Use the YOLOv5 target detection algorithm to perform sample balanced model training on data sets A, B, C, and D to obtain the target detection model M1 (whose detection categories include human body, various recorded ground objects, garbage objects, and floor display screen information);

[0078] Step 2: Use the ST-GCN skeleton behavior recognition algorithm to convert the data set E into a file suitable for training, and then perform model training to obtain the skeleton behavior recognition model M2;

[0079] Step 3: Use the FastReID-based person re-identification model (the feature extraction model of the backbone network is ResNet50, and the feature integration part uses the Attention pooling operation) to train the model and obtain the person re-identification model M3.

[0080] In the operation mode establishment and sub-module stage, the main process is as follows:

[0081] During the operation mode establishment phase:

[0082] The inherent mode is divided into indoor and outdoor modes. Since the indoor mode is in an elevator in a constant environment (i.e., the light source is constant), it is intended to determine that there is no high light or low light in the elevator, so the image enhancement algorithm is not used. The outdoor mode is that the lighting environment in the elevator will be affected by changes in natural light (sunlight), such as transparent elevators in villas, elevators with one or several sides of transparent glass in shopping malls, and elevators outside shopping malls that can be exposed to sunlight. Therefore, an image enhancement algorithm is required to convert low-light or high-light images into normal-light images, so that the low-light or high-light images are clearer and richer in information.

[0083] In the submodule phase:

[0084] The visual communication frame enhancement module in the submodule is mainly used to process low-light and high-light video frames, converting them into normal-light frames, so that more information can be extracted after enhancement, and the contrast is significantly enhanced, and the clarity is higher. When the present invention uses this method, in the process of iterative illumination adaptive image enhancement, when comparing the illumination components of the enhanced image with the range of the normal illumination components after extraction, if it is not in the reasonable range of normal illumination, a mechanism of convergence rationality judgment and trend processing is embedded to promote the optimization of the image enhancement process.

[0085] The workflow diagram of the visual communication frame enhancement module is as follows: Figure 2As shown in the figure: after acquiring the RGB frame, it is converted into an HSI frame (the operation on the I component has no effect on the H component and the S component), and then the multi-scale Gaussian filter function is used to compress the dynamic range of the image while calculating the illumination component of the image (the illumination component of the illumination image obtained is extracted by Gaussian filter functions of 4 different scales, and the weight of the illumination component extracted at each scale is defined as one-fourth, and the illumination level of the illumination image obtained is judged according to the extracted illumination component) and the illumination component is recorded, and then it is determined whether the maximum number of iterations is reached. If it is reached, it is converted into RGB frame data output, otherwise it is determined whether it is a normal illumination video frame. If it is a normal illumination video frame, it is converted into an RGB frame for output; otherwise it is determined whether it is the first time to determine whether this frame is a normal image. If not, the iteration flag position is 1 and the number of iterations is recorded, otherwise, the iteration flag position is 0; then it is determined whether it is a low-light image. If it is a low-light frame, the image illumination component flag position is 0, and the camera response model is used. Where f and h represent the brightness mapping function and exposure rate of the camera response model, respectively; a, b and γ represent the camera parameters and adjustable parameters, respectively; the camera parameters a and b can be set to -0.33 and 1.13, respectively; the model under this parameter is suitable for most cameras; under the condition that the low-light image F0 is underexposed, in order to make the virtual exposure image F1 better exposed, the optimal h value is determined---then the pixels with higher exposure in F0 are removed, and then the pixel point set V with underexposed pixels is constructed, according to V = {F0(x)|G(x)<ξ}, where G and ξ=0.5 represent the illumination component of F0 and the expected normal illumination component interval, respectively. The brightness matrix is ​​constructed using the brightness of all pixels in V to determine the h value, and the information entropy of the brightness matrix B of all pixels in V is used. (where p i Represents the probability of each gray level i appearing in the brightness matrix), and then maximizes the image entropy to determine the optimal exposure rate value h) to obtain the virtual exposure image F1; then judge whether the iteration flag is 1, if not, perform fusion processing, otherwise perform convergence rationality judgment and trend processing (compare the illumination component after the last image enhancement with the illumination component of the current image, if it is less than the illumination component after the last image enhancement, based on the positive correlation between the weight matrix and the illumination of the scene in the complex illumination image, the enhancement parameter of the adjustment weight matrix J is set to the calculated value (enhancement parameter initial value * (100% + 5% * number of iterations)), and then F0 and F1 are fused (F2 = J°F0 + (1-J°F1)), where J and operator ° respectively represent the multiplication of the corresponding position elements of the weight matrix and the matrix with a uniform number of rows and columns; based on the fact that the weight value of pixels with good exposure is higher than that of pixels with insufficient exposure, the weight matrix J is positively correlated with the illumination of the scene in the complex illumination image, so J = G τ, τ represents the parameter controlling the enhancement degree, and its value is generally 0.5), and the fused image F2 is obtained. The returned image is again extracted with the multi-scale Gaussian filter function to determine whether the image is a normal illumination image. If not, it is iteratively processed again, otherwise the image is converted to an RGB image for output; if it is a high illumination image, the image grayscale value is inverted, the image illumination component flag is set to 1, and low illumination image enhancement is performed. After processing, the grayscale value is inverted again to determine whether the image is a normal illumination image. If not, it is iteratively processed again, otherwise the image is converted to an RGB image for output.

[0086] During the operation phase, the main process is as follows:

[0087] Step 1: The operation mode setting takes effect. Before the camera is turned on, the software installer sets the operation mode to indoor or outdoor mode based on factors such as whether there is sunlight in the elevator and the installation location of the elevator itself. The default setting is outdoor mode.

[0088] Step 2: Automatically set the parameters of the car rectangular frame ground area. Automatically set the corresponding area based on the existing car rectangular frame ground area coordinate data;

[0089] Step 3: frame update;

[0090] Step 4: Determine the operating mode. If it is an outdoor mode, perform image enhancement and then proceed to the next step. If it is an indoor mode, do not perform any processing and directly proceed to the next step.

[0091] Step 5: Target detection and result recording in the rectangular floor area of ​​the car; the position and number of human bodies in the rectangular floor area of ​​the car can be used to identify the position, type and number of items on the ground, the position, type and number of garbage, and the floor information for target detection, and the relevant information is recorded;

[0092] Step 6: Determine whether the human body detection result is correct; through the data record of the process detection of whether there are passengers in the elevator car area, determine if there are passengers in the elevator car area, the performance flag position is 1, and determine whether the proportion of the human body detection frame in the elevator floor area exceeds the specified threshold (the default is 60%). If it exceeds, the flag is reset and the process returns to step 3; otherwise, proceed to the next step; if there is no person in the elevator, the performance flag position is 0 and the process proceeds to the next step;

[0093] Step 7, garbage identification: Based on the target detection result record in the rectangular ground area of ​​the car, if it is determined that there is garbage, the video frame and garbage information are sent back to the property management department to proceed to the next step; if not, directly proceed to the next step;

[0094] Step 8, two threads are used to process the statistical sorting of the relevant information of the identifiable objects placed on the ground and the unidentifiable areas; one thread performs the statistical sorting of the relevant information of the identifiable objects placed on the ground; the second thread performs frame difference between the ground area except the detection frame of people, objects placed on the ground, and garbage in the current video frame and the same ground area of ​​the stored empty ladder frame, and then the absolute value of the difference between the absolute value of the pixel difference after the frame difference and the absolute value of the adjacent difference value is checked to see whether there is a pixel area within the set threshold range. If so, there is no unidentifiable object, and the comparison of the identifiable ground object area with the convex hull area and the Garbage area, delete the same area, sort out the information of unrecognizable area and recognizable items on the ground again; if there is none, display the pixels that are not within the threshold range in the form of convex hull in the frame, remove the convex hull containing pixels less than the specified threshold, compare the recognizable leftover area with the convex hull area and the garbage area, delete the same area, sort out the information of unrecognizable area and recognizable items on the ground again; determine whether the number of recognizable items on the ground and unrecognizable area is greater than 0, if not, reset the flag bit, and return to step 3; otherwise, go to the next step;

[0095] Step 9, determine whether this identifiable ground object and this unidentifiable area information are detected for the first time today, if so, reset the flag bit and return to step 3, otherwise, proceed to the next step;

[0096] Step 10: Use a behavior recognition algorithm to determine whether someone has held the identifiable ground object or the unidentifiable area in the previous frame. If so, the object or area is preliminarily determined to be a non-left-on-the-ground object, and the process proceeds to the next step; otherwise, the object or area is preliminarily determined to be a left-on-the-ground object, and the process proceeds to the next step;

[0097] Step 11, compare the identifiable ground objects and the unidentifiable areas of the previous frame to see if the same area of ​​the current frame has disappeared. If not, use the pedestrian re-identification algorithm (pedestrian re-identification model is a self-training model), behavior recognition algorithm (behavior recognition model is a self-training model) and existing recorded information (target information) to determine whether the same person in this frame and the previous frame has the behavior of holding the identifiable ground objects or the unidentifiable area. If not, send the video frame and the information of the lost items back to the property and other relevant departments, reset the flag, and return to step 3; if yes, it is a normal person in the elevator car holding his own items, reset the flag, and return to step 3; otherwise, go to the next step;

[0098] Step 12, detect whether someone has left the elevator in this frame. If no one has left the elevator, use the behavior recognition algorithm to determine whether someone has picked up or held the shelved items or areas (picking up, moving, etc.). If so, determine whether the shelved items or unrecognizable areas in the same area of ​​the previous frame are determined to be items left on the ground. If so, the video frame of the target person and the information about the lost items are sent back to the property management department and other relevant departments, the flag is reset, and the process returns to step 3. If not, the flag is reset, and the process returns to step 3. On the contrary, if a person blocks the recognizable shelved items or unrecognizable areas, the video frame and the information about the blocked shelved items are sent back to the property management department, the flag is reset, and the process returns to step 3. If someone has left the elevator, proceed to the next step.

[0099] Step 13: Use the behavior recognition algorithm to determine whether someone has picked up the items on the shelf but has not touched the items and the unrecognizable area (picking up, moving, etc.) in the previous frame. If not, save and record the information of the person taking away the items on the shelf, reset the flag, and return to step 3; if there is such behavior, proceed to the next step;

[0100] Step 14: Determine whether the placed item or unrecognizable area in the same area of ​​the previous frame is an item left on the ground based on the record. If not, frame the video frame of the suspect who takes away the placed item and may pick up the placed item but does not touch the item and the unrecognizable area (picking up, moving, etc.), and save the information that the items of others are taken away by someone other than the suspect, reset the flag, and return to step 3; otherwise, go to the next step;

[0101] Step 15, save and record the video frame of the target person in the previous frame who picked up the placed items but did not touch the items and the unrecognizable area (picking up, moving, etc.) and the information of lost items. Save and record the information that the person may have taken the placed items, reset the flag, and return to step 3.

[0102] The above-mentioned embodiments only express the specific implementation of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for detecting objects placed on the floor in an elevator in multiple scenarios, characterized in that: The following steps are involved: S100, data collection and labeling: Collect video data in the elevator, process and label the video data accordingly, and form a floor information dataset A, a ground object dataset B, a garbage object dataset C, a human body dataset D, a behavior recognition dataset E of people in the elevator on ground objects, and a pedestrian re-identification dataset F; The step S100 further includes the following steps: S107, if the operation mode is the outdoor mode, the video data containing the objects left on the ground in the acquired video segment data DT3 without people is used to obtain the range of illumination components that can normally identify the video frame of the objects left on the ground by using an image enhancement algorithm; S108, detecting and tracking pedestrians in the elevator video data and capturing pedestrian video frames, setting the elevator user confidence threshold filtering operation and manual filtering operation, obtaining a preliminary pedestrian re-identification data set, and then using a FastReID-based model and a feature extraction component to extract pedestrian features, clustering the extracted pedestrian features, archiving frames, and manually filtering to obtain a pedestrian re-identification data set F of the corresponding ID in the corresponding folder; S200, training the labeled data: training the labeled data set to obtain a target detection model M1, a behavior recognition model M2, and a pedestrian re-identification model M3; S300, determining an operation mode: the operation mode includes an indoor mode and an outdoor mode. When the operation mode is the outdoor mode, an image enhancement algorithm is used to convert a low-light or high-light image into a normal-light image; S400, operation phase: determine whether the operation mode is indoor mode or outdoor mode according to step S300, and then detect the objects placed on the ground in the elevator using the target detection model M1, behavior recognition model M2 and pedestrian re-identification model M3 of step S200 trained according to the floor information dataset A, ground object dataset B, garbage object dataset C, human body dataset D, elevator occupant behavior recognition dataset E and pedestrian re-identification dataset F of step S100.

2. The method for detecting objects placed on the floor in an elevator in multiple scenarios according to claim 1 is characterized in that: The step S100 specifically includes the following steps: S101, obtaining video data DT1 collected by the elevator camera within a recent specified time period; S102, obtaining video data DT2 containing a floor information display screen from video data DT1, and classifying and labeling the video data DT2 according to the floor information of the display screen to form a floor information data set A; S103, perform human detection on the video data DT1 to obtain video segment data DT3 without people and video segment data DT4 with people; classify and label the video data containing objects placed on the ground in the video segment data DT3 without people according to the categories of the objects placed on the ground, to form a dataset of objects placed on the ground B; classify and label the video data containing garbage in the obtained video segment data DT3 without people according to the categories of garbage, to form a dataset of garbage objects C; S104, select and mark the ground area in the elevator in the recent day in the video segment data DT3 without people, select the car rectangular frame ground area of ​​the best picture as a single car rectangular frame ground area data set, and record the corresponding coordinate data; S105, marking the human body in the acquired video segment data DT4 data with people, to form a human body data set D; S106. Obtain the video data DT5 containing user behavior from the video segment data DT4 containing people, use the AlphaPose human posture estimation algorithm to obtain the human joint information of the people in the human detection area containing user behavior, save the skeleton diagram, and define the behavior segmentation labels of people towards the objects left on the ground, ensure that each video has one label, and the video time length is less than two seconds, and number the video folders and establish indexes of behavior labels and video files, so as to obtain the behavior recognition data set E of people in the elevator towards the objects left behind.

3. The method for detecting objects placed on the floor in an elevator in multiple scenarios according to claim 1, characterized in that: The step S200 specifically includes the following steps: S201, using the YOLOv5 target detection algorithm to perform sample balanced model training on the floor information dataset A, the ground object dataset B, the garbage object dataset C, and the human body dataset D, to obtain a target detection model M1; S202, using the ST-GCN skeleton behavior recognition algorithm to convert the behavior recognition data set E of people in the elevator to the objects left on the ground into a file suitable for training, and then perform model training to obtain a skeleton behavior recognition model M2; S203: Use the FastReID-based pedestrian re-identification model to perform model training to obtain a pedestrian re-identification model M3.

4. The method for detecting objects placed on the floor in an elevator in multiple scenarios according to claim 3 is characterized in that: In step S300, the image enhancement algorithm includes a visual communication frame enhancement module, which is used to process low-light and high-light video frames and convert them into normal-light frames. In the process of iterative illumination adaptive image enhancement, when the illumination components of the enhanced image are extracted and compared with the range of the normal illumination components, if they are not in a reasonable range of normal illumination, a convergence rationality judgment and trend processing mechanism is embedded to promote the optimization of the image enhancement process.

5. The method for detecting objects placed on the floor in an elevator in multiple scenarios according to claim 4, characterized in that: The image enhancement algorithm is used to convert low-light or high-light images into normal-light images as follows: After acquiring the RGB frame, convert it into an HSI frame, and then use the multi-scale Gaussian filter function to compress the image dynamic range while calculating the image illumination component and recording it, and then determine whether the maximum number of iterations has been reached. If so, convert it into RGB frame data output, otherwise determine whether it is a normal illumination video frame. If it is a normal illumination video frame, convert it into an RGB frame for output; Otherwise, it is determined whether it is the first time to determine whether this frame is a normal image. If not, the iteration flag position is 1 and the number of iterations is recorded. Otherwise, the iteration flag position is 0. Then it is determined whether it is a low-light image. If it is a low-light frame, the image illumination component flag position is 0, and the camera response model is used. Where f and h represent the brightness mapping function and exposure rate of the camera response model, respectively, and a, b and γ represent the camera parameters and adjustable parameters, respectively; Under the condition that the low-light image F0 is underexposed, in order to make the virtual exposure image F1 better exposed, the optimal h value is determined and then the pixels with higher exposure in F0 are removed, and then the pixel point set V with underexposed pixels is constructed according to V={F0(x)|G(x)<ξ}, where G and ξ represent the illumination component of F0 and the expected normal illumination component interval respectively; The brightness matrix is ​​constructed using the brightness of all pixels in V to determine the h value, based on the information entropy of the brightness matrix B of all pixels in V Among them, p i Represents the probability of each gray level i appearing in the brightness matrix, and then maximizes the image entropy to determine the optimal exposure rate value h and obtain the virtual exposure image F1; Then, it is judged whether the iteration flag is 1. If it is not 1, fusion processing is performed. Otherwise, convergence rationality judgment and trend processing are performed: the illumination component after the last image enhancement is compared with the illumination component of the current image. If it is smaller than the illumination component after the last image enhancement, based on the positive correlation between the weight matrix and the illumination of the scene in the complex illumination image, the enhancement parameter of the adjustment weight matrix J is set to the calculated value: Initial value of enhancement parameter * (100% + 5% * number of iterations) Then F0 and F1 are fused: where J and the operator They represent the multiplication of the weight matrix and the corresponding elements of the matrix with the uniform number of rows and columns; Since the weight value of well-exposed pixels is higher than that of underexposed pixels, the weight matrix J is positively correlated with the illumination of the scene in the complex illumination image, so J = G τ , τ represents the parameter controlling the enhancement degree, and the fused image F2 is obtained. The returned image is used to extract the illumination component again using the multi-scale Gaussian filter function to determine whether the image is a normal illumination image. If not, it is iterated again, otherwise the image is converted into an RGB image output; If it is a high-light image, the image grayscale value is inverted, the image illumination component flag is set to 1, and low-light image enhancement processing is performed. After processing, the grayscale value is inverted again to determine whether the image is a normal-light image. If not, it is iterated again, otherwise the image is converted into an RGB image output.

6. The method for detecting objects placed on the floor in an elevator in multiple scenarios according to claim 5, characterized in that: The step S400 specifically includes the following steps: S401, the operation mode setting takes effect: before the camera is turned on, the operation mode is set to indoor mode or outdoor mode, and the default is outdoor mode; S402, automatic setting of parameters of the floor area of ​​the rectangular frame in the elevator: automatically setting the corresponding area according to the existing coordinate data of the floor area of ​​the rectangular frame of the elevator; S403, frame update; S404, determining the operation mode, if it is the outdoor mode, performing image enhancement, and then proceeding to the next step; if it is the indoor mode, no processing is performed, and directly proceeding to the next step; S405, target detection and result recording in the rectangular floor area of ​​the elevator: the position and number of human bodies in the rectangular floor area of ​​the elevator can be used to identify the position, type and number of items on the floor, the position, type and number of garbage, and the floor information for target detection, and the relevant information is recorded; S406, whether there is human body detection result: through the data record of the process detection of whether there are passengers in the elevator car area, it is determined that if there are passengers in the elevator car area, the performance flag position is 1, and it is determined whether the proportion of the human body detection frame part in the elevator floor area exceeds the specified threshold. If it exceeds, the flag is reset, and the process returns to step S403; otherwise, the process proceeds to the next step; if there is no person in the elevator, the performance flag position is 0, and the process proceeds to the next step; S407, garbage identification: Based on the target detection result record in the rectangular ground area of ​​the elevator, if it is determined that there is garbage, the video frame and the garbage information are sent back to the background and the next step is entered; if not, the next step is directly entered; S408, processing statistical sorting of relevant information of identifiable items placed on the ground and unidentifiable areas in two threads: one thread performs statistical sorting of relevant information of identifiable items placed on the ground; the second thread performs frame difference between the ground area except the detection frame of people, items placed on the ground, and garbage in the current video frame and the same ground area of ​​the stored empty ladder frame, and then checks whether there is a pixel area within the set threshold range by the absolute value of the difference between the absolute value of the pixel difference after the frame difference and the absolute value of the adjacent difference value. If there is, there is no unidentifiable object, and directly enters the comparison of the identifiable ground object area with the convex hull area and the garbage area. If there is no such information, the pixels not within the threshold range are displayed in the form of a convex hull in the frame, and the convex hull containing pixels less than the specified threshold is removed. The identifiable leftover item area is compared with the convex hull area and the garbage area, and the same area is deleted. The information of the unidentifiable area and the identifiable items on the ground are sorted again. It is determined whether the number of identifiable items on the ground and the unidentifiable area is greater than 0. If not, the flag is reset, and the process returns to step S403. Otherwise, the process proceeds to the next step. S409, determining whether it is the first time to detect the identifiable ground object and the unidentifiable area information on the day, if so, reset the flag bit and return to step S403, otherwise, proceed to the next step; S410, determining by a behavior recognition algorithm whether a person has held the identifiable ground object or the unidentifiable area in the previous frame, if so, the object or area is preliminarily determined to be a non-left-on-ground object, and proceeding to the next step; otherwise, the object or area is preliminarily determined to be a left-on-ground object, and proceeding to the next step; S411, compare the identifiable items on the ground and the unidentifiable area of ​​the previous frame to see if the same area of ​​the current frame has disappeared. If not, use the pedestrian re-identification algorithm, behavior recognition algorithm and existing record information to determine whether the same person in this frame and the previous frame has the behavior of holding the identifiable items on the ground or the unidentifiable area. If not, send the video frame and the information of the lost items back to the property and other relevant departments, reset the flag bit, and return to step S403; if yes, it is a normal person in the elevator car holding his own items, reset the flag bit, and return to step S403; otherwise, go to the next step; S412, detect whether someone has left the elevator in this frame. If no one has left the elevator, use the behavior recognition algorithm to determine whether someone has picked up or held the shelf items or areas. If yes, determine whether the shelf items or unrecognizable areas in the same area of ​​the previous frame are determined to be items left on the ground. If yes, send the video frame of the target person and the information about the lost items to the background, reset the flag, and return to step S403. If not, reset the flag, and return to step S403. On the contrary, if the recognizable shelf items or unrecognizable areas are blocked by a person, send the video frame and the information about the blocked shelf items to the background, reset the flag, and return to step S403. If someone has left the elevator, proceed to the next step. S413, using the behavior recognition algorithm to determine whether someone has picked up the items on the shelf but has not touched the items and the unrecognizable area in the previous frame. If not, save and record the information of the person taking away the items on the shelf, reset the flag, and return to step S403; if there is such behavior, proceed to the next step; S414, judging whether the placed item or the unrecognizable area in the same area of ​​the previous frame is determined as an item left on the ground according to the record, if not, framing the video frame of the suspect who took away the placed item and may have picked up the placed item but did not touch the item and the unrecognizable area, and saving the information that the items of others were taken away by someone other than the suspect, resetting the flag bit, and returning to step S403; otherwise, going to the next step; S415. The video frame of the target person in the previous frame in which the target person picks up the placed items but does not touch the items and the unrecognizable area, and the information about the lost items are saved and recorded. The information about the person who may have taken the placed items is also saved and recorded. The flag is reset and the process returns to step S403.

Images