A fluid pyramid elevator foreign matter detection method based on multi-scale deep feature fusion

Abstract

The application provides a foreign matter detection method for a fluid pyramid elevator based on multi-scale deep feature fusion. The method detects whether there is a person in an elevator car by using a Faster-RCNN algorithm to determine the empty state of the elevator; when the elevator is empty, the empty elevator image is taken as a reference to extract a key frame without passengers from a to-be-detected video as a query image; a twin network is designed to extract the features of the reference image and the query image respectively and perform feature fusion; and multi-scale feature fusion is performed on the multi-layer features of the deep network based on a fluid pyramid network to accurately detect a foreign matter area. The method effectively improves the precision and noise resistance of foreign matter detection and can accurately identify foreign matters such as water accumulation, garbage and oil stains in the elevator, thereby providing a strong guarantee for safe operation of the elevator.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of elevator foreign matter detection, and specifically relates to a fluid pyramid elevator foreign matter detection method based on multi-scale deep feature fusion. BACKGROUND

[0002] With the continuous deepening development of city modernization and intelligentization, elevators have become an important tool indispensable in the daily life of residents. The existence of elevator foreign matter has a bad influence on the use safety and operation efficiency of the elevator; at the same time, the development of AI technology is liberating humans from heavy and mechanical repetitive work, so using computer technology to realize foreign matter detection has gradually become the current development trend.

[0003] With the rise of deep learning technology, its successful application in the fields of image recognition and target detection provides a new idea for elevator foreign matter detection. By constructing a deep learning model, features can be automatically learned from a large amount of data, improving the accuracy and robustness of detection. However, due to the particularity of the elevator environment, such as limited space, possibly unstable light, frequent movement of objects and personnel, etc., these will all have an impact on multi-scale image detection.

[0004] Although multi-scale image detection can handle targets of different scales, there is still a problem of insufficient detection accuracy for smaller targets. In the training model, the detailed information of large and small targets decays with the deepening of the model layers. For semantic information, since small targets are small in scale, with the increase of model layers, semantic information may be extracted in the shallow layer, and then with the continuous increase of layers, the semantic information of small targets will be quickly diluted by environmental information. Large targets are large in scale, and sufficient semantic information may be extracted in the deep layer, but at this time the semantic information of small targets has been almost lost. In the elevator environment, how to balance the detection ability of large and small targets is the problem of multi-scale image detection.

[0005] In addition, multi-scale image detection improves detection performance by fusing features of different scales, but in actual application, the fusion of different scale features may not be sufficient, resulting in the model being unable to fully utilize the feature information of each scale, thereby affecting the detection accuracy. Therefore, the existing multi-scale image detection technology cannot well solve the detection of foreign matter in the elevator scene. SUMMARY

[0006] The purpose of the present application is to provide a fluid pyramid elevator foreign matter detection method based on multi-scale deep feature fusion.

[0007] The technical solution for achieving the purpose of the present application is: a fluid pyramid elevator foreign matter detection method based on multi-scale deep feature fusion, comprising:

[0008] S1: input the continuous set frame elevator image into the empty load detection model for detection to obtain the detection result of whether there is a passenger in each frame image and the corresponding confidence, and when the detection result and the confidence of the continuous set frame image meet the set condition, step 2 is performed;

[0009] S2: select the frame with the highest definition from the continuous set frame image as a reference image; select the latest frame image detected in real time at the current moment as a query image; input the reference image and the query image into the trained foreign matter detection model to obtain a foreign matter detection result.

[0010] Preferably, the empty load detection model adopts a ResNet-18 network, the ResNet-18 network comprises, in sequence, a convolutional layer, a maximum pooling layer, a first residual block, a second residual block, a third residual block, an SE channel attention module and a fourth residual block, and an input image is sequentially subjected to the convolutional layer and the maximum pooling layer to extract low-level features; the image output by the maximum pooling layer is sequentially subjected to the first residual block, the second residual block, the third residual block and the fourth residual block to obtain a deep feature map, wherein the image output by the third residual block is subjected to the SE channel attention module to calculate the global importance of each feature channel and obtain an enhanced feature map; the enhanced feature map is input into the fourth residual block for down-sampling to obtain a final deep feature map; and the final deep feature map is output through a classifier to output the detection result of whether there is a passenger in the elevator image and the corresponding confidence.

[0011] Compared with the prior art, the present application has the following advantages:

[0012] 1. Multi-scale deep feature extraction: the present application extracts high-level semantic features and low-level texture features of an image through different scale convolution kernels, and provides rich information for subsequent feature fusion and change detection.

[0013] 2. Feature fusion and change detection: the present application introduces a fluid pyramid network to fuse the extracted multi-scale deep features, reduces the calculation amount while retaining the main features of the feature map, combines high-level features and low-level features to fully fuse the information of the high level and the low level, improves the detection accuracy and noise resistance, and realizes high-precision detection of the foreign matter area in the elevator. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 It is a structural schematic diagram of a fluid pyramid network (FPN-S).

[0015] Figure 2 It is a schematic diagram of feature alignment and preliminary fusion of a double-branch twin network.

[0016] Figure 3 It is a flowchart of a fluid pyramid elevator foreign matter detection method based on multi-scale deep feature fusion.

[0017] Figure 4 This is a schematic diagram of the system hardware deployment. Detailed Implementation

[0018] A foreign object detection method for a fluid pyramid elevator based on multi-scale deep feature fusion is shown in Figure 4. The overall process is as follows: Figure 3 As shown, it includes:

[0019] S1: Input the elevator images of consecutive frames into the empty-load detection model for detection, and obtain the detection result and corresponding confidence level of whether there are passengers in each frame. When the detection result and confidence level of each frame meet the set conditions, proceed to step 2.

[0020] In a further embodiment, the ResNet-18 network is trained using a training set specific to elevator scenarios to obtain an empty-load detection model;

[0021] Specifically, the elevator scenario-specific training set is constructed through the following processing:

[0022] (1) Data acquisition: Using a 2-megapixel CMOS industrial camera, images were acquired in the cars of 10 elevators at different times (morning peak 7:00-9:00, off-peak 12:00-14:00, evening peak 18:00-20:00, and nighttime 22:00-6:00), covering three lighting scenarios: normal lighting, strong direct light, and weak light. A total of 80,000 original images were acquired.

[0023] (2) Foreign object annotation: The LabelImg tool was used to annotate foreign objects. The annotation information included the foreign object category and bounding box coordinates. The annotation format adopted VOC XML. Blurred images were marked and removed. Finally, 65,000 valid samples were retained.

[0024] (3) Data augmentation: Perform the following augmentation operations on the elevator scene features: Light adjustment: Brightness adjustment range is 0.6-1.4 times the original brightness, contrast adjustment range is 0.8-1.2 times, to simulate light fluctuations in the elevator; Geometric transformation: Random horizontal flip, random cropping, affine transformation, to maintain the original aspect ratio, to simulate noise interference from elevator cameras;

[0025] (4) Data partitioning: The enhanced data is divided into training set, validation set and test set in a ratio of 8:1:1, and all image pixel values ​​are normalized to the [0,1] interval.

[0026] Specifically, the ResNet-18 network includes:

[0027] ResNet-18 includes 1 convolutional layer, 1 maximum pooling layer, a first residual block, a second residual block, a third residual block and a fourth residual block. The input image sequentially passes through the convolutional layer and the maximum pooling layer, and is rapidly sampled to extract low-level features and further reduce the feature map size; the output image is processed by the first residual block to combine the low-level features extracted by the convolutional layer into textures and simple patterns; the feature map is further extracted and combined by the second residual block to obtain part features of an object, the third residual block identifies the whole object, the feature map output by the third residual block is calculated by the SE channel attention module to obtain the global importance of each feature channel, different weights are given, the feature channel related to the passenger is enhanced, the irrelevant background channel is suppressed, and an enhanced feature map is obtained; the enhanced feature map is input into the fourth residual block for down-sampling to capture global scene semantic information, and finally a set of highly condensed deep feature maps are obtained.

[0028] The finally obtained deep feature map is output through a classifier to obtain a detection result of whether the elevator image has a passenger and a corresponding confidence.

[0029] In view of the problem that the traditional RPN is not adapted to the size characteristics of the elevator passenger, the anchor box parameters are optimized based on actual data statistics, the size and the aspect ratio more consistent with the passenger body shape characteristics are set, the number of anchor boxes generated by each feature map is reduced by 30% than the traditional method, and the detection efficiency is significantly improved; meanwhile, the ROI Align is used to replace the traditional ROI Pooling, the feature values in the proposal box are accurately calculated through bilinear interpolation, the quantization error is avoided, and the regression accuracy of the passenger position is improved.

[0030] In a further embodiment, the specific process of the output feature map of the third residual block through the SE channel attention module to calculate the global importance of each feature channel, give different weights, enhance the feature channel related to the passenger, and suppress the irrelevant background to obtain the enhanced feature map is as follows:

[0031] Global average pooling is performed on the feature map output by the third residual block to obtain a 1x1x256 channel statistical vector, and the formula is , wherein is the feature map of the cth channel;

[0032] A channel weight vector s is generated through two fully connected layers (FC1: 256→64, FC2: 64→256) and a Sigmoid activation function, and the formula is ;

[0033] The weight vector s is multiplied with the feature map F channel by channel, that is, the cth element of the weight vector s is multiplied with the cth channel of the feature map F element by element to obtain an enhanced feature map to boost the channel weights of passenger features and suppress background interference. The mathematical expression of this process is as follows:

[0034]

[0035] wherein, and represent the cth channel of the output and input feature map respectively, is the cth element of the weight vector s.

[0036] In further embodiments, when training the ResNet-18 network, the total loss function adopted is a weighted sum of the classification loss function and the position regression loss function, and the formula is as follows:

[0037]

[0038] wherein, is the total loss function; is the classification loss function; is the position regression loss function. is a hyperparameter for balancing the weights of the two loss terms, and its value range is 0.8-1.2.

[0039] The classification loss function aims to solve the problem of class imbalance caused by the uneven distribution of “with / without passenger” samples in the elevator. The specific formula is as follows:

[0040]

[0041] wherein, is the classification loss, is the class weight, “with passenger” samples =0.25, “without passenger” samples =0.75, to balance the difference in sample quantity; is the focusing parameter, set to 2, by assigning a low weight to easy-to-classify samples and focusing on difficult-to-classify samples; is the probability of predicting a passenger, and vice versa ;

[0042] The position regression loss function aims to optimize the bounding box regression, and the specific formula is as follows:

[0043]

[0044] wherein, is the coordinate difference between the predicted box and the real box, and L2 Loss is used when the coordinate difference is small, and L1 Loss is used when the coordinate difference is large, to improve the position regression accuracy.

[0045] After the above training, the empty state of the elevator is determined. In order to avoid single frame detection error, the "multi-frame confirmation" logic is adopted: 3 frames of elevator images are continuously collected, and are input into the empty detection model respectively; if the detection results of the 3 frames are all "no passenger", and the "no passenger" confidence of each frame is >0.9, it is determined that the elevator is in an empty state; after the empty state is determined, the frame with the highest definition is selected from the 3 frames as the reference image X, which is used for subsequent foreign matter detection.

[0046] S2: select the frame with the highest definition from the continuous set of frames as the reference image X; select the latest frame image collected in real time at the current time as the query image Y; input the reference image X and the query image Y into the foreign matter detection model to obtain the foreign matter detection result, and the specific steps are as shown in Figure 2

[0047] In further embodiments, the foreign matter detection model includes a double-branch twin network, a preliminary feature fusion module, an FPN-S network, and a foreign matter classification and positioning module.

[0048] As shown in Figure 2 The double-branch twin network is a double-branch, weight-shared lightweight convolutional neural network, each branch of which is a feature extractor based on the ResNet architecture but with lightweight design, including an initial convolutional layer, a maximum pooling layer, and four residual blocks (conv2_x to conv5_x). Both sub-networks perform a series of convolution and down-sampling operations on the input reference image X and query image Y to extract feature maps at three levels of conv3_x, conv4_x, and conv5_x, and finally output the feature maps of the reference image X: F_x3, F_x4, F_x5; and the feature maps of the query image Y: F_y3, F_y4, F_y5.

[0049] During the operation of the elevator, the camera may have slight jitter, causing the same scene area of the reference image X and the query image Y to shift. If the features are directly fused, errors will occur. To solve this problem, for each feature map level of the query image Y, the spatial position is corrected using the optical flow field. First, the Farneback algorithm is used to calculate the dense optical flow field between the reference image X and the query image Y to obtain the shift amount (u, v) of each pixel of Y relative to X. According to the calculated shift amount, the bilinear interpolation operation is performed on the feature maps (F_y3, F_y4, F_y5) of the query image Y to align them in spatial position with the feature maps of the reference image X, and the aligned query image Y feature maps: F_y3', F_y4', F_y5' are output.

[0050] ​The preliminary feature fusion module is configured to splice the feature maps of the reference image X and the aligned query image Y in a channel dimension, reduce the number of channels after splicing by using a 1x1 convolution, and realize preliminary combination of the features of the reference image and the query image. Meanwhile, a spatial attention module is introduced to generate a spatial weight map: the weight of the region where the foreign matter is likely to exist is high, and the weight of the background region is low. The weight map is multiplied by the spliced multi-scale feature map, so as to enhance the feature response of the foreign matter region and suppress the background interference. Finally, the enhanced multi-scale preliminary fusion features F_pre3, F_pre4 and F_pre5 are output.

[0051] As shown in Figure 1 , the FPN-S network receives the multi-scale preliminary fusion features F_pre3, F_pre4 and F_pre5 output from the twin network preliminary fusion module, adopts a top-down path for fusion, and the specific steps are as shown in Figure 1 .

[0052] The bilinear interpolation upsampling is performed on the deepest layer feature F_pre5 (20x20) to enlarge it to 40x40 to obtain F_up5,

[0053] The F_up5 and the F_pre4 of the same scale are respectively subjected to 1x1 convolution to unify the number of channels to 256 to obtain the first scale feature F_up5_adj and the second scale feature F_pre4_adj. The first dynamic weight w5 and the second dynamic weight w4 are generated by respectively using 1x1 convolution on the F_up5_adj and the F_pre4_adj. The first dynamic weight w5 and the second dynamic weight w4 are subjected to Softmax normalization to ensure that w5+w4=1. Finally, the first scale feature and the second scale feature are weighted and summed to output F_fuse4 (40x40). The formula of F_fuse4 is: F_fuse4=w5xF_up5_adj+w4xF_pre4_adj.

[0054] Then, the middle and low layers are fused, and the process is similar to the high layer fusion. The F_fuse4 is upsampled to 80x80 to obtain F_up4. After unifying the number of channels of the F_up4 and the F_pre3, the dynamic weights w4' and w3 are generated. After weighted fusion, F_fuse3 (80x80x128) is obtained.

[0055] Finally, the F_fuse3 is upsampled to 160x160 to obtain F_up3, and a 3x3 convolution is performed on the feature to optimize the smoothness to obtain the final fusion feature Ffinal (160x160x128).

[0056] The final fusion feature Ffinal is input into the foreign matter classification and positioning module to realize foreign matter category discrimination and position positioning. The foreign matter classification and positioning module includes a classification branch and a regression branch; the two branches share the same input feature F_final, but use different convolution layers to learn their respective tasks.

[0057] The classification branch includes 2 3x3 convolution layers and 1 1x1 convolution layer. The final fusion feature F_final is first subjected to feature extraction and deepening of the feature map via the two 3x3 convolution layers, and then is mapped to a category prediction via the 1x1 convolution layer, which predicts 9 anchor box category scores for each position. The output of the 1x1 convolution layer is a 160x160x9 feature map, representing the original scores of 9 anchor boxes belonging to each foreign matter category at each position. Finally, the 160x160x9 feature map is applied with a Softmax function in the channel dimension to convert the original scores into probabilities, so that the sum of the probabilities of each category for each anchor box is 1, and finally an 160x160x9 category probability map is output.

[0058] The regression branch includes 2 3x3 convolution layers and 1 1x1 convolution layer. The first 3x3 convolution layer extracts features suitable for position regression, the second 3x3 convolution layer further optimizes the position regression features, and finally the 1x1 convolution layer completes the 4 coordinate offsets Δxmin, Δymin, Δxmax, Δymax of the corresponding bounding box. An output of 160x160x36 bounding box offset tensor is output.

[0059] Finally, the offsets (Δx_min, Δy_min, Δx_max, Δy_max) predicted by the regression branch are applied to the corresponding preset anchor box to calculate the accurate coordinates of the final predicted bounding box in the actual image.

[0060] The vacancy detection model is trained to simultaneously optimize the three tasks of feature similarity, foreign matter classification, and position positioning, and a hybrid loss function is designed , and the specific formula is as follows:

[0061]

[0062] wherein = 0.3, = 0.4, = 0.3;

[0063] is a feature comparison loss, which is used to measure the similarity of the reference image and the query image features. If there is a foreign matter, the difference between the two image features is large, and the loss value is large; otherwise, the loss value is small. The formula is:

[0064] Wherein: y is a label, y=0 represents no foreign matter (the reference image X is similar to the query image Y feature), y=1 represents that there is foreign matter (the reference image X and the query image Y feature difference is big); 、 is the feature output by the last residual block of the double-branch twin neural network of the reference image and the query image; m is a margin value, which is set to 1.0, to ensure that the distance between the two feature vectors is less than 1.0 when there is no foreign matter, and greater than 1.0 when there is foreign matter;

[0065] is a classification loss, and the specific formula is:

[0066]

[0067] Wherein: N: total number of image pixels;

[0068] C: the number of categories;

[0069] : the true label of pixel i;

[0070] : the probability that pixel i is predicted to be category c;

[0071] is a positioning loss, and the specific formula is:

[0072]

[0073] Wherein A is a predicted bounding box, and B is a real bounding box. When A and B have no overlap, IoU=0, the loss value is 1, and the gradient is 0, which leads to training stagnation, so a penalty term is added, The formula becomes:

[0074]

[0075] Wherein: : the Euclidean distance between the center points of the predicted box and the real box;

[0076] d: the diagonal length of the smallest rectangle enclosing the two boxes;

[0077] The penalty term ensures that the loss value still has a gradient when there is no overlap, promoting model convergence.

[0078] The arrangement of the hardware device carried by the application is as shown in Figure 4 .

[0079] The application has the following characteristics:

[0080] Scene screening and then detection: first, the improved Faster-RCNN is used to screen the empty scene without passengers, to avoid false positives caused by passenger shielding, and to reduce unnecessary foreign matter detection calculation, thereby improving system efficiency;

[0081] Multi-scale feature dynamic fusion: In view of the large size difference of elevator foreign matters, a fluid pyramid network is designed to realize efficient fusion of high-level semantic features and low-level detail features through dynamic weight distribution, which is superior to the fixed weight fusion of traditional FPN;

[0082] Lightweight and precision balance: Through network pruning, quantization training and other means, the model complexity is reduced under the premise of ensuring the detection accuracy, and the computing capacity of the elevator terminal edge device is adapted.

[0083] The application is suitable for real-time detection and early warning of foreign matters in the elevator car in residential, commercial complex, office building and other scenes, can accurately identify water accumulation, garbage, oil stains, ropes, metal fragments and other foreign matters threatening the safe operation of the elevator, and solves the problem of high missing detection rate of small targets and low contrast foreign matters in traditional elevator foreign matter detection.

Claims

1. A fluid pyramid elevator foreign matter detection method based on multi-scale deep feature fusion, characterized in that, Comprise: S1: input the continuous set frame elevator image into the empty load detection model for detection, obtain the detection result of whether there is a passenger in each frame image and the corresponding confidence, when the detection result and the confidence of the continuous set frame image meet the set condition, proceed to step S2, the set condition refers to when 3 frame detection results are all no passengers, and the no passenger confidence of each frame >0.9; S2: select the frame with the highest definition from the continuous set frame image as the reference image; The latest frame image detected in real time from the current time as the query image; Input the reference image and the query image into the trained foreign matter detection model to obtain the foreign matter detection result, the foreign matter detection model comprises a double-branch twin network, a preliminary feature fusion module, an FPN-S network, a foreign matter classification and positioning module; The double-branch twin network extracts multi-scale feature maps of the reference image and the query image respectively, and aligns the multi-scale feature maps of the query image with the multi-scale feature maps of the reference image; The preliminary feature fusion module is used for one-to-one splicing of the multi-scale feature maps of the query image and the multi-scale feature maps of the reference image, and multiplying the spliced multi-scale feature maps by a weight map to obtain multi-scale preliminary fusion features; The FPN-S network is used for fusing the multi-scale preliminary fusion features to obtain final fusion features, and the specific process is as follows: After bilinear interpolation upsampling of the deepest layer features in the multi-scale preliminary fusion features, and the same scale preliminary fusion features are uniformly channelled through 1x1 convolution respectively, the corresponding first scale feature and the second scale feature are obtained; 1x1 convolution is used to generate first dynamic weight and second dynamic weight for the first scale feature and the second scale feature respectively, and the first dynamic weight and the second dynamic weight are normalized by Softmax, so that the sum of the first dynamic weight and the second dynamic weight is 1; the first dynamic weight and the second dynamic weight are used to weight and sum the first scale feature and the second scale feature to obtain new deepest layer features; The above process is repeated for the new deepest layer features until fusion with all scale preliminary fusion features is completed to obtain final fusion features; The foreign matter classification and positioning module is used for foreign matter detection according to the final fusion features to obtain a foreign matter detection frame, the foreign matter classification and positioning module comprises a classification branch and a regression branch, the classification branch comprises two 3x3 convolution layers and one 1x1 convolution layer, the final fusion feature Ffinal is first subjected to feature extraction and deepening of the feature map through two 3x3 convolution layers, and then is mapped to a class prediction through the 1x1 convolution layer, each position predicts the class scores of 9 anchor frames; the output of this 1x1 convolution layer is a 160x160x9 feature map, which represents the original scores of each position belonging to each foreign matter class for 9 anchor frames respectively; finally, the 160x160x9 feature map is applied to the channel dimension to apply the Softmax function, and the original scores are converted to probabilities, so that the sum of the probabilities of each class of each anchor frame is 1, and finally a 160x160x9 class probability map is output. The regression branch comprises two 3*3 convolution layers and one 1*1 convolution layer, the first 3*3 convolution layer extracts features suitable for position regression, the second 3*3 convolution layer further optimizes the position regression features, and finally the 1*1 convolution layer completes the four coordinate offsets Δxmin, Δymin, Δxmax and Δymax of the corresponding bounding box; an output of a 160*160*36 bounding box offset tensor is obtained; The four coordinate offsets predicted by the regression branch are applied to the corresponding preset anchor box to calculate the coordinates of the final predicted bounding box in the actual image.

2. The fluid pyramid elevator foreign object detection method based on multi-scale deep feature fusion according to claim 1, characterized in that, The empty load detection model adopts a ResNet-18 network, the ResNet-18 network comprises convolution layers, a maximum pooling layer, a first residual block, a second residual block, a third residual block, an SE channel attention module and a fourth residual block connected in sequence, and an input image sequentially passes through the convolution layers and the maximum pooling layer to extract low-level features; The image output by the maximum pooling layer is sequentially processed by the first residual block, the second residual block, the third residual block and the fourth residual block to obtain a deep feature map, wherein the image output by the third residual block is calculated by the SE channel attention module to obtain the global importance of each feature channel and an enhanced feature map; The enhanced feature map is input into the fourth residual block for down-sampling to obtain a final deep feature map; the final deep feature map is output by a classifier to output a detection result of whether the elevator image has a passenger and a corresponding confidence.

3. The method for fluid pyramid elevator foreign object detection based on multi-scale deep feature fusion according to claim 2, characterized in that, The specific process that the image output by the third residual block is calculated by the SE channel attention module to obtain the global importance of each feature channel and an enhanced feature map is as follows: Global average pooling is performed on the feature map F output by the third residual block to obtain a channel statistical vector z; The channel statistical vector z is nonlinearly transformed by two fully connected layers and a Sigmoid activation function to generate a channel weight vector s; The weight vector s is multiplied channel by channel with the feature map F output by the third residual block, that is, the cth element of the weight vector s is multiplied element by element with the cth channel of the feature map F to obtain an enhanced feature map : F' = s * F : ； wherein, and represent the c-th channel of the output and input feature map, respectively, is the c-th element of the weight vector s.

4. The method for fluid pyramid elevator foreign object detection based on multi-scale deep feature fusion according to claim 3, characterized in that, The total loss function adopted by the ResNet-18 network during training is composed of a classification loss function and a position regression loss function, and the formula is as follows: ； In the formula, is a total loss function; is a classification loss function; is a position regression loss function, is a hyperparameter balancing the weights of the two loss terms, and are specifically: ； ； wherein, is a class weight; is a probability of predicting a passenger; is a focus parameter; is a coordinate difference between a predicted box and a real box.

5. The method for fluid pyramid elevator foreign object detection based on multi-scale deep feature fusion according to claim 1, characterized in that, The loss function adopted by the foreign matter detection model during training is as follows: ； , = 0.3, = 0.4, = 0.3 is a feature contrast loss, is a classification loss, is a positioning loss, respectively, ； ； ； wherein y is a label, y=0 represents no foreign matter, y=1 represents foreign matter, , are respectively the features output by the last residual block of the double-branch twin network of the reference image and the query image, m is a margin value; N is the total number of image pixels; C is the number of categories; is the true label of pixel i; is the probability of pixel i being predicted as category c; is the Euclidean distance between the center points of the predicted box and the true box, d is the diagonal length of the smallest rectangle enclosing the two boxes, represents the center point coordinates of the predicted bounding box output by the model, represents the center point coordinates of the true bounding box in the labeled data.

Images