Escalator step gap real-time detection method and system

Abstract

The application discloses a kind of escalator pedal gap real-time detection method and system, it is related to escalator pedal detection field, the present application uses transformer module to replace the convolution activation layer in basic model with YOLOX model as basic model, using upsampling expansion module replaces the upsampling module in basic model, obtain first YOLOX model, and train first YOLOX model by including the data set of the labeled gap color block without the labeled gap block without color block, and in training process, improved YOLOX model is obtained using loss function, escalator pedal image is input to improved YOLOX model, and the gap color block detection picture is detected, the color segmentation of gap color block detection picture is carried out by HSV model, corresponding black and white grayscale image is obtained, the pixel width of black region in black and white grayscale image is obtained, i.e. the width of escalator pedal gap, it realizes real-time detection to escalator pedal gap using visual target detection and image segmentation technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of escalator step detection, in particular to an escalator step gap real-time detection method and system. BACKGROUND

[0002] Visual target detection and image segmentation are classic tasks in the field of computer vision. The demand for intelligentization in today's society is growing. Fan Qing (published in Shanghai Rail Transit Smart Subway Station Escalator Intelligentization Reconstruction [J]. Machine and Electrical Information) studied the online monitoring function of escalators applied to smart subway stations. The research first records and stores the running and fault information of the escalator, sends the information to the manufacturer's remote safety monitoring system through wireless transmission, and synchronously uploads the information to the station integrated monitoring system through wired mode, achieving the purpose of data management of escalator running and fault information. The focus of this research is to collect the running status and fault information of the escalator, analyze and summarize, form various statistical reports and equipment operation status evaluation reports, so as to better develop maintenance and repair plans, and also can automatically notify professional technicians to the scene to eliminate faults and rescue trapped passengers in time. And realize the remote supervision and technical support of the on-site maintenance personnel, ensure that they complete the maintenance work in time, accurately and efficiently. He Dongshan (published in Design of Metro Escalator Online Monitoring and Early Warning System [J]. China Elevator) and others use intelligent sensing, big data, network, clustering analysis and other advanced technologies to establish a real-time monitoring system for the running status of all escalators in the line network. The system mainly includes four components: line network cloud platform, line network central data comprehensive service system, data transmission system and station level front-end monitoring and early warning system. According to the characteristics of each key component in the escalator, the system couples the experience summarized in the long-term operation and maintenance process, and monitors the main characteristic parameters of the escalator main machine, step chain, main drive chain, handrail belt, brake, truss, lubrication system and control cabinet in operation through sensors and intelligent analysis technology.

[0003] However, the application of visual target detection and image segmentation in the field of escalator step gap detection has not been involved in the prior art. The present application realizes real-time detection of escalator step gap through computer vision technology to ensure escalator safety. SUMMARY

[0004] In order to realize real-time detection of escalator step gap based on visual target detection and image segmentation technology, the present application proposes an escalator step gap real-time detection method applied to an improved YOLOX model. Color bands are attached to both sides of the gap of the escalator step. The method comprises the following steps:

[0005] S01: Acquire images and videos of escalator steps with colored stripes affixed to both sides of the gap, and images and videos of escalator steps without colored stripes affixed. Convert both escalator step images and videos into image sets, and label the color blocks with gaps in the images with color blocks in the image sets, to obtain a dataset containing negative samples and positive samples; the positive samples represent the labeled color blocks with gaps, and the negative samples represent the unlabeled gap patches without color blocks;

[0006] S02: Using the YOLOX model as the base model, the convolutional activation layer in the base model is replaced with a transformer module, and the upsampling module in the base model is replaced with an upsampling module to obtain the first YOLOX model. The first YOLOX model is trained using a dataset. The loss curves corresponding to the object error, class error, and feature point detection error of the first YOLOX model during training are obtained through the loss function. Training is stopped when the fluctuation amplitude of each loss curve is within the corresponding preset range, resulting in an improved YOLOX model. The improved YOLOX model includes: a first YOLO detection head, a second YOLO detection head, and a third YOLO detection head, as well as multiple feature extraction branches containing a transformer module. The feature data of each detection head is provided by feature fusion of the corresponding feature extraction branches.

[0007] S03: Input the escalator tread image into the improved YOLOX model. Each detection head in the model obtains the corresponding detection features through the feature data provided by its respective feature extraction branch. The detection features of each detection head are used to obtain the color block detection image with gaps. The detection features include: category detection features, feature point detection features, and object detection features.

[0008] S04: Perform color segmentation on the color block detection image with gaps using the HSV model to obtain the corresponding black and white grayscale image, and obtain the pixel width occupied by the black area in the black and white grayscale image, which is the width of the gap in the escalator tread.

[0009] Furthermore, the loss function in step S02 includes:

[0010] The VarifocalLoss loss function is expressed as follows:

[0011]

[0012] In the formula, p is the classification score predicted by the improved YOLOX model, q is the target IoU score, for positive samples q is the IoU between the predicted box and the ground truth box, and for negative samples q is 0; α is the loss weight; p γ The sample weights are represented by VFL(p,q); VFL(p,q) represents the object error and the class error.

[0013] The loss function is expressed as follows:

[0014]

[0015] In the formula:

[0016]

[0017]

[0018] Where Intersection(A,B) represents the intersection area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model, and Union(A,B) represents the union area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model; b,b gt ρ represents the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. 2 (b,b gt ) is the Euclidean distance between the center points of the predicted box and the ground truth box, c is the diagonal distance of the smallest closure region that can simultaneously contain the predicted box and the ground truth box, and α is the loss weight. Indicates the feature point detection error; w gt h is the width of the actual bounding box. gt is the height of the ground truth bounding box, w is the width of the predicted bounding box, and h is the height of the predicted bounding box.

[0019] Furthermore, the process of training the first YOLOX model using the dataset also includes:

[0020] For the image data in the dataset, the images currently input to the first YOLOX model are randomly flipped, and Mosaic data augmentation is used. Specifically, any four images are stitched together, and the new stitched image is added to the training to expand the dataset.

[0021] Furthermore, the colored band is red, orange, yellow, green, cyan, blue, or purple.

[0022] The specific steps of S04 are as follows:

[0023] Obtain the angular range of the color of the color stripe in the OpenCV hue wheel;

[0024] Based on the angular range of the color band, the saturation and hue of the color block detection image with gaps are respectively assigned to the first preset angular range and the second preset angular range according to the color distribution of the OpenCV hue wheel.

[0025] Color segmentation is performed on the color block detection image with gaps to obtain its corresponding black and white grayscale image. The pixel width occupied by the black area in the black and white grayscale image is obtained, which is the width of the gap in the escalator tread.

[0026] Furthermore, the real-time detection method further includes the following steps:

[0027] S05: Determine whether the width of the gap between the escalator steps exceeds the set threshold. If so, issue an alarm.

[0028] This invention also proposes a real-time detection system for escalator tread gaps, applied to an improved YOLOX model, wherein colored strips are affixed to both sides of the gaps in the escalator treads, and the system includes:

[0029] The dataset module is used to acquire images and videos of escalator steps with colored stripes affixed to both sides of the gap, and images and videos of escalator steps without colored stripes affixed. Both escalator step images and videos are converted into image sets, and the colored stripes in the images with gaps in the image sets are labeled, resulting in a dataset containing negative and positive samples. The positive samples represent labeled colored stripes with gaps, and the negative samples represent unlabeled gap patches without colored stripes.

[0030] The model training module is used to take the YOLOX model as the base model, replace the convolutional activation layer in the base model with a transformer module, and replace the upsampling module in the base model with an upsampling module to obtain a first YOLOX model. The first YOLOX model is trained with a dataset. The loss curves corresponding to the object error, class error, and feature point detection error of the first YOLOX model during training are obtained through the loss function. Training is stopped when the fluctuation amplitude of each loss curve is within the corresponding preset range, resulting in an improved YOLOX model. The improved YOLOX model includes: a first YOLO detection head, a second YOLO detection head, and a third YOLO detection head, as well as multiple feature extraction branches containing a transformer module. The feature data of each detection head is provided by feature fusion of the corresponding feature extraction branches.

[0031] The detection module is used to input the escalator tread image into the improved YOLOX model. Each detection head in the model obtains the corresponding detection features through the feature data provided by its corresponding feature extraction branch. The detection features of each detection head are used to obtain the color block detection image with gaps. The detection features include: category detection features, feature point detection features, and object detection features.

[0032] The gap width acquisition module is used to perform color segmentation on the color block detection image with gaps using the HSV model to obtain the corresponding black and white grayscale image, and to obtain the pixel width occupied by the black area in the black and white grayscale image, which is the width of the gap in the escalator tread.

[0033] Furthermore, the loss function in the model training module includes:

[0034] The VarifocalLoss loss function is expressed as follows:

[0035]

[0036] In the formula, p is the classification score predicted by the improved YOLOX model, q is the target IoU score, for positive samples q is the IoU between the predicted box and the ground truth box, and for negative samples q is 0; α is the loss weight; p γ The sample weights are represented by VFL(p,q); VFL(p,q) represents the object error and the class error.

[0037] The loss function is expressed as follows:

[0038]

[0039] In the formula:

[0040]

[0041]

[0042] Where Intersection(A,B) represents the intersection area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model, and Union(A,B) represents the union area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model; b,b gt ρ represents the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. 2 (b,b gt ) is the Euclidean distance between the center points of the predicted box and the ground truth box, c is the diagonal distance of the smallest closure region that can simultaneously contain the predicted box and the ground truth box, and α is the loss weight. Indicates the feature point detection error; w gt h is the width of the actual bounding box. gt is the height of the ground truth bounding box, w is the width of the predicted bounding box, and h is the height of the predicted bounding box.

[0043] Furthermore, the process of training the first YOLOX model using the dataset also includes:

[0044] For the image data in the dataset, the images currently input to the first YOLOX model are randomly flipped, and Mosaic data augmentation is used. Specifically, any four images are stitched together, and the new stitched image is added to the training to expand the dataset.

[0045] Furthermore, the colored band is red, orange, yellow, green, cyan, blue, or purple.

[0046] The gap width acquisition module is specifically used for:

[0047] Obtain the angular range of the color of the color stripe in the OpenCV hue wheel;

[0048] Based on the angular range of the color band, the saturation and hue of the color block detection image with gaps are respectively assigned to the first preset angular range and the second preset angular range according to the color distribution of the OpenCV hue wheel.

[0049] Color segmentation is performed on the color block detection image with gaps to obtain its corresponding black and white grayscale image. The pixel width occupied by the black area in the black and white grayscale image is obtained, which is the width of the gap in the escalator tread.

[0050] Furthermore, the system also includes an alarm module for issuing an alarm when the width of the gap between the escalator steps exceeds a set threshold.

[0051] Compared with the prior art, the present invention has at least the following beneficial effects:

[0052] (1) This invention uses the YOLOX model as the base model, replaces the convolutional activation layer in the base model with the transformer module, and replaces the upsampling module in the base model with the upsampling expansion module to obtain the first YOLOX model. The first YOLOX model is trained with a dataset including labeled color blocks with gaps and unlabeled gap patches without color blocks. During the training process, the loss function is used to obtain the improved YOLOX model. The escalator step image is input into the improved YOLOX model to detect the color block detection image (visual object detection). The color block detection image is color segmented (image segmentation technology) using the HSV model to obtain the corresponding black and white grayscale image. The pixel width occupied by the black area in the black and white grayscale image is obtained, which is the width of the escalator step gap. It realizes the real-time detection of escalator step gaps by using visual object detection and image segmentation technology.

[0053] (2) The method proposed in this invention for real-time detection of escalator tread gaps using visual target detection and image segmentation technology has high real-time performance, high efficiency and accuracy. It represents a leap forward compared with traditional manual inspection, while saving a lot of manpower and material resources and greatly improving the detection accuracy.

[0054] (3) This invention uses the Varifocal Loss function to control the training of the model, so that more attention is paid to high-quality positive samples during the training process of the model, thereby improving the detection accuracy of the improved YOLOX model.

[0055] (4) This invention uses the YOLOX model as the base model, replaces the convolutional activation layer in the base model with the transformer module, and replaces the upsampling module in the base model with the upsampling expansion module to extract deeper features, thereby improving the detection accuracy of the model. Attached Figure Description

[0056] Figure 1 Flowchart of a method for real-time detection of gaps in escalator treads;

[0057] Figure 2 This is a structural diagram of the YOLOX basic model;

[0058] Figure 3 A structural diagram of the improved YOLOX model;

[0059] Figure 4 Diagram of the internal structure of the YOLO detection head;

[0060] Figure 5 This is a black and white grayscale image obtained after color segmentation of the image with gapped color patches using the HSV model.

[0061] Figure 6 This is a module diagram of a real-time detection system for gaps in escalator treads. Detailed Implementation

[0062] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0063] Example 1

[0064] To achieve real-time detection of escalator tread gaps based on visual target detection and image segmentation techniques, such as... Figure 1 As shown, this invention proposes a real-time detection method for escalator tread gaps, applied to an improved YOLOX model. Colored strips are affixed to both sides of the escalator tread gaps. The method includes the following steps:

[0065] S01: Acquire images and videos of escalator steps with colored stripes affixed to both sides of the gap, and images and videos of escalator steps without colored stripes affixed. Convert both escalator step images and videos into image sets, and label the color blocks with gaps in the images with color blocks in the image sets, to obtain a dataset containing negative samples and positive samples; the positive samples represent the labeled color blocks with gaps, and the negative samples represent the unlabeled gap patches without color blocks;

[0066] It should be noted that the escalator step images and videos in this invention are acquired by a camera installed above the escalator. This invention only requires a camera and very basic computing equipment to implement, and has real-time performance, high efficiency and accuracy.

[0067] S02: Using the YOLOX model as the base model, replace the convolutional activation layer in the base model with a transformer module (specifically, a Swin-transformer module), and replace the upsampling module in the base model with an upsampling expansion module (specifically, a "Patch Expanding module + Swin-transformer module") to obtain a first YOLOX model. Train the first YOLOX model using a dataset, and obtain the loss curves corresponding to the object error, class error, and feature point detection error of the first YOLOX model during training using a loss function. Stop training when the fluctuation amplitude of each loss curve is within the corresponding preset range to obtain an improved YOLOX model. The improved YOLOX model includes: a first YOLO detection head, a second YOLO detection head, and a third YOLO detection head, as well as multiple feature extraction branches containing a transformer module. The feature data of each detection head is provided by feature fusion of the corresponding feature extraction branches.

[0068] It should be noted that, Figure 2 This is a structural diagram of the YOLOX basic model in the existing technology. Figure 4 This is a diagram showing the internal structure of each detection head in the YOLOX basic model.

[0069] Figure 3 A structural diagram of the improved YOLOX model;

[0070] In step S02, the convolutional activation layer in the base model is replaced with the transformer module. Specifically, the "Swin-transformer module" is used to replace... Figure 2 The convolutional activation layers in the code; the replaced convolutional activation layers specifically include... Figure 2 The layers that propagate downwards from the Csp layer are: "convolutional layer + BN layer + SiLU activation function", "convolutional layer + BN layer + SiLU activation function + Csp layer", and "convolutional layer + BN layer + SiLU activation function + fully connected layer + Csp layer".

[0071] Figure 3 The first transformer module, the second transformer module, and the third transformer module are actually replacements of the Swin-transformer module. For ease of description, they are distinguished by name.

[0072] In step S02, the upsampling module in the base model is replaced with an upsampling dilation module, specifically as follows:

[0073] Replace with "Patch Expanding module + Swin-transformer module" Figure 2 The upsampling module in the middle.

[0074] Figure 3 The first upsampling expansion module and the second upsampling expansion module are actually replacements of the "PatchExpanding module + Swin-transformer module"; for ease of description, the names are distinguished.

[0075] What needs to be explained is:

[0076] like Figure 3 As shown, in the improved YOLOX model, the feature data of the first YOLOX detector head comes from its corresponding feature extraction branch, which includes:

[0077] First feature extraction branch: First transformer module - First feature fusion module;

[0078] Second feature extraction branch: Second transformer module - Second feature fusion module - 2D convolution - Second upsampling dilation module - First feature fusion module;

[0079] The third feature extraction branch consists of: the third transformer module - 2D convolution - the first upsampling dilation module - the second feature fusion module - 2D convolution - the second upsampling dilation module - the first feature fusion module;

[0080] The feature data of the second YOLO detection head comes from its corresponding feature extraction branch, which includes:

[0081] First feature extraction branch: First transformer module - First feature fusion module - Downsampling - Third feature fusion module;

[0082] Second feature extraction branch: Second transformer module - Second feature fusion module - 2D convolution - Second upsampling dilation module - First feature fusion module - Downsampling - Third feature fusion module;

[0083] Third feature extraction branch: Second transformer module - Second feature fusion module - 2D convolution - Third feature fusion module;

[0084] Fourth feature extraction branch: Third transformer module - 2D convolution - First upsampling dilation module - Second feature fusion module - 2D convolution - Second upsampling dilation module - First feature fusion module - Downsampling - Third feature fusion module;

[0085] Fifth feature extraction branch: Third transformer module - 2D convolution - First upsampling dilation module - Second feature fusion module - 2D convolution - Third feature fusion module;

[0086] The feature data of the third YOLO detector comes from its corresponding feature extraction branch, which includes:

[0087] First feature extraction branch: First transformer module - First feature fusion module - Downsampling - Third feature fusion module - Downsampling - Fourth feature fusion module;

[0088] Second feature extraction branch: Second transformer module - Second feature fusion module - 2D convolution - Second upsampling dilation module - First feature fusion module - Downsampling - Third feature fusion module - Downsampling - Fourth feature fusion module;

[0089] The third feature extraction branch: Second transformer module - Second feature fusion module - 2D convolution - Third feature fusion module - Downsampling - Fourth feature fusion module;

[0090] Fourth feature extraction branch: Third transformer module - 2D convolution - First upsampling dilation module - Second feature fusion module - 2D convolution - Second upsampling dilation module - First feature fusion module - Downsampling - Third feature fusion module - Downsampling - Fourth feature fusion module;

[0091] Fifth feature extraction branch: Third transformer module - 2D convolution - First upsampling dilation module - Second feature fusion module - 2D convolution - Third feature fusion module - Downsampling - Fourth feature fusion module;

[0092] The sixth feature extraction branch consists of the third transformer module, a two-dimensional convolution, and the fourth feature fusion module.

[0093] It should be noted that this invention uses the "Swin-transformer module" instead. Figure 2 The convolutional activation layer in the transformer module performs downsampling processes of 8x, 16x, and 32x respectively through the first transformer module, the second transformer module, and the third transformer module.

[0094] In addition, this invention uses the "Patch Expanding module + Swin-transformer module" to replace Figure 2The upsampling module in the model, which uses the "Patch Expanding module + Swin-transformer module", upsamples the feature tensors in the model to twice their original size, reduces the number of channels corresponding to the feature tensors to 1 / 4, and increases the width W and height H of the feature tensors to twice their original size, so that they are aligned with the downsampled features of the corresponding downsampling factor.

[0095] This invention uses the YOLOX model as the base model, replaces the convolutional activation layer in the base model with the transformer module, and replaces the upsampling module in the base model with the upsampling dilation module to extract deeper features, which greatly improves the detection accuracy of the model.

[0096] The process of training the first YOLOX model using the dataset also includes:

[0097] For the image data in the dataset, the images currently input to the first YOLOX model are randomly flipped, and Mosaic data augmentation is used. Specifically, any four images are stitched together, and the new stitched image is added to the training to expand the dataset.

[0098] In this embodiment, step S01 further includes dividing the dataset into a training set and a validation set. Step S02 specifically involves training a first YOLOX model using the training set. Between stopping training and obtaining the improved YOLOX model, the following steps are included: validating the trained model using the validation set after training stops. During validation, the commonly used evaluation metric mAP in object detection is used as the model's evaluation standard. AP represents the detection accuracy for a specific category; a higher AP indicates fewer detection errors. The average AP across all categories yields mAP. When the model's mAP exceeds a set value, the current model is saved as the improved YOLOX model; when the model's mAP falls below the set value, training the first YOLOX model continues.

[0099] The loss function in step S02 includes:

[0100] The VarifocalLoss loss function is expressed as follows:

[0101]

[0102] In the formula, p is the classification score predicted by the improved YOLOX model, q is the target IoU score, for positive samples q is the IoU between the predicted box and the ground truth box, and for negative samples q is 0; α is the loss weight; p γ The sample weights are represented by VFL(p,q); VFL(p,q) represents the object error and the class error.

[0103] The loss function is expressed as follows:

[0104]

[0105] In the formula:

[0106]

[0107]

[0108] Where Intersection(A,B) represents the intersection area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model, and Union(A,B) represents the union area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model; b,b gt ρ represents the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. 2 (b,b gt ) is the Euclidean distance between the center points of the predicted box and the ground truth box, c is the diagonal distance of the smallest closure region that can simultaneously contain the predicted box and the ground truth box, and α is the loss weight. Indicates the feature point detection error; w gt h is the width of the actual bounding box. gt is the height of the ground truth bounding box, w is the width of the predicted bounding box, and h is the height of the predicted bounding box.

[0109] In this embodiment, the Focal Loss loss function, which was originally used for class loss and confidence loss in the YOLOX basic model to evaluate target boxes and prediction boxes, is replaced with the Varifocal Loss loss function. This utilizes the imbalanced weighting of positive and negative samples to highlight positive samples and improve the detection performance of the model.

[0110] In this invention, the following is used The regression branch of the model is trained using the loss function. The target and class branches of the model are trained using the VarifocalLoss loss function, thereby improving the accuracy of the model detection through the loss function. The higher the accuracy of the model, the lower the value of the loss function.

[0111] This invention uses the Varifocal Loss function to control the training of the model, which makes more attention paid to high-quality positive samples during the training process, thereby improving the detection accuracy of the improved YOLOX model.

[0112] S03: Input the escalator tread image into the improved YOLOX model. Each detection head in the model obtains the corresponding detection features through the feature data provided by its respective feature extraction branch. The detection features of each detection head are used to obtain the color block detection image with gaps. The detection features include: category detection features, feature point detection features, and object detection features.

[0113] S04: Perform color segmentation on the color block detection image with gaps using the HSV model to obtain the corresponding black and white grayscale image, and obtain the pixel width occupied by the black area in the black and white grayscale image, which is the width of the gap in the escalator tread.

[0114] The colored band is red, orange, yellow, green, cyan, blue, or purple.

[0115] The specific steps of S04 are as follows:

[0116] Obtain the angular range of the color of the color stripe in the OpenCV hue wheel;

[0117] Based on the angular range of the color band, the saturation and hue of the color block detection image with gaps are respectively assigned to the first preset angular range and the second preset angular range according to the color distribution of the OpenCV hue wheel.

[0118] Color segmentation is performed on the hue of the image with gapped color patches to obtain its corresponding black and white grayscale image (see...). Figure 5 In the image, black represents the gap and white represents the segmented color blocks. The pixel width of the black area in the grayscale image is obtained, which is the width of the gap in the escalator tread.

[0119] This embodiment uses green or yellow color bands as examples for illustration:

[0120] The HSV model is divided into H (Hue), S (Saturation), and V (Title), which are also known as Hue, Saturation, and Value.

[0121] In this embodiment, yellow and green are located at 11-34° and 35-99° respectively in the OpenCV hue wheel.

[0122] Based on the angular range of yellow or green in the OpenCV hue wheel, the saturation and hue of the gapped color block detection image are assigned to 43-255° and 46-255° respectively according to the color distribution of the OpenCV hue wheel.

[0123] Color segmentation is performed on the hue of the color patch detection image with gaps to obtain its corresponding black and white grayscale image.

[0124] The real-time detection method further includes the following steps:

[0125] S05: Determine whether the width of the gap between the escalator steps exceeds the set threshold. If so, issue an alarm.

[0126] This invention uses the YOLOX model as the base model, replaces the convolutional activation layer in the base model with a transformer module, and replaces the upsampling module in the base model with an upsampling dilation module to obtain a first YOLOX model. The first YOLOX model is trained using a dataset including labeled gapped color patches and unlabeled gapped patches. During training, an improved YOLOX model is obtained using a loss function. Escalator tread images are input into the improved YOLOX model to detect gapped color patch images (visual object detection). The gapped color patch detection images are then color segmented using an HSV model (image segmentation technique) to obtain the corresponding black and white grayscale images. The pixel width of the black area in the black and white grayscale images is obtained, which is the width of the escalator tread gap. This invention achieves real-time detection of escalator tread gaps by utilizing visual object detection and image segmentation techniques.

[0127] Example 2

[0128] like Figure 6 As shown, this invention also proposes a real-time escalator tread gap detection system, applied to an improved YOLOX model, wherein colored strips are affixed to both sides of the escalator tread gap, and the system includes:

[0129] The dataset module is used to acquire images and videos of escalator steps with colored stripes affixed to both sides of the gap, and images and videos of escalator steps without colored stripes affixed. Both escalator step images and videos are converted into image sets, and the colored stripes in the images with gaps in the image sets are labeled, resulting in a dataset containing negative and positive samples. The positive samples represent labeled colored stripes with gaps, and the negative samples represent unlabeled gap patches without colored stripes.

[0130] The model training module is used to take the YOLOX model as the base model, replace the convolutional activation layer in the base model with a transformer module, and replace the upsampling module in the base model with an upsampling module to obtain a first YOLOX model. The first YOLOX model is trained with a dataset. The loss curves corresponding to the object error, class error, and feature point detection error of the first YOLOX model during training are obtained through the loss function. Training is stopped when the fluctuation amplitude of each loss curve is within the corresponding preset range, resulting in an improved YOLOX model. The improved YOLOX model includes: a first YOLO detection head, a second YOLO detection head, and a third YOLO detection head, as well as multiple feature extraction branches containing a transformer module. The feature data of each detection head is provided by feature fusion of the corresponding feature extraction branches.

[0131] The loss function in the model training module includes:

[0132] The VarifocalLoss loss function is expressed as follows:

[0133]

[0134] In the formula, p is the classification score predicted by the improved YOLOX model, q is the target IoU score, for positive samples q is the IoU between the predicted box and the ground truth box, and for negative samples q is 0; α is the loss weight; p γ The sample weights are represented by VFL(p,q); VFL(p,q) represents the object error and the class error.

[0135] The loss function is expressed as follows:

[0136]

[0137] In the formula:

[0138]

[0139]

[0140] Where Intersection(A,B) represents the intersection area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model, and Union(A,B) represents the union area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model; b,b gt ρ represents the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. 2 (b,b gt ) is the Euclidean distance between the center points of the predicted box and the ground truth box, c is the diagonal distance of the smallest closure region that can simultaneously contain the predicted box and the ground truth box, and α is the loss weight. Indicates the feature point detection error; w gt h is the width of the actual bounding box. gt is the height of the ground truth bounding box, w is the width of the predicted bounding box, and h is the height of the predicted bounding box.

[0141] The process of training the first YOLOX model using the dataset also includes:

[0142] For the image data in the dataset, the images currently input to the first YOLOX model are randomly flipped, and Mosaic data augmentation is used. Specifically, any four images are stitched together, and the new stitched image is added to the training to expand the dataset.

[0143] The detection module is used to input the escalator tread image into the improved YOLOX model. Each detection head in the model obtains the corresponding detection features through the feature data provided by its corresponding feature extraction branch. The detection features of each detection head are used to obtain the color block detection image with gaps. The detection features include: category detection features, feature point detection features, and object detection features.

[0144] The gap width acquisition module is used to perform color segmentation on the color block detection image with gaps using the HSV model to obtain the corresponding black and white grayscale image, and to obtain the pixel width occupied by the black area in the black and white grayscale image, which is the width of the gap in the escalator tread.

[0145] The colored band is red, orange, yellow, green, cyan, blue, or purple.

[0146] The gap width acquisition module is specifically used for:

[0147] Obtain the angular range of the color of the color stripe in the OpenCV hue wheel;

[0148] Based on the angular range of the color band, the saturation and hue of the color block detection image with gaps are respectively assigned to the first preset angular range and the second preset angular range according to the color distribution of the OpenCV hue wheel.

[0149] Color segmentation is performed on the color block detection image with gaps to obtain its corresponding black and white grayscale image. The pixel width occupied by the black area in the black and white grayscale image is obtained, which is the width of the gap in the escalator tread.

[0150] The system also includes an alarm module, which issues an alarm when the width of the gap between the escalator steps exceeds a set threshold.

[0151] The method for real-time detection of escalator tread gaps proposed in this invention, which utilizes visual target detection and image segmentation technology, has high real-time performance, high efficiency and accuracy. It represents a significant improvement over traditional manual inspection, while saving a large amount of manpower and material resources and greatly improving the detection accuracy.

[0152] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0153] Furthermore, in this invention, descriptions involving terms such as "first," "second," and "a" are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0154] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0155] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

Claims

1. A method for real-time detection of gaps in escalator treads, characterized in that, Applied to the improved YOLOX model, the escalator treads have colored strips affixed to both sides of the gaps. The method includes the following steps: S01: Acquire images and videos of escalator steps with colored stripes affixed to both sides of the gap, and images and videos of escalator steps without colored stripes affixed. Convert both escalator step images and videos into image sets, and label the color blocks with gaps in the images with color blocks in the image sets, to obtain a dataset containing negative samples and positive samples; the positive samples represent the labeled color blocks with gaps, and the negative samples represent the unlabeled gap patches without color blocks; S02: Using the YOLOX model as the base model, the convolutional activation layer in the base model is replaced with a transformer module, and the upsampling module in the base model is replaced with an upsampling module to obtain the first YOLOX model. The first YOLOX model is trained using a dataset. The loss curves corresponding to the object error, class error, and feature point detection error of the first YOLOX model during training are obtained through the loss function. Training is stopped when the fluctuation amplitude of each loss curve is within the corresponding preset range, resulting in an improved YOLOX model. The improved YOLOX model includes: a first YOLO detection head, a second YOLO detection head, and a third YOLO detection head, as well as multiple feature extraction branches containing a transformer module. The feature data of each detection head is provided by feature fusion of the corresponding feature extraction branches. The loss function in step S02 includes: The VarifocalLoss loss function is expressed as follows: In the formula, p is the classification score predicted by the improved YOLOX model, q is the target IoU score, for positive samples q is the IoU between the predicted box and the ground truth box, and for negative samples q is 0; α is the loss weight; p γ The sample weights are represented by VFL(p,q); VFL(p,q) represents the object error and the class error. The loss function is expressed as follows: In the formula: Where Intersection(A,B) represents the intersection area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model, and Union(A,B) represents the union area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model; b,b gt ρ represents the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. 2 (b,b gt ) is the Euclidean distance between the center points of the predicted box and the ground truth box, c is the diagonal distance of the smallest closure region that can simultaneously contain the predicted box and the ground truth box, and α is the loss weight. Indicates the feature point detection error; w gt h is the width of the actual bounding box. gt w is the height of the ground truth bounding box, w is the width of the predicted bounding box, and h is the height of the predicted bounding box. S03: Input the escalator tread image into the improved YOLOX model. Each detection head in the model obtains the corresponding detection features through the feature data provided by its respective feature extraction branch. The detection features of each detection head are used to obtain the color block detection image with gaps. The detection features include: category detection features, feature point detection features, and object detection features. S04: Perform color segmentation on the image with gapped color blocks using the HSV model to obtain the corresponding grayscale image. Then, obtain the pixel width of the black area in the grayscale image, which is the width of the escalator tread gap. Specifically, step S04 involves: Obtain the angular range of the color of the color stripe in the OpenCV hue wheel; Based on the angular range of the color band, the saturation and hue of the color block detection image with gaps are respectively assigned to the first preset angular range and the second preset angular range according to the color distribution of the OpenCV hue wheel. Color segmentation is performed on the color block detection image with gaps to obtain its corresponding black and white grayscale image. The pixel width occupied by the black area in the black and white grayscale image is obtained, which is the width of the gap in the escalator tread.

2. The method for real-time detection of escalator tread gaps according to claim 1, characterized in that, The process of training the first YOLOX model using the dataset also includes: For the image data in the dataset, the images currently input to the first YOLOX model are randomly flipped, and Mosaic data augmentation is used. Specifically, any four images are stitched together, and the new stitched image is added to the training to expand the dataset.

3. The method for real-time detection of escalator tread gaps according to claim 2, characterized in that, The colored bands are red, orange, yellow, green, cyan, blue, or purple.

4. The method for real-time detection of escalator tread gaps according to claim 3, characterized in that, The real-time detection method further includes the following steps: S05: Determine whether the width of the gap between the escalator steps exceeds the set threshold. If so, issue an alarm.

5. A real-time detection system for escalator tread gaps, characterized in that, Applied to the improved YOLOX model, the escalator treads have colored strips affixed to both sides of the gaps. The system includes: The dataset module is used to acquire images and videos of escalator steps with colored stripes affixed to both sides of the gap, and images and videos of escalator steps without colored stripes affixed. Both escalator step images and videos are converted into image sets, and the colored stripes in the images with gaps in the image sets are labeled, resulting in a dataset containing negative and positive samples. The positive samples represent labeled colored stripes with gaps, and the negative samples represent unlabeled gap patches without colored stripes. The model training module is used to take the YOLOX model as the base model, replace the convolutional activation layer in the base model with a transformer module, and replace the upsampling module in the base model with an upsampling module to obtain a first YOLOX model. The first YOLOX model is trained with a dataset. The loss curves corresponding to the object error, class error, and feature point detection error of the first YOLOX model during training are obtained through the loss function. Training is stopped when the fluctuation amplitude of each loss curve is within the corresponding preset range, resulting in an improved YOLOX model. The improved YOLOX model includes: a first YOLO detection head, a second YOLO detection head, and a third YOLO detection head, as well as multiple feature extraction branches containing a transformer module. The feature data of each detection head is provided by feature fusion of the corresponding feature extraction branches. The loss function in the model training module includes: The VarifocalLoss loss function is expressed as follows: In the formula, p is the classification score predicted by the improved YOLOX model, q is the target IoU score, for positive samples q is the IoU between the predicted box and the ground truth box, and for negative samples q is 0; α is the loss weight; p γ The sample weights are represented by VFL(p,q); VFL(p,q) represents the object error and the class error. The loss function is expressed as follows: In the formula: Where Intersection(A,B) represents the intersection area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model, and Union(A,B) represents the union area of ​​the predicted bounding box A and the target bounding box B in the improved YOLOX model; b,b gt ρ represents the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. 2 (b,b gt ) is the Euclidean distance between the center points of the predicted box and the ground truth box, c is the diagonal distance of the smallest closure region that can simultaneously contain the predicted box and the ground truth box, and α is the loss weight. Indicates the feature point detection error; w gt h is the width of the actual bounding box. gt w is the height of the ground truth bounding box, w is the width of the predicted bounding box, and h is the height of the predicted bounding box. The detection module is used to input the escalator tread image into the improved YOLOX model. Each detection head in the model obtains the corresponding detection features through the feature data provided by its corresponding feature extraction branch. The detection features of each detection head are used to obtain the color block detection image with gaps. The detection features include: category detection features, feature point detection features, and object detection features. The gap width acquisition module is used to perform color segmentation on the color block detection image with gaps using the HSV model to obtain the corresponding black and white grayscale image, and to obtain the pixel width occupied by the black area in the black and white grayscale image, which is the width of the escalator tread gap; the gap width acquisition module is specifically used for: Obtain the angular range of the color of the color stripe in the OpenCV hue wheel; Based on the angular range of the color band, the saturation and hue of the color block detection image with gaps are respectively assigned to the first preset angular range and the second preset angular range according to the color distribution of the OpenCV hue wheel. Color segmentation is performed on the color block detection image with gaps to obtain its corresponding black and white grayscale image. The pixel width occupied by the black area in the black and white grayscale image is obtained, which is the width of the gap in the escalator tread.

6. The real-time detection system for escalator tread gaps according to claim 5, characterized in that, The process of training the first YOLOX model using the dataset also includes: For the image data in the dataset, the images currently input to the first YOLOX model are randomly flipped, and Mosaic data augmentation is used. Specifically, any four images are stitched together, and the new stitched image is added to the training to expand the dataset.

7. The real-time detection system for escalator tread gaps according to claim 6, characterized in that, The colored bands are red, orange, yellow, green, cyan, blue, or purple.

8. The real-time detection system for escalator tread gaps according to claim 7, characterized in that, The system also includes an alarm module, which issues an alarm when the width of the gap between the escalator steps exceeds a set threshold.

Images