Training methods, anomaly monitoring methods, equipment and media for anomaly classification models

Abstract

This application relates to a training method for an anomaly classification model, an anomaly monitoring method for escalators, equipment, and media, belonging to the field of image recognition technology. The method includes: acquiring sample images of the escalator and at least one corresponding mask label image; inputting the sample images into a pre-created convolutional neural network to obtain a network classification image; inputting the network classification image and the corresponding mask label image into a preset loss function to obtain a loss result; and adjusting the network parameters of the convolutional neural network based on the loss result to train an anomaly classification model. This method can solve the problems of slow inference speed and high hardware requirements of traditional escalator anomaly monitoring methods due to the use of deep learning network models. It can make the anomaly classification model more lightweight, improve the inference speed of the anomaly classification model, and reduce the hardware requirements of the equipment.

Description

[Technical Field]

[0001] This application relates to a training method for an anomaly classification model, an anomaly monitoring method, equipment, and medium for escalators, and belongs to the field of image recognition technology. [Background Technology]

[0002] Escalators can transport passengers upwards or downwards at an angle. With the widespread use of escalators, safety accidents during their operation are frequent. Therefore, it is necessary to monitor any abnormalities on escalators.

[0003] Traditional methods for monitoring escalator anomalies include inputting captured images of the escalator into a pre-trained neural network model to obtain monitoring results. The neural network model is typically a deep learning network model such as a residual network (ResNet) or a dense convolutional network (DenseNet).

[0004] However, deep learning network models have complex network structures, numerous model parameters, and large memory consumption, which leads to slow inference speed of monitoring results and high hardware requirements for devices. [Summary of the Invention]

[0005] This application provides a training method for an anomaly classification model, an anomaly monitoring method for escalators, equipment, and media. It addresses the problems of slow inference speed and high hardware requirements associated with traditional escalator anomaly monitoring methods that rely on deep learning network models. The technical solution provided in this application is as follows:

[0006] Firstly, a training method for an anomaly classification model is provided, the method comprising:

[0007] Obtain a sample image of an escalator and at least one mask label image corresponding to the sample image, wherein the mask label image is used to indicate whether the sample image includes an abnormal target of the corresponding type;

[0008] The sample image is input into a pre-created convolutional neural network to obtain a network classification image. The convolutional neural network includes an input layer, multiple cascaded feature extraction structures connected to the input layer, a nonlinear transformation structure connected to the last feature extraction structure, and an output layer connected to the nonlinear transformation structure. The nonlinear transformation structure is used to convert the feature pixel values ​​of the feature map output by the feature extraction structure into mask pixel values ​​that match the pixel value range of the mask label image, thereby obtaining the network classification image.

[0009] The mask label images corresponding to the network classification image and the sample image are input into a preset loss function to obtain the loss result;

[0010] The network parameters of the convolutional neural network are adjusted based on the loss result to train an anomaly classification model, which is then used for anomaly monitoring of escalators.

[0011] Optionally, each feature extraction structure includes n cascaded convolutional structures, m dilated convolutional structures connected to the last convolutional structure, and convolutional blocks connected to the dilated convolutional structures; where n and m are both positive integers.

[0012] Each convolutional structure includes multiple cascaded convolutional blocks and a max-pooling layer connected to the last convolutional block; the cascaded convolutional blocks are used to extract features from the input of the previous layer.

[0013] Optionally, n is 3 and m is 1;

[0014] The first convolutional structure consists of two cascaded convolutional blocks, each with 3 input channels and 32 output channels, and a kernel size of 5×5.

[0015] The second convolutional structure consists of three cascaded convolutional blocks, each with 32 input channels, 64 output channels, and a kernel size of 5×5.

[0016] The third convolutional structure consists of four cascaded convolutional blocks, each with 64 input channels and 64 output channels, and a kernel size of 5×5.

[0017] The dilated convolutional structure has 64 input channels, 1024 output channels, a kernel size of 3×3, and an expansion rate of 4.

[0018] The convolution block connected to the dilated convolution structure has 1024 input channels and the number of output channels equal to the number of classifications of the abnormal targets. The convolution kernel size is 1×1.

[0019] Optionally, the abnormal targets include at least two types, and the acquisition of sample images of the escalator and at least one mask label image corresponding to the sample images includes:

[0020] For each sample image, a labeling tool is used to annotate each abnormal target in the sample image to obtain a label file;

[0021] The label file is parsed to set the pixel values ​​corresponding to the marked areas in the label file as the label pixel values, thereby obtaining the sample mask feature image corresponding to the sample image. Different abnormal targets correspond to different label pixel values.

[0022] Based on the label pixel values, the sample mask feature image is divided into mask label images corresponding to various abnormal targets. The mask label image corresponding to each abnormal target indicates the area where the abnormal target is located through a first mask pixel value and indicates other areas besides the abnormal target through a second mask pixel value. Both the first mask pixel value and the second mask pixel value are located within the pixel value range of the mask label image.

[0023] Optionally, before inputting the mask label images corresponding to the network classification image and the sample image into a preset loss function to obtain the loss result, the method further includes:

[0024] The connected components of abnormal targets are determined in the mask label image;

[0025] If the number of pixels in the identified connected components is greater than or equal to a threshold, the mask label image is determined to indicate the presence of the abnormal target.

[0026] If the number of pixels in the identified connected component is less than a threshold, the mask label image is determined to indicate that the abnormal target is not present.

[0027] Optionally, the preset loss function includes a mean squared error loss function, and correspondingly, the loss result includes the mean squared error loss result output by the mean squared error loss function.

[0028] On the other hand, a method for monitoring anomalies in escalators is provided, the method comprising:

[0029] Acquire the target image obtained by image acquisition of the escalator to be monitored;

[0030] The target image is input into the anomaly classification model provided above to obtain at least one mask feature image;

[0031] Determine whether the mask feature image contains any abnormal target connected components;

[0032] If present, the target image includes an abnormal target and an alarm is triggered.

[0033] Optionally, the anomalous target includes a first anomalous target and a second anomalous target, the mask feature image includes a first mask feature image and a second mask feature image, and the output layer of the anomalous target classification model includes a first output layer and a second output layer. The first mask feature image output by the first output layer is used to indicate whether the first anomalous target is included; the second output layer outputs a second mask feature image to indicate whether the second anomalous target is included.

[0034] The method further includes:

[0035] If the first mask feature image indicates that the first anomalous target is included, it is determined that the target image includes the first anomalous target;

[0036] If the second mask feature image indicates that the second anomalous target is included, then the target image is determined to include the second anomalous target.

[0037] In a second aspect, an electronic device is provided, the device including a processor and a memory; the memory stores a program, the program being loaded and executed by the processor to implement the training method for the anomaly classification model provided in the first aspect, or to implement the anomaly monitoring method for escalators provided in the second aspect.

[0038] Thirdly, a computer-readable storage medium is provided, wherein a program is stored in the storage medium, and when executed by a processor, the program is used to implement the training method for the anomaly classification model provided in the first aspect, or to implement the anomaly monitoring method for escalators provided in the second aspect.

[0039] The beneficial effects of this application include at least the following: acquiring sample images of escalators and at least one corresponding mask label image, the mask label image indicating whether the sample image contains an abnormal target of the corresponding type; inputting the sample images into a pre-created convolutional neural network to obtain a network classification image, the convolutional neural network including an input layer, multiple cascaded feature extraction structures connected to the input layer, a nonlinear transformation structure connected to the last feature extraction structure, and an output layer connected to the nonlinear transformation structure; inputting the network classification image and the mask label image corresponding to the sample images into a preset loss function to obtain a loss result; adjusting the network parameters of the convolutional neural network based on the loss result to train an anomaly classification model, and using the anomaly classification model for escalator classification. This method enables anomaly monitoring and solves the problems of slow inference speed and high hardware requirements associated with traditional escalator anomaly monitoring methods that rely on deep learning network models. The nonlinear transformation structure converts the feature map output by the feature extraction structure into mask pixel values ​​that match the pixel value range of the mask label image. This allows the anomaly classification model to classify abnormal targets in the target image at the feature map level. Compared to traditional neural network models, the computational cost of the nonlinear transformation structure is far less than that of the feature fusion layer and classification layer after the feature extraction structure in traditional neural network models. This makes the anomaly classification model more lightweight, improves its inference speed, and reduces hardware requirements.

[0040] Furthermore, feature extraction can be achieved by setting the feature extraction structure as a simple cascade of network layers, which simplifies the network structure. Simultaneously, by incorporating max pooling layers to reduce the resolution of the feature maps, the computational cost of the feature extraction structure can be reduced, improving computational speed. Additionally, by combining dilated convolutions to expand the receptive field of the feature maps, the accuracy of the feature maps can be improved.

[0041] In addition, by adjusting the network parameters of the convolutional neural network using the mean squared error loss function to obtain an anomaly classification model, the difference between the feature map output by the model and the mask label image is expressed as a difference at the image pixel level, which can improve the accuracy of the model in generating mask feature images of abnormal targets.

[0042] In addition, by automatically dividing the sample mask feature images corresponding to various abnormal targets in the sample images into mask label images corresponding to each abnormal target, the generation efficiency of mask label images can be improved.

[0043] Furthermore, by acquiring target images from the escalator to be monitored, and inputting these images into a pre-trained anomaly classification model to obtain at least one mask feature image, the model can determine whether an anomaly is included. This addresses the problems of slow inference speed and high hardware requirements associated with traditional escalator anomaly monitoring methods that rely on deep learning network models. The nonlinear transformation structure converts the feature pixel values ​​of the feature map into label pixel values ​​that match the pixel value range of the mask label image. This allows the anomaly classification model to classify anomalies in the target image at the feature map level. Compared to traditional neural network models, the computational cost of the nonlinear transformation structure is significantly lower than that of the feature fusion layer and classification layer after feature extraction in traditional neural network models. This makes the anomaly classification model more lightweight, improves its inference speed, and reduces hardware requirements. Simultaneously, by triggering alarms when anomaly targets are present in connected components, users can promptly perceive anomalies, improving the timeliness of anomaly handling.

[0044] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings. [Attached Image Description]

[0045] Figure 1 This is a flowchart of a training method for an anomaly classification model provided in one embodiment of this application;

[0046] Figure 2 This is a schematic diagram of a mask label image provided in one embodiment of this application;

[0047] Figure 3 This is a schematic diagram of a convolutional neural network provided in one embodiment of this application;

[0048] Figure 4 This is a schematic diagram of a dilated convolution process provided in one embodiment of this application;

[0049] Figure 5 This is a schematic diagram of a convolutional block provided in one embodiment of this application;

[0050] Figure 6 This is a schematic diagram illustrating the changes in mean squared error loss and intersection-union ratio as the number of iterations increases, provided in one embodiment of this application.

[0051] Figure 7 This is a flowchart of an escalator anomaly monitoring method provided in one embodiment of this application;

[0052] Figure 8 This is a block diagram of a training apparatus for an anomaly classification model provided in one embodiment of this application;

[0053] Figure 9 This is a block diagram of an escalator anomaly monitoring device provided in one embodiment of this application;

[0054] Figure 10 This is a block diagram of an electronic device provided in one embodiment of this application.

Detailed Implementation Methods

[0055] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0056] Optionally, this application uses the abnormal monitoring method of escalator provided in various embodiments in an electronic device as an example for illustration. The electronic device is a terminal or a server. The terminal can be a mobile phone, computer, tablet computer, scanner, electronic eye, surveillance camera, etc. This embodiment does not limit the type of electronic device.

[0057] Figure 1 This is a flowchart of a training method for an anomaly classification model provided in one embodiment of this application. The method includes at least the following steps:

[0058] Step 101: Obtain a sample image of the escalator and at least one mask label image corresponding to the sample image. The mask label image is used to indicate whether the sample image includes an abnormal target of the corresponding type.

[0059] In one example, acquiring sample images includes: obtaining data samples from surveillance video of escalators; dividing the data samples into a training set and a validation set according to a preset ratio, wherein the data samples in the training set are the sample images. The preset ratio can be 8:2, 7:3, etc., and this embodiment does not limit the value of the preset ratio.

[0060] After obtaining the training set, the electronic device can also perform data augmentation on the data in the training set to avoid the problem of relatively simple collection scenarios of sample images and poor performance of the trained model, thereby improving the sample richness of the sample images and the generalization ability of the model. Among them, data augmentation methods include, but are not limited to: translation, rotation, mirroring, adding noise and / or color dithering. This embodiment does not limit the data augmentation methods.

[0061] Furthermore, since the image size processed by the anomaly classification model is relatively fixed, this embodiment also requires adjusting the image size of the sample images to match the anomaly classification model. At this time, the electronic device uses bilinear interpolation to scale each sample image so that the sample image size is a preset size that matches the anomaly classification model. The preset size can be 512*512 pixels or 256*256 pixels; this embodiment does not limit the value of the preset size. Figure 2 The example below uses a preset size of 512*512 pixels.

[0062] After obtaining the sample image, the sample image is labeled to obtain the coordinate position of the abnormal target, thereby obtaining the category label of the abnormal target.

[0063] In one example, the anomalous targets include at least two types. The process involves obtaining sample images of an escalator and at least one corresponding mask label image, including: for each sample image, using a labeling tool to annotate each anomalous target in the sample image to obtain a label file; parsing the label file to set the pixel values ​​corresponding to the annotated areas in the label file as label pixel values ​​to obtain a sample mask feature image corresponding to the sample image, where different anomalous targets have different label pixel values; dividing the sample mask feature image into mask label images corresponding to each type of anomalous target based on the label pixel values; the mask label image corresponding to each anomalous target indicates the area where the anomalous target is located through a first mask pixel value and indicates other areas besides the anomalous target through a second mask pixel value, where both the first and second mask pixel values ​​are within the pixel value range of the mask label image.

[0064] Schematic, the pixel value range of the mask label image is [0, 1], the first mask pixel value is 1, and the second mask pixel value is 0.

[0065] For example, using Labelme as the labeling tool and large luggage and strollers as the anomalous targets, the electronic device uses Labelme to label these two types of anomalous targets in each sample image, generating a .json file. Then, parsing the generated .json file yields the sample image and its corresponding label file. This label file converts the pixel values ​​within the marked areas into label values: background pixels are set to 0, large luggage pixels to 1, and stroller pixels to 2, thus generating a sample mask feature image for that sample image. Finally, each sample mask feature image is divided into two mask label images: the first represents whether there is large luggage (if so, the pixel value in the corresponding area is 1, and the other background areas are 0); the second represents whether there is a stroller (if so, the pixel value in the corresponding area is 1, and the other background areas are 0).

[0066] refer to Figure 2 After labeling the sample images, two sample mask feature images are obtained. The first image represents whether there is large luggage, and the second image represents whether there is a stroller. If so, the pixel value of the area corresponding to the large luggage and stroller is 1, and the pixel value of other areas is 0. Correspondingly, the monitoring results obtained by the anomaly classification model trained using the sample mask feature images also include two mask feature images: the first image represents whether there is large luggage, and the second image represents whether there is a stroller, as described below.

[0067] Step 102: Input the sample image into a pre-created convolutional neural network to obtain a network classification image. The convolutional neural network includes an input layer, multiple cascaded feature extraction structures connected to the input layer, a nonlinear transformation structure connected to the last feature extraction structure, and an output layer connected to the nonlinear transformation structure. The nonlinear transformation structure is used to convert the feature pixel values ​​of the feature map output by the feature extraction structure into mask pixel values ​​that match the pixel value range of the mask label image to obtain the network classification image.

[0068] Traditional escalator anomaly monitoring methods use neural network models that typically connect multiple network layers after the feature extraction structure, such as multi-scale feature fusion layers and / or classification layers. In this case, even if the feature extraction structure uses a lightweight network structure, the overall neural network model is still large because there are still many network layers after the feature extraction structure.

[0069] Based on this, this embodiment creatively replaces each network layer after the feature extraction structure with a nonlinear transformation structure, converting the feature pixel values ​​of the feature map into mask pixel values ​​that match the pixel value range of the mask label image. This enables the anomaly classification model trained using a convolutional neural network to classify abnormal targets at the feature map level. At the same time, compared with the traditional neural network model mentioned above, this anomaly classification model is more lightweight.

[0070] Among them, the nonlinear transformation structure is a network layer implemented based on a nonlinear function, and its computational cost is much less than that of the feature fusion layer, classification layer, etc. after the feature extraction structure in traditional neural network models.

[0071] Schematic example: taking a mask label image with pixel values ​​in the range [0, 1] and each pixel value in the feature map output by the feature extraction structure being greater than 0, the nonlinear transformation structure is a hyperbolic tangent function (tanh). According to the curve characteristics of the tanh function, when the input value is greater than 0, the output value ranges from [0, 1]. Therefore, after inputting each pixel value in the feature map output by the feature extraction structure into the tanh function, the resulting pixel score values ​​all belong to the range [0, 1].

[0072] Optionally, to further reduce the weight of the anomaly classification model, each feature extraction structure includes n cascaded convolutional structures, m dilated convolutional structures connected to the last convolutional structure, and convolutional blocks connected to the dilated convolutional structures; n and m are both positive integers.

[0073] Each convolutional structure includes multiple cascaded convolutional blocks and a max-pooling layer connected to the last convolutional block; the cascaded convolutional blocks are used to extract features from the input of the previous layer; the max-pooling layer is used to reduce the resolution of the feature maps extracted by the convolutional blocks.

[0074] Dilated convolutional structures are used to expand the receptive field of the feature map output from the previous layer and obtain multi-scale semantic information without changing the resolution.

[0075] The convolutional block connected to the dilated convolutional structure is used to reduce the number of channels in the feature map output by the dilated convolutional structure. The number of channels after reduction is determined based on the number of classifications of abnormal targets.

[0076] In this embodiment, the feature extraction structure can achieve feature extraction simply by cascading network layers, thus simplifying the network structure. Simultaneously, by incorporating max pooling layers to reduce the resolution of the feature map, the computational cost of the feature extraction structure can be reduced, improving computational speed. Furthermore, by combining dilated convolutions to expand the receptive field of the feature map, the accuracy of the feature map can be improved.

[0077] In one example, the structural reference of the anomaly classification model Figure 3 As shown, the anomaly classification model includes a feature extraction structure 31 and a nonlinear transformation structure 32. Figure 3 Taking feature extraction structure 31 with n=3 and m=1 as an example, the first convolutional structure includes two cascaded convolutional blocks, each with 3 input channels and 32 output channels, and a kernel size of 5×5. Specifically, the input of the first convolutional structure can be an RGB three-channel image. The second convolutional structure includes three cascaded convolutional blocks, each with 32 input channels and 64 output channels, and a kernel size of 5×5. The third convolutional structure includes four cascaded convolutional blocks, each with 64 input channels and 64 output channels, and a kernel size of 5×5. The dilated convolutional structure has 64 input channels and 1024 output channels, a kernel size of 3×3, and a dilation rate of 4. The convolutional block connected to the dilated convolutional structure has 1024 input channels and the number of output channels is the number of abnormal target classifications, with a kernel size of 1×1. Finally, the feature map output by feature extraction structure 31 is processed by nonlinear structure 32 to obtain the network classification image.

[0078] according to Figure 3 As can be seen, when there are at least two types of anomalous targets, each type of anomalous target corresponds to a network classification image, and different types of anomalous targets correspond to different network classification images. The network classification image corresponding to each anomalous target is used to indicate whether the corresponding anomalous target exists. Accordingly, there are at least two output layers, and each output layer outputs a network classification image corresponding to one type of anomalous target, with different types of anomalous targets corresponding to different output layers.

[0079] Figure 3 Taking large luggage and strollers as examples of abnormal targets, the number of output channels of a 1×1 convolutional block is 2. In other embodiments, abnormal targets can also be set to other types, such as: foreign objects in the gap between the escalator skirt and the steps, foreign objects between the upper and lower steps of the escalator, passengers' feet in contact with the brushes on the escalator, and / or the distance between the passenger's standing position and the escalator skirt is less than a preset distance threshold, etc. This embodiment does not limit the way abnormal targets are classified.

[0080] refer to Figure 4 The calculation process of dilated convolution is shown. A feature map 41 with an input size of 5 pixels × 5 pixels is convolved into a dilated convolution 42 with a sum of 3 × 3. The output feature map 43 still has a size of 5 pixels × 5 pixels, but its receptive field is larger.

[0081] Optionally, refer to Figure 5Each convolutional block includes a convolutional layer 51, a batch normalization layer 52, and a non-linear rectified linear unit (ReLU) layer 53 connected in sequence. The convolutional layer 51 is used for feature extraction, and the batch normalization layer 52 and the non-linear ReLU layer 53 are used to improve the convergence speed of the anomaly classification model during training.

[0082] according to Figure 5 The convolutional block shown includes a ReLU layer. Since the ReLU function takes values ​​greater than or equal to 0, the pixel values ​​of each pixel in the feature map output by the feature extraction layer are all greater than or equal to 0. In this case, the pixel values ​​in the feature map may be greater than 1, which does not match the pixel value range of the mask label image. Therefore, in this embodiment, the tanh function is used to calculate the pixel score of the feature map, ensuring that the converted pixel values ​​all fall within the pixel value range of the mask label image.

[0083] Step 103: Input the mask label images corresponding to the network classification image and the sample image into the preset loss function to obtain the loss result.

[0084] Since the mask label image is obtained by pixel transformation of the labeled regions in the sample image, its size is the same as the sample image. However, the presence of the max-pooling layer in the convolutional neural network reduces the size of the sample image, resulting in the final output network classification image being smaller than the sample image. Therefore, before inputting the mask label image corresponding to the sample image into the preset loss function, the mask label image needs to be reduced in size according to the proportion by which the convolutional neural network reduces the sample image. For example: in Figure 3 The network model shown will reduce the input image by 1 / 8. Therefore, before inputting the mask label image corresponding to the sample image into the preset loss function, the mask label image will be reduced to 1 / 8 of the original image.

[0085] Specifically, since each abnormal target corresponds to a network classification image, for each network classification image, the network classification image and the corresponding mask label image are input into a preset loss function to calculate the loss result of each network classification image.

[0086] In one example, the preset loss function includes the mean squared error loss function, and correspondingly, the loss result includes the mean squared error loss result output by the mean squared error loss function.

[0087] Schematic, the mean squared error loss function is expressed by the following equation:

[0088]

[0089] Where N represents the total number of pixels in the network classification image and the mask label image, x i y represents the pixel score value of each pixel in the image classified by the network. i This represents the label value of the corresponding pixel in the mask label image.

[0090] In other embodiments, the preset loss function may also include other types of loss functions, such as L1 norm loss, cross-entropy loss, etc. This embodiment does not limit the implementation of the preset loss function.

[0091] Optionally, since the sample mask feature images obtained using the labeling tool may contain labeling errors, leading to the misselected area on the sample image being used as the labeled area for an abnormal target, a mask label image corresponding to the abnormal target will be generated. To improve the accuracy of the mask label images, it is necessary to filter the mask label images generated by the erroneous operation. Since the misselected area is usually small, based on this, before inputting the mask label images corresponding to the network classification image and the sample image into a preset loss function and obtaining the loss result, the following steps are also included: determining the connected components of the abnormal target in the mask label image; if the number of pixels in the determined connected component is greater than or equal to a threshold, the mask label image is determined to indicate the presence of an abnormal target; if the number of pixels in the determined connected component is less than the threshold, the mask label image is determined to indicate the absence of an abnormal target.

[0092] Optionally, the connected component determination of the abnormal target in the mask label image includes: using the Two-Pass algorithm to calculate the connected components of foreground pixels with the same pixel value and adjacent positions in the mask label image, thus obtaining the connected components of the abnormal target. Here, the pixel value of the foreground pixel is the first mask pixel value, and the pixel value of the background pixels other than the foreground pixel is the second mask pixel value. For example, if the first mask pixel value is 1 and the second mask pixel value is 0, then the pixels with a pixel value of 1 constitute the connected components of the abnormal target.

[0093] In addition, if the connected components of anomalous targets in the masked label image are not determined, the masked label image is used to indicate that there are no anomalous targets.

[0094] Step 104: Adjust the network parameters of the convolutional neural network based on the loss results to train an anomaly classification model, and use the anomaly classification model for anomaly monitoring of escalators.

[0095] Specifically, the electronic device uses the minimum batch method to calculate the loss result and update the weights each time. During training, the Adam optimization algorithm and StepLR learning rate strategy are used to iteratively train the convolutional neural network to obtain the anomaly classification model. During the iterative training process, the difference between the feature map output by the model (i.e., the network classification image) and the mask label image is expressed as the difference at the image pixel level. By continuously fitting the mask label image, the pixel score value in the feature map gradually approaches the label value (i.e., pixel value) in the mask label image, thereby achieving the segmentation of abnormal targets in the image. Furthermore, the pixel value of the feature map indicates whether the sample image contains an abnormal target.

[0096] Optionally, the electronic device can also use a validation set to validate the current network model after training it.

[0097] Optionally, after the training process has reached a preset number of iterations, the performance of the currently trained network model can also be obtained. Illustratively, when the training process has reached a preset number of iterations, the electronic device calculates the ratio of the intersection to the union of the network classification images and the ground truth values ​​for each anomalous target, obtaining the Intersection-Over-Union (IoU). The model performance is then determined based on this IoU. The IoU is positively correlated with the model performance.

[0098] refer to Figure 6 Taking a preset number of times of 100 as an example, according to Figure 6 It can be seen that after training for 100 batches (epochs), the mean squared error loss of the training set gradually decreased to about 0.001, and the IoU score increased to about 0.99; the validation set loss gradually decreased to about 0.005, and the IoU score increased to about 0.83.

[0099] Optionally, after training the anomaly classification model, the electronic device can also use an independent test set to test the network performance of the anomaly classification model to obtain the classification results and detection speed of the anomaly classification model.

[0100] Specifically, the images in the test set are scaled to a preset size of 512*512 to match the anomaly classification model, then input into the trained anomaly classification model, and the network's detection speed is output. In this embodiment, the model storage footprint can be reduced to 58.9M. The model inference time for a single image on a GTX1660 GPU is shown in Table 1 below, indicating a relatively short inference time.

[0101] Table 1:

[0102] Batch_size 1 2 4 8 16 Min_time(ms) 2.997 1.449 0.750 0.375 0.125 Max_time(ms) 6.004 3.501 2.500 0.625 0.188 Avg_time(ms) 3.884 1.989 1.012 0.446 0.144

[0103] In summary, the training method for the anomaly classification model provided in this embodiment involves acquiring sample images of escalators and at least one corresponding mask label image. The mask label image indicates whether the sample image contains an anomaly of the corresponding type. The sample images are input into a pre-created convolutional neural network to obtain a network classification image. The convolutional neural network includes an input layer, multiple cascaded feature extraction structures connected to the input layer, a nonlinear transformation structure connected to the last feature extraction structure, and an output layer connected to the nonlinear transformation structure. The network classification image and the mask label images corresponding to the sample images are input into a preset loss function to obtain a loss result. The network parameters of the convolutional neural network are adjusted based on the loss result to train an anomaly classification model for use. This method enables anomaly monitoring of escalators. It addresses the issues of slow inference speed and high hardware requirements associated with traditional escalator anomaly monitoring methods that rely on deep learning network models. The nonlinear transformation structure converts the feature map output by the feature extraction structure into mask pixel values ​​that match the pixel value range of the mask label image. This allows the anomaly classification model to classify abnormal targets in the target image at the feature map level. Compared to traditional neural network models, the computational cost of the nonlinear transformation structure is significantly lower than that of the feature fusion layer and classification layer following the feature extraction structure. This makes the anomaly classification model more lightweight, improves its inference speed, and reduces hardware requirements.

[0104] Furthermore, feature extraction can be achieved by setting the feature extraction structure as a simple cascade of network layers, which simplifies the network structure. Simultaneously, by incorporating max pooling layers to reduce the resolution of the feature maps, the computational cost of the feature extraction structure can be reduced, improving computational speed. Additionally, by combining dilated convolutions to expand the receptive field of the feature maps, the accuracy of the feature maps can be improved.

[0105] In addition, by adjusting the network parameters of the convolutional neural network using the mean squared error loss function to obtain an anomaly classification model, the difference between the feature map output by the model and the mask label image is expressed as a difference at the image pixel level, which can improve the accuracy of the model in generating mask feature images of abnormal targets.

[0106] In addition, by automatically dividing the sample mask feature images corresponding to various abnormal targets in the sample images into mask label images corresponding to each abnormal target, the generation efficiency of mask label images can be improved.

[0107] Figure 7 This is a flowchart of an escalator anomaly monitoring method provided in one embodiment of this application. The method includes at least the following steps:

[0108] Step 701: Obtain the target image obtained by image acquisition of the escalator to be monitored.

[0109] The target image can be an image captured from an escalator or a frame from a video stream captured from an escalator. This embodiment does not limit the source of the target image.

[0110] Optionally, the target image may include the entire operating area of ​​the escalator to be monitored, or it may include only a portion of the operating area of ​​the escalator to be monitored, such as only the middle part of the operating area of ​​the escalator to be monitored. This embodiment does not limit the scope of the escalator to be collected.

[0111] Each target image may include one escalator, or may include at least two escalators. This embodiment does not limit the number of escalators included in each target image.

[0112] Step 702: Input the target image into the pre-trained anomaly classification model to obtain at least one mask feature image.

[0113] The anomaly classification model was trained using the training method provided in the above embodiments.

[0114] Specifically, the anomaly classification model has the same model structure as the convolutional neural network model in the above embodiments, including an input layer, multiple cascaded feature extraction structures connected to the input layer, a nonlinear transformation structure connected to the last feature extraction structure, and an output layer connected to the nonlinear transformation structure. The nonlinear transformation structure is used to convert the feature pixel values ​​of the feature map output by the feature extraction structure into mask pixel values ​​that match the pixel value range of the mask label image. Since the pixel value range of the mask label image is the same as the pixel value range of the mask feature image, the nonlinear transformation structure can convert the feature pixel values ​​of the feature map output by the feature extraction structure into mask pixel values ​​that match the pixel value range of the mask feature image.

[0115] Step 703: Determine whether there are abnormal target connected components in the mask feature image.

[0116] Optionally, determining whether an abnormal target connected component exists in the mask feature image includes: determining the connected components of the mask feature image corresponding to the abnormal target; if the number of pixels in the determined connected component is greater than or equal to a preset threshold, determining that an abnormal target connected component exists, and executing step 704; if the number of pixels in the determined connected component is less than the preset threshold, determining that no abnormal target connected component exists, thereby determining that no abnormal target exists in the mask feature image. In this case, no alarm processing is performed for the abnormal target.

[0117] Among them, determining the connected components of the mask feature image corresponding to the abnormal target includes: using the Two-Pass algorithm to calculate the connected components of foreground pixels with the same pixel value and adjacent positions in the mask feature image.

[0118] Step 704: If an abnormal target exists, the target image includes an abnormal target and an alarm is triggered.

[0119] Alarm handling methods include, but are not limited to: audio output alarm, light output alarm, and / or message notification alarm, etc. This embodiment does not limit the implementation method of alarm handling.

[0120] Optionally, if the abnormal monitoring result indicates that the target image includes at least two abnormal targets, the electronic device determines the alarm mode corresponding to at least two abnormal targets, wherein the alarm mode corresponding to different abnormal targets is different; and outputs an abnormal alarm according to the alarm mode.

[0121] For example: if the abnormal target is a large piece of luggage, the alarm handling method is to output audio 1; if the abnormal target is a stroller, the alarm handling method is to output audio 2; if the abnormal target is both a large piece of luggage and a stroller, then audio 3 is output and a light prompt is given. In other embodiments, the alarm handling method can also be implemented in other ways, which will not be listed here.

[0122] Optionally, in the presence of an anomalous target, the electronic device can also output a classification of the anomalous target. Specifically, the anomalous targets include a first anomalous target (such as large luggage) and a second anomalous target (such as a stroller), and correspondingly, the mask feature image includes a first mask feature image and a second mask feature image. In this case, the output layer of the anomalous target classification model includes a first output layer and a second output layer. The first mask feature image output by the first output layer is used to indicate whether the first anomalous target is included; the second output layer outputs a second mask feature image to indicate whether the second anomalous target is included. In this case, if the first mask feature image indicates that the first anomalous target is included, the target image is determined to include the first anomalous target; if the second mask feature image indicates that the second anomalous target is included, the target image is determined to include the second anomalous target.

[0123] In this context, "first" and "second" simply indicate differences and do not limit the number of targets. In actual implementation, there can be two or more categories of abnormal targets. Correspondingly, the output layer also includes two or more categories. The specific classification principle of abnormal targets is the same as that in the above embodiment, and will not be repeated here.

[0124] In summary, the escalator anomaly monitoring method provided in this embodiment acquires a target image of the escalator to be monitored through image acquisition; inputs the target image into a pre-trained anomaly classification model to obtain at least one mask feature image, and uses at least one mask feature image to determine whether an abnormal target is included. This solves the problem that traditional escalator anomaly monitoring methods, due to the use of deep learning network models, have slow inference speeds and high hardware requirements. Because the nonlinear transformation structure converts the feature pixel values ​​of the feature map into label pixel values ​​that match the pixel value range of the mask label image, the anomaly classification model can classify abnormal targets in the target image at the feature map level. Compared with traditional neural network models, the computational cost of the nonlinear transformation structure is much lower than that of the feature fusion layer and classification layer after the feature extraction structure in traditional neural network models. This makes the anomaly classification model more lightweight, improves the inference speed of the anomaly classification model, and reduces the hardware requirements of the equipment.

[0125] In addition, by issuing alarms when abnormal target connected components exist, users can be aware of the anomalies in a timely manner, thereby improving the timeliness of anomaly handling.

[0126] In addition, by setting different alarm handling methods for different abnormal targets, users can determine the type of abnormality based on the corresponding alarm handling method, thereby improving the timeliness of abnormality identification.

[0127] In addition, by determining the connected components of the mask feature image corresponding to the abnormal target, alarm processing is only performed when the number of pixels in the connected component is greater than or equal to a preset threshold. This can avoid the problem of false alarms caused by the possibility of missegmentation of small regions indicating abnormal targets in the mask feature image in the abnormal monitoring results, and can improve the accuracy of abnormal alarms.

[0128] Figure 8 This is a block diagram of a training apparatus for an anomaly classification model provided in one embodiment of this application. The apparatus includes at least the following modules: a data acquisition module 810, an image calculation module 820, a loss calculation module 830, and a model training module 840.

[0129] The data acquisition module 810 is used to acquire a sample image of the escalator and at least one mask label image corresponding to the sample image, wherein the mask label image is used to indicate whether the sample image includes an abnormal target of the corresponding type;

[0130] Image processing module 820 is used to input the sample image into a pre-created convolutional neural network to obtain a network classification image. The convolutional neural network includes an input layer, multiple cascaded feature extraction structures connected to the input layer, a nonlinear transformation structure connected to the last feature extraction structure, and an output layer connected to the nonlinear transformation structure. The nonlinear transformation structure is used to convert the feature pixel values ​​of the feature map output by the feature extraction structure into mask pixel values ​​that match the pixel value range of the mask label image to obtain the network classification image.

[0131] The loss calculation module 830 is used to input the mask label images corresponding to the network classification image and the sample image into a preset loss function to obtain the loss result;

[0132] The model training module 840 is used to adjust the network parameters of the convolutional neural network based on the loss result to train an anomaly classification model, and to use the anomaly classification model for anomaly monitoring of escalators.

[0133] For relevant details, please refer to the above method implementation examples.

[0134] It should be noted that the training device for the anomaly classification model provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the anomaly classification model training device can be divided into different functional modules to complete all or part of the functions described above. In addition, the anomaly classification model training device and the anomaly classification model training method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0135] Figure 9 This is a block diagram of an escalator anomaly monitoring device according to an embodiment of this application. The device includes at least the following modules: an image acquisition module 910, an image processing module 920, a connected component determination module 930, and an alarm processing module 940.

[0136] The image acquisition module 910 is used to acquire the target image obtained by image acquisition of the escalator to be monitored;

[0137] Image processing module 920 is used to input the target image into a pre-trained anomaly classification model to obtain at least one mask feature image;

[0138] Connectivity determination module 930 is used to determine whether there are abnormal target connected components in the mask feature image;

[0139] The alarm processing module 940 is used to perform alarm processing if the target image contains an abnormal target.

[0140] For relevant details, please refer to the above method implementation examples.

[0141] It should be noted that the escalator anomaly monitoring device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the escalator anomaly monitoring device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the escalator anomaly monitoring device and the escalator anomaly monitoring method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0142] Figure 10 This is a block diagram of an electronic device provided in one embodiment of this application. The device includes at least a processor 1001 and a memory 1002.

[0143] Processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0144] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store at least one instruction, which is executed by the processor 1001 to implement the training method for the anomaly classification model and the anomaly monitoring method for escalators provided in the method embodiments of this application.

[0145] In some embodiments, the electronic device may also optionally include a peripheral device interface and at least one peripheral device. The processor 1001, memory 1002, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to, radio frequency circuits, touch displays, audio circuits, and power supplies.

[0146] Of course, electronic devices may also include fewer or more components, and this embodiment does not limit this.

[0147] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the training method for the anomaly classification model and the anomaly monitoring method for escalators in the above method embodiments.

[0148] Optionally, this application also provides a computer product including a computer-readable storage medium storing a program, which is loaded and executed by a processor to implement the training method of the anomaly classification model and the anomaly monitoring method of the escalator in the above method embodiments.

[0149] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0150] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A training method for an anomaly classification model, characterized in that, The method includes: Obtain a sample image of an escalator and at least one mask label image corresponding to the sample image, wherein the mask label image is used to indicate whether the sample image includes an abnormal target of the corresponding type; The sample image is input into a pre-created convolutional neural network to obtain a network-classified image. The convolutional neural network includes an input layer, multiple cascaded feature extraction structures connected to the input layer, a non-linear transformation structure connected to the last feature extraction structure, and an output layer connected to the non-linear transformation structure. Each feature extraction structure includes n cascaded convolutional structures, m dilated convolutional structures connected to the last convolutional structure, and convolutional blocks connected to the dilated convolutional structures; where n and m are both positive integers. Each convolutional structure includes multiple cascaded convolutional blocks and a max-pooling layer connected to the last convolutional block. The cascaded convolutional blocks are used to process the input from the previous layer... The network performs feature extraction; the max pooling layer reduces the resolution of the feature map extracted by the convolutional block; the dilated convolutional structure expands the receptive field of the feature map output by the previous layer and obtains multi-scale semantic information without changing the resolution; the convolutional block connected to the dilated convolutional structure reduces the number of channels in the feature map output by the dilated convolutional structure, and the reduced number of channels is determined based on the number of abnormal target classifications; the nonlinear transformation structure converts the feature pixel values ​​of the feature map output by the feature extraction structure into mask pixel values ​​that match the pixel value range of the mask label image, thereby obtaining the network classification image; the nonlinear transformation structure is a network layer implemented based on a nonlinear function. The mask label images corresponding to the network classification image and the sample image are input into a preset loss function to obtain the loss result; The network parameters of the convolutional neural network are adjusted based on the loss result to train an anomaly classification model, which is then used for anomaly monitoring of escalators.

2. The method according to claim 1, characterized in that, The value of n is 3, and the value of m is 1; The first convolutional structure consists of two cascaded convolutional blocks, each with 3 input channels and 32 output channels, and a kernel size of 5×5. The second convolutional structure consists of three cascaded convolutional blocks, each with 32 input channels, 64 output channels, and a kernel size of 5×5. The third convolutional structure consists of four cascaded convolutional blocks, each with 64 input channels and 64 output channels, and a kernel size of 5×5. The dilated convolutional structure has 64 input channels, 1024 output channels, a kernel size of 3×3, and an expansion rate of 4. The convolution block connected to the dilated convolution structure has 1024 input channels and the number of output channels equal to the number of classifications of the abnormal targets. The convolution kernel size is 1×1.

3. The method according to claim 1, characterized in that, The abnormal targets include at least two types, and the acquisition of sample images of the escalator and at least one mask label image corresponding to the sample images includes: For each sample image, a labeling tool is used to annotate each abnormal target in the sample image to obtain a label file; The label file is parsed to set the pixel values ​​corresponding to the marked areas in the label file as the label pixel values, thereby obtaining the sample mask feature image corresponding to the sample image. Different abnormal targets correspond to different label pixel values. Based on the label pixel values, the sample mask feature image is divided into mask label images corresponding to various abnormal targets. The mask label image corresponding to each abnormal target indicates the area where the abnormal target is located through a first mask pixel value and indicates other areas besides the abnormal target through a second mask pixel value. Both the first mask pixel value and the second mask pixel value are located within the pixel value range of the mask label image.

4. The method according to claim 1, characterized in that, Before inputting the mask label images corresponding to the network classification image and the sample image into a preset loss function to obtain the loss result, the method further includes: The connected components of abnormal targets are determined in the mask label image; If the number of pixels in the identified connected components is greater than or equal to a threshold, the mask label image is determined to indicate the presence of the abnormal target. If the number of pixels in the identified connected component is less than a threshold, the mask label image is determined to indicate that the abnormal target is not present.

5. The method according to claim 1, characterized in that, The preset loss function includes the mean squared error loss function, and correspondingly, the loss result includes the mean squared error loss result output by the mean squared error loss function.

6. A method for monitoring anomalies in escalators, characterized in that, The method includes: Acquire the target image obtained by image acquisition of the escalator to be monitored; The target image is input into the anomaly classification model as described in any one of claims 1-5 to obtain at least one mask feature image; Determine whether the mask feature image contains any abnormal target connected components; If present, the target image includes an abnormal target and an alarm is triggered.

7. The method according to claim 6, characterized in that, The abnormal target includes a first abnormal target and a second abnormal target, the mask feature image includes a first mask feature image and a second mask feature image, the output layer of the abnormal target classification model includes a first output layer and a second output layer, and the first mask feature image output by the first output layer is used to indicate whether the first abnormal target is included; The second output layer outputs a second mask feature image to indicate whether the second anomalous target is included; The method further includes: If the first mask feature image indicates that the first anomalous target is included, it is determined that the target image includes the first anomalous target; If the second mask feature image indicates that the second anomalous target is included, then the target image is determined to include the second anomalous target.

8. An electronic device, characterized in that, The device includes a processor and a memory; the memory stores a program, which is loaded and executed by the processor to implement the training method for the anomaly classification model as described in any one of claims 1 to 5; or, to implement the anomaly monitoring method for escalators as described in claim 6 or 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement the training method for the anomaly classification model as described in any one of claims 1 to 5; or to implement the anomaly monitoring method for escalators as described in claim 6 or 7.

Images