Escalator operation status identification method, device, electronic equipment and storage medium

By adding spatial and timing feature extraction modules to the 2D convolutional residual neural network, the problem of large calculations of 3D convolutional neural network is solved, and the escalator running state recognition with lower cost and higher robustness is achieved to meet real-time Identify requirements.

CN114758265BActive Publication Date: 2024-11-19SHENZHEN XUMI YUNTU SPACE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210219707.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-08
Publication Date
2024-11-19
Estimated Expiration
2042-03-08

AI Technical Summary

Technical Problem

In the prior art, 3D convolutional neural network is used to identify the operating state of the escalator, which has a large amount of calculation and high calculation cost, making it difficult to meet the needs of real-time identification.

Method used

Using the improved 2D convolutional residual neural network, the parameter quantity and calculation quantity are reduced by adding spatial feature extraction modules and/or timing feature extraction modules in the second, third and fifth convolutional layers, while maintaining the same as the traditional 3D volume. The accuracy of the neural network approximation.

Benefits of technology

It achieves lower computing costs and higher robustness, and can effectively identify the operating status of the escalator, replacing traditional 3D convolutional neural networks, and meeting the needs of real-time identification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114758265B_ABST
    Figure CN114758265B_ABST
Patent Text Reader

Abstract

The present disclosure relates to the field of image processing technology, and provides an escalator operation status recognition method, device, electronic device and storage medium. The method includes: obtaining a video image to be analyzed; inputting the video image into an improved residual neural network, and outputting the recognition result of the escalator operation status; the improved residual neural network is a 2D convolutional neural network, including the first, second, third, fourth, and fifth convolutional layers and a fully connected layer connected in sequence; the second convolutional layer includes a first residual module and a spatial feature extraction module; the third convolutional layer includes a second residual module, a spatial feature extraction module and a temporal feature extraction module; the fourth convolutional layer includes a third residual module; the fifth convolutional layer includes a fourth residual module and a temporal feature extraction module. The improved residual neural network provided by the present disclosure can achieve an accuracy similar to that of a traditional 3D convolutional neural network while reducing the amount of calculation, and has better robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an escalator operation status recognition method, device, electronic equipment and storage medium. Background Art

[0002] Since the recognition of escalator operation status has high real-time requirements, the current video-based escalator operation status recognition technology generally uses 3D convolutional neural networks. Although the use of 3D convolutional neural networks can learn more significant spatiotemporal features and achieve relatively high accuracy, this method has a large number of parameters, a large amount of calculations, and a high computational cost. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide an escalator operating status identification method, device, electronic device and storage medium to solve the problem of large amount of calculation and high cost in the prior art of using 3D convolutional neural network to identify the escalator operating status.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a method for identifying an escalator operation state, comprising:

[0005] Acquire a video image to be analyzed;

[0006] The video image is input into the improved residual neural network, and the recognition result of the escalator operation status is output;

[0007] The improved residual neural network is a 2D convolutional neural network, and the improved residual neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer and a fully connected layer connected in sequence;

[0008] The second convolutional layer includes a first residual module and a spatial feature extraction module;

[0009] The third convolutional layer includes a second residual module, a spatial feature extraction module and a temporal feature extraction module;

[0010] The fourth convolutional layer includes a third residual module;

[0011] The fifth convolutional layer includes a fourth residual module and a temporal feature extraction module.

[0012] According to a second aspect of the embodiments of the present disclosure, there is provided an escalator operation status identification device, comprising:

[0013] An image acquisition module, configured to acquire a video image to be analyzed;

[0014] An image recognition module is configured to input the video image into the improved residual neural network and output a recognition result of the escalator operation state;

[0015] The improved residual neural network is a 2D convolutional neural network, and the improved residual neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer and a fully connected layer connected in sequence;

[0016] The second convolutional layer includes a first residual module and a spatial feature extraction module;

[0017] The third convolutional layer includes a second residual module, a spatial feature extraction module and a temporal feature extraction module;

[0018] The fourth convolutional layer includes a third residual module;

[0019] The fifth convolutional layer includes a fourth residual module and a temporal feature extraction module.

[0020] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0021] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0022] Compared with the prior art, the beneficial effects of the disclosed embodiment at least include: the disclosed embodiment obtains a video image to be analyzed; inputs the video image into the improved residual neural network, and outputs the recognition result of the escalator operation status; wherein the improved residual neural network is a 2D convolutional neural network, and the improved residual neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer and a fully connected layer connected in sequence; the second convolutional layer includes a first residual module and a spatial feature extraction module; the third convolutional layer includes a second residual module, a spatial feature extraction module and a temporal feature extraction module; the fourth convolutional layer includes a third residual module; the fifth convolutional layer includes a fourth residual module and a temporal feature extraction module. The improved residual neural network adopted in the present invention is a 2D convolutional neural network. By adding spatial feature extraction modules and / or temporal feature extraction modules in the second, third and fifth convolutional layers, it can not only effectively reduce the number of parameters and the amount of calculation, but also achieve an accuracy similar to that of a traditional 3D convolutional neural network. It can replace the traditional 3D convolutional neural network for identifying the operating status of an escalator. At the same time, the improved residual neural network has better robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 It is a flow chart of a method for identifying the running state of an escalator provided in an embodiment of the present disclosure;

[0025] Figure 2 is a schematic diagram of the structure of an improved residual neural network in the escalator operation state recognition method provided by an embodiment of the present disclosure;

[0026] Figure 3 is a structural schematic diagram of a second convolutional layer in the escalator operation status recognition method provided by an embodiment of the present disclosure;

[0027] Figure 4 is a structural schematic diagram of a third convolutional layer in the escalator operation status recognition method provided by an embodiment of the present disclosure;

[0028] Figure 5 is a structural schematic diagram of a fifth convolutional layer in the escalator operation status recognition method provided by an embodiment of the present disclosure;

[0029] Figure 6 It is a flow chart of extracting a feature matrix using a spatial feature extraction module in the escalator operation state identification method provided by an embodiment of the present disclosure;

[0030] Figure 7 It is a flow chart of extracting a feature matrix using a time series feature extraction module in the escalator operation state identification method provided by an embodiment of the present disclosure;

[0031] Figure 8 It is a structural schematic diagram of an escalator operation status identification device provided by an embodiment of the present disclosure;

[0032] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0034] A method and device for identifying the running state of an escalator according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0035] Figure 1 FIG. 1 is a flow chart of a method for identifying the running status of an escalator provided by an embodiment of the present disclosure. Figure 1 As shown, the escalator operation status identification method includes:

[0036] Step S101, obtaining a video image to be analyzed.

[0037] In one embodiment, a video stream to be analyzed may be obtained first, and the video stream may be video data containing an escalator collected by a monitoring camera device (such as a camera). Then, the monitoring camera device transmits the video stream to a server (such as a background server or a cloud server, etc.). After receiving the video stream, the server may first perform frame processing on the video stream to obtain the video image to be analyzed. For example, if a video stream with a duration of 1 second is obtained, and the acquisition frequency of the video stream is 25 frames per second, then the 1-second video stream may be divided into 25 frames of video images. Similarly, if a video stream with a duration of 2 seconds is obtained, then the video stream may be divided into 50 frames of video images.

[0038] Step S102, input the video image into the improved residual neural network, and output the recognition result of the escalator operation status.

[0039] In one embodiment, the basic network architecture of the improved residual neural network disclosed in the present invention is similar to the basic network architecture of the Resnet50 residual network, both of which include five convolutional layers connected in sequence and one full connection. The difference between the two mainly lies in the specific network structure of the second, third and fifth convolutional layers.

[0040] Combination Figure 2 The improved residual neural network disclosed in the present invention is a 2D convolutional neural network, including a first convolutional layer 201, a second convolutional layer 202, a third convolutional layer 203, a fourth convolutional layer 204, a fifth convolutional layer 205 and a fully connected layer 206 connected in sequence. Among them, the second convolutional layer 202 includes a first residual module 2021 and a spatial feature extraction module 2022; the third convolutional layer 203 includes a second residual module 2031, a spatial feature extraction module 2032 and a temporal feature extraction module 2033; the fourth convolutional layer 204 includes a third residual module 2041; the fifth convolutional layer 205 includes a fourth residual module 2051 and a temporal feature extraction module 2052.

[0041] In one embodiment, the first residual module 2021, the second residual module 2031, the third residual module 2041 and the fourth residual module 2051 may be residual modules with the same structure. The spatial feature extraction module 2022 and the spatial feature extraction module 2032 may be modules with the same structure. The temporal feature extraction module 2033 and the temporal feature extraction module 2052 may be modules with the same structure.

[0042] Combination Figure 3 In one embodiment, the network structure of the second convolutional layer 202 may include a Conv2D (two-dimensional convolutional layer), a spatial feature extraction module (SGM), and a Conv2D (two-dimensional convolutional layer) connected in sequence.

[0043] Combination Figure 4 In one embodiment, the network structure of the third convolutional layer 203 may include a Conv2D (two-dimensional convolutional layer), a spatial feature extraction module (SGM, Spatial Gating Model), a temporal feature extraction module (TIAM) and a Conv2D (two-dimensional convolutional layer) connected in sequence.

[0044] Combination Figure 5 In one embodiment, the network structure of the fifth convolutional layer 205 may include a Conv2D (two-dimensional convolutional layer), a temporal feature extraction module (TIAM, temporal interaction-and-aggression module) and a Conv2D (two-dimensional convolutional layer) connected in sequence.

[0045] The technical solution provided by the embodiment of the present disclosure improves the existing Resnet50 residual network. Specifically, a spatial feature extraction module and / or a temporal feature extraction module are added to the second, third, and fifth convolutional layers of the existing residual network. This can not only effectively reduce the amount of parameters and the amount of calculation, but also achieve an accuracy similar to that of a traditional 3D convolutional neural network. It can replace the traditional 3D convolutional neural network for identifying the operating status of an escalator. At the same time, the improved residual neural network has better robustness.

[0046] In some embodiments, the above step S102 includes:

[0047] Use the first convolutional layer to perform feature extraction on the video image to obtain a first feature matrix;

[0048] Use the second convolutional layer to perform secondary feature extraction on the first feature matrix to obtain a second feature matrix;

[0049] Use the third convolutional layer to perform three feature extractions on the second feature matrix to obtain a third feature matrix;

[0050] Use the fourth convolutional layer to perform four feature extractions on the third feature matrix to obtain a fourth feature matrix;

[0051] Use the fifth convolutional layer to perform five feature extractions on the fourth feature matrix to obtain a fifth feature matrix;

[0052] The fifth feature matrix is ​​input into the fully connected layer, and the recognition result of the escalator operation status is output.

[0053] In one embodiment, assuming that a video stream with a duration of 1 second is acquired, 25 frames of video images can be obtained by performing frame processing on the video stream, and then the video images to be analyzed are extracted from the 25 frames of video images according to a preset frame extraction rule (for example, how many frames are used to extract a frame of video image). For example, the preset frame extraction rule is to extract a frame of video image to be analyzed every 3 frames, so 8 frames of video images to be analyzed can be extracted from the 25 frames of video images. Then, the 8 frames of video images to be analyzed are input into the improved residual neural network disclosed in the present invention, and the first convolution layer performs feature extraction on the 8 frames of video images to be analyzed to obtain a first feature matrix.

[0054] For example, assume that the dimensions of the 8 frames of video images to be analyzed are [512, 8, 224, 224], where 512, 8, 224, and 224 represent the number of channels (C), the number of extracted video frames (T), the height of the image (H), and the width of the image (W), respectively. After the video image [512, 8, 224, 224] is subjected to a feature extraction by the first convolutional layer, a first feature matrix with a dimension of [64, 8, 112, 112] ([C, T, H, W]) can be obtained, where 64, 8, 112, and 112 represent the number of channels, the number of extracted video frames, the height of the image, and the width of the image, respectively.

[0055] In one embodiment, a second feature extraction is performed on the first feature matrix using a second convolutional layer to obtain a second feature matrix, specifically including:

[0056] Using the first residual module to perform feature extraction on the first feature matrix to obtain an extracted feature matrix;

[0057] The spatial feature extraction module is used to perform spatial feature extraction processing on the extracted feature matrix to obtain a second feature matrix.

[0058] In combination with the above example, the first feature matrix with a dimension of [64, 8, 112, 112] ([C, T, H, W]) obtained after processing by the first convolutional layer is input into the first residual module for feature extraction, and an extracted feature matrix with a dimension of [256, 8, 56, 56] ([C, T, H, W]) can be obtained. Then, the spatial feature extraction module of the second convolutional layer is used to perform spatial feature extraction processing on the extracted feature matrix with a dimension of [256, 8, 56, 56] ([C, T, H, W]), and a second feature matrix with a dimension of [256, 8, 56, 56] ([C, T, H, W]) can be obtained.

[0059] In one embodiment, a spatial feature extraction module is used to perform spatial feature extraction processing on the extracted feature matrix to obtain a second feature matrix, including:

[0060] Performing matrix transformation on the extracted feature matrix to obtain a transformed feature matrix;

[0061] The transformed feature matrix is ​​averaged to obtain an average processing matrix;

[0062] Perform convolution operation and activation function operation on the average processing matrix to generate the escalator position weight matrix;

[0063] According to the escalator position weight matrix and the extracted feature matrix, a second feature matrix is ​​calculated.

[0064] Combination Figure 6 , combined with the above example, the extracted feature matrix with the dimension of [256,8,56,56]([C,T,H,W]) can be first subjected to matrix transformation (such as processing with the reshape function) in the module (Reshape, Pooling, squeeze) to obtain a transformed feature matrix with the dimension of [256,56,56,8]([C,H,W,T]). Among them, the reshape function is a function in MATLAB that transforms a specified matrix into a matrix of a specific dimension, and the number of elements in the matrix remains unchanged. The function can readjust the number of rows, columns, and dimensions of the matrix.

[0065] Next, the transformation feature matrix of the above dimension [256, 56, 56, 8] ([C, H, W, T]) can be input into the module (Norm, average) to perform average processing on the transformation feature matrix to obtain an average processing matrix. Specifically, in the time dimension, the transformation feature matrix is ​​averaged and pooled to obtain an average pooling matrix; in the dimension of the number of channels, the average pooling matrix is ​​feature averaged to obtain an average processing matrix.

[0066] Combined with the above example, in the time dimension, the transformation feature matrix of the dimension [256,56,56,8]([C,H,W,T]) is averaged and processed to obtain an average pooling matrix of the dimension [256,56,56]([C,H,W]). In the channel number dimension, the feature average processing is performed on the dimension [256,56,56]([C,H,W]) to obtain an average processing matrix of the dimension [56,56]([H,W]).

[0067] The technical solution provided by the embodiment of the present disclosure can reduce the interference of frame-level features and obtain higher accuracy by averaging the features.

[0068] Next, in the input module (Conv1D (one-dimensional convolution), ReLU (linear rectification function), Conv1D), the above average processing matrix of dimension [56,56]([H,W]) is subjected to one-dimensional convolution operation and activation function (ReLU, linear rectification function) operation, and the operation result is then subjected to sigmoid function operation to generate the escalator position weight matrix. Finally, the escalator position weight matrix is ​​multiplied by the above extracted feature matrix of dimension [256,8,56,56]([C,T,H,W]) to calculate the second feature matrix of dimension [256,8,56,56]([C,T,H,W]).

[0069] In some embodiments, the third convolutional layer is used to perform three feature extractions on the second feature matrix to obtain a third feature matrix, including:

[0070] Using a second residual module to perform feature extraction on the second feature matrix to obtain a first extracted feature matrix;

[0071] Using a spatial feature extraction module to perform feature extraction on the first extracted feature matrix to obtain a second extracted feature matrix;

[0072] The time series feature extraction module is used to perform feature extraction on the second extracted feature matrix to obtain a third feature matrix.

[0073] In one embodiment, in combination with the above example, the second feature matrix with the dimension of [256, 8, 56, 56] ([C, T, H, W]) is input into the third convolutional layer, and the second residual module is used to extract features from the second feature matrix with the dimension of [256, 8, 56, 56] ([C, T, H, W]) to obtain a first extracted feature matrix with the dimension of [512, 8, 28, 28] ([C, T, H, W]). Then, the spatial feature extraction module is used to extract features from the first extracted feature matrix with the dimension of [512, 8, 28, 28] ([C, T, H, W]) to obtain a second extracted feature matrix with the dimension of [512, 8, 28, 28] ([C, T, H, W]). Afterwards, the time series feature extraction module is used to perform feature extraction on the second extraction feature matrix with a dimension of [512, 8, 28, 28] ([C, T, H, W]) to obtain a third feature matrix with a dimension of [512, 8, 28, 28] ([C, T, H, W]).

[0074] It should be noted that the use of the spatial feature extraction module to perform feature extraction on the first extracted feature matrix to obtain the second extracted feature matrix is ​​similar to the specific calculation process of performing spatial feature extraction processing on the extracted feature matrix using the spatial feature extraction module to obtain the second feature matrix, which will not be repeated here.

[0075] In one embodiment, a time series feature extraction module is used to perform feature extraction on the second extracted feature matrix to obtain a third feature matrix, including:

[0076] Performing global spatial average pooling processing on the second extracted feature matrix to obtain an average pooling feature matrix;

[0077] The average pooling feature matrix is ​​split according to the time dimension to obtain a first split matrix, a second split matrix, a third split matrix and a fourth split matrix;

[0078] A third characteristic matrix is ​​calculated according to the first splitting matrix, the second splitting matrix, the third splitting matrix and the fourth splitting matrix.

[0079] Combination Figure 7, combined with the above example, the second extracted feature matrix input module (Pooling) with the dimension [512,8,28,28]([C,T,H,W]) is subjected to global spatial average pooling processing to obtain an average pooled feature matrix with the dimension [512,8]([C,T]). Subsequently, operations are performed based on the compressed one-dimensional time information (the temporal feature extraction module TIAM only learns the time dimension, and the spatial information is learned by 2D convolution), and the average pooled feature matrix is ​​split to obtain the first split matrix t1, the second split matrix t2, the third split matrix t3 and the fourth split matrix t4. Among them, the dimensions of the first split matrix t1, the second split matrix t2, the third split matrix t3 and the fourth split matrix t4 are all [C,T / 4] (that is, the dimension is [512,2]).

[0080] In one embodiment, the third characteristic matrix is ​​calculated according to the first splitting matrix, the second splitting matrix, the third splitting matrix and the fourth splitting matrix, including:

[0081] Performing a one-dimensional convolution operation and an activation function operation on the first split matrix to obtain a first time series feature matrix;

[0082] Performing a one-dimensional convolution operation and an activation function operation on the first time series feature matrix and the second split matrix to obtain a second time series feature matrix;

[0083] Performing a one-dimensional convolution operation and an activation function operation on the second time series feature matrix and the third split matrix to obtain a third time series feature matrix;

[0084] Performing a one-dimensional convolution operation and an activation function operation on the three time series feature matrices and the fourth split matrix to obtain a fourth time series feature matrix;

[0085] A third characteristic matrix is ​​calculated based on the first time series characteristic matrix, the second time series characteristic matrix, the third time series characteristic matrix and the fourth time series characteristic matrix.

[0086] In combination with the above example, a one-dimensional convolution operation (Conv1D) and an activation function operation (ReLU) are performed on the first split matrix of dimension [512,2]([C,T / 4]) to obtain a short-term first time series feature matrix T1 of dimension [512,2]([C,T / 4]). The first time series feature matrix T1 is added to the second split matrix t2, and the same one-dimensional convolution operation and activation function operation are performed to obtain a second time series feature matrix T2 of dimension [512,2]([C,T / 4]). The second time series feature matrix T2 combines the time series features of the first time series feature matrix T1 and the second split matrix t2 in two time periods. Similarly, through the above steps, a third time series feature matrix T3 of dimension [512,2]([C,T / 4]) and a fourth time series feature matrix T4 of dimension [512,2]([C,T / 4]) can be obtained.

[0087] In this embodiment, through one-dimensional convolution operation and activation function operation (Conv1D+ReLU), sufficient nonlinearity can be added under the premise of using as little computation as possible, so that the time series feature extraction module has a stronger time series feature fitting ability. In addition, T1, T2, T3, and T4 represent the features of different time series lengths respectively. This structural form represents the time series features in a finer granularity, has a stronger multi-scale extraction capability, and can obtain both short-term features and long-term features.

[0088] In some embodiments, the third characteristic matrix is ​​calculated based on the first timing characteristic matrix, the second timing characteristic matrix, the third timing characteristic matrix, and the fourth timing characteristic matrix, including:

[0089] Merging the first time series feature matrix, the second time series feature matrix, the third time series feature matrix and the fourth time series feature matrix to obtain a merged feature matrix;

[0090] Scaling the merged feature matrix to obtain a scaled feature matrix;

[0091] A third feature matrix is ​​calculated based on the scaled feature matrix and the second extracted features.

[0092] Next, the first time series feature matrix T1, the second time series feature matrix T2, the third time series feature matrix T3 and the fourth time series feature matrix T4 are merged to obtain a merged feature matrix T′ with a dimension of [512,8]([C,T]). Then, the merged feature matrix is ​​operated by the sigmod function, and then the dimension of the copy space is scaled from [512,8]([C,T]) of T′ to a scaled feature matrix T″ with a dimension of [512,8,28,28]([C,T,H,W]). Then, the scaled feature matrix T″ is element-wise multiplied with the second extracted feature matrix to obtain a third feature matrix with a dimension of [512,8,28,28]([C,T,H,W]).

[0093] In one embodiment, the third feature matrix of the dimension [512, 8, 28, 28] ([C, T, H, W]) is input into the fourth convolutional layer, and the third residual module performs four feature extractions to obtain a fourth feature matrix of the dimension [1024, 8, 14, 14] ([C, T, H, W]). After that, the fourth feature matrix is ​​input into the fifth convolutional layer, and the fourth residual module and the temporal feature extraction module of the fifth convolutional layer are processed to obtain a fifth feature matrix of the dimension 2048, 8, 7, 7]. Finally, the fifth feature matrix is ​​input into the fully connected layer, and the recognition result of the escalator operation state is output. The recognition result is the probability values ​​representing the two operation states of "movement" and "stationary", and the sum of the two probability values ​​is 1. According to the probability value, it can be determined whether the operation state of the escalator is moving or stationary. For example, if the output recognition result is a probability value of 0.8 representing "movement" and a probability value of 0.2 representing "stationary", then it can be determined that the operation state of the escalator is moving.

[0094] It should be noted that the specific process of inputting the above-mentioned fourth feature matrix into the fifth convolutional layer and processing it by the fourth residual module and the temporal feature extraction module of the fifth convolutional layer is similar to the processing process of the second residual module and the temporal feature extraction module in the processing process of the third convolutional layer in the previous text, and will not be repeated here.

[0095] The technical solution provided by the disclosed embodiment can enhance the network's ability to extract and aggregate spatial features by adding a spatial feature extraction module to the existing residual neural network; and, by adding a temporal feature extraction module, the network can have stronger temporal feature extraction and fitting capabilities, and finally, the network can fully integrate sensitive spatial information (such as the position information of the escalator) and finer temporal feature information in the video image to be analyzed, thereby improving the accuracy of identifying the escalator's operating status and making the network more robust.

[0096] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0097] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0098] Figure 8 Schematic diagram of an escalator operation status identification device provided by an embodiment of the present disclosure. Figure 8 As shown, the escalator operation status identification device includes:

[0099] An image acquisition module 801 is configured to acquire a video image to be analyzed;

[0100] The image recognition module 802 is configured to input the video image into the improved residual neural network and output the recognition result of the escalator operation state;

[0101] The improved residual neural network is a 2D convolutional neural network, and the improved residual neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer and a fully connected layer connected in sequence;

[0102] The second convolutional layer includes a first residual module and a spatial feature extraction module;

[0103] The third convolutional layer includes a second residual module, a spatial feature extraction module and a temporal feature extraction module;

[0104] The fourth convolutional layer includes a third residual module;

[0105] The fifth convolutional layer includes a fourth residual module and a temporal feature extraction module.

[0106] The technical solution provided by the embodiment of the present disclosure obtains the video image to be analyzed through the image acquisition module 801; the image recognition module 802 inputs the video image into the improved residual neural network, and outputs the recognition result of the escalator operation status; wherein the improved residual neural network is a 2D convolutional neural network, and the improved residual neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer and a fully connected layer connected in sequence; the second convolutional layer includes a first residual module and a spatial feature extraction module; the third convolutional layer includes a second residual module, a spatial feature extraction module and a temporal feature extraction module; the fourth convolutional layer includes a third residual module; the fifth convolutional layer includes a fourth residual module and a temporal feature extraction module. The improved residual neural network adopted in the present invention is a 2D convolutional neural network. By adding spatial feature extraction modules and / or temporal feature extraction modules in the second, third and fifth convolutional layers, it can not only effectively reduce the number of parameters and the amount of calculation, but also achieve an accuracy similar to that of a traditional 3D convolutional neural network. It can replace the traditional 3D convolutional neural network for identifying the operating status of an escalator. At the same time, the improved residual neural network has better robustness.

[0107] In some embodiments, the image recognition module 802 includes:

[0108] A first extraction unit is configured to perform a feature extraction on the video image using a first convolutional layer to obtain a first feature matrix;

[0109] A second extraction unit is configured to perform secondary feature extraction on the first feature matrix using a second convolutional layer to obtain a second feature matrix;

[0110] A third extraction unit is configured to use a third convolutional layer to perform three feature extractions on the second feature matrix to obtain a third feature matrix;

[0111] a fourth extraction unit, configured to perform feature extraction four times on the third feature matrix using a fourth convolutional layer to obtain a fourth feature matrix;

[0112] a fifth extraction unit, configured to use a fifth convolutional layer to perform five feature extractions on the fourth feature matrix to obtain a fifth feature matrix;

[0113] The recognition unit is configured to input the fifth feature matrix into the fully connected layer and output a recognition result of the escalator operation state.

[0114] In some embodiments, the second extraction unit may be specifically configured as follows:

[0115] Using the first residual module to perform feature extraction on the first feature matrix to obtain an extracted feature matrix;

[0116] The spatial feature extraction module is used to perform spatial feature extraction processing on the extracted feature matrix to obtain a second feature matrix.

[0117] In some embodiments, the spatial feature extraction module is used to perform spatial feature extraction processing on the extracted feature matrix to obtain a second feature matrix, including:

[0118] Performing matrix transformation on the extracted feature matrix to obtain a transformed feature matrix;

[0119] The transformed feature matrix is ​​averaged to obtain an average processing matrix;

[0120] Perform convolution operation and activation function operation on the average processing matrix to generate the escalator position weight matrix;

[0121] According to the escalator position weight matrix and the extracted feature matrix, a second feature matrix is ​​calculated.

[0122] In some embodiments, the above-mentioned averaging process of the transformed feature matrix to obtain the averaged processing matrix includes:

[0123] In the time dimension, the transformation feature matrix is ​​average pooled to obtain an average pooling matrix;

[0124] In the dimension of the number of channels, the average pooling matrix is ​​subjected to feature averaging to obtain an average processing matrix.

[0125] In some embodiments, the third extraction unit may be specifically configured as follows:

[0126] Using a second residual module to perform feature extraction on the second feature matrix to obtain a first extracted feature matrix;

[0127] Using a spatial feature extraction module to perform feature extraction on the first extracted feature matrix to obtain a second extracted feature matrix;

[0128] The time series feature extraction module is used to perform feature extraction on the second extracted feature matrix to obtain a third feature matrix.

[0129] In some embodiments, the above-mentioned use of the time series feature extraction module to extract features from the second extracted feature matrix to obtain a third feature matrix includes:

[0130] Performing global spatial average pooling processing on the second extracted feature matrix to obtain an average pooling feature matrix;

[0131] The average pooling feature matrix is ​​split according to the time dimension to obtain a first split matrix, a second split matrix, a third split matrix and a fourth split matrix;

[0132] A third characteristic matrix is ​​calculated according to the first splitting matrix, the second splitting matrix, the third splitting matrix and the fourth splitting matrix.

[0133] In some embodiments, the third characteristic matrix is ​​calculated based on the first split matrix, the second split matrix, the third split matrix and the fourth split matrix, including:

[0134] Performing a one-dimensional convolution operation and an activation function operation on the first split matrix to obtain a first time series feature matrix;

[0135] Performing a one-dimensional convolution operation and an activation function operation on the first time series feature matrix and the second split matrix to obtain a second time series feature matrix;

[0136] Performing a one-dimensional convolution operation and an activation function operation on the second time series feature matrix and the third split matrix to obtain a third time series feature matrix;

[0137] Performing a one-dimensional convolution operation and an activation function operation on the three time series feature matrices and the fourth split matrix to obtain a fourth time series feature matrix;

[0138] A third characteristic matrix is ​​calculated based on the first time series characteristic matrix, the second time series characteristic matrix, the third time series characteristic matrix and the fourth time series characteristic matrix.

[0139] In some embodiments, the third characteristic matrix is ​​calculated based on the first timing characteristic matrix, the second timing characteristic matrix, the third timing characteristic matrix and the fourth timing characteristic matrix, including:

[0140] Merging the first time series feature matrix, the second time series feature matrix, the third time series feature matrix and the fourth time series feature matrix to obtain a merged feature matrix;

[0141] Scaling the merged feature matrix to obtain a scaled feature matrix;

[0142] A third feature matrix is ​​calculated based on the scaled feature matrix and the second extracted features.

[0143] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0144] Fig. 9 is a schematic diagram of an electronic device 900 provided in an embodiment of the present disclosure. Fig. 9As shown, the electronic device 900 of this embodiment includes: a processor 901, a memory 902, and a computer program 903 stored in the memory 902 and executable on the processor 901. When the processor 901 executes the computer program 903, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 901 executes the computer program 903, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0145] Exemplarily, the computer program 903 may be divided into one or more modules / units, which are stored in the memory 902 and executed by the processor 901 to complete the present disclosure. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 903 in the electronic device 900.

[0146] The electronic device 900 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 900 may include, but is not limited to, a processor 901 and a memory 902. Those skilled in the art will appreciate that Fig. 9 It is only an example of the electronic device 900 and does not constitute a limitation of the electronic device 900. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0147] The processor 901 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0148] The memory 902 may be an internal storage unit of the electronic device 900, for example, a hard disk or memory of the electronic device 900. The memory 902 may also be an external storage device of the electronic device 900, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 900. Further, the memory 902 may also include both an internal storage unit of the electronic device 900 and an external storage device. The memory 902 is used to store computer programs and other programs and data required by the electronic device. The memory 902 may also be used to temporarily store data that has been output or is to be output.

[0149] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0150] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0151] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0152] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0156] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A method for identifying the running state of an escalator, characterized in that: include: Acquire a video image to be analyzed; Inputting the video image into the improved residual neural network and outputting the recognition result of the escalator operation state; The improved residual neural network is a 2D convolutional neural network, and the improved residual neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer and a fully connected layer connected in sequence; The second convolutional layer includes a first residual module and a spatial feature extraction module; The third convolutional layer includes a second residual module, a spatial feature extraction module and a temporal feature extraction module; The fourth convolutional layer includes a third residual module; The fifth convolutional layer includes a fourth residual module and a temporal feature extraction module; The step of inputting the video image into the improved residual neural network and outputting the recognition result of the escalator operation state comprises: Using the first convolutional layer to perform feature extraction on the video image to obtain a first feature matrix; Using the second convolutional layer to perform secondary feature extraction on the first feature matrix to obtain a second feature matrix; Using the third convolutional layer to perform three feature extractions on the second feature matrix to obtain a third feature matrix; Using the fourth convolutional layer to perform four feature extractions on the third feature matrix to obtain a fourth feature matrix; Using the fifth convolutional layer to perform five feature extractions on the fourth feature matrix to obtain a fifth feature matrix; Inputting the fifth feature matrix into the fully connected layer, and outputting the recognition result of the escalator operation state; The using the second convolutional layer to perform secondary feature extraction on the first feature matrix to obtain a second feature matrix includes: Using the first residual module to perform feature extraction on the first feature matrix to obtain an extracted feature matrix; Using the spatial feature extraction module to perform spatial feature extraction processing on the extracted feature matrix to obtain a second feature matrix; The step of using the third convolutional layer to perform three feature extractions on the second feature matrix to obtain a third feature matrix includes: Using a second residual module to perform feature extraction on the second feature matrix to obtain a first extracted feature matrix; Using the spatial feature extraction module to perform feature extraction on the first extracted feature matrix to obtain a second extracted feature matrix; The time series feature extraction module is used to perform feature extraction on the second extracted feature matrix to obtain a third feature matrix.

2. The method according to claim 1, characterized in that: The using the spatial feature extraction module to perform spatial feature extraction processing on the extracted feature matrix to obtain a second feature matrix includes: Performing matrix transformation on the extracted feature matrix to obtain a transformed feature matrix; Performing an averaging process on the transformation feature matrix to obtain an average processing matrix; Performing convolution operation and activation function operation on the average processing matrix to generate an escalator position weight matrix; A second feature matrix is ​​calculated based on the escalator position weight matrix and the extracted feature matrix.

3. The method according to claim 2, characterized in that The step of performing averaging processing on the transformation feature matrix to obtain an average processing matrix includes: In the time dimension, the transformation feature matrix is ​​average pooled to obtain an average pooling matrix; In the dimension of the number of channels, feature averaging processing is performed on the average pooling matrix to obtain an average processing matrix.

4. The method according to claim 1, characterized in that: The step of using the time series feature extraction module to extract features from the second extracted feature matrix to obtain a third feature matrix includes: Performing global spatial average pooling processing on the second extracted feature matrix to obtain an average pooling feature matrix; Splitting the average pooling feature matrix according to the time dimension to obtain a first split matrix, a second split matrix, a third split matrix and a fourth split matrix; A third characteristic matrix is ​​calculated based on the first splitting matrix, the second splitting matrix, the third splitting matrix and the fourth splitting matrix.

5. The method according to claim 4, characterized in that The step of calculating a third characteristic matrix according to the first split matrix, the second split matrix, the third split matrix and the fourth split matrix includes: Performing a one-dimensional convolution operation and an activation function operation on the first split matrix to obtain a first time series feature matrix; Performing a one-dimensional convolution operation and an activation function operation on the first time series feature matrix and the second split matrix to obtain a second time series feature matrix; Performing a one-dimensional convolution operation and an activation function operation on the second time series feature matrix and the third split matrix to obtain a third time series feature matrix; Performing a one-dimensional convolution operation and an activation function operation on the three time series feature matrices and the fourth split matrix to obtain a fourth time series feature matrix; A third characteristic matrix is ​​calculated based on the first timing characteristic matrix, the second timing characteristic matrix, the third timing characteristic matrix and the fourth timing characteristic matrix.

6. The method according to claim 5, characterized in that The step of calculating a third characteristic matrix according to the first timing characteristic matrix, the second timing characteristic matrix, the third timing characteristic matrix and the fourth timing characteristic matrix comprises: Merging the first time series feature matrix, the second time series feature matrix, the third time series feature matrix and the fourth time series feature matrix to obtain a merged feature matrix; Scaling the combined feature matrix to obtain a scaled feature matrix; A third feature matrix is ​​calculated based on the scaled feature matrix and the second extracted features.

7. An escalator operation status identification device, characterized in that: include: An image acquisition module, configured to acquire a video image to be analyzed; An image recognition module is configured to input the video image into the improved residual neural network and output a recognition result of the escalator operation state; The improved residual neural network is a 2D convolutional neural network, and the improved residual neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer and a fully connected layer connected in sequence; The second convolutional layer includes a first residual module and a spatial feature extraction module; The third convolutional layer includes a second residual module, a spatial feature extraction module and a temporal feature extraction module; The fourth convolutional layer includes a third residual module; The fifth convolutional layer includes a fourth residual module and a temporal feature extraction module; The step of inputting the video image into the improved residual neural network and outputting the recognition result of the escalator operation state comprises: Using the first convolutional layer to perform feature extraction on the video image to obtain a first feature matrix; Using the second convolutional layer to perform secondary feature extraction on the first feature matrix to obtain a second feature matrix; Using the third convolutional layer to perform three feature extractions on the second feature matrix to obtain a third feature matrix; Using the fourth convolutional layer to perform four feature extractions on the third feature matrix to obtain a fourth feature matrix; Using the fifth convolutional layer to perform five feature extractions on the fourth feature matrix to obtain a fifth feature matrix; Inputting the fifth feature matrix into the fully connected layer, and outputting the recognition result of the escalator operation state; The using the second convolutional layer to perform secondary feature extraction on the first feature matrix to obtain a second feature matrix includes: Using the first residual module to perform feature extraction on the first feature matrix to obtain an extracted feature matrix; Using the spatial feature extraction module to perform spatial feature extraction processing on the extracted feature matrix to obtain a second feature matrix; The step of using the third convolutional layer to perform three feature extractions on the second feature matrix to obtain a third feature matrix includes: Using a second residual module to perform feature extraction on the second feature matrix to obtain a first extracted feature matrix; Using the spatial feature extraction module to perform feature extraction on the first extracted feature matrix to obtain a second extracted feature matrix; The time series feature extraction module is used to perform feature extraction on the second extracted feature matrix to obtain a third feature matrix.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.