Belt running state detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a belt running state detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring video frame sequence data containing a to-be-detected belt; inputting the video frame sequence data containing the to-be-detected belt into a pre-trained belt running state detection model to obtain a belt running state detection result; wherein the belt running state detection model comprises at least one convolution module, a convolution block attention module, at least one full connection layer and a classifier, and the convolution block attention module obtains a channel attention feature map and a space attention feature map based on the feature map of each video frame in the video frame sequence data. According to the technical scheme, the channel attention feature map and the space attention feature map are obtained through the convolution block attention module, so that the model pays more attention to the belt area, the attention to irrelevant background areas is reduced, and the accuracy of the belt running state detection result is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a belt running state detection method, device, electronic equipment and storage medium. Background Art

[0002] Belt conveyors are commonly used as material transportation tools and are widely used in coal mines.

[0003] An empty belt can easily lead to unnecessary belt wear and large amounts of electrical energy consumption, reducing the belt's service life while also wasting resources and increasing production costs.

[0004] In the process of implementing the present invention, it is found that there are at least the following technical problems in the prior art: the existing belt running status detection scheme has the problem of inaccurate detection results. Summary of the invention

[0005] The present invention provides a belt running state detection method, device, electronic equipment and storage medium to improve the accuracy of belt running state detection results.

[0006] According to one aspect of the present invention, a method for detecting a belt running state is provided, comprising:

[0007] Acquire video frame sequence data containing the belt to be detected;

[0008] Inputting the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain a belt running state detection result;

[0009] Among them, the belt running status detection model includes at least one convolution module, a convolution block attention module, at least one fully connected layer and a classifier, and the convolution block attention module obtains a channel attention feature map and a spatial attention feature map based on the feature map of each video frame in the video frame sequence data.

[0010] According to another aspect of the present invention, there is provided a belt running state detection device, comprising:

[0011] A video frame sequence data acquisition module, used to acquire video frame sequence data containing the belt to be detected;

[0012] A belt running state detection module, used for inputting the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain a belt running state detection result;

[0013] Among them, the belt running status detection model includes at least one convolution module, a convolution block attention module, at least one fully connected layer and a classifier, and the convolution block attention module obtains a channel attention feature map and a spatial attention feature map based on the feature map of each video frame in the video frame sequence data.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor;

[0016] and a memory communicatively coupled to the at least one processor;

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the belt running state detection method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the belt running state detection method described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention obtains the video frame sequence data containing the belt to be detected, and then inputs the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain the belt running state detection result; wherein the belt running state detection model includes at least one convolution module, a convolution block attention module, at least one fully connected layer and a classifier, and the convolution block attention module obtains the channel attention feature map and the spatial attention feature map based on the feature map of each video frame in the video frame sequence data. The above technical solution obtains the channel attention feature map and the spatial attention feature map through the convolution block attention module, so that the model pays more attention to the belt area and reduces the attention to irrelevant background areas, thereby improving the accuracy of the belt running state detection result.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a belt running state detection method provided according to Embodiment 1 of the present invention;

[0023] Figure 2 is a flow chart of a belt running state detection method provided according to Embodiment 2 of the present invention;

[0024] Figure 3 is a structural schematic diagram of a belt running state detection model provided according to an embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of the structure of a convolution module provided according to an embodiment of the present invention;

[0026] Figure 5 is a structural schematic diagram of a convolutional block attention module provided according to an embodiment of the present invention;

[0027] Figure 6 is a flow chart of a belt running status detection method provided according to Embodiment 3 of the present invention;

[0028] Figure 7 is a flow chart of a belt no-load operation alarm method provided according to an embodiment of the present invention;

[0029] Figure 8 is a structural schematic diagram of a belt running state detection device provided according to a fourth embodiment of the present invention;

[0030] Fig. 9 It is a structural schematic diagram of an electronic device for implementing the belt running state detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0033] Embodiment 1

[0034] Figure 1 This is a flow chart of a belt running state detection method provided in the first embodiment of the present invention. This embodiment is applicable to the case where the belt running state is automatically detected based on a video frame sequence. The method can be executed by a belt running state detection device. The belt running state detection device can be implemented in the form of hardware and / or software. The belt running state detection device can be configured in a terminal and / or a server. Figure 1 As shown, the method includes:

[0035] S110, obtaining video frame sequence data containing the belt to be detected.

[0036] The video frame sequence data may be understood as a collection of frame images in the video arranged in the order in which the video is played.

[0037] Exemplarily, the video frame sequence data containing the belt to be detected can be pre-stored locally in the electronic device, in other storage devices associated with the electronic device, or in the cloud, and the video frame sequence data containing the belt to be detected can be directly obtained from the corresponding storage location when needed.

[0038] S120, inputting the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain a belt running state detection result.

[0039] The belt running state detection result can be belt no-load running, belt loaded running, belt no-load shutdown or belt loaded shutdown. The belt running state detection model includes at least one convolution module, a convolution block attention module, at least one fully connected layer and a classifier. The convolution block attention module obtains a channel attention feature map and a spatial attention feature map based on the feature map of each video frame in the video frame sequence data.

[0040] In this embodiment, the channel attention feature map may include features of the belt area, and the spatial attention feature map may include location information of the belt and / or material, so that the model pays more attention to the belt area and reduces attention to irrelevant background areas, thereby improving the accuracy of the belt running status detection results.

[0041] In an optional implementation of an embodiment of the present application, before obtaining the video frame sequence data containing the belt to be detected, it also includes: obtaining frame sequence data of a sample video containing the belt to be detected, and labels of the frame sequence data of the sample video containing the belt to be detected, wherein the categories of labels include belt no-load operation, belt loaded operation, belt no-load shutdown and belt loaded shutdown; using the frame sequence data of the sample video containing the belt to be detected, and the labels of the frame sequence data of the sample video containing the belt to be detected as sample data to train the initially established neural network model to obtain a belt operation status detection model.

[0042] Exemplarily, the training data set may include multiple sample videos, each sample video includes a label category, each sample video has a duration of T seconds, and the frame rate of the sample video may be F frames, then the length of the sample video N = T*F. In order to further identify the running status of the belt, the sample video segment is divided into T / t segments, each segment contains F*t frames, the segments are uniformly sampled into 16 frames, and each 16-frame segment is used as the frame sequence data of the sample video and sent to the neural network model for training. By continuously adjusting the parameter values ​​of the model parameters, the distance deviation between the belt running status prediction result output by the model and the label is gradually reduced until the distance deviation tends to be stable, and a trained belt running status detection model is obtained.

[0043] It should be noted that the belt running state detection model of the embodiment of the present invention can regard the moving material (such as ore, etc.) as a moving target, and can make a significant distinction between the static background, the belt unloaded state when stopped, and the loaded state when stopped. Starting from the movement information of the material and using the characteristics of the material on the belt, the belt running state detection model can accurately distinguish whether the belt is stopped or running and can distinguish whether it is loaded or unloaded, thereby improving the accuracy of the belt running state detection results.

[0044] The technical solution of the embodiment of the present invention obtains the video frame sequence data containing the belt to be detected, and then inputs the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain the belt running state detection result; wherein the belt running state detection model includes at least one convolution module, a convolution block attention module, at least one fully connected layer and a classifier, and the convolution block attention module obtains the channel attention feature map and the spatial attention feature map based on the feature map of each video frame in the video frame sequence data. The above technical solution obtains the channel attention feature map and the spatial attention feature map through the convolution block attention module, so that the model pays more attention to the belt area and reduces the attention to irrelevant background areas, thereby improving the accuracy of the belt running state detection result.

[0045] Embodiment 2

[0046] Figure 2A flowchart of a belt running status detection method provided in the second embodiment of the present invention, the method of this embodiment can be combined with the various optional schemes in the belt running status detection method provided in the above embodiments. The belt running status detection method provided in this embodiment has been further optimized. Optionally, the convolution module includes a first convolution module, a second convolution module, a third convolution module and a fourth convolution module, the convolution block attention module includes a channel attention submodule and a spatial attention submodule, and the fully connected layer includes a first fully connected layer, a second fully connected layer and a third fully connected layer; accordingly, the video frame sequence data containing the belt to be detected is input into a pre-trained belt running status detection model to obtain a belt running status detection result, including: inputting the video frame sequence data containing the belt to be detected into the first convolution module to obtain a first feature map of each video frame in the video frame sequence data; the video frame sequence The first feature map of each video frame in the column data is input into the second convolution module to obtain the second feature map of each video frame in the video frame sequence data; the second feature map of each video frame in the video frame sequence data is input into the third convolution module to obtain the third feature map of each video frame in the video frame sequence data; the third feature map of each video frame in the video frame sequence data is input into the channel attention submodule to obtain the channel attention feature map of each video frame in the video frame sequence data; the channel attention feature map of each video frame in the video frame sequence data is multiplied by the third feature map of each video frame in the video frame sequence data to obtain the video The first dot product feature map of each video frame in the frame sequence data; the first dot product feature map of each video frame in the video frame sequence data is input into the spatial attention submodule to obtain the spatial attention feature map of each video frame in the video frame sequence data; the spatial attention feature map of each video frame in the video frame sequence data is dot-multiplied with the first dot product feature map of each video frame in the video frame sequence data to obtain the second dot product feature map of each video frame in the video frame sequence data; the second dot product feature map of each video frame in the video frame sequence data is input into the fourth convolution module to obtain the fourth feature map of each video frame in the video frame sequence data; the video frame The fourth feature map of each video frame in the sequence data is input into the first fully connected layer to obtain the first fully connected feature map of each video frame in the video frame sequence data; the first fully connected feature map of each video frame in the video frame sequence data is input into the second fully connected layer to obtain the second fully connected feature map of each video frame in the video frame sequence data; the second fully connected feature map of each video frame in the video frame sequence data is input into the third fully connected layer to obtain the third fully connected feature map of each video frame in the video frame sequence data; the third fully connected feature map of each video frame in the video frame sequence data is input into the classifier to obtain the belt running status detection result.

[0047] like Figure 2 As shown, the method includes:

[0048] S201, obtaining video frame sequence data containing a belt to be detected.

[0049] S202: Input the video frame sequence data containing the belt to be detected into the first convolution module to obtain a first feature map of each video frame in the video frame sequence data.

[0050] S203. Input the first feature map of each video frame in the video frame sequence data into the second convolution module to obtain the second feature map of each video frame in the video frame sequence data.

[0051] S204. Input the second feature map of each video frame in the video frame sequence data into the third convolution module to obtain a third feature map of each video frame in the video frame sequence data.

[0052] S205. Input the third feature map of each video frame in the video frame sequence data into the channel attention submodule to obtain the channel attention feature map of each video frame in the video frame sequence data.

[0053] S206. Multiply the channel attention feature map of each video frame in the video frame sequence data by the third feature map of each video frame in the video frame sequence data to obtain a first dot product feature map of each video frame in the video frame sequence data.

[0054] S207. Input the first dot product feature map of each video frame in the video frame sequence data into the spatial attention submodule to obtain the spatial attention feature map of each video frame in the video frame sequence data.

[0055] S208. Multiply the spatial attention feature map of each video frame in the video frame sequence data by the first dot product feature map of each video frame in the video frame sequence data to obtain the second dot product feature map of each video frame in the video frame sequence data.

[0056] S209. Input the second dot product feature map of each video frame in the video frame sequence data into the fourth convolution module to obtain a fourth feature map of each video frame in the video frame sequence data.

[0057] S210. Input the fourth feature map of each video frame in the video frame sequence data into the first fully connected layer to obtain a first fully connected feature map of each video frame in the video frame sequence data.

[0058] S211. Input the first fully connected feature map of each video frame in the video frame sequence data into the second fully connected layer to obtain the second fully connected feature map of each video frame in the video frame sequence data.

[0059] S212. Input the second fully connected feature map of each video frame in the video frame sequence data into the third fully connected layer to obtain a third fully connected feature map of each video frame in the video frame sequence data.

[0060] S213. Input the third fully connected feature map of each video frame in the video frame sequence data into the classifier to obtain a belt running state detection result.

[0061] For example, Figure 3 1 is a structural diagram of a belt running state detection model provided according to an embodiment of the present invention. Specifically, the belt running state detection model includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a convolution block attention module, a first fully connected layer, a second fully connected layer, a third fully connected layer and a classifier.

[0062] Figure 4 1 is a schematic diagram of a convolution module according to an embodiment of the present invention. Specifically, the convolution module may include a three-dimensional convolution unit (Conv3d), a rectified linear unit (relu) and a three-dimensional maximum pooling unit (maxpool3d).

[0063] Among them, the three-dimensional convolution unit can simultaneously capture the spatial information and temporal information of each video frame in the video frame sequence data. Exemplarily, the convolution kernel of the three-dimensional convolution unit can be 3×3×3, and the step size of the three-dimensional convolution unit can be 1. The number of channels output by the first convolution module, the second convolution module, the third convolution module, and the fourth convolution module are 16, 32, 64, and 128, respectively. The number of channels output by the first fully connected layer, the second fully connected layer, and the third fully connected layer are 2048, 1024, and 4, respectively.

[0064] Specifically, the first convolution module includes a first three-dimensional convolution unit, a first rectified linear unit, and a first three-dimensional maximum pooling unit, the convolution kernel size of the first three-dimensional maximum pooling unit is 1×2×2, and the step size of the first three-dimensional maximum pooling unit is 1×2×2. The second convolution module includes a second three-dimensional convolution unit, a second rectified linear unit, and a second three-dimensional maximum pooling unit, the convolution kernel size of the second three-dimensional maximum pooling unit is 2×2×2, and the step size of the second three-dimensional maximum pooling unit is 2×2×2; the third convolution module includes a third three-dimensional convolution unit, a third rectified linear unit, and a third three-dimensional maximum pooling unit, the convolution kernel size of the third three-dimensional maximum pooling unit is 2×2×2, and the step size of the third three-dimensional maximum pooling unit is 2×2×2; the fourth convolution module includes a fourth three-dimensional convolution unit, a fourth rectified linear unit, and a fourth three-dimensional maximum pooling unit, the convolution kernel size of the fourth three-dimensional maximum pooling unit is 2×2×2, and the step size of the fourth three-dimensional maximum pooling unit is 2×2×2.

[0065] It should be noted that the convolution kernel size of the first three-dimensional maximum pooling unit is set to 1×2×2, and the step size is set to 1×2×2 to adapt to the input of video frame sequence data.

[0066] Figure 5 It is a structural diagram of a convolutional block attention module provided according to an embodiment of the present invention. The convolutional block attention module includes a channel attention submodule and a spatial attention submodule, wherein the channel attention submodule can make the model more focused on the characteristics of the belt area, and the spatial attention submodule can make the model more focused on the position information of the belt and the material, so that the model pays more attention to the belt material area, thereby improving the accuracy of the belt running status detection result.

[0067] Specifically, the third feature map can be input into the channel attention submodule to obtain a channel attention feature map, and then the channel attention feature map is dot-multiplied with the third feature map to obtain a first dot-product feature map. Furthermore, the first dot-product feature map can be input into the spatial attention submodule to obtain a spatial attention feature map, and then the spatial attention feature map is dot-multiplied with the first dot-product feature map to obtain a second dot-product feature map.

[0068] In this embodiment of the present invention, the calculation formula of the first dot product feature map may be:

[0069]

[0070] Mc(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)));

[0071] Among them, F represents the third feature map, Mc(F) represents the channel attention feature map, denotes dot product, Fc denotes the first dot product feature map, AvgPool(·) denotes the average pooling layer, MaxPool(·) denotes the maximum pooling layer, MLP(·) denotes the multilayer perceptron, and σ(·) denotes the sigmoid activation function.

[0072] The calculation formula of the second point multiplication feature map can be:

[0073]

[0074] Ms(Fc)=σ(f 5*5 ([AvgPool(Fc);MaxPool(Fc)]);

[0075] Among them, Ms(Fc) represents the spatial attention feature map, Fsc represents the second point product feature map, and f 5*5It indicates a convolution with a convolution kernel of 5*5, and [AvgPool(Fc);MaxPool(Fc)] means concatenating AvgPool(Fc) and MaxPool(Fc) based on the channel dimension.

[0076] Optionally, the classifier may be a normalized classifier or other classifiers. Exemplarily, the normalized classifier may be a softmax classifier.

[0077] The technical solution of the embodiment of the present invention, by inputting the video frame into the first convolution module, the second convolution module, the third convolution module, the fourth convolution module, the convolution block attention module, the first fully connected layer, the second fully connected layer, the third fully connected layer and the classifier, makes the model pay more attention to the belt area and reduce the attention to irrelevant background areas, thereby improving the accuracy of the belt running status detection results.

[0078] Embodiment 3

[0079] Figure 6 A flowchart of a belt running state detection method provided in Embodiment 3 of the present invention, the method of this embodiment can be combined with each optional scheme in the belt running state detection method provided in the above embodiments. The belt running state detection method provided in this embodiment is further optimized. Optionally, after the video frame sequence data containing the belt to be detected is input into the pre-trained belt running state detection model to obtain the belt running state detection result, it also includes: if the belt running state detection result is that the belt is running without load, and the confidence score is greater than the first preset confidence threshold, then the belt no-load running alarm information is generated; if the belt running state detection result is that the belt is running without load, and the confidence score is less than the first preset confidence threshold and greater than the second preset confidence threshold, then the number of video frames with a confidence score greater than the second preset confidence threshold is counted; if the number of video frames with a confidence score greater than the second preset confidence threshold exceeds the preset number threshold, then the belt no-load running alarm information is generated.

[0080] like Figure 6 As shown, the method includes:

[0081] S310, obtaining video frame sequence data containing the belt to be detected.

[0082] S320, inputting the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain a belt running state detection result.

[0083] S330: If the belt running state detection result is that the belt is running without load, and the confidence score is greater than a first preset confidence threshold, a belt no-load running alarm message is generated.

[0084] S340. If the belt running status detection result is that the belt is running without load, and the confidence score is less than the first preset confidence threshold and greater than the second preset confidence threshold, then the number of video frames with a confidence score greater than the second preset confidence threshold is counted; if the number of video frames with the confidence score greater than the second preset confidence threshold exceeds the preset number threshold, then a belt no-load running alarm message is generated.

[0085] In the embodiment of the present invention, the belt no-load operation refers to the state where the belt is running without any load. The confidence score refers to an indicator for measuring the credibility of the belt running state detection result.

[0086] For example, Figure 7 It is a flowchart of a belt no-load running alarm method provided according to an embodiment of the present invention. Specifically, if the belt running state detection result corresponding to the current video frame in the video frame sequence data is that the belt is running without load, and the confidence score is greater than L1, then a belt no-load running alarm message is generated. If the belt running state detection result corresponding to the current video frame in the video frame sequence data is that the belt is running without load and the confidence score is not greater than L1, then it is determined whether the confidence score is greater than L2. If the confidence score is greater than L2, then the number of video frames with confidence scores greater than L2 in the video frame sequence data is counted; if the confidence score is not greater than L2, then the belt running state detection result corresponding to the next video frame is returned to continue the alarm judgment. If the number of video frames with confidence scores greater than L2 exceeds M, then a belt no-load running alarm message is generated; if the number of video frames with confidence scores greater than L2 does not exceed M, then the belt running state detection result corresponding to the next video frame is returned to continue the alarm judgment until the detection process ends. Among them, L1 represents the first preset confidence threshold, L2 represents the second preset confidence threshold, and M represents the preset number threshold. The specific values ​​of L1, L2 and M are not specifically limited here, and users can customize the settings based on experience or experimental results.

[0087] The technical solution of the embodiment of the present invention improves the accuracy of belt no-load running alarm detection by setting multi-level alarm rules.

[0088] Embodiment 4

[0089] Figure 8 This is a schematic diagram of the structure of a belt running state detection device provided by the fourth embodiment of the present invention. Figure 8 As shown, the device comprises:

[0090] The video frame sequence data acquisition module 410 is used to acquire the video frame sequence data containing the belt to be detected;

[0091] The belt running state detection module 420 is used to input the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain a belt running state detection result;

[0092] Among them, the belt running status detection model includes at least one convolution module, a convolution block attention module, at least one fully connected layer and a classifier, and the convolution block attention module obtains a channel attention feature map and a spatial attention feature map based on the feature map of each video frame in the video frame sequence data.

[0093] The technical solution of the embodiment of the present invention obtains the video frame sequence data containing the belt to be detected, and then inputs the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain the belt running state detection result; wherein the belt running state detection model includes at least one convolution module, a convolution block attention module, at least one fully connected layer and a classifier, and the convolution block attention module obtains the channel attention feature map and the spatial attention feature map based on the feature map of each video frame in the video frame sequence data. The above technical solution obtains the channel attention feature map and the spatial attention feature map through the convolution block attention module, so that the model pays more attention to the belt area and reduces the attention to irrelevant background areas, thereby improving the accuracy of the belt running state detection result.

[0094] In some optional embodiments, the convolution module includes a first convolution module, a second convolution module, a third convolution module and a fourth convolution module, the convolution block attention module includes a channel attention submodule and a spatial attention submodule, and the fully connected layer includes a first fully connected layer, a second fully connected layer and a third fully connected layer;

[0095] Accordingly, the belt running state detection module 420 is specifically used for:

[0096] Inputting the video frame sequence data containing the belt to be detected into the first convolution module to obtain a first feature map of each video frame in the video frame sequence data;

[0097] Inputting the first feature map of each video frame in the video frame sequence data into the second convolution module to obtain the second feature map of each video frame in the video frame sequence data;

[0098] Inputting the second feature map of each video frame in the video frame sequence data into the third convolution module to obtain a third feature map of each video frame in the video frame sequence data;

[0099] Inputting the third feature map of each video frame in the video frame sequence data into the channel attention submodule to obtain the channel attention feature map of each video frame in the video frame sequence data;

[0100] Perform a dot product of the channel attention feature map of each video frame in the video frame sequence data and the third feature map of each video frame in the video frame sequence data to obtain a first dot product feature map of each video frame in the video frame sequence data;

[0101] Inputting the first dot product feature map of each video frame in the video frame sequence data into the spatial attention submodule to obtain the spatial attention feature map of each video frame in the video frame sequence data;

[0102] Multiplying the spatial attention feature map of each video frame in the video frame sequence data by the first dot product feature map of each video frame in the video frame sequence data to obtain a second dot product feature map of each video frame in the video frame sequence data;

[0103] Inputting the second dot product feature map of each video frame in the video frame sequence data into the fourth convolution module to obtain a fourth feature map of each video frame in the video frame sequence data;

[0104] Inputting the fourth feature map of each video frame in the video frame sequence data into the first fully connected layer to obtain a first fully connected feature map of each video frame in the video frame sequence data;

[0105] Inputting the first fully connected feature map of each video frame in the video frame sequence data into the second fully connected layer to obtain a second fully connected feature map of each video frame in the video frame sequence data;

[0106] Inputting the second fully connected feature map of each video frame in the video frame sequence data into the third fully connected layer to obtain a third fully connected feature map of each video frame in the video frame sequence data;

[0107] The third fully connected feature map of each video frame in the video frame sequence data is input into the classifier to obtain the belt running state detection result.

[0108] In some optional embodiments, the first convolution module includes a first three-dimensional convolution unit, a first rectified linear unit and a first three-dimensional maximum pooling unit, the convolution kernel size of the first three-dimensional maximum pooling unit is 1×2×2, and the step size of the first three-dimensional maximum pooling unit is 1×2×2.

[0109] In some optional embodiments, the second convolution module includes a second three-dimensional convolution unit, a second rectified linear unit, and a second three-dimensional maximum pooling unit, the convolution kernel size of the second three-dimensional maximum pooling unit is 2×2×2, and the step size of the second three-dimensional maximum pooling unit is 2×2×2; the third convolution module includes a third three-dimensional convolution unit, a third rectified linear unit, and a third three-dimensional maximum pooling unit, the convolution kernel size of the third three-dimensional maximum pooling unit is 2×2×2, and the step size of the third three-dimensional maximum pooling unit is 2×2×2; the fourth convolution module includes a fourth three-dimensional convolution unit, a fourth rectified linear unit, and a fourth three-dimensional maximum pooling unit, the convolution kernel size of the fourth three-dimensional maximum pooling unit is 2×2×2, and the step size of the fourth three-dimensional maximum pooling unit is 2×2×2.

[0110] In some optional embodiments, the classifier is a normalized classifier.

[0111] In some optional implementations, the belt running state detection device further includes:

[0112] The no-load running alarm module is used to generate a belt no-load running alarm message if the belt running status detection result is that the belt is running no-load and the confidence score is greater than a first preset confidence threshold; if the belt running status detection result is that the belt is running no-load and the confidence score is less than the first preset confidence threshold and greater than the second preset confidence threshold, then count the number of video frames whose confidence scores are greater than the second preset confidence threshold; if the number of video frames whose confidence scores are greater than the second preset confidence threshold exceeds a preset number threshold, a belt no-load running alarm message is generated.

[0113] In some optional implementations, the belt running state detection device further includes:

[0114] A belt running state detection model training module, used to obtain frame sequence data of a sample video containing a belt to be detected, and labels of the frame sequence data of the sample video containing the belt to be detected, wherein the categories of labels include belt no-load operation, belt loaded operation, belt no-load shutdown and belt loaded shutdown;

[0115] The frame sequence data of the sample video containing the belt to be detected and the label of the frame sequence data of the sample video containing the belt to be detected are used as sample data to train the initially established neural network model to obtain a belt running state detection model.

[0116] The belt running state detection device provided in the embodiment of the present invention can execute the belt running state detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0117] Embodiment 5

[0118] Fig. 9 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0119] like Fig. 9 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The I / O interface 15 is also connected to the bus 14.

[0120] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0121] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a belt running state detection method, which includes:

[0122] Acquire video frame sequence data containing the belt to be detected;

[0123] Inputting the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain a belt running state detection result;

[0124] Among them, the belt running status detection model includes at least one convolution module, a convolution block attention module, at least one fully connected layer and a classifier, and the convolution block attention module obtains a channel attention feature map and a spatial attention feature map based on the feature map of each video frame in the video frame sequence data.

[0125] In some embodiments, the belt running state detection method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the belt running state detection method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the belt running state detection method in any other appropriate manner (for example, by means of firmware).

[0126] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0128] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0131] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0132] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0133] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A belt running status detection method, characterized in that: include: Acquire video frame sequence data containing the belt to be detected; Inputting the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain a belt running state detection result; Among them, the belt running status detection model includes at least one convolution module, a convolution block attention module, at least one fully connected layer and a classifier, and the convolution block attention module obtains a channel attention feature map and a spatial attention feature map based on the feature map of each video frame in the video frame sequence data.

2. The method according to claim 1, characterized in that The convolution module includes a first convolution module, a second convolution module, a third convolution module and a fourth convolution module, the convolution block attention module includes a channel attention submodule and a spatial attention submodule, and the fully connected layer includes a first fully connected layer, a second fully connected layer and a third fully connected layer; Correspondingly, the video frame sequence data containing the belt to be detected is input into a pre-trained belt running state detection model to obtain a belt running state detection result, including: Inputting the video frame sequence data containing the belt to be detected into the first convolution module to obtain a first feature map of each video frame in the video frame sequence data; Inputting the first feature map of each video frame in the video frame sequence data into the second convolution module to obtain the second feature map of each video frame in the video frame sequence data; Inputting the second feature map of each video frame in the video frame sequence data into the third convolution module to obtain a third feature map of each video frame in the video frame sequence data; Inputting the third feature map of each video frame in the video frame sequence data into the channel attention submodule to obtain the channel attention feature map of each video frame in the video frame sequence data; Perform a dot product of the channel attention feature map of each video frame in the video frame sequence data and the third feature map of each video frame in the video frame sequence data to obtain a first dot product feature map of each video frame in the video frame sequence data; Inputting the first dot product feature map of each video frame in the video frame sequence data into the spatial attention submodule to obtain the spatial attention feature map of each video frame in the video frame sequence data; Multiplying the spatial attention feature map of each video frame in the video frame sequence data by the first dot product feature map of each video frame in the video frame sequence data to obtain a second dot product feature map of each video frame in the video frame sequence data; Inputting the second dot product feature map of each video frame in the video frame sequence data into the fourth convolution module to obtain a fourth feature map of each video frame in the video frame sequence data; Inputting the fourth feature map of each video frame in the video frame sequence data into the first fully connected layer to obtain a first fully connected feature map of each video frame in the video frame sequence data; Inputting the first fully connected feature map of each video frame in the video frame sequence data into the second fully connected layer to obtain a second fully connected feature map of each video frame in the video frame sequence data; Inputting the second fully connected feature map of each video frame in the video frame sequence data into the third fully connected layer to obtain a third fully connected feature map of each video frame in the video frame sequence data; The third fully connected feature map of each video frame in the video frame sequence data is input into the classifier to obtain the belt running state detection result.

3. The method according to claim 2, characterized in that The first convolution module includes a first three-dimensional convolution unit, a first rectified linear unit and a first three-dimensional maximum pooling unit, the convolution kernel size of the first three-dimensional maximum pooling unit is 1×2×2, and the step size of the first three-dimensional maximum pooling unit is 1×2×2.

4. The method according to claim 2, characterized in that: The second convolution module includes a second three-dimensional convolution unit, a second rectified linear unit and a second three-dimensional maximum pooling unit, the convolution kernel size of the second three-dimensional maximum pooling unit is 2×2×2, and the step size of the second three-dimensional maximum pooling unit is 2×2×2; the third convolution module includes a third three-dimensional convolution unit, a third rectified linear unit and a third three-dimensional maximum pooling unit, the convolution kernel size of the third three-dimensional maximum pooling unit is 2×2×2, and the step size of the third three-dimensional maximum pooling unit is 2×2×2; the fourth convolution module includes a fourth three-dimensional convolution unit, a fourth rectified linear unit and a fourth three-dimensional maximum pooling unit, the convolution kernel size of the fourth three-dimensional maximum pooling unit is 2×2×2, and the step size of the fourth three-dimensional maximum pooling unit is 2×2×2.

5. The method according to claim 2, characterized in that: The classifier is a normalized classifier.

6. The method according to claim 1, characterized in that After inputting the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain the belt running state detection result, the method further includes: If the belt running state detection result is that the belt is running without load, and the confidence score is greater than a first preset confidence threshold, a belt no-load running alarm message is generated; If the belt running status detection result is that the belt is running without load, and the confidence score is less than the first preset confidence threshold and greater than the second preset confidence threshold, the number of video frames with a confidence score greater than the second preset confidence threshold is counted; if the number of video frames with a confidence score greater than the second preset confidence threshold exceeds the preset number threshold, a belt no-load running alarm message is generated.

7. The method according to any one of claims 1 to 6, characterized in that: Before obtaining the video frame sequence data containing the belt to be detected, it also includes: Acquire frame sequence data of a sample video containing a belt to be detected, and labels of the frame sequence data of the sample video containing the belt to be detected, wherein the categories of the labels include belt no-load operation, belt loaded operation, belt no-load stop, and belt loaded stop; The frame sequence data of the sample video containing the belt to be detected and the label of the frame sequence data of the sample video containing the belt to be detected are used as sample data to train the initially established neural network model to obtain a belt running state detection model.

8. A belt running status detection device, characterized in that: include: A video frame sequence data acquisition module, used to acquire video frame sequence data containing the belt to be detected; A belt running state detection module, used for inputting the video frame sequence data containing the belt to be detected into a pre-trained belt running state detection model to obtain a belt running state detection result; Among them, the belt running status detection model includes at least one convolution module, a convolution block attention module, at least one fully connected layer and a classifier, and the convolution block attention module obtains a channel attention feature map and a spatial attention feature map based on the feature map of each video frame in the video frame sequence data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the belt running state detection method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the belt running state detection method according to any one of claims 1 to 7 when executed.

Images