Conveying device anomaly detection method and device, electronic equipment and storage medium

Abstract

The application provides a conveying device anomaly detection method and device, electronic equipment and a computer readable storage medium. The conveying device anomaly detection method comprises: acquiring a preceding state image and a subsequent state image when a conveying device is in a working state; performing optical flow map prediction processing according to the preceding state image and the subsequent state image to obtain a first optical flow map of the conveying device; and performing detection processing according to the preceding state image, the subsequent state image and the first optical flow map to obtain anomaly information of the conveying device. The application can improve the timeliness and accuracy of conveying device anomaly detection.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a method, apparatus, electronic device, and computer-readable storage medium for detecting anomalies in a transmission device. Background Technology

[0002] Conveying devices such as belt conveyors can be used to transport various materials such as cartons and bags. Applying conveying devices to production lines and sorting lines can effectively improve work efficiency.

[0003] However, during the material conveying process, the conveying device is prone to jamming at inclines, declines, turns, and corners. This can cause minor damage to the material or, in severe cases, short circuits in the conveying device, leading to safety accidents.

[0004] In existing technologies, the main method is to detect whether the conveyor is operating normally through sensors built into the conveyor itself, and to force a shutdown if the conveyor malfunctions. However, this method can only detect abnormalities when the conveyor is severely stalled, which has a certain lag and can easily lead to safety hazards. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and computer-readable storage medium for detecting abnormalities in a transmission device, aiming to solve the problem that abnormalities in a transmission device can only be detected when the transmission device is severely stalled.

[0006] In a first aspect, this application provides a method for detecting anomalies in a transmission device, the method comprising:

[0007] Acquire the prior state image and the subsequent state image when the transmission device is in operation;

[0008] Based on the prior state image and the subsequent state image, optical flow map prediction processing is performed to obtain the first optical flow map of the transmission device.

[0009] Based on the prior state image, the subsequent state image, and the first optical flow map, detection processing is performed to obtain the abnormal information of the transmission device.

[0010] In one possible implementation of this application, the step of performing detection processing based on the prior state image, the subsequent state image, and the first optical flow map to obtain the abnormal information of the transmission device includes:

[0011] The prior state image, the subsequent state image, and the first optical flow map are overlaid to obtain a first fused image of the transmission device.

[0012] The trained anomaly detection network is invoked to perform feature extraction processing on the first fused image to obtain the first image feature information of the first fused image, and prediction processing is performed on the first image feature information to obtain the anomaly information of the transmission device, wherein the anomaly information includes the coordinate information and confidence level of the anomaly region of the transmission device.

[0013] In one possible implementation of this application, the anomaly information includes coordinate information of the anomaly region of the transmission device. The step of obtaining the anomaly information of the transmission device by performing detection processing based on the prior state image, the subsequent state image, and the first optical flow map further includes:

[0014] The state detection process is performed based on the coordinate information of the abnormal area to obtain the congestion level of the transmission device, wherein the congestion level is used to indicate whether the transmission device is in an abnormal state.

[0015] In one possible implementation of this application, the step of performing state detection processing based on the coordinate information of the abnormal region to obtain the congestion level of the transmission device includes:

[0016] The trained congestion classification network is invoked to perform state detection processing based on the coordinate information of the abnormal area to obtain the congestion level of the transmission device.

[0017] In one possible implementation of this application, the step of performing optical flow graph prediction processing based on the prior state image and the subsequent state image to obtain a first optical flow graph of the transmission device includes:

[0018] The prior state image and the subsequent state image are superimposed to obtain the second fused image of the transmission device;

[0019] Based on the second fused image, feature extraction processing is performed to obtain the second image feature information of the second fused image;

[0020] Based on the second image feature information, a prediction process is performed to obtain the first optical flow map of the transmission device.

[0021] In one possible implementation of this application, the step of performing feature extraction processing based on the second fused image to obtain the second image feature information of the second fused image includes:

[0022] The feature extraction subnetwork in the trained optical flow map prediction network is invoked, and feature extraction processing is performed based on the second fusion map to obtain the second image feature information of the second fusion map;

[0023] The step of performing prediction processing based on the second image feature information to obtain the first optical flow map of the transmission device includes:

[0024] The prediction subnetwork in the trained optical flow map prediction network is invoked to perform prediction processing based on the second image feature information, thereby obtaining the first optical flow map of the transmission device.

[0025] In one possible implementation of this application, the method further includes:

[0026] Acquire a training image, wherein the training image includes a first state image and a second state image when the transmission device is in operation, and the training image has been labeled with the actual optical flow map;

[0027] The first state image and the second state image are superimposed to obtain the training fusion image of the transmission device;

[0028] The feature extraction subnetwork in the preset optical flow map prediction network is invoked, and feature extraction processing is performed based on the training fusion map to obtain the feature information of the training fusion map;

[0029] The prediction subnetwork in the preset optical flow map prediction network is invoked, and prediction processing is performed based on the feature information of the training fusion map to obtain the second optical flow map of the transmission device.

[0030] Based on the second optical flow map and the actual optical flow map, the preset optical flow map prediction network is trained to obtain the trained optical flow map prediction network.

[0031] Secondly, this application provides a transmission device malfunction detection device, the transmission device malfunction detection device comprising:

[0032] The acquisition unit is used to acquire the prior state image and the subsequent state image when the transmission device is in working state.

[0033] The prediction unit is used to perform optical flow map prediction processing based on the prior state image and the subsequent state image to obtain the first optical flow map of the transmission device.

[0034] The detection unit is used to perform detection processing based on the prior state image, the subsequent state image, and the first optical flow map to obtain abnormal information of the transmission device.

[0035] In one possible implementation of this application, the detection unit is further configured to:

[0036] The prior state image, the subsequent state image, and the first optical flow map are overlaid to obtain a first fused image of the transmission device.

[0037] The trained anomaly detection network is invoked to perform feature extraction processing on the first fused image to obtain the first image feature information of the first fused image, and prediction processing is performed on the first image feature information to obtain the anomaly information of the transmission device, wherein the anomaly information includes the coordinate information and confidence level of the anomaly region of the transmission device.

[0038] In one possible implementation of this application, the anomaly information includes coordinate information of the anomaly region of the transmission device. After the step of obtaining the anomaly information of the transmission device by performing detection processing based on the prior state image, the subsequent state image, and the first optical flow map, the detection unit is further specifically used for:

[0039] The state detection process is performed based on the coordinate information of the abnormal area to obtain the congestion level of the transmission device, wherein the congestion level is used to indicate whether the transmission device is in an abnormal state.

[0040] In one possible implementation of this application, the detection unit is further configured to:

[0041] The trained congestion classification network is invoked to perform state detection processing based on the coordinate information of the abnormal area to obtain the congestion level of the transmission device.

[0042] In one possible implementation of this application, the prediction unit is further configured to:

[0043] The prior state image and the subsequent state image are superimposed to obtain the second fused image of the transmission device;

[0044] Based on the second fused image, feature extraction processing is performed to obtain the second image feature information of the second fused image;

[0045] Based on the second image feature information, a prediction process is performed to obtain the first optical flow map of the transmission device.

[0046] In one possible implementation of this application, the prediction unit is further configured to:

[0047] The feature extraction subnetwork in the trained optical flow map prediction network is invoked, and feature extraction processing is performed based on the second fusion map to obtain the second image feature information of the second fusion map;

[0048] The prediction subnetwork in the trained optical flow map prediction network is invoked to perform prediction processing based on the second image feature information, thereby obtaining the first optical flow map of the transmission device.

[0049] In one possible implementation of this application, the transmission device anomaly detection device further includes a training unit, which is specifically used for:

[0050] Acquire a training image, wherein the training image includes a first state image and a second state image when the transmission device is in operation, and the training image has been labeled with the actual optical flow map;

[0051] The first state image and the second state image are superimposed to obtain the training fusion image of the transmission device;

[0052] The feature extraction subnetwork in the preset optical flow map prediction network is invoked, and feature extraction processing is performed based on the training fusion map to obtain the feature information of the training fusion map;

[0053] The prediction subnetwork in the preset optical flow map prediction network is invoked, and prediction processing is performed based on the feature information of the training fusion map to obtain the second optical flow map of the transmission device.

[0054] Based on the second optical flow map and the actual optical flow map, the preset optical flow map prediction network is trained to obtain the trained optical flow map prediction network.

[0055] Thirdly, this application also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it executes the steps in any of the abnormal detection methods for transmission devices provided in this application.

[0056] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the steps in the transmission device anomaly detection method.

[0057] This application acquires a prior state image and a subsequent state image of a conveyor device when it is in operation; performs optical flow map prediction processing based on the prior state image and the subsequent state image to obtain a first optical flow map of the conveyor device; and performs detection processing based on the prior state image, the subsequent state image, and the first optical flow map to obtain abnormal information of the conveyor device. Firstly, leveraging the characteristic that optical flow maps can capture object motion information, this application uses reverse thinking, employing optical flow information (i.e., the first optical flow map) to mark moving objects, and detects abnormal conveyor devices (such as congested conveyor belts) by detecting non-moving cargo conveyor devices (such as conveyor belts). Secondly, by fusing the state image and optical flow map of the conveyor device instead of simple image prediction, the expressive power of features is enhanced, improving the accuracy of conveyor device abnormality detection. Thirdly, since detection is based on images and does not rely on sensor sensing, it can promptly detect conveyor devices in abnormal states, avoiding the lag problem of existing sensor detection schemes. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of a scenario for the transmission device anomaly detection system provided in an embodiment of this application;

[0060] Figure 2 This is a flowchart illustrating a method for detecting anomalies in a transmission device provided in an embodiment of this application.

[0061] Figure 3 This is a schematic diagram illustrating an application scenario of the transmission device anomaly detection method provided in the embodiments of this application;

[0062] Figure 4 This is a schematic flowchart of an embodiment of step 20 provided in this application;

[0063] Figure 5 This is a schematic flowchart of an embodiment of step 30 provided in this application;

[0064] Figure 6 This is a schematic diagram of an embodiment of the transmission device anomaly detection device provided in this application.

[0065] Figure 7 This is a schematic diagram of an embodiment of the electronic device provided in this application. Detailed Implementation

[0066] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0068] To enable any person skilled in the art to implement and use this application, the following description is provided. In this description, details are set forth for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known processes will not be described in detail to avoid obscuring the description of the embodiments of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0069] This application provides a method, apparatus, electronic device, and computer-readable storage medium for detecting anomalies in a transmission device. The anomaly detection apparatus can be integrated into an electronic device, which may be a server or a terminal, etc.

[0070] First, before introducing the embodiments of this application, let's first introduce the relevant content of the application background of the embodiments of this application.

[0071] Conveying devices (such as belt conveyors, or simply belt conveyors) are widely used in various scenarios because they can transport various materials such as cartons and bags. Belt conveyors provide smooth transport, with no relative movement between the material and the conveyor belt, thus avoiding damage to the transported items. Belt conveyors are characterized by their simple structure, ease of maintenance, low energy consumption, and low operating costs. Therefore, belt conveyors are widely used in logistics transfer stations. During the transport of goods, belt conveyors are prone to jamming at inclines, declines, turns, and corners. This can lead to minor damage such as crushing and damaging the goods, or even serious accidents such as short circuits in the conveyor motor.

[0072] In existing technology, belt conveyors are equipped with sensors to detect whether the belt conveyor is working properly. If the belt conveyor is malfunctioning, it will be forced to stop. However, this method can only detect abnormalities in the event of severe stall, which has a certain lag and poses a safety hazard.

[0073] Based on the aforementioned deficiencies in existing related technologies, this application provides a method for detecting abnormalities in a transmission device, which at least to some extent overcomes the deficiencies in existing related technologies.

[0074] The execution subject of the transmission device anomaly detection method in this application embodiment can be the transmission device anomaly detection device provided in this application embodiment, or different types of electronic devices such as server equipment, physical host, or user equipment (UE) that integrate the transmission device anomaly detection device. The transmission device anomaly detection device can be implemented in hardware or software. The UE can specifically be a terminal device such as a smartphone, tablet computer, laptop computer, handheld computer, desktop computer, or personal digital assistant (PDA).

[0075] The electronic device can operate independently or in a cluster. By applying the abnormal detection method for the transmission device provided in this application, the accuracy of abnormal detection of the transmission device can be improved. It does not rely on sensor sensing, so it can detect the transmission device in an abnormal state in a timely manner, avoiding the lag problem of existing sensor detection schemes.

[0076] The abnormal detection method for the transmission device provided in this application embodiment can be implemented using computer vision technology.

[0077] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0078] In this embodiment, the so-called conveyor device anomaly detection refers to extracting image features from an image containing a conveyor device, and identifying whether the conveyor device in the image is malfunctioning based on these features (e.g., extracting features from an image of a sorting area containing a belt conveyor, and identifying whether the belt conveyor in the image is congested; or extracting features from an image of a production area containing a belt conveyor, and identifying whether the belt conveyor in the image is damaged). The results of this conveyor device anomaly detection have positive implications for certain application scenarios, such as improving production efficiency and safety in industrial production, and improving sorting efficiency and safety in express mail sorting.

[0079] See Figure 1 , Figure 1 This is a schematic diagram of a scenario for a transmission device anomaly detection system provided in an embodiment of this application. The system may include an electronic device 100, which integrates a transmission device anomaly detection device. For example, the electronic device can acquire a prior state image and a subsequent state image of the transmission device when it is in operation; perform optical flow map prediction processing based on the prior state image and the subsequent state image to obtain a first optical flow map of the transmission device; and perform detection processing based on the prior state image, the subsequent state image, and the first optical flow map to obtain anomaly information of the transmission device.

[0080] In addition, such as Figure 1 As shown, the abnormal detection system of the conveying device may also include a memory 200 for storing data, such as image data and video data, for example, image data and video data captured by cameras deployed in a specific scene area (such as a sorting area) over a period of time.

[0081] It should be noted that, Figure 1 The schematic diagram of the transmission device anomaly detection system shown is merely an example. The transmission device anomaly detection system and scenario described in this application embodiment are for the purpose of more clearly illustrating the technical solutions of this application embodiment and do not constitute a limitation on the technical solutions provided in this application embodiment. As those skilled in the art will know, with the evolution of transmission device anomaly detection systems and the emergence of new business scenarios, the technical solutions provided in the embodiments of this invention are also applicable to similar technical problems.

[0082] The following describes the abnormal detection method for the transmission device provided in the embodiments of this application. In the embodiments of this application, an electronic device is used as the execution subject. For the sake of simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments.

[0083] Reference Figure 2 , Figure 2 This is a flowchart illustrating a method for detecting anomalies in a transmission device according to an embodiment of this application. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here. The method for detecting anomalies in a transmission device includes steps S10 to S30, wherein:

[0084] S10. Acquire the prior state image and the subsequent state image when the transmission device is in operation.

[0085] Among them, the conveying device is a device used for bulk transport of items, such as a belt conveyor, a rubber conveyor, etc. The method of this application embodiment is applicable to the anomaly detection of conveying devices in various application scenarios; for example, a rubber conveyor used to transport production materials in a production line. Another example is a belt conveyor used to transport items for security checks in a subway security checkpoint. Yet another example is a belt conveyor used to transport logistics packages in the sorting area of ​​a logistics transfer station. Figure 3 As shown, Figure 3 This is a schematic diagram illustrating an application scenario of the abnormal detection method for the transmission device provided in the application embodiments.

[0086] The preceding state image refers to the state image of the transmission device when it is in operation. The following state image refers to the state image of the transmission device when it is in operation. The difference between the preceding and following state images is that the preceding state image is the state image of the transmission device at the first moment, while the following state image is the state image of the transmission device at the second moment; the second moment lags behind the first moment.

[0087] Specifically, in practical applications, the electronic device using the abnormal detection method for the conveying device provided in the embodiments of this application may directly include a camera in the scene area where the conveying device is located (such as a subway security check area or a sorting area in a logistics transfer station). This camera is mainly used to capture images containing the conveying device, and the images captured by the camera are stored locally and can be directly read inside the electronic device. Alternatively, the electronic device may establish a network connection with the camera and obtain the images captured by the camera online according to the network connection. Alternatively, the electronic device may also read the images captured by the camera from a relevant storage medium that stores the images captured by the camera. The specific acquisition method is not limited here.

[0088] The camera can capture images according to preset shooting methods, such as setting the shooting height, shooting direction, or shooting distance. The specific shooting method can be adjusted according to the camera itself and is not limited here. Multiple frames captured by the camera can be combined into a video through a timeline.

[0089] For ease of understanding, the following example will be used to check whether the belt conveyor (hereinafter referred to as belt conveyor) used to transport sorted goods in the sorting area of ​​the logistics transfer center is abnormal.

[0090] S20. Perform optical flow map prediction processing based on the prior state image and the subsequent state image to obtain the first optical flow map of the transmission device.

[0091] The first optical flow diagram refers to the optical flow diagram of the transmission device obtained through prediction.

[0092] Please refer to Figure 4 , Figure 4 This is a schematic flowchart of one embodiment of step 20 provided in this application. In some embodiments of this application, step S20 may specifically include the following steps S21 to S23, wherein:

[0093] S21. The prior state image and the subsequent state image are superimposed to obtain the second fused image of the transmission device.

[0094] The second fusion image refers to the fusion image obtained by overlaying the prior state image and the subsequent state image.

[0095] For example, the first state image [R1,G1,B1] and the second state image [R2,G2,B2] are superimposed to form a 6-channel fused image [R1,G1,B1,R2,G2,B2].

[0096] S22. Perform feature extraction processing based on the second fused image to obtain the second image feature information of the second fused image.

[0097] The second image feature information refers to the image feature information obtained after feature extraction processing of the second fused image.

[0098] Specifically, in some embodiments, a deep-learned optical flow graph prediction network can be used to predict the first optical flow graph of the transmission device; wherein, the optical flow graph prediction network (unless otherwise specified, the optical flow graph prediction network referred to herein as the trained optical flow graph prediction network) can be trained through the following steps A1 to A5.

[0099] Optical flow map prediction networks can include feature extraction subnetworks and prediction subnetworks.

[0100] The feature extraction subnetwork is used to output the second image feature information of the second fused image based on the second fused image. The feature extraction subnetwork takes the second fused image as input and performs convolution operations on the second fused image to extract features from the second fused image and obtain the second image feature information.

[0101] The prediction subnetwork is used to output a first optical flow map of the transmission device based on the second image feature information. The prediction subnetwork takes the second image feature information as input, performs optical flow map prediction processing based on the second image feature information, and determines the first optical flow map of the transmission device.

[0102] At this point, step S22 can be implemented by the feature extraction subnetwork in the optical flow map prediction network. Specifically, step S22 may include: calling the feature extraction subnetwork in the trained optical flow map prediction network, performing feature extraction processing based on the second fusion map, and obtaining the second image feature information of the second fusion map.

[0103] For example, the second fused image of the belt conveyor is input into the optical flow map prediction network to call the feature extraction subnetwork in the optical flow map prediction network, and perform convolution operation based on the second fused image to extract the second image feature information of the second fused image.

[0104] S23. Perform prediction processing based on the second image feature information to obtain the first optical flow map of the transmission device.

[0105] Correspondingly, step S23 can be implemented by the prediction sub-network in the optical flow map prediction network. Specifically, step S23 may include: calling the prediction sub-network in the trained optical flow map prediction network, performing prediction processing based on the second image feature information, and obtaining the first optical flow map of the transmission device.

[0106] To facilitate understanding, let's continue with the example from step S22 above. For instance, the feature extraction subnetwork in the optical flow map prediction network extracts features from the second fused map to obtain the second image feature information of the second fused map. Then, it inputs the second image feature information into the prediction subnetwork of the optical flow map prediction network to call the prediction subnetwork in the optical flow map prediction network. Based on the second image feature information, it performs optical flow map prediction processing to determine the optical flow map of the belt conveyor, thereby obtaining the first optical flow map of the belt conveyor.

[0107] As can be seen from the above, since the optical flow map prediction network is trained using deep learning, the trained network can fully learn the feature relationships between two consecutive images of the transmission device over a period of time and the optical flow map of the transmission device, thus ensuring the prediction accuracy of the first optical flow map. Because the optical flow map can reflect the motion of an object to a certain extent, an accurate first optical flow map can more precisely reflect the motion of items on the transmission device, providing precise data for subsequent determination of whether the transmission device is malfunctioning.

[0108] S30. Based on the prior state image, the subsequent state image, and the first optical flow map, detection processing is performed to obtain the abnormal information of the transmission device.

[0109] Among them, abnormal information is relevant information used to characterize whether the transmission device is experiencing congestion or other abnormal conditions. Specifically, it may include one or more of the following: an indication of whether the transmission device is in an abnormal state; the identification number of the transmission device in an abnormal state; the coordinate information and confidence level of the abnormal area of ​​the transmission device; and the acquisition time of the corresponding subsequent state image.

[0110] In some embodiments of this application, step S30 may specifically include: overlaying the prior state image, the subsequent state image, and the first optical flow map to obtain a first fused image of the transmission device; performing feature extraction processing based on the first fused image to obtain first image feature information of the first fused image; and performing prediction processing based on the first image feature information to obtain abnormal information of the transmission device.

[0111] Specifically, anomaly information of the transmission device can be obtained by performing detection processing based on the prior state image, the subsequent state image, and the first optical flow map after deep learning. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic flowchart of one embodiment of step 30 provided in this application. Specifically, in some embodiments of this application, step S30 may include steps S31 to S32, wherein:

[0112] S31. The prior state image, the subsequent state image, and the first optical flow map are superimposed to obtain the first fused image of the transmission device.

[0113] The first fusion map refers to the fusion map obtained by superimposing the prior state image, the subsequent state image, and the first optical flow map.

[0114] For example, the first state image [R1,G1,B1], the second state image [R2,G2,B2] and the first optical flow map [R3,G3,B3] are superimposed to form a 9-channel fusion map [R1,G1,B1,R2,G2,B2,R3,G3,B3].

[0115] S32. The trained anomaly detection network is invoked to perform feature extraction processing based on the first fusion image to obtain the first image feature information of the first fusion image, and prediction processing is performed based on the first image feature information to obtain the anomaly information of the transmission device.

[0116] The abnormal information includes the coordinates and confidence level of the abnormal area of ​​the transmission device.

[0117] The first image feature information refers to the image feature information obtained after feature extraction processing of the first fused image.

[0118] For example, firstly, a pre-defined anomaly detection network is trained based on a training dataset (including fused images of multiple conveyor belts). This allows the network to learn the feature relationships between the fused image of the conveyor belt (pre-state image, post-state image, and optical flow image) and abnormal states (such as congestion). The trained anomaly detection network can then detect whether congestion occurs on the conveyor belt based on the fused image. The pre-defined anomaly detection network can include, but is not limited to, convolutional neural networks, such as YOLOv3 or SSD networks. Specifically, an open-source detection network with default model parameters can be used as the pre-defined anomaly detection network.

[0119] As can be seen from the above, since the anomaly detection network is trained through deep learning, it can fully learn the characteristic relationship between the fusion graph of the transmission device and the abnormal state (such as whether it is congested). Through the trained anomaly detection network, it can be guaranteed that the abnormal state of the transmission device can be accurately detected.

[0120] Then, the first fused image is input into the trained anomaly detection network. The network performs feature extraction based on the first fused image to obtain its first image feature information. Based on this first image feature information, prediction processing is performed to obtain the anomaly information of the transmission device. For example, the network can ultimately determine and output: the transmission device is in an abnormal state, its identification number, the coordinates and confidence level of the abnormal area, and the acquisition time of the corresponding subsequent state image. This allows relevant management personnel to promptly address any abnormalities in the transmission device, preventing serious losses or dangers.

[0121] As can be seen from the above, by acquiring the prior and subsequent state images of the conveyor device when it is in operation; performing optical flow map prediction processing based on the prior and subsequent state images to obtain the first optical flow map of the conveyor device; and performing detection processing based on the prior, subsequent, and first optical flow maps to obtain the abnormal information of the conveyor device. Firstly, leveraging the characteristic that optical flow maps can capture object motion information, by using reverse thinking, optical flow information (i.e., the first optical flow map) is used to mark moving objects, and by detecting non-moving cargo conveyor devices (such as belt conveyors), abnormal conveyor devices (such as congested belt conveyors) can be detected. Secondly, by fusing the state image and optical flow map of the conveyor device instead of simple image prediction, the expressive power of features is enhanced, improving the accuracy of conveyor device abnormality detection. Thirdly, since detection is based on images and does not rely on sensor sensing, it can promptly detect conveyor devices in abnormal states, avoiding the lag problem of existing sensor detection schemes.

[0122] In step S32, the anomaly detection network can only initially detect two states of the transmission device: either it is in an abnormal state or it is not in an abnormal state.

[0123] To further improve the accuracy of anomaly detection in the transmission device, in some embodiments of this application, the congestion level of the anomaly area is further detected based on the coordinate information of the anomaly area of ​​the transmission device determined in step S30. That is, after step S30 or step S32, the anomaly detection method for the transmission device further includes: performing state detection processing based on the coordinate information of the anomaly area to obtain the congestion level of the transmission device.

[0124] The abnormal information determined in step S30 or step S32 includes the coordinate information of the abnormal area of ​​the transmission device.

[0125] The congestion level is used to indicate whether the conveyor is in an abnormal state. For example, the congestion level can be divided into five categories: belt conveyor empty, belt conveyor carrying goods, belt conveyor slightly congested, belt conveyor moderately congested, and belt conveyor heavily congested. A congestion level of slightly congested, moderately congested, or heavily congested indicates that the conveyor is in an abnormal state. A congestion level of empty or conveyor carrying goods indicates that the conveyor is not in an abnormal state.

[0126] Specifically, a congestion classification network trained by deep learning can be used to perform state detection processing based on the coordinate information of abnormal areas to obtain the congestion level of the transmission device. That is, the step of "performing state detection processing based on the coordinate information of the abnormal areas to obtain the congestion level of the transmission device" can specifically include: calling the trained congestion classification network to perform state detection processing based on the coordinate information of the abnormal areas to obtain the congestion level of the transmission device.

[0127] For example, firstly, a pre-defined congestion classification network is trained based on a training dataset (including multiple images of locations containing congestion at the transmission device). This allows the network to learn the features of the transmission device at different congestion levels, resulting in a trained congestion classification network. At this point, the network can detect the congestion level of a specific area containing the transmission device from an image. The pre-defined congestion classification network can be, but is not limited to, a convolutional neural network, such as ResNet or VGG. Specifically, an open-source classification network with default model parameters can be used as the pre-defined congestion classification network.

[0128] During the training process of the pre-defined congestion classification network, it is necessary to determine the model parameters of the pre-defined congestion classification network based on the congestion level predicted by the network according to the coordinate information of the abnormal area and the actual labeled congestion level, and update the network according to the total loss value of the network. The total loss value of the pre-defined congestion classification network can be determined by the following formula (1).

[0129]

[0130] Where L represents the total loss value of the preset congestion classification network, M represents the number of congestion categories, and y ic p represents the indicator variable (0 or 1, 1 if sample i is predicted to be in the same class as sample c, 0 otherwise). ic This represents the predicted probability that observed sample i belongs to category c.

[0131] Then, the coordinate information of the abnormal area is input into the trained congestion classification network, so that the trained congestion classification network can be invoked to perform feature extraction processing based on the coordinate information of the abnormal area to obtain the feature information of the abnormal area; and classification regression processing is performed based on the feature information of the abnormal area to determine the congestion level of the transmission device.

[0132] Let's illustrate this with a specific example. Considering the potential for false detections in the detection network (the anomaly detection network trained in step S32 predicts the anomaly information of the conveyor), to further improve the accuracy of the solution, the congested areas in the fused image are cropped using the coordinates of the congested conveyor belt (i.e., the coordinate information of the anomaly areas detected in step S32), and the size of the cropped image is adjusted. The cropped and adjusted image is then fed into the trained ResNet classification network (i.e., the congestion level classification network), finally outputting five categories: empty conveyor belt, conveyor belt transporting goods, slightly congested conveyor belt, moderately congested conveyor belt, and heavily congested conveyor belt.

[0133] As can be seen from the above, by further analyzing the coordinate information of the abnormal area of ​​the transmission device to detect the degree of congestion in the abnormal area, the accuracy of transmission device anomaly detection can be improved. This avoids the problem of high false positive rates caused by only being able to preliminarily detect two states of the transmission device—either abnormal or not—based on the image.

[0134] In some embodiments of this application, the optical flow map prediction network is trained through the following steps A1 to A5, wherein:

[0135] A1. Obtain the image to be trained.

[0136] The images to be trained have been labeled with actual optical flow maps.

[0137] The training images include a first-state image and a second-state image when the transmission device is in operation. The first-state image is the image used for training when the transmission device is in operation. The second-state image is the image used for training when the transmission device is in operation. The acquisition time of the first-state image lags behind the acquisition time of the second-state image.

[0138] The transmission device mentioned in the network model training phase and the transmission device mentioned in the optical flow graph prediction network practical application phase can be the same type of different transmission device. For example, belt conveyor 1 is used in the network model training phase and is used to predict the optical flow graph of the belt conveyor in the optical flow graph prediction network practical application phase. In order to improve the prediction accuracy of the optical flow graph prediction network, the same transmission device is used in the network model training phase and the transmission device anomaly detection phase in this embodiment.

[0139] For example, images of two consecutive states of a conveyor belt within a given time period can be collected as training images (i.e., the first state image and the second state image, respectively). Furthermore, the actual operating speed of the conveyor belt can be considered to set a specific time interval for the data collection, allowing the images to fully reflect any anomalies such as congestion. For instance, a 3-second time interval could be set when the conveyor belt operates at a relatively high speed, while a 6-second time interval could be set when the actual operating speed is relatively low.

[0140] A2. Overlay the first state image and the second state image to obtain the training fusion image of the transmission device.

[0141] The training fusion image refers to the fusion image obtained by superimposing the first state image and the second state image.

[0142] For example, the first state image [R1,G1,B1] and the second state image [R2,G2,B2] are superimposed to form a 6-channel fused image [R1,G1,B1,R2,G2,B2].

[0143] A3. Call the feature extraction subnetwork in the preset optical flow map prediction network, and perform feature extraction processing based on the training fusion map to obtain the feature information of the training fusion map.

[0144] The preset optical flow graph prediction network includes, but is not limited to, convolutional neural networks, such as FlowNetSimple, FlowNetCorr, and FlowNet2.0.

[0145] In some embodiments of this application, the preset optical flow map prediction network includes a feature extraction subnetwork and a prediction subnetwork. The functions of the feature extraction subnetwork and the prediction subnetwork, as well as the network's input and output, can be referred to the explanation in step S22 above, and will not be repeated here.

[0146] For example, the training fusion map is input into a preset optical flow map prediction network to call the feature extraction subnetwork in the preset optical flow map prediction network, and convolution operation is performed on the training fusion map to extract the feature information of the training fusion map.

[0147] A4. Call the prediction sub-network in the preset optical flow map prediction network, and perform prediction processing based on the feature information of the training fusion map to obtain the second optical flow map of the transmission device.

[0148] The second optical flow graph refers to the optical flow graph of the transmission device predicted by a preset optical flow graph prediction network during model training.

[0149] To facilitate understanding, let's continue with the example from step A3 above. For instance, the feature extraction subnetwork in the preset optical flow map prediction network extracts features from the training fusion map to obtain its feature information. This feature information is then input into the prediction subnetwork of the preset optical flow map prediction network. The prediction subnetwork then performs optical flow map prediction processing based on the feature information of the training fusion map to determine the optical flow map of the transmission device, thereby obtaining the second optical flow map of the transmission device.

[0150] A5. Based on the second optical flow map and the actual optical flow map, train the preset optical flow map prediction network to obtain the trained optical flow map prediction network.

[0151] Specifically, firstly, based on the second optical flow map and the actual optical flow map, the training loss value of the preset optical flow map prediction network is determined. Then, based on the training loss value, the model parameters of the preset optical flow map prediction network are updated until the preset optical flow map prediction network converges, resulting in the trained optical flow map prediction network. At this point, the optical flow map prediction network can predict the optical flow map of the transmission device based on two consecutive images of the transmission device within a certain time period.

[0152] For example, the training loss value of the preset optical flow graph prediction network can be determined by the following formula (2).

[0153]

[0154] Where L1 represents the training loss value of the preset optical flow graph prediction network, (u, v) represents the second optical flow graph, (u g v g ) represents the actual optical flow diagram.

[0155] As can be seen from the above, the model is iteratively trained by using the image to be trained as the input feature and the second optical flow map of the corresponding transmission device as the desired output feature, until the iteration stops, resulting in a trained optical flow map prediction network. This allows the trained optical flow map prediction network to be directly called subsequently to predict the optical flow map of the transmission device based on two consecutive images of the transmission device within a certain time period. Since the trained optical flow map prediction network is obtained through deep learning, it can fully learn the feature information of the transmission device, thus ensuring the accuracy of the optical flow map prediction.

[0156] To better implement the transmission device anomaly detection method in the embodiments of this application, based on the transmission device anomaly detection method, the embodiments of this application also provide a transmission device anomaly detection device, such as... Figure 6The diagram shown is a structural schematic of one embodiment of the transmission device anomaly detection device 600 in this application. The transmission device anomaly detection device 600 includes:

[0157] The acquisition unit 601 is used to acquire the prior state image and the subsequent state image when the transmission device is in the working state.

[0158] Prediction unit 602 is used to perform optical flow map prediction processing based on the prior state image and the subsequent state image to obtain the first optical flow map of the transmission device;

[0159] The detection unit 603 is used to perform detection processing based on the prior state image, the subsequent state image and the first optical flow map to obtain abnormal information of the transmission device.

[0160] In one possible implementation of this application, the detection unit 603 is further configured to:

[0161] The prior state image, the subsequent state image, and the first optical flow map are overlaid to obtain a first fused image of the transmission device.

[0162] The trained anomaly detection network is invoked to perform feature extraction processing on the first fused image to obtain the first image feature information of the first fused image, and prediction processing is performed on the first image feature information to obtain the anomaly information of the transmission device, wherein the anomaly information includes the coordinate information and confidence level of the anomaly region of the transmission device.

[0163] In one possible implementation of this application, the anomaly information includes the coordinate information of the anomaly region of the transmission device. After the step of obtaining the anomaly information of the transmission device by performing detection processing based on the prior state image, the subsequent state image, and the first optical flow map, the detection unit 603 is further specifically used for:

[0164] The state detection process is performed based on the coordinate information of the abnormal area to obtain the congestion level of the transmission device, wherein the congestion level is used to indicate whether the transmission device is in an abnormal state.

[0165] In one possible implementation of this application, the detection unit 603 is further configured to:

[0166] The trained congestion classification network is invoked to perform state detection processing based on the coordinate information of the abnormal area to obtain the congestion level of the transmission device.

[0167] In one possible implementation of this application, the prediction unit 602 is further configured to:

[0168] The prior state image and the subsequent state image are superimposed to obtain the second fused image of the transmission device;

[0169] Based on the second fused image, feature extraction processing is performed to obtain the second image feature information of the second fused image;

[0170] Based on the second image feature information, a prediction process is performed to obtain the first optical flow map of the transmission device.

[0171] In one possible implementation of this application, the prediction unit 602 is further configured to:

[0172] The feature extraction subnetwork in the trained optical flow map prediction network is invoked, and feature extraction processing is performed based on the second fusion map to obtain the second image feature information of the second fusion map;

[0173] The prediction subnetwork in the trained optical flow map prediction network is invoked to perform prediction processing based on the second image feature information, thereby obtaining the first optical flow map of the transmission device.

[0174] In one possible implementation of this application, the transmission device anomaly detection device further includes a training unit (not shown in the figure), the training unit being specifically used for:

[0175] Acquire a training image, wherein the training image includes a first state image and a second state image when the transmission device is in operation, and the training image has been labeled with the actual optical flow map;

[0176] The first state image and the second state image are superimposed to obtain the training fusion image of the transmission device;

[0177] The feature extraction subnetwork in the preset optical flow map prediction network is invoked, and feature extraction processing is performed based on the training fusion map to obtain the feature information of the training fusion map;

[0178] The prediction subnetwork in the preset optical flow map prediction network is invoked, and prediction processing is performed based on the feature information of the training fusion map to obtain the second optical flow map of the transmission device.

[0179] Based on the second optical flow map and the actual optical flow map, the preset optical flow map prediction network is trained to obtain the trained optical flow map prediction network.

[0180] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0181] Because the abnormality detection device of the transmission device can perform the functions described in this application, Figures 1 to 5 Corresponding to the steps in the abnormal detection method of the transmission device in any embodiment, the present application can be implemented as described above. Figures 1 to 5 The beneficial effects that the transmission device anomaly detection method can achieve in any embodiment are detailed in the preceding description and will not be repeated here.

[0182] Furthermore, to better implement the transmission device anomaly detection method in the embodiments of this application, based on the transmission device anomaly detection method, the embodiments of this application also provide an electronic device, see below. Figure 7 , Figure 7 This illustration shows a structural diagram of an electronic device according to an embodiment of this application. Specifically, the electronic device provided in this embodiment includes a processor 701, which executes a computer program stored in a memory 702 to implement, for example... Figures 1 to 5 Corresponding to each step of the abnormal detection method for the transmission device in any embodiment; or, when the processor 701 executes the computer program stored in the memory 702, it implements as follows: Figure 7 The functions of each unit in the corresponding embodiment.

[0183] For example, a computer program can be divided into one or more modules / units, one or more of which are stored in memory 702 and executed by processor 701 to complete the embodiments of this application. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device.

[0184] The electronic device may include, but is not limited to, processor 701 and memory 702. Those skilled in the art will understand that the illustrations are merely examples of an electronic device and do not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc., and processor 701, memory 702, input / output devices, and network access devices are connected via a bus.

[0185] The processor 701 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.

[0186] The memory 702 can be used to store computer programs and / or modules. The processor 701 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 702 and by calling data stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, video data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0187] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described transmission device anomaly detection device, electronic equipment, and their corresponding units can be found in, for example... Figures 1 to 5 The description of the abnormal detection method for the transmission device in any embodiment will not be repeated here.

[0188] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0189] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the present application. Figures 1 to 5 For the steps in the abnormal detection method of the transmission device in any embodiment, the specific operation can be referred to as follows: Figures 1 to 5 The description of the abnormal detection method of the transmission device in any embodiment will not be repeated here.

[0190] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0191] Because of the instructions stored in the computer-readable storage medium, the present application can be executed as described above. Figures 1 to 5 Corresponding to the steps in the abnormal detection method of the transmission device in any embodiment, the present application can be implemented as described above. Figures 1 to 5 The beneficial effects that the transmission device anomaly detection method can achieve in any embodiment are detailed in the preceding description and will not be repeated here.

[0192] The present application provides a detailed description of a method, apparatus, electronic device, and computer-readable storage medium for detecting anomalies in a transmission device. Specific examples have been used to illustrate the principles and implementation methods of the present application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present application. Therefore, the content of this specification should not be construed as a limitation of the present application.

Claims

1. A method for detecting malfunctions in a conveying device, characterized in that, The method includes: Acquire the prior state image and the subsequent state image when the transmission device is in operation; Optical flow map prediction processing is performed based on the prior state image and the subsequent state image to obtain a first optical flow map of the conveying device; the first optical flow map can reflect the movement of the items on the conveying device. Based on the prior state image, the subsequent state image, and the first optical flow map, detection processing is performed to obtain the abnormal information of the transmission device; the abnormal information includes the coordinate information of the abnormal area of ​​the transmission device. The state detection process is performed based on the coordinate information of the abnormal area to obtain the congestion level of the transmission device, wherein the congestion level is used to indicate whether the transmission device is in an abnormal state.

2. The method for detecting abnormalities in a conveying device according to claim 1, characterized in that, The step of detecting and processing the anomaly information of the transmission device based on the prior state image, the subsequent state image, and the first optical flow map includes: The prior state image, the subsequent state image, and the first optical flow map are overlaid to obtain a first fused image of the transmission device. The trained anomaly detection network is invoked to perform feature extraction processing on the first fused image to obtain the first image feature information of the first fused image, and prediction processing is performed on the first image feature information to obtain the anomaly information of the transmission device, wherein the anomaly information also includes the confidence level of the anomaly region of the transmission device.

3. The method for detecting abnormalities in a conveying device according to claim 1, characterized in that, The step of performing state detection processing based on the coordinate information of the abnormal area to obtain the congestion level of the transmission device includes: The trained congestion classification network is invoked to perform state detection processing based on the coordinate information of the abnormal area to obtain the congestion level of the transmission device.

4. The method for detecting abnormalities in a conveying device according to claim 1, characterized in that, The step of performing optical flow map prediction processing based on the prior state image and the subsequent state image to obtain the first optical flow map of the transmission device includes: The prior state image and the subsequent state image are superimposed to obtain the second fused image of the transmission device; Based on the second fused image, feature extraction processing is performed to obtain the second image feature information of the second fused image; Based on the second image feature information, a prediction process is performed to obtain the first optical flow map of the transmission device.

5. The method for detecting abnormalities in a conveying device according to claim 4, characterized in that, The step of performing feature extraction processing based on the second fused image to obtain the second image feature information of the second fused image includes: The feature extraction subnetwork in the trained optical flow map prediction network is invoked, and feature extraction processing is performed based on the second fusion map to obtain the second image feature information of the second fusion map; The step of performing prediction processing based on the second image feature information to obtain the first optical flow map of the transmission device includes: The prediction subnetwork in the trained optical flow map prediction network is invoked to perform prediction processing based on the second image feature information, thereby obtaining the first optical flow map of the transmission device.

6. The method for detecting abnormalities in a conveying device according to claim 5, characterized in that, The method further includes: Acquire a training image, wherein the training image includes a first state image and a second state image when the transmission device is in operation, and the training image has been labeled with the actual optical flow map; The first state image and the second state image are superimposed to obtain the training fusion image of the transmission device; The feature extraction subnetwork in the preset optical flow map prediction network is invoked, and feature extraction processing is performed based on the training fusion map to obtain the feature information of the training fusion map; The prediction subnetwork in the preset optical flow map prediction network is invoked, and prediction processing is performed based on the feature information of the training fusion map to obtain the second optical flow map of the transmission device. Based on the second optical flow map and the actual optical flow map, the preset optical flow map prediction network is trained to obtain the trained optical flow map prediction network.

7. A fault detection device for a conveying device, characterized in that, The abnormality detection device for the transmission device includes: The acquisition unit is used to acquire the prior state image and the subsequent state image when the transmission device is in working state. The prediction unit is used to perform optical flow map prediction processing based on the prior state image and the subsequent state image to obtain a first optical flow map of the conveying device; the first optical flow map can reflect the movement of the items on the conveying device. The detection unit is configured to perform detection processing based on the prior state image, the subsequent state image, and the first optical flow map to obtain abnormal information of the transmission device; and to perform state detection processing based on the coordinate information of the abnormal region to obtain the congestion level of the transmission device, wherein the congestion level is used to indicate whether the transmission device is in an abnormal state; the abnormal information includes the coordinate information of the abnormal region of the transmission device.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the transmission device anomaly detection method as described in any one of claims 1 to 6 when it invokes the computer program in the memory.

9. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to execute the steps in the transmission device anomaly detection method according to any one of claims 1 to 6.

Images