Transmission belt state identification method and device, electronic equipment and storage medium

Abstract

The invention provides a transmission band state identification method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the target detection of continuous M frames of images, obtaining a transmission band common detection frame corresponding to the continuous M frames of images, and M is a natural number greater than or equal to 2; fusing the continuous M frames of images to obtain a first fused image; according to the transmission band common detection frame, obtaining a target transmission band area map from the first fusion image; and state prediction is carried out on the target transmission band area map, and the target state of the transmission band is determined. According to the method, continuous multi-frame images are fused, the fused images are judged, judgment can be carried out in the two dimensions of time and space, the accuracy of determining the target state of the conveying belt is improved, compared with a judgment method in the prior art, other auxiliary equipment is not needed for judgment, only an image collection device is needed, and the judgment efficiency is improved. The method can reduce the judgment cost, and is not liable to be affected by the outside.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as image processing, image recognition, and deep learning, and more particularly to a method, device, electronic device, and storage medium for identifying the state of a conveyor belt. Background Art

[0002] The belt operation identification system monitors and analyzes the status and behavior of the belt transmission system, identifies the operation of the belt, detects anomalies and faults, and ensures the normal operation of the system. In industrial production, logistics and transportation, the belt transmission system is an important component, and any anomaly will lead to reduced production efficiency or safety accidents.

[0003] In current technology, belt operation recognition is usually used in industrial production lines, mines, ports, logistics centers and other places where the belt operation status needs to be monitored in real time. Methods based on traditional image algorithms, such as background modeling, optical flow or feature point matching, are easily affected by external interference, resulting in unsatisfactory recognition effects and safety hazards. Summary of the invention

[0004] The present disclosure provides a method, device, electronic device and storage medium for identifying the status of a transmission belt.

[0005] According to a first aspect of the present disclosure, a method for identifying a transmission belt status is provided, comprising: performing target detection on M consecutive frame images to obtain a transmission belt common detection frame corresponding to the M consecutive frame images, where M is a natural number greater than or equal to 2; fusing the M consecutive frame images to obtain a first fused image; obtaining a target transmission belt area map from the first fused image based on the transmission belt common detection frame; and performing state prediction on the target transmission belt area map to determine a target state of the transmission belt.

[0006] According to a second aspect of the present disclosure, a device for identifying the state of a conveyor belt is provided, comprising: a detection module, used to perform target detection on M consecutive frame images, and obtain a conveyor belt common detection frame corresponding to the M consecutive frame images, wherein M is a natural number greater than or equal to 2; a fusion module, used to fuse the M consecutive frame images to obtain a first fused image; an acquisition module, used to obtain a target conveyor belt area map from the first fused image according to the conveyor belt common detection frame; and a prediction module, used to perform state prediction on the target conveyor belt area map to determine the target state of the conveyor belt.

[0007] According to the third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for identifying the status of a transmission belt described in the above-mentioned first aspect embodiment.

[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, on which a computer program / instructions are stored, and the computer instructions are used to enable the computer to execute the method for identifying the status of a transmission belt described in the above-mentioned embodiment.

[0009] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the method for identifying the state of a transmission belt as described in the above-mentioned first embodiment.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

[0011] By fusing multiple consecutive frames of images and making judgments on the fused images, judgments can be made in both time and space dimensions, thereby improving the accuracy of determining the target state of the conveyor belt. Compared with the judgment methods in current technologies, no other auxiliary equipment is required for judgment, only an image acquisition device is required, which can reduce the judgment cost and is not easily affected by the outside world. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 A schematic diagram of a flow chart of a method for identifying a conveyor belt status provided by an embodiment of the present disclosure;

[0014] Figure 2 A schematic flow chart of another method for identifying a conveyor belt state provided by an embodiment of the present disclosure;

[0015] Figure 3 A schematic flow chart of another method for identifying a conveyor belt state provided by an embodiment of the present disclosure;

[0016] Figure 4 A schematic flow chart of another method for identifying a conveyor belt state provided by an embodiment of the present disclosure;

[0017] Figure 5A schematic diagram of the overall process of a method for identifying a conveyor belt state provided by an embodiment of the present disclosure;

[0018] Figure 6 A schematic diagram of the structure of a device for identifying the state of a conveyor belt provided in an embodiment of the present disclosure;

[0019] Figure 7 The present invention is a block diagram of an electronic device for a method of identifying a transmission belt status according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] The following describes a method, device and electronic device for identifying the state of a transmission belt according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0022] Deep Learning (DL) is a new research direction in the field of Machine Learning (ML). It is introduced into machine learning to make it closer to its original goal - artificial intelligence. Deep learning is the inherent laws and representation levels of learning sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far exceed previous related technologies.

[0023] Image processing technology is a technology that uses computers to analyze images to achieve the desired results. It is also called image processing. Image processing generally refers to digital image processing. A digital image refers to a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of the array are called pixels, and their values ​​are called grayscale values. Image processing technology generally includes image compression, enhancement and restoration, matching, description and recognition.

[0024] Image recognition refers to the technology of using computers to process, analyze and understand images to identify targets and objects of various different patterns. It is a practical application of deep learning algorithms. At present, image recognition technology is generally divided into face recognition and product recognition. Face recognition is mainly used in security inspections, identity verification and mobile payments; product recognition is mainly used in the circulation of goods, especially in unmanned retail fields such as unmanned shelves and smart retail cabinets. The generation and updating of image recognition technology has become a very important development direction at present, and it has shown good development prospects. Image recognition technology has been widely used in information collection, medical treatment and product safety, and has played a very important role.

[0025] At present, belt running status recognition algorithms are mainly divided into two categories: one is based on traditional image algorithms or uses deep learning detection or classification algorithms for single-frame images; the second is based on other auxiliary equipment, such as infrared, laser, etc.

[0026] Methods based on traditional image algorithms, such as background modeling, optical flow, or feature point matching, are easily affected by external interference, resulting in unsatisfactory recognition results and potential safety hazards.

[0027] Based on the characteristics of deep learning and machine learning, it is believed that setting rule comparison requires manual setting of comparison rules, such as feature matrix, similarity threshold, etc. This rule needs to be summarized through a large number of manual experiments and has poor robustness.

[0028] Using deep learning detection or classification algorithms based on single-frame images, it is difficult to determine the movement of the belt using only the information of a single frame, and the effect is poor.

[0029] Judging based on other auxiliary equipment, such as infrared, laser, etc., this solution is expensive, requires customized auxiliary equipment, and is easily affected by external factors, such as light and shadow, human operation, machine operating status, etc.

[0030] In order to solve the above problems, the present disclosure proposes a method for identifying the status of a conveyor belt. Figure 1 A flowchart of a method for identifying a conveyor belt status provided in an embodiment of the present disclosure.

[0031] like Figure 1 As shown, the method for identifying the state of the transmission belt may include:

[0032] S101, performing target detection on M consecutive image frames to obtain a transmission belt common detection frame corresponding to the M consecutive image frames, where M is a natural number greater than or equal to 2.

[0033] It should be noted that M can be changed according to actual design requirements and is not limited here. In a possible implementation, the accuracy of the conveyor belt state recognition can be achieved by controlling the size of M.

[0034] In the embodiment of the present disclosure, M consecutive frames of images may be acquired by an image acquisition device. The image acquisition device may be of various types, which are not limited here, for example, may be a camera, a still camera, etc.

[0035] It should be noted that the M frames of images may be collected periodically or randomly, and no limitation is made here.

[0036] The detection frame is a detection frame that can capture the image of the object to be detected. In the embodiment of the present disclosure, when the conveyor belt is in operation, the position and size of the detection frame corresponding to the M frame image in the image may be different.

[0037] In the embodiment of the present disclosure, the detection frames corresponding to the M frames of images may be spliced ​​to generate a transmission belt common detection frame corresponding to the M frames of images.

[0038] In a possible implementation, if a detection frame is not detected in one or several frames of the M frames, calculations are performed based on other images in which detection frames are detected to generate a common detection frame.

[0039] S102: Fusing M consecutive image frames to obtain a first fused image.

[0040] It should be noted that there are many methods for fusing M consecutive image frames, which are not limited here.

[0041] In one possible implementation, the M consecutive frames of images may be processed by an image fusion model to generate a first fused image. The image fusion model may be pre-trained, used to fuse images, and may be stored in a storage space of an electronic device for easy retrieval when needed.

[0042] S103, acquiring a target transmission belt area map from the first fused image according to the transmission belt common detection frame.

[0043] In the disclosed embodiment, the first fused image may be intercepted by a common detection frame to obtain a target transmission belt area map, whereby the target transmission belt area map may carry all image information containing the transmission belt in the M frames of images.

[0044] S104, predicting the state of the target transmission belt area map to determine the target state of the transmission belt.

[0045] It should be noted that the target state of the conveyor belt can be various, and no limitation is made here. For example, the target state can be a stationary state, a moving state, an abnormal state, etc.

[0046] In the disclosed embodiment, there are many methods for predicting the state of the target conveyor belt area map, which are not limited here. In one possible implementation, the target conveyor belt area map can be predicted by a state recognition model to determine the target state of the conveyor belt. It should be noted that the state recognition model is trained in advance and stored in the storage space of the electronic device to facilitate retrieval and use when needed.

[0047] In another possible implementation, the target state of the conveyor belt can be determined by processing the pixels in the target conveyor belt area map through a state prediction algorithm and calculating a set judgment value. The prediction algorithm is designed in advance and can be changed according to actual design needs, and no limitation is made here. For example, an offset value can be calculated based on the positions of different pixels at the same position, and the offset value can be used as a judgment value.

[0048] In the disclosed embodiment, firstly, target detection is performed on M consecutive frames of images to obtain a common detection frame of the transmission belt corresponding to the M consecutive frames of images, where M is a natural number greater than or equal to 2. Then, the M consecutive frames of images are fused to obtain a first fused image. Then, according to the common detection frame of the transmission belt, a target transmission belt area map is obtained from the first fused image. Finally, the state of the target transmission belt area map is predicted to determine the target state of the transmission belt. Thus, by fusing multiple consecutive frames of images and judging the fused image, judgment can be made in two dimensions of time and space, thereby improving the accuracy of determining the target state of the transmission belt. Compared with the judgment method in the current technology, no other auxiliary equipment is required for judgment, only an image acquisition device is required, which can reduce the judgment cost and is not easily affected by the outside world.

[0049] In the above embodiment, target detection is performed on M consecutive image frames to obtain a transmission belt common detection frame corresponding to the M consecutive image frames. Figure 2 Explaining further, the method includes:

[0050] S201, obtaining a candidate transmission belt detection frame of each frame of M consecutive image frames.

[0051] In the above embodiment, the candidate conveyor belt detection frame is a detection frame that can capture the object image to be detected in each frame of image. In actual operation, when the conveyor belt is in operation, since the posture of the conveyor belt is constantly changing, the position and size of the detection frame corresponding to the M frame image may be different in the image.

[0052] In the disclosed embodiment, the candidate belt detection frame of each frame of the continuous M frames can be obtained by an image recognition model. The image recognition model is trained in advance, and the M frames of images can be input into the image recognition model to determine the candidate belt detection frame of each frame of the image.

[0053] S202, determining a minimum circumscribed rectangle of a candidate transmission belt detection frame of each frame image as a common transmission belt detection frame.

[0054] In the disclosed embodiment, the candidate transmission belt detection frame of each frame of the continuous M frames is first obtained, and then the minimum circumscribed rectangle including the candidate transmission belt detection frame of each frame is determined as the common transmission belt detection frame. Thus, the transmission belt common detection frame is generated by the candidate transmission belt detection frame of each frame, which can ensure that when the image is subsequently captured through the transmission belt common detection frame, the image information containing the transmission belt in the fused image is captured as much as possible, providing as complete reference data as possible for subsequent prediction.

[0055] In the above embodiment, to fuse M consecutive image frames, each image frame in the M consecutive image frames can be first converted from the RGB mode into a grayscale image, where R represents the red channel, G represents the green channel, and B represents the blue channel, and then the channel images of the B channel, the G channel, and the R channel are determined from the grayscale image, and finally the first fused image in the BGR mode is obtained according to the channel image corresponding to each color channel. It should be noted that the first fused image in the BGR mode is obtained according to the channel image corresponding to each color channel, and the channel images of each color channel can be spliced ​​in the order of the B channel, the G channel, and the R channel to obtain the first fused image in the BGR mode.

[0056] It should be noted that, to determine the channel images of the B channel, the G channel and the R channel from the grayscale image, the grayscale image can be first grouped according to the time sequence of the continuous M frame images to obtain a set of grayscale images corresponding to each color channel. The grouping method can be multiple and is not limited here. In one possible implementation method, the M frame images can be divided into images corresponding to the B channel, the G channel and the R channel in sequence. Then, for each color channel, its own channel image is determined from the set of grayscale images corresponding to the color channel. Therefore, by fusing the continuous M frame images to generate the first fused image, the image elements of the continuous M frame images can be fused into the same image, which can provide a more comprehensive data basis for subsequent predictions.

[0057] In the above embodiment, according to the transmission belt common detection frame, the target transmission belt area map is obtained from the first fused image, and the target transmission belt area map can also be obtained by Figure 3 Explaining further, the method includes:

[0058] S301, performing transmission belt positioning on the first fused image according to the transmission belt common detection frame to determine the area where the transmission belt is located.

[0059] S302, obtaining an initial transmission belt area map from the first fused image, and preprocessing the initial transmission belt area map to obtain a target transmission belt area map.

[0060] In the embodiment of the present disclosure, the initial transmission band area map is preprocessed by first converting the initial transmission band area map from BGR (BLUE GREEN RED) mode to RGB (RED GREEN BLUE) mode to obtain a first transmission band area map, then resizing the first transmission band area map to obtain a second transmission band area map, and performing pixel correction on the pixels of the second transmission band area map to obtain a third transmission band area map, and finally performing channel conversion on the third transmission band area map to obtain a target transmission band area map.

[0061] It should be noted that in image processing, the main difference between BGR mode and RGB mode lies in the order of color channels. BGR mode is the default color space arrangement of some libraries or frameworks (such as OpenCV), while RGB mode is a more common color representation method, especially widely used in web design and image editing software.

[0062] It should be noted that there are many methods for converting the initial transmission band area map from the BGR mode to the RGB mode, which are not limited here. For example, the initial transmission band area map can be converted by a conversion application, and the initial transmission band area map can be input into a conversion model to complete the conversion from the BGR mode to the RGB mode.

[0063] In a possible implementation, the initial transmission belt area map is converted from the BGR mode to the RGB mode by first obtaining a channel image of each color channel from the initial transmission belt area map, and then splicing the channel image of each color channel in the order of the R channel, the G channel, and the B channel to obtain the first transmission belt area map in the RGB mode. Thus, by converting the initial transmission belt area map from the BGR mode to the RGB mode, subsequent processing can be facilitated.

[0064] It should be noted that there are many methods for performing pixel correction on the pixels of the second transmission belt area map, which are not limited here. In one possible implementation, the pixels of the second transmission belt area map can be normalized first, and the pixel mean and standard deviation of the second transmission belt area map can be obtained. Then, based on the pixel mean and standard deviation, the second transmission belt area map is pixel corrected to obtain the third transmission belt area map. Therefore, by performing pixel correction on the pixels of the second transmission belt area map, the accuracy and efficiency of subsequent image processing can be improved, thereby improving the accuracy of the final prediction.

[0065] In the disclosed embodiment, firstly, the transmission belt is located on the first fused image according to the transmission belt common detection frame to determine the area where the transmission belt is located, and then an initial transmission belt area map is obtained from the first fused image, and the initial transmission belt area map is preprocessed to obtain a target transmission belt area map. Thus, by performing image capture through the transmission belt common detection frame, image information containing the transmission belt in the fused image can be captured as much as possible, providing as complete reference data as possible for subsequent predictions.

[0066] In the above embodiment, the state of the target transmission belt area map is predicted to determine the target state of the transmission belt. Figure 4 Explaining further, the method includes:

[0067] S401, inputting the target transmission belt area map into a pre-trained target classification model, and obtaining the predicted state type of the target transmission belt area map by the target classification model.

[0068] It should be noted that the target classification model is a model for classifying the state of the conveyor belt in the image. The target classification model is designed in advance and stored in the storage space of the electronic device to facilitate retrieval and use when needed.

[0069] In one possible implementation, the target classification model can be a ViT-Base model. Vision Transformer (ViT)-Base is a visual classification model based on the Transformer architecture. It converts image data into a sequence form through an embedding layer, then extracts features and models them through multiple Transformer encoder layers, and finally outputs the classification result through a classification head.

[0070] Specifically, the Embedding layer of ViT-Base uses linear projection and position encoding to divide the input image into fixed-size image patches. Each image patch is transformed into a one-dimensional vector through linear projection and added to the sequence through position encoding information, thereby retaining the spatial information of the image.

[0071] Specifically, the Transformer Encoder layer of ViT-Base stacks multiple standard Transformer encoders, which are composed of a multi-head self-attention mechanism and a feed-forward neural network. In these layers, the multi-head self-attention mechanism can capture the global relationship between image blocks, while the feed-forward neural network can perform nonlinear transformations on the extracted features, further improving the representation ability of the model.

[0072] Specifically, the classification head of ViT-Base uses a standard fully connected layer. After being processed by multiple Transformer Encoder layers, the final features are output as category labels through the fully connected layer. During the training process, ViT-Base uses the standard cross-entropy loss function for optimization, and uses large-scale pre-training data to improve the generalization ability of the model.

[0073] The advantage of the ViT-Base model lies in its powerful feature extraction capability and flexible model structure. It captures the global information of the image through the Transformer architecture and effectively improves the classification accuracy.

[0074] In an embodiment of the present disclosure, the target transmission band area map can be first input into a pre-trained target classification model, the target transmission band area map can be divided into image blocks of fixed sizes, and then the image blocks can be linearly projected through an embedding layer in the target classification model to obtain a representation vector of the image blocks, and the position coding information of the image blocks can be obtained. Finally, the representation vector and the position coding information of the image blocks can be feature fused to obtain fused features, and the fused features can be input into the predicted state type of the target transmission band area map output by the fully connected layer in the target classification model.

[0075] S402, determining a target state of the transmission belt according to the predicted state types of N consecutive target transmission belt area graphs.

[0076] In the embodiments of the present disclosure, due to the complex working conditions at the work site and the accuracy of the image acquisition device, there may be a situation where the predicted state type finally generated has errors. Therefore, the present disclosure determines the target state of the conveyor belt by the predicted state types of N consecutive target conveyor belt area maps, which can avoid the occurrence of the above situation. In a possible implementation method, the proportion of each state type in the predicted state types of N target conveyor belt area maps can be counted, and the predicted state type with the largest proportion can be used as the target state of the conveyor belt.

[0077] In another possible implementation, the target state of the conveyor belt is determined based on the predicted state types of N consecutive target conveyor belt area maps. The judgment can be made by the ratio of motion state types of the N consecutive predicted state types. In response to the predicted state types of the N target conveyor belt area maps being all motion state types, the target state is determined to be a motion state. In response to the presence of a non-motion state type in the predicted state types of the N target conveyor belt area maps, the target state is determined to be an abnormal state.

[0078] In the disclosed embodiment, the target conveyor belt area map is first input into the pre-trained target classification model, and the target classification model obtains the predicted state type of the target conveyor belt area map, and then the target state of the conveyor belt is determined according to the predicted state types of N consecutive target conveyor belt area maps. Thus, by determining the target state of the conveyor belt according to the predicted state types of N consecutive target conveyor belt area maps, it is possible to avoid the situation where the predicted state type finally generated may have errors due to the complex working conditions at the work site and the accuracy of the image acquisition device.

[0079] In the disclosed embodiment, in response to the target state of the conveyor belt being an abnormal state, identification information of the image acquisition device associated with the image is determined, and based on the identification information, early warning information of the image acquisition device is generated. It should be noted that the early warning information may be of various types, and no limitation is made here. For example, the early warning information may be a text message warning, a sound warning, and the like. To generate the early warning information, the installation location of the image acquisition device may be determined first based on the identification information, and then the early warning information of the image acquisition device may be generated based on the installation location and the abnormal state. In this way, by detecting and determining the abnormal state and generating an early warning, the operator can be promptly reminded to confirm or take corresponding measures, thereby improving the safety of the conveyor belt operation.

[0080] Figure 5 The overall process diagram of the conveyor belt status recognition disclosed in the present invention is as follows: Figure 5As shown, firstly, target detection is performed on M consecutive image frames to obtain a candidate transmission belt detection frame of each frame in the M consecutive image frames, and then the minimum circumscribed rectangle including the candidate transmission belt detection frame of each frame is determined as a common transmission belt detection frame, and the M consecutive image frames are fused at the same time to obtain a first fused image, and a target transmission belt area map is obtained from the first fused image according to the transmission belt common detection frame, and after preprocessing the target transmission belt area map, the target transmission belt area map is input into a pre-trained target classification model, and the target classification model obtains the predicted state type of the target transmission belt area map, and the target state of the transmission belt is determined according to the predicted state types of N consecutive target transmission belt area maps.

[0081] Corresponding to the transmission belt status identification methods provided in the above-mentioned embodiments, an embodiment of the present disclosure further provides a transmission belt status identification device. Since the transmission belt status identification device provided in the embodiment of the present disclosure corresponds to the transmission belt status identification methods provided in the above-mentioned embodiments, the implementation methods of the above-mentioned transmission belt status identification methods are also applicable to the transmission belt status identification device provided in the embodiment of the present disclosure, and will not be described in detail in the following embodiments.

[0082] Figure 6 The schematic diagram of the structure of a transmission belt state recognition device provided by the embodiment of the present disclosure is as follows: The transmission belt state recognition device 600 comprises: a detection module 610 , a fusion module 620 , an acquisition module 630 and a prediction module 640 .

[0083] The detection module 610 is used to perform target detection on M consecutive image frames to obtain a transmission belt common detection frame corresponding to the M consecutive image frames, where M is a natural number greater than or equal to 2.

[0084] The fusion module 620 is used to fuse M consecutive image frames to obtain a first fused image.

[0085] The acquisition module 630 is used to acquire a target transmission belt area map from the first fused image according to the transmission belt common detection frame.

[0086] The prediction module 640 is used to perform state prediction on the target transmission belt area map to determine the target state of the transmission belt.

[0087] In the embodiment of the present disclosure, the detection module 610 is further used to: obtain a candidate transmission belt detection frame for each frame of the continuous M frames of images; determine the minimum circumscribed rectangle including the candidate transmission belt detection frame of each frame of the image as a common transmission belt detection frame.

[0088] In the embodiment of the present disclosure, the fusion module 620 is also used to: convert each frame of the continuous M frame images from the RGB mode into a grayscale image, where R represents the red channel, G represents the green channel, and B represents the blue channel; determine the channel images of the B channel, the G channel, and the R channel respectively from the grayscale image; and obtain the first fused image in the BGR mode according to the channel image corresponding to each color channel.

[0089] In the embodiment of the present disclosure, the fusion module 620 is further used to: splice the channel images of each color channel in the order of the B channel, the G channel and the R channel to obtain a first fused image in the BGR mode.

[0090] In the disclosed embodiment, the fusion module 620 is further used to: group the grayscale images according to the time sequence of the continuous M frames of images to obtain a set of grayscale images corresponding to each color channel; and for each color channel, determine its own channel image from the set of grayscale images corresponding to the color channel.

[0091] In the embodiment of the present disclosure, the acquisition module 630 is also used to: locate the transmission belt in the first fused image according to the common detection frame of the transmission belt, and determine the area where the transmission belt is located; obtain an initial transmission belt area map from the first fused image, and pre-process the initial transmission belt area map to obtain a target transmission belt area map.

[0092] In the embodiment of the present disclosure, the acquisition module 630 is also used to: convert the initial transmission band area map from the BGR mode to the RGB mode to obtain a first transmission band area map; resize the first transmission band area map to obtain a second transmission band area map, and perform pixel correction on the pixels of the second transmission band area map to obtain a third transmission band area map; perform channel conversion on the third transmission band area map to obtain a target transmission band area map.

[0093] In the embodiment of the present disclosure, the acquisition module 630 is also used to: obtain the channel image of each color channel from the initial transmission band area map; splice the channel image of each color channel in the order of R channel, G channel and B channel to obtain the first transmission band area map in RGB mode.

[0094] In the embodiment of the present disclosure, the acquisition module 630 is also used to: normalize the pixels of the second transmission belt area map and obtain the pixel mean and standard deviation of the second transmission belt area map; perform pixel correction processing on the second transmission belt area map based on the pixel mean and standard deviation to obtain a third transmission belt area map.

[0095] In the embodiment of the present disclosure, the prediction module 640 is also used to: input the target transmission belt area map into a pre-trained target classification model, and obtain the predicted state type of the target transmission belt area map from the target classification model; determine the target state of the transmission belt based on the predicted state types of N consecutive target transmission belt area maps.

[0096] In the embodiment of the present disclosure, the prediction module 640 is also used to: input the target transmission band area map into a pre-trained target classification model, divide the target transmission band area map into image blocks of fixed sizes; linearly project the image blocks through the embedding layer in the target classification model to obtain the representation vector of the image blocks, and obtain the position coding information of the image blocks; perform feature fusion on the representation vector and the position coding information of the image blocks to obtain fused features, and input the fused features into the predicted state type of the target transmission band area map output by the fully connected layer in the target classification model.

[0097] In the embodiment of the present disclosure, the prediction module 640 is also used to: in response to the predicted state types of the N target transmission belt area maps being all motion state types, determine that the target state is a motion state; in response to the presence of a non-motion state type in the predicted state types of the N target transmission belt area maps, determine that the target state is an abnormal state.

[0098] In the disclosed embodiment, the prediction module 640 is further used to: in response to the target state of the conveyor belt being an abnormal state, determine identification information of the image acquisition device associated with the image; and generate warning information of the image acquisition device according to the identification information.

[0099] In the embodiment of the present disclosure, the prediction module 640 is further used to: determine the installation location of the image acquisition device according to the identification information; and generate early warning information of the image acquisition device according to the installation location and the abnormal state.

[0100] By fusing multiple consecutive frames of images and making judgments on the fused images, judgments can be made in both time and space dimensions, thereby improving the accuracy of determining the target state of the conveyor belt. Compared with the judgment methods in current technologies, no other auxiliary equipment is required for judgment, only an image acquisition device is required, which can reduce the judgment cost and is not easily affected by the outside world.

[0101] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0102] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0103] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0104] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program / instruction stored in a read-only memory (ROM) 702 or a computer program / instruction loaded from a storage unit 707 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 706 is also connected to the bus 704.

[0105] A number of components in the device 700 are connected to the I / O interface 706, including: an input unit 707 such as a keyboard, a mouse, etc.; an output unit 707 such as various types of displays, speakers, etc.; a storage unit 708 such as a disk, an optical disk, etc.; and a communication unit 709 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0106] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the identification method of the conveyor belt state. For example, in some embodiments, the identification method of the conveyor belt state may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 707. In some embodiments, part or all of the computer program / instructions may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program / instructions are loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the identification method of the conveyor belt state described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the method for identifying the state of the transmission belt in any other appropriate manner (for example, by means of firmware).

[0107] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs / instructions that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0109] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0111] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0112] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs / instructions running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0113] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the disclosure can be achieved, and this document does not limit them here.

[0114] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for identifying the state of a conveyor belt, in, The method comprises: Performing target detection on M consecutive image frames to obtain a transmission belt common detection frame corresponding to the M consecutive image frames, where M is a natural number greater than or equal to 2; Fusing the M consecutive image frames to obtain a first fused image; According to the transmission belt common detection frame, acquiring a target transmission belt area map from the first fused image; The state of the target transmission belt area map is predicted to determine the target state of the transmission belt.

2. The method according to claim 1, in, The performing target detection on the continuous M-frame images to obtain a transmission belt common detection frame corresponding to the continuous M-frame images includes: Obtaining a candidate transmission belt detection frame for each frame of the continuous M frames of images; A minimum circumscribed rectangle of the candidate transmission belt detection frame of each frame image is determined as the common transmission belt detection frame.

3. The method according to claim 1, in, The step of fusing the continuous M frames of images to obtain a first fused image includes: Convert each frame of the continuous M frames of images from an RGB mode to a grayscale image, wherein R represents a red channel, G represents a green channel, and B represents a blue channel; Determine the channel images of the B channel, the G channel and the R channel respectively from the grayscale image; According to the channel image corresponding to each color channel, the first fused image in BGR mode is obtained.

4. The method according to claim 3, in, The step of obtaining the first fused image in the BGR mode according to the channel image corresponding to each color channel includes: The channel images of each color channel are spliced ​​in the order of B channel, G channel and R channel to obtain the first fused image in BGR mode.

5. The method according to claim 3, in, Determining the channel images of the B channel, the G channel and the R channel from the grayscale image includes: According to the time sequence of the continuous M frames of images, the grayscale images are grouped to obtain a set of grayscale images corresponding to each color channel; For each color channel, its own channel image is determined from the grayscale image set corresponding to the color channel.

6. The method according to claim 1, in, The step of acquiring a target transmission belt area map from the first fused image according to the transmission belt common detection frame includes: According to the transmission belt common detection frame, positioning the transmission belt on the first fused image to determine the area where the transmission belt is located; An initial transmission belt area map is acquired from the first fused image, and the initial transmission belt area map is preprocessed to obtain the target transmission belt area map.

7. The method according to claim 6, in, The preprocessing of the initial transmission belt area map to obtain the target transmission belt area map includes: Converting the initial transmission belt area map from BGR mode to RGB mode to obtain a first transmission belt area map; resizing the first transmission belt area map to obtain a second transmission belt area map, and performing pixel correction on pixels of the second transmission belt area map to obtain a third transmission belt area map; Channel conversion is performed on the third transmission belt area map to obtain the target transmission belt area map.

8. The method according to claim 7, in, The converting of the initial transmission band area map from the BGR mode to the RGB mode comprises: Obtaining a channel image of each color channel from the initial transmission band region map; The channel images of each color channel are spliced ​​in the order of R channel, G channel and B channel to obtain the first transmission belt area map in RGB mode.

9. The method according to claim 7, in, The step of performing pixel correction on the pixels of the second transmission belt area map to obtain a third transmission belt area map comprises: Normalizing the pixels of the second transmission belt area map, and obtaining a pixel mean and a standard deviation of the second transmission belt area map; The second transmission belt area map is subjected to pixel correction processing based on the pixel mean and standard deviation to obtain the third transmission belt area map.

10. The method according to any one of claims 1 to 9, in, The step of predicting the state of the target transmission belt area map to determine the target state of the transmission belt includes: Inputting the target transmission belt area map into a pre-trained target classification model, and obtaining the predicted state type of the target transmission belt area map by the target classification model; The target state of the transmission belt is determined according to the predicted state types of N consecutive target transmission belt area maps.

11. The method according to claim 10, in, The step of inputting the target transmission belt area map into a pre-trained target classification model, and obtaining the predicted state type of the target transmission belt area map by the target classification model, comprises: Inputting the target transmission belt area map into a pre-trained target classification model, and dividing the target transmission belt area map into image blocks of fixed sizes; Performing linear projection on the image block through the embedding layer in the target classification model to obtain a representation vector of the image block, and acquiring position encoding information of the image block; Feature fusion is performed on the representation vector and position encoding information of the image block to obtain a fused feature, and the fused feature is input into the predicted state type of the target transmission belt area map output by the fully connected layer in the target classification model.

12. The method according to claim 10, in, Determining the target state of the transmission belt according to the predicted state types of N consecutive target transmission belt area maps includes: In response to the predicted state types of the N target transmission belt area maps being all motion state types, determining that the target state is a motion state; In response to the presence of a non-motion state type in the predicted state types of the N target transmission belt area maps, the target state is determined to be an abnormal state.

13. The method according to claim 12, in, The method further comprises: In response to the target state of the conveyor belt being an abnormal state, determining identification information of an image acquisition device associated with the image; According to the identification information, early warning information of the image acquisition device is generated.

14. The method according to claim 13, in, Generating the warning information of the image acquisition device according to the identification information includes: Determining the installation position of the image acquisition device according to the identification information; According to the installation position and the abnormal state, early warning information of the image acquisition device is generated.

15. A device for identifying the state of a conveyor belt, include: A detection module, used to perform target detection on M consecutive image frames, and obtain a transmission belt common detection frame corresponding to the M consecutive image frames, where M is a natural number greater than or equal to 2; A fusion module, used for fusing the M consecutive image frames to obtain a first fused image; An acquisition module, configured to acquire a target transmission belt area map from the first fused image according to the transmission belt common detection frame; The prediction module is used to perform state prediction on the target transmission belt area map to determine the target state of the transmission belt.

16. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

17. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to enable the computer to execute the method for identifying the state of a conveyor belt according to any one of claims 1-14.

18. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instruction is executed by a processor, the method for identifying the state of a conveyor belt according to any one of claims 1 to 14 is implemented.

Images