Belt conveyor state analysis method, storage medium, electronic device, and program product

Abstract

This application provides a method, storage medium, electronic device, and program product for belt conveyor state analysis, relating to the field of deep learning. The method includes: acquiring a video to be detected; processing the video to obtain a feature matrix corresponding to the video; inputting the feature matrix into a pre-trained belt conveyor state analysis model to obtain multiple channel output data of the model, wherein the multiple channel output data includes at least belt conveyor state channel data, optical flow field channel data, and instance representation channel data; and determining the state information of the belt conveyor contained in the video based on the multiple channel output data. This application overcomes the deficiency of belt conveyor segmentation and optical flow calculation being separate in traditional methods, achieving comprehensive, real-time, and accurate monitoring of the belt conveyor's operating status.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, specifically to a belt conveyor state analysis method, storage medium, electronic device, and program product. Background Technology

[0002] In modern logistics hubs, due to the dense arrangement of production lines and the interlacing of belt conveyors, on-site managers face the challenge of comprehensively monitoring the operation of belt conveyors, but this solution cannot meet the requirements of real-time performance and accuracy.

[0003] Currently, the commonly used solution is to first determine the pixel positions of the conveyor belt in the image through manual annotation or algorithms; then, optical flow is calculated based on the working video, and the motion state of the conveyor belt is inferred by combining the pixel positions. The drawback of this method is that it separates conveyor belt segmentation from optical flow calculation and relies on pre-annotation or time-consuming algorithms. Summary of the Invention

[0004] In view of this, embodiments of this application provide a belt conveyor condition analysis method, storage medium, electronic device, and program product.

[0005] In a first aspect, one embodiment of this application provides a belt conveyor state analysis method, comprising: acquiring a video to be detected; processing the video to be detected to obtain a feature matrix corresponding to the video to be detected; inputting the feature matrix into a pre-trained belt conveyor state analysis model to obtain multiple channel output data of the belt conveyor state analysis model, wherein the multiple channel output data includes at least belt conveyor state channel data, optical flow field channel data, and instance characterization channel data; and determining the state information of the belt conveyor contained in the video to be detected based on the multiple channel output data.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the video to be detected is processed to obtain the feature matrix corresponding to the video to be detected, including: extracting several video segments from the video to be detected; extracting multiple frames of images from each video segment and dividing the multiple frames of images into color images and grayscale images; stacking the color images and grayscale images to form the feature matrix corresponding to the video segment, thereby obtaining the feature matrix corresponding to the video to be detected.

[0007] In conjunction with the first aspect, in certain implementations of the first aspect, the state information of the conveyor belt contained in the video to be detected is determined based on multiple channel output data, including: processing the conveyor belt state channel data using an image segmentation algorithm to obtain the conveyor belt moving area and / or the conveyor belt stationary area; processing the instance representation channel data using a clustering algorithm to obtain multiple independent moving conveyor belt segments corresponding to the moving area of ​​the conveyor belt, and / or multiple independent stationary conveyor belt segments corresponding to the stationary area of ​​the conveyor belt; and processing the optical flow field channel data using a mean algorithm to obtain the respective motion directions of the multiple independent moving conveyor belt segments.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the belt conveyor state analysis model includes multiple input channels, an encoder, a decoder, and multiple output channels; there are skip connections between the encoder and decoder; the input channels include color image input channels and grayscale image input channels; the multiple output channels include belt conveyor state output channels, optical flow field output channels, and instance representation output channels; the belt conveyor state analysis model is trained based on target sample data and the loss function corresponding to each output channel; the target sample data includes multiple video segments and instance annotation data and optical flow calculation data corresponding to each video segment.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the conveyor belt status output channel includes a background area output channel, a conveyor belt motion area output channel, and a conveyor belt stationary area output channel; the loss function corresponding to the conveyor belt status output channel is the cross-entropy loss function, which is used to calculate the difference between the output value and the supervised value of each of the background area output channel, the conveyor belt motion area output channel, and the conveyor belt stationary area output channel.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the optical flow field output channel includes an optical flow field horizontal component output channel and an optical flow field vertical component output channel; the loss function corresponding to the optical flow field output channel is a mean square error loss function, which is used to calculate the difference between the output value and the supervision value of each of the optical flow field horizontal component output channel and the optical flow field vertical component output channel.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the instance representation channel includes a multi-dimensional vector used to distinguish the instance category to which the pixels in the animation belong; the loss function corresponding to the instance representation channel is a logistic function used to minimize the differences in pixel representations within the same instance and / or maximize the differences in pixel representations between different instances.

[0012] Secondly, one embodiment of this application provides a belt conveyor state analysis device, comprising: a first acquisition module for acquiring a video to be detected; a processing module for processing the video to be detected to obtain a feature matrix corresponding to the video to be detected; a second acquisition module for inputting the feature matrix into a pre-trained belt conveyor state analysis model to acquire multiple channel output data of the belt conveyor state analysis model, wherein the multiple channel output data includes at least belt conveyor state channel data, optical flow field channel data, and instance characterization channel data; and a determination module for determining the state information of the belt conveyor contained in the video to be detected based on the multiple channel output data.

[0013] Thirdly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the methods described in the first and second aspects.

[0014] Fourthly, one embodiment of this application provides an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to perform the methods described in the first and second aspects.

[0015] Fifthly, one embodiment of this application provides a computer program product, characterized in that the computer program product includes instructions that, when executed on an electronic device, cause the electronic device to implement the methods described in the first and second aspects.

[0016] In this application, specifically in this embodiment, the belt conveyor state analysis model can analyze the instance representation and optical flow field data of the belt conveyor in real time and accurately, overcoming the shortcomings of traditional methods that separate belt conveyor segmentation from optical flow calculation. This meets the real-time monitoring needs of modern logistics transit hubs and achieves comprehensive, real-time, and accurate monitoring of the belt conveyor's operating status. Therefore, by utilizing the belt conveyor motion state analysis model, the state information of the panoramic belt conveyor can be obtained more quickly, accurately, and comprehensively. Attached Figure Description

[0017] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 The diagram shown is a flowchart of a belt conveyor state analysis method provided in an embodiment of this application.

[0019] Figure 2The diagram shown is a structural schematic of a belt conveyor condition analysis device provided in an embodiment of this application.

[0020] Figure 3 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] As described in the background section, existing commonly used solutions have some obvious limitations. First, this method separates belt conveyor segmentation from optical flow calculation, resulting in insufficient clarity in the segmentation of continuous belt segments. However, moving and stationary belt segments differ significantly in their optical flow characteristics. If these two steps can be combined, richer information can be provided, thus facilitating more accurate segmentation. Second, the methods mentioned in the background section rely on pre-annotation or complex algorithms to determine the pixel positions of each belt conveyor segment in the image. This is not only time-consuming and labor-intensive but also difficult to adapt to changes in the scene, has high computational costs, and cannot meet real-time requirements. These shortcomings limit the applicability and reliability of the method in variable environments. Therefore, this application is proposed to overcome these shortcomings.

[0023] Specifically, Figure 1 The diagram shown is a flowchart illustrating a belt conveyor condition analysis method according to an embodiment of this application. For example, as shown... Figure 1 As shown, the method includes the following steps.

[0024] Step S110: Obtain the video to be tested.

[0025] For example, the video to be detected can be acquired via wired or wireless network. This video can be a live stream or a previously stored historical video. Furthermore, the video format includes, but is not limited to, common video formats such as MP4 and AVI.

[0026] Step S120: Process the video to be detected to obtain the feature matrix corresponding to the video to be detected.

[0027] Specifically, the feature matrix is ​​a multi-channel data structure that integrates color image information and grayscale image information from the video to be detected, as well as other possible auxiliary information (such as optical flow field information), providing richer input data for the belt conveyor state analysis model.

[0028] Step S130: Input the feature matrix into the pre-trained belt conveyor state analysis model to obtain the output data of multiple channels of the belt conveyor state analysis model.

[0029] Specifically, the output data from multiple channels includes at least the belt conveyor status channel data, the optical flow field channel data, and the instance characterization channel data.

[0030] The belt conveyor status channel data is used to identify and distinguish different states of the belt conveyor, i.e., which parts are moving and which parts are stationary. The optical flow field channel data provides detailed information about the belt conveyor's motion trends, such as the direction and speed of motion. The instance characterization channel data is used to distinguish which instance each pixel in the image belongs to, i.e., to identify and distinguish different belt conveyor segments in the image.

[0031] Step S140: Based on the output data from multiple channels, determine the status information of the belt conveyor contained in the video to be detected.

[0032] For example, firstly, the moving and stationary belt regions are segmented using the belt conveyor status channel data. Then, these regions are further subdivided into independent belt conveyor segments using instance characterization channel data. Based on this, the motion direction of each independent moving belt conveyor segment is calculated by combining the optical flow field channel data. In this way, by analyzing the output data of multiple channels, the status information of the belt conveyor contained in the video to be detected can be comprehensively determined.

[0033] Accordingly, the status information of the belt conveyor includes the belt conveyor moving area, the belt conveyor stationary area, multiple independent moving belt conveyor segments corresponding to the belt conveyor moving area, and / or the movement directions of the multiple independent stationary belt conveyor segments and the multiple independent moving belt conveyor segments corresponding to the belt conveyor stationary area.

[0034] In this embodiment, the belt conveyor state analysis model can analyze the instance representation and optical flow field data of the belt conveyor in real time and accurately, making up for the shortcomings of traditional methods that separate belt conveyor segmentation and optical flow calculation. This meets the real-time monitoring needs of modern logistics transit hubs and achieves comprehensive, real-time, and accurate monitoring of the belt conveyor's operating status. Therefore, using the belt conveyor motion state analysis model, the state information of the panoramic belt conveyor can be obtained more quickly, accurately, and comprehensively.

[0035] Regarding step S120, in some implementations, firstly, several video segments are extracted from the video to be detected; then, multiple frames of images are extracted from each video segment, and the multiple frames of images are divided into color images and grayscale images; then, the color images and grayscale images are stacked to form the feature matrix corresponding to the video segment, thereby obtaining the feature matrix corresponding to the video to be detected.

[0036] Specifically, a video clip refers to a portion of a certain length and content extracted from the video of the conveyor belt in operation. It contains continuous frames of the video and can range from a few seconds to a few minutes, depending on the needs of analysis or processing. For example, in this embodiment, the length of the video clip is 3 seconds.

[0037] Next, determine the frame extraction interval, i.e., how many frames every second to extract one image frame. This interval can be set according to the frame rate of the video clip and the required level of detail for analysis. For example, if the frame rate of the video clip is 30 frames per second, and we want to extract one frame per second, then the interval is set to 30. Then, starting from the first frame of each video clip, extract image frames at equal intervals according to the set intervals.

[0038] For example, when extracting multiple frames, the first frame is kept in its original color state so that it can provide more visual information, which is helpful for subsequent analysis and recognition. The remaining extracted frames are converted to grayscale images. Grayscale images only contain brightness information and not color information, which reduces the amount of data and is sufficient for some analyses that only focus on brightness changes. For example, four frames are extracted from each video segment; the first frame remains a color image, and the second through fourth frames remain grayscale images.

[0039] After extraction and transformation, these images are stacked into a feature matrix. For example, each row of this feature matrix represents an image. The first row might be a color image, and the remaining rows might correspond to grayscale images. In this way, a feature matrix containing both color and grayscale information can be constructed, which not only preserves key visual information but also reduces computational complexity and improves processing efficiency by using grayscale images.

[0040] In this embodiment, by capturing video clips, the operating status of the conveyor belt within a specific time period can be obtained in a targeted manner, which helps to focus on analyzing performance within a specific event or cycle. Next, multiple frames are extracted from these clips and divided into color and grayscale images, thus preserving visual information while reducing data redundancy.

[0041] Regarding step S140, in some implementations, an image segmentation algorithm is used to process the conveyor state channel data to obtain the conveyor moving area and / or the conveyor stationary area; a clustering algorithm is used to process the instance representation channel data to obtain multiple independent moving conveyor segments corresponding to the conveyor moving area, and / or multiple independent stationary conveyor segments corresponding to the conveyor stationary area; and a mean algorithm is used to process the optical flow field channel data to obtain the respective motion directions of the multiple independent moving conveyor segments.

[0042] For example, an image segmentation algorithm is used to further process the belt conveyor status channel data, including using a clustering algorithm to assign pixels in the image to different categories, such as moving areas and stationary areas, to obtain the belt conveyor moving area and / or belt conveyor stationary area.

[0043] For example, the K-means clustering algorithm is used to process the instance representation channel data. The K-means clustering algorithm divides n samples into K clusters, such that each sample belongs to the cluster represented by the nearest mean (i.e., cluster center), thereby minimizing the sum of the distances from the sample to its cluster center.

[0044] Each cluster center represents an independent conveyor belt segment, and sample points (i.e., different parts of the conveyor belt) are assigned to corresponding clusters based on their characteristics. By iteratively optimizing the location of the cluster centers, the moving and stationary areas of the conveyor belt can be subdivided into multiple independent conveyor belt segments, each with similar motion characteristics. This method can identify and distinguish different conveyor belt segments based on their actual motion state, rather than solely on location information, thus facilitating more accurate monitoring and maintenance of the conveyor belt segment's operating status.

[0045] For example, optical flow field data for each independent moving conveyor belt segment is first acquired from the optical flow field output channel. Based on this, the displacement of each independent moving conveyor belt segment is calculated. Then, the average displacement component of all pixels in each moving conveyor belt segment is calculated. Specifically, for the horizontal displacement component, the sum of the horizontal displacement values ​​of all pixels within the moving conveyor belt segment is calculated, and then divided by the total number of pixels in the moving conveyor belt segment to obtain the average horizontal displacement value of the moving conveyor belt segment. The same method is used to calculate the average value of the vertical displacement component. These two average values ​​together define a vector whose direction represents the average movement direction of the moving conveyor belt segment, and whose magnitude is proportional to the magnitude of the average displacement. In this way, a clear direction vector can be determined for each independent moving conveyor belt segment, thereby accurately identifying the movement trend of each segment.

[0046] This embodiment provides detailed views of the moving and stationary areas of the belt conveyor, and can identify and distinguish multiple independent belt conveyor segments, whether moving or stationary. By determining the direction of movement of each independent belt conveyor segment, the operating status of the belt conveyor can be monitored more accurately, abnormalities can be detected in a timely manner, and potential failures can be prevented, thereby improving the operating efficiency and safety of the belt conveyor.

[0047] In some embodiments, the conveyor belt state analysis model includes multiple input channels, an encoder, a decoder, and multiple output channels; the encoder and decoder are connected in a skip connection. The input channels include a color image input channel and a grayscale image input channel; the multiple output channels include a conveyor belt state output channel, an optical flow field output channel, and an instance representation output channel; the conveyor belt state analysis model is trained based on target sample data and the loss function corresponding to each output channel. The target sample data includes multiple video clips and instance annotation data and optical flow calculation data corresponding to each video clip.

[0048] Specifically, the structural features of the belt conveyor state analysis model are mainly reflected in its symmetrical U-shaped design. The encoder and decoder correspond to each other through skip connections. This skip connection mechanism allows the belt conveyor state analysis model to utilize feature information from different levels, thereby improving the accuracy and robustness of instance segmentation. In addition, the encoder is responsible for progressive downsampling to capture the contextual information of the animation, while the decoder progressively upsampling to recover the details and spatial information of the animation.

[0049] For example, the number of input channels is related to the number of input images. For instance, if four images are input, with the first image being a color image and the second to fourth images being grayscale images, then the belt conveyor state analysis model includes six input channels.

[0050] For example, the number of output channels is equal to the sum of the number of conveyor state output channels, optical flow field output channels, and instance representation output channels. For instance, if there are 3 conveyor state output channels, 2 optical flow field output channels, and 3 instance representation output channels, then the conveyor state analysis model includes 8 input channels.

[0051] Understandably, color images provide rich color information, which is very useful for identifying different objects and materials on a conveyor belt. For example, materials transported on a conveyor belt may have different colors, and color images can help models distinguish these materials, as well as identify possible contamination or anomalies, such as oil stains or watermarks. On the other hand, grayscale images focus on the texture and shape information contained in the animation, especially when performing image processing tasks such as edge detection and feature extraction.

[0052] During the training phase of the belt conveyor state analysis model, each video segment in the target sample data corresponds to instance annotation data and optical flow calculation data. Optical flow calculation is used to estimate the operating information of the belt conveyor in each video segment of the target sample data. Its purpose is to find the motion vector of each pixel or feature point in the video segment from one frame to another; these motion vectors constitute the optical flow field data. The instance annotation data outlines the contours of each belt conveyor in each image frame of the video segment and marks their motion state. Then, the optical flow field data of each image frame of the video segment is combined with the instance annotation data to obtain the target sample data.

[0053] Therefore, in this embodiment, by comprehensively using both color and grayscale image input channels, the belt conveyor condition analysis model can utilize both color and texture information simultaneously, improving the overall understanding of the belt conveyor's condition. This multi-channel input design allows the belt conveyor condition analysis model to adapt to different scenarios and conditions, maintaining high analytical performance regardless of changes in lighting or material color. Thus, this design enhances the belt conveyor condition analysis model's ability to identify changes in belt conveyor condition in complex industrial environments.

[0054] Next, we further explore the design of the conveyor state output channels and their corresponding loss functions in the conveyor state analysis model. These channels are specifically designed for different states and motion characteristics of the conveyor to achieve more detailed state monitoring and analysis. Specifically, in some embodiments, the conveyor state output channels include a background region output channel, a conveyor motion region output channel, and a conveyor stationary region output channel. The loss function corresponding to the conveyor state output channels is the cross-entropy loss function, used to calculate the difference between the output values ​​and supervised values ​​of each of the three output channels: the background region output channel, the conveyor motion region output channel, and the conveyor stationary region output channel.

[0055] Specifically, the background area output channel is responsible for identifying and outputting environmental information around the belt conveyor, the belt conveyor moving area output channel focuses on identifying and outputting the areas on the belt conveyor that are in motion, and the belt conveyor stationary area output channel is used to identify and output the stationary or abnormally stopped parts of the belt conveyor. Through this, the belt conveyor status output channel can precisely distinguish and identify different aspects of the belt conveyor's operating status.

[0056] The cross-entropy loss function offers significant advantages in handling classification problems. First, it directly measures the difference between the output value and the true value of the conveyor belt's state output channel, which is natural and intuitive for classification tasks. Second, the cross-entropy loss function is a convex function with a unique global minimum, making it easier to find the optimal solution when using optimization algorithms such as gradient descent, thus ensuring the stability and effectiveness of the conveyor belt state analysis model training. Finally, when the output value and the true value are the same, the cross-entropy loss function approaches 0, indicating that the initial network model's prediction is accurate; when the output value and the true value are different, the cross-entropy loss function value is large, indicating that the initial network model's prediction has errors. In other words, the smaller the value of the cross-entropy loss function, the more accurate the conveyor belt state analysis model's prediction, which directly reflects the degree of fit of the conveyor belt state analysis model to the class distribution. Therefore, by minimizing the cross-entropy loss function, the initial network model can learn more accurate conveyor belt state predictions, thereby improving the accuracy and reliability of conveyor belt state analysis.

[0057] Next, we further explore the design of the optical flow field output channel and its corresponding loss function. This involves how the belt conveyor state analysis model analyzes and predicts the motion characteristics of the belt conveyor, including motion vectors in the horizontal and vertical directions. Specifically, in some embodiments, the optical flow field output channel includes a horizontal component output channel and a vertical component output channel. The loss function corresponding to the optical flow field output channel is the mean square error loss function, used to calculate the difference between the output values ​​and supervised values ​​of the horizontal and vertical component output channels.

[0058] First, the optical flow field describes the motion vector of each pixel in the image from one frame to another, which can be decomposed into horizontal and vertical components. The horizontal component output channel captures the displacement changes of the pixels in the horizontal direction, i.e., the movement of the conveyor belt in the horizontal direction; the vertical component output channel captures the displacement changes of the pixels in the vertical direction, i.e., the movement of the conveyor belt in the vertical direction. This decomposition helps to analyze and understand the motion characteristics of the conveyor belt in greater detail, thus providing richer dynamic information for conveyor belt state analysis.

[0059] Furthermore, the supervised values ​​are obtained from the optical flow field data, representing the actual values ​​of the motion vectors of each pixel in the horizontal and vertical directions. To make these actual values ​​of motion vectors applicable to the training process, they are first truncated to a range of -10 to 10; that is, any values ​​outside this range are restricted to this interval, which reduces the impact of outliers on the training process. Next, these values ​​are normalized to adjust them to a uniform scale, making the initial network model easier to learn and generalize. For example, they are scaled to between -1 and 1.

[0060] The goal of the belt conveyor state analysis model is to accurately output the motion vector of each pixel in the horizontal and vertical directions, which is a regression problem. The mean squared error loss function provides a quantification of this difference by calculating the square of the difference between the output value and the true value and then averaging it. This metric squares the error, making the impact of larger errors more significant, thus more effectively guiding the update of the original network model parameters during training to reduce these larger errors. Furthermore, the mean squared error loss function is stable during optimization, insensitive to outliers, and computationally simple and easy to implement. Therefore, the mean squared error loss function is a suitable choice for the optical flow field output channel.

[0061] Next, we further explore the design of the instance representation channel and its corresponding loss function to facilitate the complex task of identifying and distinguishing individual objects in an animation. Specifically, in some embodiments, the instance representation channel includes a multi-dimensional vector used to distinguish the instance category to which a pixel in the animation belongs. The loss function corresponding to the instance representation channel is a logistic function, used to minimize the differences in pixel representations within the same instance and / or maximize the differences in pixel representations between different instances.

[0062] Specifically, the multidimensional vector for each pixel represents the probability or confidence level of its belonging to a specific instance category. These vectors typically have a much higher dimension than the original image, enabling them to capture more subtle and complex image features. For example, the instance representation channel includes a 3-dimensional vector.

[0063] In the instance representation channel, multidimensional vectors are used to distinguish the instance category to which pixels in the animation belong. The logistic function can convert the output of these three-dimensional vectors into a probabilistic form for classification. Furthermore, when used as a loss function, the logistic function minimizes the differences in pixel representations within the same instance while maximizing the differences in pixel representations between different instances. This helps the conveyor belt state analysis model better distinguish different instance categories. In this way, the conveyor belt state analysis model can learn discriminative features to differentiate between different instance categories, thus accurately classifying pixels in new images in practical applications.

[0064] The method embodiments of this application have been described in detail above. The following will be combined with… Figure 2 The present application provides a detailed description of the apparatus embodiments. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0065] Figure 2 The diagram shown is a structural schematic of a belt conveyor state analysis model training device provided in an embodiment of this application. Figure 2As shown, the belt conveyor state analysis model training device 20 provided in this embodiment includes:

[0066] The first acquisition module 210 is used to acquire the video to be detected;

[0067] Processing module 220 is used to process the video to be detected to obtain the feature matrix corresponding to the video to be detected;

[0068] The second acquisition module 230 is used to input the feature matrix into the pre-trained belt conveyor state analysis model and acquire multiple channel output data of the belt conveyor state analysis model. The multiple channel output data includes at least belt conveyor state channel data, optical flow field channel data and instance characterization channel data.

[0069] The determination module 240 is used to determine the belt conveyor contained in the video to be detected based on the output data from multiple channels.

[0070] In one embodiment of this application, the processing module 220 is further configured to: extract several video segments from the video to be detected; extract multiple frames of images from each video segment; divide the multiple frames of images into color images and grayscale images; stack the color images and grayscale images into a feature matrix corresponding to the video segment; and thus obtain the feature matrix corresponding to the video to be detected.

[0071] In one embodiment of this application, the determining module 240 is further configured to: process the conveyor state channel data using an image segmentation algorithm to obtain the conveyor moving area and / or the conveyor stationary area; process the instance characterization channel data using a clustering algorithm to obtain multiple independent moving conveyor segments corresponding to the conveyor moving area, and / or multiple independent stationary conveyor segments corresponding to the conveyor stationary area; and process the optical flow field channel data using a mean algorithm to obtain the respective motion directions of the multiple independent moving conveyor segments.

[0072] In one embodiment of this application, the belt conveyor state analysis model includes multiple input channels, an encoder, a decoder, and multiple output channels; the encoder and decoder are connected in a skip connection; the input channels include a color image input channel and a grayscale image input channel; the multiple output channels include a belt conveyor state output channel, an optical flow field output channel, and an instance characterization output channel;

[0073] The belt conveyor condition analysis model is trained based on target sample data and the loss function corresponding to each output channel. The target sample data includes multiple video segments and instance annotation data and optical flow calculation data corresponding to each video segment.

[0074] In one embodiment of this application, the conveyor state output channel includes a background area output channel, a conveyor motion area output channel, and a conveyor stationary area output channel; the loss function corresponding to the conveyor state output channel is the cross-entropy loss function, which is used to calculate the difference between the output value and the supervision value of each of the background area output channel, the conveyor motion area output channel, and the conveyor stationary area output channel.

[0075] In one embodiment of this application, the optical flow field output channel includes a horizontal component output channel and a vertical component output channel; the loss function corresponding to the optical flow field output channel is a mean square error loss function, used to calculate the difference between the output value and the supervised value of each of the horizontal and vertical component output channels.

[0076] In one embodiment of this application, the instance representation channel includes a multi-dimensional vector used to distinguish the instance category to which the pixels in the animation belong; the loss function corresponding to the instance representation channel is a logistic function used to minimize the difference in pixel representation within the same instance and / or maximize the difference in pixel representation between different instances.

[0077] Below, for reference Figure 3 This describes an electronic device according to embodiments of the present application. Figure 3 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.

[0078] like Figure 3 As shown, the electronic device 30 includes one or more processors 301 and memory 302.

[0079] The processor 301 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 30 to perform desired functions.

[0080] The memory 302 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 301 may execute the program instructions to implement the methods of the various embodiments of this application described above and / or other desired functions. The computer-readable storage medium may also store various content such as the video to be detected, feature matrix, multiple channel output data, and conveyor belt status information.

[0081] In one example, the electronic device 30 may also include an input device 303 and an output device 304, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0082] The input device 303 may include, for example, a keyboard, a mouse, etc.

[0083] The output device 304 can output various information to the outside, including the video to be detected, feature matrix, multiple channel output data, conveyor belt status information, etc. The output device 304 may include, for example, a display, speaker, printer, and communication network and its connected remote output devices, etc.

[0084] Of course, for the sake of simplicity, Figure 3 Only some of the components of the electronic device 30 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 30 may include any other suitable components depending on the specific application.

[0085] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods described above according to various embodiments of this application.

[0086] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0087] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods described above according to various embodiments of this application.

[0088] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0089] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0090] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0091] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0092] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0093] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for analyzing the condition of a belt conveyor, characterized in that, include: Obtain the video to be tested; The video to be detected is processed to obtain the feature matrix corresponding to the video to be detected; The feature matrix is ​​input into a pre-trained belt conveyor state analysis model to obtain multiple channel output data of the belt conveyor state analysis model. The multiple channel output data includes at least belt conveyor state channel data, optical flow field channel data, and instance characterization channel data. Based on the output data from the multiple channels, the status information of the belt conveyor contained in the video to be detected is determined.

2. The belt conveyor condition analysis method according to claim 1, characterized in that, The process of processing the video to be detected to obtain the feature matrix corresponding to the video to be detected includes: Extract several video segments from the video to be detected; Multiple frames are extracted from each video segment, and the multiple frames are divided into color images and grayscale images; The color image and the grayscale image are stacked to form the feature matrix corresponding to the video segment, thereby obtaining the feature matrix corresponding to the video to be detected.

3. The belt conveyor condition analysis method according to claim 1, characterized in that, The step of determining the status information of the belt conveyor contained in the video to be detected based on the output data of the multiple channels includes: The image segmentation algorithm is used to process the belt conveyor status channel data to obtain the belt conveyor moving area and / or belt conveyor stationary area; Clustering algorithms are used to process the instance characterization channel data to obtain multiple independent moving belt conveyor segments corresponding to the moving area of ​​the belt conveyor, and / or multiple independent stationary belt conveyor segments corresponding to the stationary area of ​​the belt conveyor. The optical flow field channel data is processed using a mean value algorithm to obtain the motion direction of each of the multiple independent moving belt conveyor sections.

4. The belt conveyor condition analysis method according to any one of claims 1 to 3, characterized in that, The belt conveyor state analysis model includes multiple input channels, an encoder, a decoder, and multiple output channels; the encoder and the decoder are connected in a jump connection, and the input channels include a color image input channel and a grayscale image input channel; The multiple output channels include a belt conveyor status output channel, an optical flow field output channel, and an instance characterization output channel; The belt conveyor state analysis model is trained based on target sample data and the loss function corresponding to each output channel. The target sample data includes multiple video segments and instance annotation data and optical flow calculation data corresponding to each video segment.

5. The belt conveyor condition analysis method according to claim 4, characterized in that, The belt conveyor status output channel includes a background area output channel, a belt conveyor moving area output channel, and a belt conveyor stationary area output channel; The loss function corresponding to the belt conveyor status output channel is the cross-entropy loss function, which is used to calculate the difference between the output value and the supervision value of the background region output channel, the belt conveyor moving region output channel and the belt conveyor stationary region output channel.

6. The belt conveyor condition analysis method according to claim 4, characterized in that, The optical flow field output channel includes an optical flow field horizontal component output channel and an optical flow field vertical component output channel; The loss function corresponding to the optical flow field output channel is the mean square error loss function, which is used to calculate the difference between the output value and the supervision value of the horizontal component output channel and the vertical component output channel of the optical flow field.

7. The belt conveyor condition analysis method according to claim 4, characterized in that, The instance representation channel includes a multi-dimensional vector used to distinguish the instance category to which the pixels in the animation belong; The loss function corresponding to the instance representation channel is the logistic function, which is used to minimize the difference in pixel representation within the same instance and / or maximize the difference in pixel representation between different instances.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the belt conveyor condition analysis method according to any one of claims 1 to 7.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is used to execute the belt conveyor condition analysis method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes instructions that, when executed on an electronic device, cause the electronic device to implement the belt conveyor condition analysis method according to any one of claims 1 to 7.

Images