Coal state recognition method and device for scraper conveyer

Abstract

The invention provides a coal state identification method and device for a scraper conveyer. The method comprises the following steps: receiving video stream data; preprocessing a current image in the video stream data to obtain a current effective image; performing optical flow analysis on the current effective image and a previous frame image corresponding to the current effective image to obtain a motion amplitude feature corresponding to the current effective image; according to the motion amplitude feature corresponding to the current effective image, obtaining a flow area and a transmission band area corresponding to the current effective image; and according to the area of the flowing area corresponding to the current effective image and the projection length in the movement direction, the area of the conveying belt area corresponding to the current effective image and the projection length in the movement direction, and a state judgment rule, obtaining a coal state recognition result of the scraper conveyor. The device is used for executing the method. According to the coal state recognition method and device of the scraper conveyer, the reliability of coal state recognition is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent coal mines, and in particular to a method and device for identifying the coal state of a scraper conveyor. Background Art

[0002] The fully mechanized coal mining face is a key place for coal mining, and the scraper conveyor undertakes the important task of coal transportation. In the production process, due to many factors such as uneven coal cutting by the coal mining machine, coal wall shedding, and unreasonable transfer point design, it is easy to cause local coal accumulation on the scraper conveyor.

[0003] If the coal pile situation of the scraper conveyor is not discovered and handled in time, there is a risk of causing serious accidents such as conveyor belt tearing, coal spilling, motor overload, and even fire, and there is also a risk of endangering the lives of operators. At present, there are the following ways to monitor coal piles: (1) Manual inspection method, which requires personnel to patrol regularly. Not only is the labor intensity high and there are safety risks, but it is also impossible to achieve real-time monitoring; (2) Traditional sensor monitoring, which requires multiple sensors to be installed on the equipment, is easily affected by dust and has high maintenance costs; (3) Traditional visual inspection has poor anti-interference ability and unstable accuracy, and is difficult to adapt to the complex environment underground. Therefore, how to more reliably monitor the coal status of scraper conveyors has become an important issue that needs to be solved in this field. Summary of the invention

[0004] In view of the problems in the prior art, an embodiment of the present invention provides a method and device for identifying the coal state of a scraper conveyor, which can at least partially solve the problems in the prior art.

[0005] In a first aspect, the present invention provides a method for identifying a coal state of a scraper conveyor, comprising:

[0006] Receiving video stream data of a scraper conveyor;

[0007] Preprocessing the current image in the video stream data to obtain a current valid image;

[0008] Perform optical flow analysis on the current effective image and the previous frame image corresponding to the current effective image to obtain the motion amplitude feature corresponding to the current effective image;

[0009] According to the motion amplitude characteristics corresponding to the current effective image, the flow area and the transmission belt area corresponding to the current effective image are obtained;

[0010] The coal state recognition result of the scraper conveyor is obtained based on the area of the flow area corresponding to the current effective image and the projection length in the moving direction, the area of the conveyor belt area corresponding to the current effective image and the projection length in the moving direction, and the state judgment rules.

[0011] In a second aspect, the present invention provides a coal state recognition device for a scraper conveyor, comprising:

[0012] a receiving module, configured to receive video stream data of the scraper conveyor;

[0013] a preprocessing module, configured to preprocess the current image in the video stream data to obtain a current valid image;

[0014] an optical flow analysis module, configured to perform optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image to obtain a motion amplitude feature corresponding to the current valid image;

[0015] an obtaining module, configured to obtain a flow region and a conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image;

[0016] a recognition module, configured to obtain a coal state recognition result of the scraper conveyor according to the area and the projection length in the motion direction of the flow region corresponding to the current valid image, the area and the projection length in the motion direction of the conveyor belt region corresponding to the current valid image, and a state judgment rule.

[0017] In a third aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the program to implement the coal state recognition method for the scraper conveyor according to any one of the above embodiments.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program / instructions, and when the computer program / instructions are executed by a processor, the coal state recognition method for the scraper conveyor according to any one of the above embodiments is implemented.

[0019] In a fifth aspect, the present invention provides a computer program product comprising a computer program / instructions, and when the computer program / instructions are executed by a processor, the coal state recognition method for the scraper conveyor according to any one of the above embodiments is implemented.

[0020] The coal state recognition method and device for a scraper conveyor provided by an embodiment of the present invention receive video stream data of the scraper conveyor; preprocess the current image in the video stream data to obtain a current valid image; perform optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image to obtain the motion amplitude feature corresponding to the current valid image; obtain the flow region and the conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image; and obtain the coal state recognition result of the scraper conveyor according to the area and the projection length in the motion direction of the flow region corresponding to the current valid image, the area and the projection length in the motion direction of the conveyor belt region corresponding to the current valid image, and the state judgment rule, thereby improving the reliability of coal state recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0022] Figure 1 is a schematic flowchart of the coal state recognition method for a scraper conveyor provided by the first embodiment of the present invention.

[0023] Figure 2 is a schematic flowchart of the coal state recognition method for a scraper conveyor provided by the second embodiment of the present invention.

[0024] Figure 3 is a schematic flowchart of the coal state recognition method for a scraper conveyor provided by the third embodiment of the present invention.

[0025] Figure 4 is a schematic flowchart of the coal state recognition method for a scraper conveyor provided by the fourth embodiment of the present invention.

[0026] Figure 5 is a schematic structural diagram of the coal state recognition device for a scraper conveyor provided by the fifth embodiment of the present invention.

[0027] Figure 6 is a schematic structural diagram of the coal state recognition device for a scraper conveyor provided by the sixth embodiment of the present invention.

[0028] Figure 7 is a schematic structural diagram of the coal state recognition device for a scraper conveyor provided by the seventh embodiment of the present invention.

[0029] Figure 8It is a schematic structural diagram of a coal state recognition device for a scraper conveyor provided by the eighth embodiment of the present invention.

[0030] Figure 9 It is a schematic structural diagram of a coal state recognition device for a scraper conveyor provided by the ninth embodiment of the present invention.

[0031] Figure 10 It is a schematic structural diagram of a coal state recognition device for a scraper conveyor provided by the tenth embodiment of the present invention.

[0032] Figure 11 It is a schematic physical structure diagram of a computer device provided by the eleventh embodiment of the present invention. Detailed implementation manners

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer and more understandable, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the schematic embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In the technical solutions of the present application, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of laws and regulations. The user information in the embodiments of the present application is obtained through legal and compliant channels, and the acquisition, storage, use, processing, etc. of the user information are authorized and consented to by the customers.

[0034] The following takes the server as the execution subject to illustrate the specific implementation process of the coal state recognition method for the scraper conveyor provided by the embodiments of the present invention. It can be understood that the execution subject of the coal state recognition method for the scraper conveyor provided by the embodiments of the present invention is not limited to the server.

[0035] Figure 1 It is a flowchart of the coal state recognition method for the scraper conveyor provided by the first embodiment of the present invention. As Figure 1 shown, the coal state recognition method for the scraper conveyor provided by the embodiments of the present invention includes:

[0036] S101. Receive the video stream data of the scraper conveyor;

[0037] Specifically, multiple video acquisition devices can be configured for the scraper conveyor for monitoring. Each industrial camera will send the video of the scraper conveyor collected to the server in the form of a video stream. The server will receive the video stream data of the scraper conveyor. Among them, the video acquisition devices include, but are not limited to, industrial cameras.

[0038] S102. Preprocess the current image in the video stream data to obtain the current valid image;

[0039] Specifically, the video stream data includes a series of consecutive image frames, and the server preprocesses each frame of the image to obtain a valid image corresponding to each frame of the image. A frame of the image being processed by the server is used as the current image. The preprocessing includes, but is not limited to, scaling processing, Gamma correction, normalization processing, etc., which are set according to actual needs and are not limited in the embodiments of the present invention.

[0040] For example, scale the current image to a resolution of 360×480; then perform Gamma correction on the scaled image to enhance the image contrast; and then perform normalization processing on the Gamma-corrected image to eliminate the influence of light changes.

[0041] For example, the normalization processing may include grayscale conversion, histogram equalization processing, and contrast normalization processing. The grayscale conversion and histogram equalization processing first convert the color image into a grayscale image, and then enhance the contrast of the image through histogram equalization, making the grayscale distribution of the image more uniform, thereby reducing the influence of uneven illumination to a certain extent. The contrast normalization processing calculates the mean and standard deviation of the local area of the image, and then performs normalization processing on each pixel to make the contrast of the image more consistent in different local areas and reduce the influence of light changes.

[0042] S103. Perform optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image to obtain the motion amplitude feature corresponding to the current valid image;

[0043] Specifically, the server performs optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image, and can obtain the motion amplitude feature corresponding to the current valid image. The motion amplitude feature includes a motion direction and an amplitude feature, and the motion amplitude feature corresponding to the current valid image may include the motion direction and amplitude feature of each pixel point in the current valid image.

[0044] For example, the RAFT model can be used to perform optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image, calculate the motion features of adjacent frame images, and obtain the motion amplitude feature. The optical flow algorithm used for optical flow analysis can be selected from other algorithms such as FlowNet2, PWC-Net, etc., which are selected according to actual needs and are not limited in the embodiments of the present invention.

[0045] S104. Obtain the flow region and the conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image;

[0046] Specifically, the server identifies the flow area and the conveyor belt area in the current valid image according to the motion amplitude characteristics corresponding to the current valid image, and can obtain the flow area and the conveyor belt area corresponding to the current valid image. The flow area corresponding to the current valid image is the area where the coal is located on the conveyor belt of the scraper conveyor in the current valid image; the conveyor belt area corresponding to the current valid image is the area where the conveyor belt of the scraper conveyor is located in the current valid image.

[0047] S105. Obtain the coal state recognition result of the scraper conveyor according to the area and the projection length in the motion direction of the flow area corresponding to the current valid image, the area and the projection length in the motion direction of the conveyor belt area corresponding to the current valid image, and the state judgment rule.

[0048] Specifically, the server projects the flow area corresponding to the current valid image in the motion direction, and can obtain the projection length of the flow area corresponding to the current valid image in the motion direction. The server projects the conveyor belt area corresponding to the current valid image in the motion direction, and can obtain the projection length of the conveyor belt area corresponding to the current valid image in the motion direction. The server can obtain the area of the flow area corresponding to the current valid image based on the flow area corresponding to the valid image, and the server can obtain the area of the conveyor belt area corresponding to the current valid image based on the conveyor belt area corresponding to the valid image. The server combines the state judgment rule based on the area of the flow area corresponding to the current valid image, the area of the conveyor belt area corresponding to the current valid image, the projection length of the flow area corresponding to the current valid image in the motion direction, and the projection length of the conveyor belt area corresponding to the current valid image in the motion direction, and can obtain the coal state recognition result of the scraper conveyor. The coal state recognition result can be divided into a normal state, a blocked state, and a risk of coal accumulation. The normal state indicates that the coal on the conveyor belt of the scraper conveyor is being transported normally; the blocked state indicates that the coal on the conveyor belt of the scraper conveyor has accumulated; the risk of coal accumulation indicates that there is a risk of coal accumulation on the conveyor belt of the scraper conveyor. Among them, the state judgment rule is preset and is set according to the actual situation, which is not limited in the embodiments of the present invention.

[0049] The coal state recognition method for a scraper conveyor provided by an embodiment of the present invention receives video stream data of the scraper conveyor; preprocesses the current image in the video stream data to obtain a current valid image; performs optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image to obtain the motion amplitude feature corresponding to the current valid image; obtains the flow region and the conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image; and obtains the coal state recognition result of the scraper conveyor according to the area of the flow region corresponding to the current valid image and the projection length in the motion direction, the area of the conveyor belt region corresponding to the current valid image and the projection length in the motion direction, and the state judgment rule, improving the reliability of coal state recognition.

[0050] Figure 2 is a schematic flowchart of the coal state recognition method for a scraper conveyor provided by the second embodiment of the present invention. As Figure 2 shown, on the basis of the above embodiments, further, the obtaining the flow region and the conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image includes:

[0051] S201. Obtain the motion region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image and a first threshold;

[0052] Specifically, the motion amplitude feature corresponding to the current valid image includes the motion amplitude feature of each pixel point in the current valid image, and the motion amplitude feature of the pixel point reflects the motion amplitude of the pixel. The server compares the motion amplitude feature of each pixel point with the first threshold. If the motion amplitude feature of the pixel point is greater than the first threshold, it indicates that the pixel point is in a motion state, and the pixel point is used as a pixel point in the motion region. If the motion amplitude feature of the pixel point is less than or equal to the first threshold, the pixel point is regarded as in a stationary state and will not be used as a pixel point in the motion region. After the motion amplitude features of all pixel points in the current valid image are compared with the first threshold, all pixel points in the motion state constitute the motion region corresponding to the current valid image. The first threshold is set according to actual experience and is not limited in the embodiment of the present invention.

[0053] S202. If it is judged that the area of the motion region corresponding to the current valid image is less than or equal to the area threshold and the image frame corresponding to the current valid image is not the first frame, obtain the weighted fusion feature corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image and the historical motion amplitude feature corresponding to the current valid image; wherein, the area threshold is determined based on the area of the current valid image;

[0054] Specifically, based on the motion area corresponding to the current valid image, the server can obtain the area of the motion area corresponding to the current valid image. Then, the area of the motion area corresponding to the current valid image is compared with the area threshold. If the area of the motion area corresponding to the current valid image is less than or equal to the area threshold, it indicates that the video capture device corresponding to the current valid image did not move when capturing the image frame corresponding to the current valid image. If the area of the motion area corresponding to the current valid image is greater than the area threshold, it indicates that the video capture device corresponding to the current valid image moved within a large range when capturing the image frame corresponding to the current valid image, and the historical motion amplitude feature is no longer applicable. Among them, the area threshold is determined based on the area of the current valid image. For example, 70% of the area of the current valid image is used as the area threshold.

[0055] For the image frame corresponding to the current valid image, it will be determined whether the image frame corresponding to the current valid image is the first frame. The first frame refers to the image frame corresponding to the first picture captured by the video capture device corresponding to the current valid image when switching to the current task, or the image frame corresponding to the first picture captured after initializing the parameters in the case of the video capture device moving after switching to the current task (the method for determining the movement of the video capture device is: the area of the motion area corresponding to the current valid image is greater than the area threshold); the current task refers to the task of judging the coal state of the scraper conveyor. If the image frame corresponding to the current valid image is the first frame, then the historical motion amplitude feature corresponding to the current valid image does not exist. If the image frame corresponding to the current valid image is not the first frame, then the current valid image has a corresponding historical motion amplitude feature. The historical motion amplitude feature corresponding to the current valid image refers to the motion amplitude feature obtained based on the motion amplitude feature corresponding to the first frame of valid image to the motion amplitude feature corresponding to the previous frame of valid image corresponding to the current valid image. For example, the average value of each motion amplitude feature from the motion amplitude feature corresponding to the first frame of valid image to the motion amplitude feature corresponding to the previous frame of valid image corresponding to the current valid image is calculated as the historical motion amplitude feature corresponding to the current valid image.

[0056] After the server determines that the area of the motion region corresponding to the current valid image is less than or equal to the area threshold and the image frame corresponding to the current valid image is not the first frame, it will obtain the historical motion amplitude feature corresponding to the current valid image, and then calculate the sum of the products of the motion amplitude feature corresponding to the current valid image and the historical motion amplitude feature corresponding to the current valid image and their respective weights as the weighted fusion feature corresponding to the current valid image. The weights of the motion amplitude feature corresponding to the current valid image and the historical motion amplitude feature corresponding to the current valid image are set according to actual experience and are not limited in the embodiments of the present invention. By using the motion amplitude feature corresponding to the current valid image and the historical motion amplitude feature corresponding to the current valid image, the weighted fusion feature corresponding to the current valid image is calculated to reduce the influence of random motion and personnel interference and improve the stability of recognition.

[0057] For example, if the motion amplitude feature corresponding to the current valid image is a and the corresponding weight is 0.4, and the historical motion amplitude feature corresponding to the current valid image is b and the corresponding weight is 0.6, then the weighted fusion feature corresponding to the current valid image is: 0.4a + 0.6b.

[0058] S203. Obtain the pending conveyor belt area corresponding to the current valid image according to the weighted fusion feature corresponding to the current valid image and the second threshold; wherein, the second threshold is greater than the first threshold.

[0059] Specifically, the weighted fusion feature corresponding to the current valid image includes the weighted fusion feature of each pixel point in the current valid image. The server compares the weighted fusion feature of each pixel point in the current valid image with the second threshold. If the weighted fusion feature of the pixel point is greater than the second threshold, then the pixel point is used as a pixel point of the pending conveyor belt area. After the weighted fusion features of all pixel points in the current valid image have been compared with the second threshold, all pixel points with weighted fusion features greater than the second threshold constitute the pending conveyor belt area corresponding to the current valid image. The second threshold is set according to actual experience and is not limited in the embodiments of the present invention. The second threshold is greater than the first threshold.

[0060] S204. Obtain the conveyor belt area corresponding to the current valid image according to the pending conveyor belt area corresponding to the current valid image and the constant conveyor belt area; wherein, the constant conveyor belt area is obtained in advance.

[0061] Specifically, after the server obtains the pending conveyor belt area corresponding to the current valid image, it takes the union of the pending conveyor belt area corresponding to the current valid image and the constant conveyor belt area corresponding to the current valid image, that is, combines the pending conveyor belt area and the constant conveyor belt area, to obtain the conveyor belt area corresponding to the current valid image. Among them, the constant conveyor belt area corresponding to the current valid image is obtained in advance. The constant conveyor belt area is the determined conveyor belt area. By correcting the pending conveyor belt area with the constant conveyor belt area, the conveyor belt area corresponding to the current valid image can be determined more accurately.

[0062] For example, the constant conveyor belt area can be determined by the pending conveyor belt areas of consecutive n frames of images. The part that is the pending conveyor belt area in all of the consecutive n frames of images is used as the pending conveyor belt area. The value of n is set according to actual needs and is not limited in the embodiments of the present invention.

[0063] S205. If it is judged that the cumulative number of frames corresponding to the current valid image is greater than the frame number threshold, then according to the conveyor belt area corresponding to the current valid image and the motion area corresponding to the current valid image, obtain the flow area corresponding to the current valid image.

[0064] Specifically, the server compares the cumulative number of frames corresponding to the current valid image with the frame number threshold. If the cumulative number of frames corresponding to the current valid image is greater than the frame number threshold, then based on the conveyor belt area corresponding to the current valid image and the motion area corresponding to the current valid image, the flow area corresponding to the current valid image can be obtained. Among them, the cumulative number of frames corresponding to the current valid image refers to the total number of frames from the first frame to the image frame corresponding to the current valid image.

[0065] For example, the intersection of the conveyor belt area corresponding to the current valid image and the motion area corresponding to the current valid image can be taken, and the area jointly owned by the conveyor belt area corresponding to the current valid image and the motion area corresponding to the current valid image is used as the flow area corresponding to the current valid image.

[0066] On the basis of the above embodiments, further, the coal state recognition method for a scraper conveyor provided by the embodiments of the present invention further includes:

[0067] If it is judged that the area of the motion area corresponding to the current valid image is greater than the area threshold, then initialize the coal state recognition parameters.

[0068] Specifically, the server compares the area of the motion region corresponding to the current valid image with the area threshold. If the area of the motion region corresponding to the current valid image is greater than the area threshold, it indicates that the video acquisition device corresponding to the current valid image has moved within a large range when capturing the image frame corresponding to the current valid image. In this case, it is necessary to initialize the coal state recognition parameters of the video acquisition device corresponding to the current valid image. By comparing the area of the motion region corresponding to the current valid image with the area threshold and initializing the coal state recognition parameters when the area is greater than the area threshold, it is possible to effectively avoid the interference of the large-range movement of the video acquisition device on coal state recognition. The coal state recognition parameters include historical motion amplitude features, cumulative number of frames, the first frame, etc.

[0069] Figure 3 It is a schematic flowchart of the coal state recognition method for a scraper conveyor provided in the third embodiment of the present invention. As Figure 3 shown, on the basis of the above embodiments, further, the coal state recognition method for a scraper conveyor provided by an embodiment of the present invention further includes:

[0070] S301: If it is determined that the area of the motion region corresponding to the current valid image is less than or equal to the area threshold and the image frame corresponding to the current valid image is the first frame, then based on the current valid image and the semantic segmentation model, obtain the recognized conveyor belt region of the current valid image; wherein, the semantic segmentation model is established in advance;

[0071] Specifically, after the server determines that the area of the motion region corresponding to the current valid image is less than or equal to the area threshold and the image frame corresponding to the current valid image is the first frame, it will input the current valid image into the semantic segmentation model to obtain the recognized conveyor belt region of the current valid image, that is, the region where the conveyor belt is located in the current valid image recognized by the semantic segmentation model.

[0072] Among them, the semantic segmentation model is established in advance. The semantic segmentation model can be trained based on the historical conveyor belt video data of the scraper conveyor through a deep learning model. For example, the deep learning model adopts the DeepLab model. It is also possible to use other methods disclosed in the prior art to establish a semantic segmentation model capable of recognizing the conveyor belt region and select according to actual needs. The embodiments of the present invention do not make any limitations.

[0073] S302: Compare the recognized conveyor belt region with each historical conveyor belt region to obtain the overlap degree between the recognized conveyor belt region and each historical conveyor belt region; wherein, the historical conveyor belt regions are stored in advance;

[0074] Specifically, the server compares the identified conveyor belt area with each historical conveyor belt area and calculates the overlap degree between the identified conveyor belt area and each historical conveyor belt area. The overlap degree is used to determine the similarity between the identified conveyor belt area and the historical conveyor belt area.

[0075] S303. If it is judged that the overlap degree between the identified conveyor belt area and the historical conveyor belt area is greater than the overlap degree threshold, then the flow area and the conveyor belt area corresponding to the historical conveyor belt area with an overlap degree greater than the overlap degree threshold are used as the flow area and the conveyor belt area corresponding to the current valid image.

[0076] Specifically, the server compares the overlap degree between the identified conveyor belt area and the historical conveyor belt area with the overlap degree threshold. If the overlap degree between the identified conveyor belt area and the historical conveyor belt area is greater than the overlap degree threshold, then the flow area and the conveyor belt area corresponding to the historical conveyor belt area with an overlap degree greater than the overlap degree threshold are used as the flow area and the conveyor belt area corresponding to the current valid image.

[0077] By comparing the identified conveyor belt area with the historical conveyor belt area to obtain the flow area and the conveyor belt area corresponding to the historical conveyor belt area with an overlap degree greater than the overlap degree threshold as the flow area and the conveyor belt area corresponding to the current valid image, the flow area and the conveyor belt area corresponding to the current valid image can be quickly obtained, improving the efficiency of coal state recognition for the scraper conveyor.

[0078] Based on the above embodiments, further, the step of using the flow area and the conveyor belt area corresponding to the historical conveyor belt area with an overlap degree greater than the overlap degree threshold as the flow area and the conveyor belt area corresponding to the current valid image includes:

[0079] Using the flow area and the conveyor belt area corresponding to the first historical conveyor belt area with an overlap degree greater than the overlap degree threshold as the flow area and the conveyor belt area corresponding to the current valid image.

[0080] Specifically, when there are multiple overlap degrees between the identified conveyor belt area and the historical conveyor belt area, the server will compare the overlap degree between the identified conveyor belt area and the historical conveyor belt area with the overlap degree threshold one by one. When the first overlap degree greater than the overlap degree threshold is obtained, the flow area and the conveyor belt area corresponding to the first historical conveyor belt area with an overlap degree greater than the overlap degree threshold are used as the flow area and the conveyor belt area corresponding to the current valid image, which can avoid comparing the overlap degrees between the remaining uncompared identified conveyor belt areas and the historical conveyor belt areas with the overlap degree threshold, thus facilitating the improvement of the recognition efficiency of the coal state of the scraper conveyor.

[0081] Based on the above embodiments, further, the step of using the flow region and the conveyor belt region corresponding to the historical conveyor belt region with an overlap degree greater than the overlap degree threshold as the flow region and the conveyor belt region corresponding to the current valid image includes:

[0082] If there are multiple overlap degrees greater than the overlap degree threshold, use the flow region and the conveyor belt region corresponding to the historical conveyor belt region with the largest overlap degree as the flow region and the conveyor belt region corresponding to the current valid image.

[0083] Specifically, the server traverses the overlap degree between the recognized conveyor belt region corresponding to the current valid image and each historical conveyor belt region, and compares each overlap degree with the overlap degree threshold. If there are multiple overlap degrees greater than the overlap degree threshold, then the largest overlap degree will be compared from these multiple overlap degrees greater than the overlap degree threshold, and the historical conveyor belt region corresponding to the largest overlap degree, that is, the historical conveyor belt region with the largest overlap degree, will be obtained. Use the flow region and the conveyor belt region corresponding to the historical conveyor belt region with the largest overlap degree as the flow region and the conveyor belt region corresponding to the current valid image. By obtaining the flow region and the conveyor belt region corresponding to the historical conveyor belt region with the largest overlap degree as the flow region and the conveyor belt region corresponding to the current valid image, it is beneficial to improve the accuracy of coal state recognition of the scraper conveyor.

[0084] Based on the above embodiments, further, the coal state recognition method for the scraper conveyor provided by the embodiments of the present invention further includes:

[0085] If it is judged that the overlap degree between the recognized conveyor belt region and the historical conveyor belt region is less than or equal to the overlap degree threshold, initialize the coal state recognition parameters.

[0086] Specifically, the server compares the overlap degree between the recognized conveyor belt region and the historical conveyor belt region with the overlap degree threshold. If the overlap degree between the recognized conveyor belt region and the historical conveyor belt region is less than or equal to the overlap degree threshold, it indicates that the recognized conveyor belt region corresponding to the current valid image is quite different from the historical conveyor belt region, and the coal state recognition parameters of the video acquisition device corresponding to the current valid image will be initialized. The coal state recognition parameters include historical motion amplitude characteristics, cumulative number of frames, first frame, etc.

[0087] Based on the above embodiments, further, the coal state recognition method for the scraper conveyor provided by the embodiments of the present invention further includes:

[0088] If it is judged that the cumulative number of frames corresponding to the current valid image is less than the frame number threshold, obtain the next frame of image as the current image.

[0089] Specifically, the server compares the cumulative number of frames corresponding to the current valid image with the frame number threshold. If the cumulative number of frames is less than the frame number threshold, the server will obtain the next frame image as the current image and perform the processing from step S101 to step S105. The frame number threshold is set according to actual needs and is not limited in the embodiments of the present invention.

[0090] Based on the above embodiments, further, the obtaining of the coal state recognition result of the scraper conveyor according to the area and the projection length in the movement direction of the flow region corresponding to the current valid image, the area and the projection length in the movement direction of the conveyor belt region corresponding to the current valid image, and the state judgment rule includes:

[0091] If it is judged that the first ratio of the area of the flow region corresponding to the current valid image to the area of the conveyor belt region corresponding to the current valid image is greater than the first preset value and the second ratio of the projection length of the flow region corresponding to the current valid image in the movement direction to the projection length of the conveyor belt region corresponding to the current valid image in the movement direction is greater than the second preset value, then the coal state recognition result is normal;

[0092] If it is judged that the first ratio is less than the third preset value and the second ratio is less than the fourth preset value, then the coal state recognition result is blocked;

[0093] If it is judged that the first ratio is greater than or equal to the third preset value and less than or equal to the first preset value, and the second ratio is greater than or equal to the fourth preset value and less than or equal to the second preset value, then the coal state recognition result is that there is a risk of blockage.

[0094] Specifically, the server calculates the first ratio of the area of the flow region corresponding to the current valid image to the area of the conveyor belt region corresponding to the current valid image based on the area of the flow region corresponding to the current valid image and the area of the conveyor belt region corresponding to the current valid image. The server calculates the second ratio of the projection length of the flow region corresponding to the current valid image in the movement direction to the projection length of the conveyor belt region corresponding to the current valid image in the movement direction based on the projection length of the flow region corresponding to the current valid image in the movement direction and the projection length of the conveyor belt region corresponding to the current valid image in the movement direction.

[0095] If the first ratio is greater than the first preset value and the second ratio is greater than the second preset value, then the coal state recognition result is normal. If the first ratio is less than the third preset value and the second ratio is less than the fourth preset value, then the coal state recognition result is blocked. If the first ratio is greater than or equal to the third preset value and less than or equal to the first preset value, and the second ratio is greater than or equal to the fourth preset value and less than or equal to the second preset value, then the coal state recognition result is that there is a risk of blockage.

[0096] The motion direction of the current valid image can be determined by the directions of the optical flow vectors of the respective pixel points of the current valid image.

[0097] Among them, the first preset value, the second preset value, the third preset value, and the fourth preset value are set according to actual experience, and are not limited in the embodiments of the present invention.

[0098] For example, the first preset value is taken as 0.6, the second preset value is taken as 0.9, the third preset value is taken as 0.2, and the third preset value is taken as 0.5.

[0099] The following uses a specific embodiment to illustrate the specific implementation process of the coal state recognition method for the scraper conveyor provided by the embodiments of the present invention.

[0100] In a fully mechanized coal mining face of a certain coal mine, coal is transported by a scraper conveyor. An industrial camera is set for the scraper conveyor to monitor the coal state, and the frame rate of video acquisition by the industrial camera is 6fps. Each industrial camera will send the collected video to the server, and the server will perform coal state recognition based on the coal state recognition method for the scraper conveyor provided by the embodiments of the present invention. The following takes the video data collected by industrial camera A as an example for illustration.

[0101] As Figure 4 shown, the specific process of the server performing coal state recognition based on the video data collected by industrial camera A is as follows:

[0102] The first step is to receive video stream data. Industrial camera A collects the video data of the scraper conveyor and sends it to the server in the form of a video, and the server will receive the video stream data. The server will obtain a frame of image from the video stream data as the current image.

[0103] The second step is to perform preprocessing. The server preprocesses the current image, including scaling the current image to a resolution of 360×480, then performing Gamma correction, and finally performing normalization processing.

[0104] The third step is to perform optical flow analysis. After the server preprocesses the current image, it will obtain the current valid image corresponding to the current image, and then perform optical flow analysis based on the current valid image and the previous frame image corresponding to the current valid image, and can obtain the motion amplitude feature corresponding to the current valid image.

[0105] The fourth step is to obtain the motion area. The server obtains the motion area corresponding to the current valid image based on the motion amplitude feature corresponding to the current valid image and the first threshold.

[0106] Step 5: Determine the proportion of the moving area. The server determines whether the area of the moving area corresponding to the current valid image is greater than the area threshold. If the area of the moving area is greater than the area threshold, then proceed to Step 12; if the area of the moving area is less than or equal to the area threshold, then proceed to Step 6. The area threshold is set to 70% of the area of the current valid image.

[0107] Step 6: Determine whether it is the first frame. The server determines whether the image frame corresponding to the current valid image is the first frame. If the image frame corresponding to the current valid image is not the first frame, then proceed to Step 7; if the image frame corresponding to the current valid image is the first frame, then proceed to Step 10.

[0108] Step 7: Calculate the weighted fusion feature. The server calculates the sum of the product of the motion amplitude feature corresponding to the current valid image and the corresponding weight of 0.4, and the product of the historical motion amplitude feature corresponding to the current valid image and the corresponding weight of 0.6, to obtain the weighted fusion feature.

[0109] Step 8: Obtain the conveyor belt area. The server obtains the pending conveyor belt area corresponding to the current valid image based on the weighted fusion feature corresponding to the current valid image and the second threshold, and then takes the union of the pending conveyor belt area corresponding to the current valid image and the constant conveyor belt area corresponding to the current valid image, that is, merges the pending conveyor belt area and the constant conveyor belt area, to obtain the conveyor belt area corresponding to the current valid image. The constant conveyor belt area is pre-obtained, and the pending conveyor belt areas corresponding to consecutive n frames of images are used to obtain the constant conveyor belt area. It is understandable that if the constant conveyor belt area is empty, then the pending conveyor belt area corresponding to the current valid image will be directly used as the conveyor belt area corresponding to the current valid image.

[0110] Step 9: Determine whether the cumulative number of frames is greater than 50. The server starts accumulating the number of frames from the first frame to obtain the cumulative number of frames, and compares the cumulative number of frames with 50. If the cumulative number of frames is greater than 50, then proceed to Step 14; if the cumulative number of frames is less than or equal to 50, then return to Step 1 to obtain the next frame of the current image for processing. The identification of the flowing area is only carried out when the cumulative number of frames is greater than 50, in order to smooth the optical flow result and filter out the interference of the optical flow results generated by objects moving for a short time on the identification of the conveyor belt area. Objects moving for a short time include, for example: people, water mist, lights, etc.

[0111] Step 10: Perform semantic recognition. The server will identify the conveyor belt area in the current valid image according to the current valid image and the semantic segmentation model, to obtain the identified conveyor belt area of the current valid image.

[0112] Step 11: Determine whether the overlap degree is greater than 80%. The server compares the recognized conveyor belt area of the current valid image with each historical conveyor belt area, and calculates the overlap degree between the recognized conveyor belt area and each historical conveyor belt area. Then, it compares the calculated overlap degree with 80%. If there is an overlap degree greater than 80%, then proceed to Step 13; if there is no overlap degree greater than 80%, then proceed to Step 12. The historical conveyor belt areas are pre-stored.

[0113] Step 12: Initialize parameters. Since it is the first-frame image and there is no historical conveyor belt area with a high overlap degree available for utilization, the historical motion amplitude feature, the cumulative number of frames, and the first frame corresponding to industrial camera A will be initialized.

[0114] Step 13: Reuse historical data. The flow area and the conveyor belt area corresponding to the historical conveyor belt area with an overlap degree greater than the overlap degree threshold are used as the flow area and the conveyor belt area corresponding to the current valid image.

[0115] Step 14: Obtain the flow area. The server finds the intersection of the conveyor belt area corresponding to the current valid image and the motion area corresponding to the current valid image, and uses the area that is common to the conveyor belt area corresponding to the current valid image and the motion area corresponding to the current valid image as the flow area corresponding to the current valid image.

[0116] Step 15: Calculate motion indicators. The server calculates the ratio of the area of the flow area corresponding to the current valid image to the area of the conveyor belt area corresponding to the current valid image as the first ratio; the server calculates the ratio of the projected length of the flow area corresponding to the current valid image in the motion direction to the projected length of the conveyor belt area corresponding to the current valid image in the motion direction as the second ratio. The first ratio and the second ratio are used as motion indicators.

[0117] Step 16: Identify the coal state. If the first ratio is greater than 0.6 and the second ratio is greater than 0.9, then the coal state recognition result is normal; if the first ratio is greater than or equal to 0.2 and less than or equal to 0.6, and the second ratio is greater than 0.5 and less than or equal to 0.9, then the coal state recognition result is that there is a risk of blockage, and a risk prompt can be given; if the first ratio is less than 0.2 and the second ratio is less than 0.5, then the coal state recognition result is blockage, and a blockage warning is required.

[0118] The present invention realizes the intelligent monitoring of the coal stacking state in the fully mechanized coal mining face, and while improving the detection accuracy and reliability, significantly improves the anti-interference ability and operation efficiency.

[0119] This application first applies the optical flow algorithm to the coal stacking monitoring in fully mechanized coal mining faces. By analyzing the motion characteristics of coal on the conveyor belt, this method judges coal stacking, creating a brand-new monitoring idea. Compared with traditional sensor monitoring and static image analysis, this method based on motion analysis can more accurately and reliably reflect the coal stacking state, with significant technological breakthroughs and innovation value.

[0120] This application constructs a coal stacking state evaluation mechanism with multiple criteria. This mechanism comprehensively considers three indicators: the projection length ratio, area ratio, and motion direction consistency of the motion area, and establishes a hierarchical early warning standard. The system also designs a parameter dynamic adjustment mechanism, which can automatically optimize the judgment parameters according to the operating state, ensuring the accuracy and adaptability of detection.

[0121] This application proposes a motion feature analysis method based on the RAFT optical flow algorithm. This method not only calculates the motion information between adjacent frames but also designs a temporal smoothing mechanism. By weighted fusion of the motion characteristics of multiple consecutive frames, the stability of analysis is effectively improved. At the same time, the system effectively filters out interferences such as personnel activities through global motion monitoring and direction consistency judgment.

[0122] This application designs a fast verification method based on deep learning. This method uses the DeepLab semantic segmentation model to identify the conveyor belt area and innovatively introduces a historical parameter verification mechanism. By calculating the area overlap degree, it judges the usability of historical parameters. When the overlap degree exceeds the preset threshold, the system directly reuses the historical parameters, significantly improving the system startup efficiency.

[0123] Figure 5 It is a schematic structural diagram of a coal state recognition device for a scraper conveyor provided by the fifth embodiment of the present invention, as Figure 5 shown, the coal state recognition device for a scraper conveyor provided by the embodiment of the present invention includes a receiving module 501, a preprocessing module 502, an optical flow analysis module 503, an obtaining module 504, and an identification module 505, where:

[0124] The receiving module 501 is used to receive the video stream data of the scraper conveyor; the preprocessing module 502 is used to preprocess the current image in the video stream data to obtain the current valid image; the optical flow analysis module 503 is used to perform optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image to obtain the motion amplitude feature corresponding to the current valid image; the obtaining module 504 is used to obtain the flow region and the conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image; the recognition module 505 is used to obtain the coal state recognition result of the scraper conveyor according to the area of the flow region corresponding to the current valid image and the projection length in the motion direction, the area of the conveyor belt region corresponding to the current valid image and the projection length in the motion direction, and the state judgment rule.

[0125] Specifically, multiple video acquisition devices can be configured for the scraper conveyor for monitoring. Each industrial camera will send the video of the scraper conveyor collected to the receiving module 501 in the form of a video stream. The receiving module 501 will receive the video stream data of the scraper conveyor. Among them, the video acquisition devices include but are not limited to industrial cameras.

[0126] The video stream data includes a series of consecutive image frames. The preprocessing module 502 will preprocess each frame of image to obtain the valid image corresponding to each frame of image. The frame of image being processed by the preprocessing module 502 is used as the current image. The preprocessing includes but is not limited to scaling processing, Gamma correction, normalization processing, etc., which are set according to actual needs and are not limited in the embodiments of the present invention.

[0127] The optical flow analysis module 503 performs optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image, and can obtain the motion amplitude feature corresponding to the current valid image. The motion amplitude feature includes the motion direction and the amplitude feature. The motion amplitude feature corresponding to the current valid image may include the motion direction and the amplitude feature of each pixel point in the current valid image.

[0128] The obtaining module 504 identifies the flow region and the conveyor belt region in the current valid image according to the motion amplitude feature corresponding to the current valid image, and can obtain the flow region and the conveyor belt region corresponding to the current valid image. The flow region corresponding to the current valid image is the region where the coal is located on the conveyor belt of the scraper conveyor in the current valid image; the conveyor belt region corresponding to the current valid image is the region where the conveyor belt of the scraper conveyor is located in the current valid image.

[0129] The recognition module 505 projects the flow region corresponding to the current valid image in the motion direction, and the projected length of the flow region corresponding to the current valid image in the motion direction can be obtained. The recognition module 505 projects the conveyor belt region corresponding to the current valid image in the motion direction, and the projected length of the conveyor belt region corresponding to the current valid image in the motion direction can be obtained. The recognition module 505 can obtain the area of the flow region corresponding to the current valid image based on the flow region corresponding to the valid image. The server can obtain the area of the conveyor belt region corresponding to the current valid image based on the conveyor belt region corresponding to the valid image. Based on the area of the flow region corresponding to the current valid image, the area of the conveyor belt region corresponding to the current valid image, the projected length of the flow region corresponding to the current valid image in the motion direction, and the projected length of the conveyor belt region corresponding to the current valid image in the motion direction, and combining the state judgment rule, the coal state recognition result of the scraper conveyor can be obtained. The coal state recognition result can be divided into a normal state, a blocked state, and a risk of coal accumulation. The normal state indicates that the coal on the conveyor belt of the scraper conveyor is being transported normally; the blocked state indicates that the coal on the conveyor belt of the scraper conveyor has accumulated; the risk of coal accumulation indicates that there is a risk of coal accumulation on the conveyor belt of the scraper conveyor. Among them, the state judgment rule is preset and is set according to the actual situation, and the embodiments of the present invention do not make limitations.

[0130] The coal state recognition device of the scraper conveyor provided in the embodiment of the present invention receives the video stream data of the scraper conveyor; preprocesses the current image in the video stream data to obtain the current valid image; performs optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image to obtain the motion amplitude feature corresponding to the current valid image; obtains the flow region and the conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image; and obtains the coal state recognition result of the scraper conveyor according to the area and the projected length in the motion direction of the flow region corresponding to the current valid image, the area and the projected length in the motion direction of the conveyor belt region corresponding to the current valid image, and the state judgment rule, improving the reliability of coal state recognition.

[0131] Figure 6 is a schematic structural diagram of the coal state recognition device of the scraper conveyor provided in the sixth embodiment of the present invention, as Figure 6 shown. On the basis of the above embodiments, further, the obtaining module 504 includes a first obtaining unit 5041, a second obtaining unit 5042, a third obtaining unit 5043, a fourth obtaining unit 5044, and a fifth obtaining unit 5045, where:

[0132] The first acquisition unit 5041 is configured to acquire a motion region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image and a first threshold value; the second acquisition unit 5042 is configured to, if it is determined that the area of the motion region corresponding to the current valid image is less than or equal to an area threshold value and the image frame corresponding to the current valid image is not the first frame, acquire a weighted fusion feature corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image and the historical motion amplitude feature corresponding to the current valid image; wherein, the area threshold value is determined based on the area of the current valid image; the third acquisition unit 5043 is configured to acquire a pending conveyor belt region corresponding to the current valid image according to the weighted fusion feature corresponding to the current valid image and a second threshold value; the fourth acquisition unit 5044 is configured to acquire a conveyor belt region corresponding to the current valid image according to the pending conveyor belt region corresponding to the current valid image and a constant conveyor belt region; wherein, the constant conveyor belt region is acquired in advance; the fifth acquisition unit 5045 is configured to, after it is determined that the cumulative number of frames corresponding to the current valid image is greater than a frame threshold value, acquire a flow region corresponding to the current valid image according to the conveyor belt region corresponding to the current valid image and the motion region corresponding to the current valid image.

[0133] Figure 7 It is a schematic structural diagram of a coal state recognition device for a scraper conveyor provided by the seventh embodiment of the present invention, as Figure 7 shown. On the basis of the above embodiments, further, the acquisition module 504 further includes an initialization unit 5046, wherein:

[0134] The initialization unit 5046 is configured to initialize the coal state recognition parameters after it is determined that the area of the motion region corresponding to the current valid image is greater than the area threshold value.

[0135] Figure 8 It is a schematic structural diagram of a coal state recognition device for a scraper conveyor provided by the eighth embodiment of the present invention, as Figure 8 shown. On the basis of the above embodiments, further, the acquisition module 504 further includes a first judgment unit 5047, a comparison unit 5048 and a second judgment unit 5049, wherein:

[0136] The first judgment unit 5047 is configured to, after judging that the area of the motion region corresponding to the current valid image is less than or equal to the area threshold and the image frame corresponding to the current valid image is the first frame, obtain the recognized conveyor belt region of the current valid image based on the current valid image and the semantic segmentation model; the comparison unit 5048 is configured to compare the recognized conveyor belt region with each historical conveyor belt region to obtain the overlap degree between the recognized conveyor belt region and each historical conveyor belt region; wherein, the historical conveyor belt regions are pre-stored; the second judgment unit 5049 is configured to, after judging that the overlap degree between the recognized conveyor belt region and the historical conveyor belt region is greater than the overlap degree threshold, use the flow region and the conveyor belt region corresponding to the historical conveyor belt region with the overlap degree greater than the overlap degree threshold as the flow region and the conveyor belt region corresponding to the current valid image.

[0137] Figure 9 is a schematic structural diagram of a coal state recognition device for a scraper conveyor provided by the ninth embodiment of the present invention, as Figure 9 shown. On the basis of the above embodiments, further, the obtaining module 504 further includes a third judgment unit 50410, wherein:

[0138] The third judgment unit 50410 is configured to initialize the coal state recognition parameters if it is judged that the overlap degree between the recognized conveyor belt region and the historical conveyor belt region is less than or equal to the overlap degree threshold.

[0139] Figure 10 is a schematic structural diagram of a coal state recognition device for a scraper conveyor provided by the tenth embodiment of the present invention, as Figure 10 shown. On the basis of the above embodiments, further, the obtaining module 504 further includes an obtaining unit 50411, wherein:

[0140] The obtaining unit 50411 is configured to obtain the next frame of image as the current image after judging that the cumulative number of frames corresponding to the current valid image is less than the frame number threshold.

[0141] On the basis of the above embodiments, further, the state judgment rule includes:

[0142] If it is judged that the first ratio of the area of the flow region corresponding to the current valid image to the area of the conveyor belt region corresponding to the current valid image is greater than the first preset value and the second ratio of the projection length of the flow region corresponding to the current valid image in the motion direction to the projection length of the conveyor belt region corresponding to the current valid image in the motion direction is greater than the second preset value, then the coal state recognition result is normal;

[0143] If it is judged that the first ratio is less than the third preset value and the second ratio is less than the fourth preset value, then the coal state recognition result is blocked;

[0144] If it is determined that the first ratio is greater than or equal to the third preset value and less than or equal to the first preset value, and the second ratio is greater than or equal to the fourth preset value and less than or equal to the second preset value, then the coal state recognition result is that there is a risk of blockage.

[0145] The embodiments of the device provided by the embodiments of the present invention can specifically be used to execute the processing procedures of the above method embodiments, and their functions will not be elaborated here. Reference can be made to the detailed descriptions of the above method embodiments.

[0146] Figure 11 is a schematic physical structure diagram of a computer device provided by an embodiment of the present invention. As Figure 11 shown, the computer device may include: a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104. Among them, the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104. The processor 1101 can call the logical instructions in the memory 1103 to execute the methods provided by the above method embodiments, for example, including: receiving video stream data of a scraper conveyor; preprocessing the current image in the video stream data to obtain a current valid image; performing optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image to obtain the motion amplitude feature corresponding to the current valid image; obtaining the flow area and the conveyor belt area corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image; obtaining the coal state recognition result of the scraper conveyor according to the area of the flow area corresponding to the current valid image and the projection length in the motion direction, the area of the conveyor belt area corresponding to the current valid image and the projection length in the motion direction, and the state judgment rule.

[0147] In addition, when the logical instructions in the above-mentioned memory 1103 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0148] This embodiment discloses a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided in the above method embodiments, for example, including: receiving video stream data of a scraper conveyor; preprocessing the current image in the video stream data to obtain a current valid image; performing optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image to obtain a motion amplitude feature corresponding to the current valid image; obtaining a flow region and a conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image; obtaining a coal state recognition result of the scraper conveyor according to the area and the projection length in the motion direction of the flow region corresponding to the current valid image, the area and the projection length in the motion direction of the conveyor belt region corresponding to the current valid image, and a state judgment rule.

[0149] This embodiment provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program enables the computer to execute the methods provided in the above method embodiments, for example, including: receiving video stream data of a scraper conveyor; preprocessing the current image in the video stream data to obtain a current valid image; performing optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image to obtain a motion amplitude feature corresponding to the current valid image; obtaining a flow region and a conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image; obtaining a coal state recognition result of the scraper conveyor according to the area and the projection length in the motion direction of the flow region corresponding to the current valid image, the area and the projection length in the motion direction of the conveyor belt region corresponding to the current valid image, and a state judgment rule.

[0150] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0152] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0154] In the description of this specification, the description with reference to terms such as "one embodiment", "a specific embodiment", "some embodiments", "for example", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0155] The above-mentioned specific embodiments further elaborate on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying the coal state of a scraper conveyor, characterized in that, Including: Receiving video stream data of a scraper conveyor; Preprocessing a current image in the video stream data to obtain a current valid image; Performing optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image to obtain a motion amplitude feature corresponding to the current valid image; Obtaining a flow region and a conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image; Obtaining a coal state recognition result of the scraper conveyor according to the area and the projection length in the motion direction of the flow region corresponding to the current valid image, the area and the projection length in the motion direction of the conveyor belt region corresponding to the current valid image, and a state judgment rule; 2. The method according to claim 1, wherein The obtaining of the flow region and the conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image includes: Obtaining a motion region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image and a first threshold; If it is judged that the area of the motion region corresponding to the current valid image is less than or equal to an area threshold and the image frame corresponding to the current valid image is not the first frame, obtaining a weighted fusion feature corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image and the historical motion amplitude feature corresponding to the current valid image; wherein, the area threshold is determined based on the area of the current valid image; Obtaining a to-be-determined conveyor belt region corresponding to the current valid image according to the weighted fusion feature corresponding to the current valid image and a second threshold; Obtaining a conveyor belt region corresponding to the current valid image according to the to-be-determined conveyor belt region corresponding to the current valid image and a constant conveyor belt region; wherein, the constant conveyor belt region is obtained in advance; If it is judged that the cumulative number of frames corresponding to the current valid image is greater than a frame number threshold, obtaining a flow region corresponding to the current valid image according to the conveyor belt region corresponding to the current valid image and the motion region corresponding to the current valid image; 3. The method according to claim 2, characterized in that, Also including: If it is judged that the area of the motion region corresponding to the current valid image is greater than the area threshold, initializing coal state recognition parameters; 4. The method according to claim 2, wherein Also including: If it is judged that the area of the motion region corresponding to the current valid image is less than or equal to the area threshold and the image frame corresponding to the current valid image is the first frame, obtaining a recognized conveyor belt region of the current valid image based on the current valid image and a semantic segmentation model; Comparing the recognized conveyor belt region with each historical conveyor belt region to obtain an overlap degree between the recognized conveyor belt region and each historical conveyor belt region; wherein, the historical conveyor belt regions are stored in advance; If it is judged that the overlap degree between the recognized conveyor belt region and the historical conveyor belt region is greater than an overlap degree threshold, using the flow region and the conveyor belt region corresponding to the historical conveyor belt region with the overlap degree greater than the overlap degree threshold as the flow region and the conveyor belt region corresponding to the current valid image; 5. The method according to claim 4, characterized in that Also including: If it is judged that the overlap degree between the recognized conveyor belt region and the historical conveyor belt region is less than or equal to the overlap degree threshold, initializing coal state recognition parameters; 6. The method according to claim 2, wherein Also including: If it is judged that the cumulative number of frames corresponding to the current valid image is less than the frame number threshold, acquiring the next frame image as the current image; 7. The method according to any one of claims 1 to 6, characterized in that, The state judgment rule includes: If it is judged that the first ratio of the area of the flow region corresponding to the current valid image to the area of the conveyor belt region corresponding to the current valid image is greater than the first preset value and the second ratio of the projected length of the flow region corresponding to the current valid image in the moving direction to the projected length of the conveyor belt region corresponding to the current valid image in the moving direction is greater than the second preset value, then the coal state recognition result is normal; If it is judged that the first ratio is less than the third preset value and the second ratio is less than the fourth preset value, then the coal state recognition result is blockage; If it is judged that the first ratio is greater than or equal to the third preset value and less than or equal to the first preset value, and the second ratio is greater than or equal to the fourth preset value and less than or equal to the second preset value, then the coal state recognition result is the risk of blockage.

8. A coal state recognition device for a scraper conveyor, characterized in that, It includes: A receiving module, configured to receive the video stream data of the scraper conveyor; A preprocessing module, configured to preprocess the current image in the video stream data to obtain the current valid image; An optical flow analysis module, configured to perform optical flow analysis on the current valid image and the previous frame image corresponding to the current valid image to obtain the motion amplitude feature corresponding to the current valid image; An obtaining module, configured to obtain the flow region and the conveyor belt region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image; An identification module, configured to obtain the coal state recognition result of the scraper conveyor according to the area and the projected length in the moving direction of the flow region corresponding to the current valid image, the area and the projected length in the moving direction of the conveyor belt region corresponding to the current valid image, and the state judgment rule.

9. The device according to claim 8, characterized in that, The obtaining module includes: A first obtaining unit, configured to obtain the motion region corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image and the first threshold; A second obtaining unit, configured to, if it is judged that the area of the motion region corresponding to the current valid image is less than or equal to the area threshold and the image frame corresponding to the current valid image is not the first frame, obtain the weighted fusion feature corresponding to the current valid image according to the motion amplitude feature corresponding to the current valid image and the historical motion amplitude feature corresponding to the current valid image; wherein, the area threshold is determined based on the area of the current valid image; A third obtaining unit, configured to obtain the pending conveyor belt region corresponding to the current valid image according to the weighted fusion feature corresponding to the current valid image and the second threshold; A fourth obtaining unit, configured to obtain the conveyor belt region corresponding to the current valid image according to the pending conveyor belt region corresponding to the current valid image and the constant conveyor belt region; wherein, the constant conveyor belt region is obtained in advance; A fifth obtaining unit, configured to, after it is judged that the cumulative number of frames corresponding to the current valid image is greater than the frame number threshold, obtain the flow region corresponding to the current valid image according to the conveyor belt region corresponding to the current valid image and the motion region corresponding to the current valid image.

10. The device according to claim 9, characterized in that, It further includes: An initialization unit, configured to initialize the coal state recognition parameters after it is judged that the area of the motion region corresponding to the current valid image is greater than the area threshold.

11. The device according to claim 9, characterized in that, It further includes: A first judgment unit, configured to, after judging that the area of the motion region corresponding to the current valid image is less than or equal to the area threshold and the image frame corresponding to the current valid image is the first frame, obtain the recognized conveyor belt region of the current valid image based on the current valid image and the semantic segmentation model; A comparison unit, configured to compare the recognized conveyor belt region with each historical conveyor belt region to obtain the overlap degree between the recognized conveyor belt region and each historical conveyor belt region; wherein, the historical conveyor belt regions are pre-stored; A second judgment unit, configured to, after judging that the overlap degree between the recognized conveyor belt region and the historical conveyor belt region is greater than the overlap degree threshold, use the flow region and the conveyor belt region corresponding to the historical conveyor belt region with an overlap degree greater than the overlap degree threshold as the flow region and the conveyor belt region corresponding to the current valid image.

12. The device according to claim 11, characterized in that, Further included: A third judgment unit, configured to initialize the coal state recognition parameters if it is judged that the overlap degree between the recognized conveyor belt region and the historical conveyor belt region is less than or equal to the overlap degree threshold.

13. The device according to claim 9, characterized in that, Further included: An acquisition unit, configured to, after judging that the cumulative number of frames corresponding to the current valid image is less than the frame number threshold, acquire the next frame of image as the current image.

14. The device according to any one of claims 8 to 13, characterized in that, The state judgment rule includes: If it is judged that the first ratio of the area of the flow region corresponding to the current valid image to the area of the conveyor belt region corresponding to the current valid image is greater than the first preset value and the second ratio of the projection length of the flow region corresponding to the current valid image in the motion direction to the projection length of the conveyor belt region corresponding to the current valid image in the motion direction is greater than the second preset value, then the coal state recognition result is normal; If it is judged that the first ratio is less than the third preset value and the second ratio is less than the fourth preset value, then the coal state recognition result is blocked; If it is judged that the first ratio is greater than or equal to the third preset value and less than or equal to the first preset value, and the second ratio is greater than or equal to the fourth preset value and less than or equal to the second preset value, then the coal state recognition result is at risk of blockage.

15. A computer device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program / instructions, and when the computer program / instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

17. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images