Control Method, Device, Electronic Equipment and Medium of Scraper Conveyor

Abstract

The present application provides a control method, device, electronic device and medium for a scraper conveyor, relating to the technical field of image processing. The method includes: monitoring the scraper conveyor by using a plurality of visual detection devices to obtain a first image collected by the plurality of visual detection devices at a first moment and a second image collected at a second moment; for any one of the visual detection devices, inputting the first image and the second image collected by the visual detection device into a depth estimation model respectively to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image; determining the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth map and the second depth map; determining the total coal volume conveyed by the scraper conveyor at the first moment according to the coal volumes of the plurality of visual detection devices, and controlling the transportation speed of the scraper conveyor according to the total coal volume. Thus, the transportation speed of the scraper conveyor can be automatically controlled.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular, to a control method, device, electronic device, and medium for a scraper conveyor. Background Art

[0002] In the process of coal production, a scraper conveyor is needed to transport coal. However, when the amount of coal transported by the scraper conveyor is small while the transport speed of the scraper conveyor is fast, the capacity of the scraper conveyor cannot be fully utilized, resulting in large energy losses and easy economic losses; when the amount of coal transported by the scraper conveyor is large while the transport speed of the scraper conveyor is slow, the power of the scraper conveyor is insufficient, and the coal falling from the shearer into the scraper conveyor cannot be transported out in time, and coal accumulation is likely to occur. How to control the scraper conveyor is very important. Summary of the Invention

[0003] The purpose of the present application is to solve at least one of the above technical problems to some extent.

[0004] To this end, the present application proposes a control method for a scraper conveyor. By using multiple visual detection devices to monitor the scraper conveyor, a first image collected by the multiple visual detection devices at a first moment and a second image collected at a second moment are obtained; for any one of the visual detection devices, the first image and the second image collected by the visual detection device are respectively input into a depth estimation model to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image; according to the first depth map and the second depth map, the coal volume of the monitoring area corresponding to the visual detection device at the first moment is determined; according to the coal volumes of the multiple visual detection devices, the total coal volume transported by the scraper conveyor at the first moment is determined, and the transport speed of the scraper conveyor is controlled according to the total coal volume. Thus, based on deep learning technology, according to the first image and the second image obtained by monitoring the scraper conveyor by multiple visual detection devices, the first depth map corresponding to the first image and the second depth map corresponding to the second image can be obtained. Further, according to each first depth map and each second depth map, the total coal volume transported by the scraper conveyor at the first moment can be automatically predicted, so that the transport speed of the scraper conveyor can be controlled based on the total coal volume.

[0005] A first aspect embodiment of the present application proposes a control method for a scraper conveyor, the method including:

[0006] Using multiple visual detection devices to monitor the scraper conveyor to obtain a first image collected by the multiple visual detection devices at a first moment and a second image collected at a second moment, wherein the scraper conveyor is in a coal state at the first moment, and the scraper conveyor is in a coal-free state at the second moment;

[0007] For any one of the visual detection devices, input the first image and the second image collected by the visual detection device into a depth estimation model respectively, so as to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image;

[0008] According to the first depth map and the second depth map, determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment;

[0009] According to the coal volumes of the multiple visual detection devices, determine the total coal volume transported by the scraper conveyor at the first moment, and control the transportation speed of the scraper conveyor according to the total coal volume.

[0010] Optionally, as the first possible implementation manner of the first aspect, the determining the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth map and the second depth map includes: performing semantic segmentation on the first image to obtain a first coal area in the first image; according to the first position of the first coal area in the first image, determine a first target area matching the first position from the first depth map, and determine a second target area matching the first position from the second depth map; according to the first depths of the first pixel units in the first target area and the second depths of the second pixel units in the second target area, determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment.

[0011] Optionally, as the second possible implementation manner of the first aspect, the performing semantic segmentation on the first image to obtain a first coal area in the first image includes: using a trained target semantic segmentation model to perform semantic segmentation on the first image to obtain the first coal area; wherein, the target semantic segmentation model is trained by the following steps: obtaining at least one first training image, wherein each first training image is labeled with a second coal area; for any one of the first training images, input the first training image into an initial semantic segmentation model for semantic segmentation to obtain a coal segmentation area; based on the difference between the second position of the second coal area in the second training image and the third position of the coal segmentation area in the second training image, train the initial semantic segmentation model to obtain the target semantic segmentation model.

[0012] Optionally, as a third possible implementation manner of the first aspect, determining the coal volume of the monitoring area corresponding to the vision detection device at the first moment according to the first depths of the first pixel units in the first target area and the second depths of the second pixel units in the second target area includes: for any one of the first pixel units in the first target area, determining a second pixel unit in the second target area that matches the position of the first pixel unit; determining the coal height corresponding to the first pixel unit based on the first depth of the first pixel unit and the second depth of the matched second pixel unit; and determining the coal volume of the monitoring area corresponding to the vision detection device at the first moment according to the coal areas and the coal heights corresponding to the first pixel units in the first target area.

[0013] Optionally, as a fourth possible implementation manner of the first aspect, determining the total coal volume conveyed by the scraper conveyor at the first moment according to the coal volumes of the multiple vision detection devices includes: determining a target coefficient according to the sum of the coal volumes of the multiple vision detection devices; obtaining a correction coefficient, where the correction coefficient is determined according to the field of view overlap range of adjacent vision detection devices; and correcting the target coefficient based on the correction coefficient to obtain the total coal volume conveyed by the scraper conveyor at the first moment.

[0014] Optionally, as a fifth possible implementation manner of the first aspect, when the vision detection device is a binocular camera device, inputting the first image collected by the vision detection device into a depth estimation model to obtain a first depth map corresponding to the first image includes: inputting the first image into the depth estimation model to obtain a first disparity map corresponding to the first image, where the first image is a left-eye image or a right-eye image collected by the binocular camera device; and obtaining the first depth map corresponding to the first image according to the first disparity map, the baseline, and the focal length corresponding to the binocular camera device that collected the first image.

[0015] Optionally, as a sixth possible implementation manner of the first aspect, the depth estimation model is trained by the following steps: A sample binocular camera device is used to monitor a sample scraper conveyor to obtain a training image pair; wherein, the training image pair includes a left-eye training image and a right-eye training image collected by the sample binocular camera device; the first reference training image is input into an initial depth estimation model to obtain a first sample disparity map corresponding to the first reference training image; wherein, the first reference training image is the left-eye training image or the right-eye training image; based on the first sample disparity map, back-project the second reference training image other than the first reference training image in the training image pair to obtain a back-projected image; according to the difference between the first reference training image and the back-projected image, train the initial depth estimation model to obtain the trained depth estimation model.

[0016] In the control method of the scraper conveyor according to the embodiment of the present application, by using a plurality of visual detection devices to monitor the scraper conveyor, a first image collected by the plurality of visual detection devices at a first moment and a second image collected at a second moment are obtained; for any one of the visual detection devices, the first image and the second image collected by the visual detection device are respectively input into a depth estimation model to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image; according to the first depth map and the second depth map, determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment; according to the coal volumes of the plurality of visual detection devices, determine the total coal volume conveyed by the scraper conveyor at the first moment, and control the transportation speed of the scraper conveyor according to the total coal volume. Thus, based on deep learning technology, according to the first image and the second image obtained by monitoring the scraper conveyor by a plurality of visual detection devices, the first depth map corresponding to the first image and the second depth map corresponding to the second image can be obtained. Further, according to each first depth map and each second depth map, the total coal volume conveyed by the scraper conveyor at the first moment can be automatically predicted, so that the transportation speed of the scraper conveyor can be controlled based on the total coal volume.

[0017] An embodiment of the second aspect of the present application provides a control device for a scraper conveyor, the device includes:

[0018] A first monitoring module, configured to use a plurality of visual detection devices to monitor the scraper conveyor to obtain a first image collected by the plurality of visual detection devices at a first moment and a second image collected at a second moment, wherein the scraper conveyor is in a coal-containing state at the first moment, and the scraper conveyor is in a coal-free state at the second moment;

[0019] A first input module, for any one of the visual detection devices, inputting the first image and the second image collected by the visual detection device into a depth estimation model respectively, so as to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image;

[0020] A first determination module, for determining the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth map and the second depth map;

[0021] A second determination module, for determining the total coal volume transported by the scraper conveyor at the first moment according to the coal volumes of the multiple visual detection devices;

[0022] A control module, for controlling the transportation speed of the scraper conveyor according to the total coal volume.

[0023] Optionally, as the first possible implementation manner of the second aspect, the first determination module is configured to: perform semantic segmentation on the first image to obtain a first coal area in the first image; according to the first position of the first coal area in the first image, determine a first target area matching the first position from the first depth map, and determine a second target area matching the first position from the second depth map; determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth of each first pixel unit in the first target area and the second depth of each second pixel unit in the second target area.

[0024] Optionally, as the second possible implementation manner of the second aspect, the first determination module is configured to: use a trained target semantic segmentation model to perform semantic segmentation on the first image to obtain the first coal area; wherein, the target semantic segmentation model is trained by the following steps: obtaining at least one first training image, wherein each first training image is labeled with a second coal area; for any one of the first training images, inputting the first training image into an initial semantic segmentation model for semantic segmentation to obtain a coal segmentation area; training the initial semantic segmentation model based on the difference between the second position of the second coal area in the second training image and the third position of the coal segmentation area in the second training image to obtain the target semantic segmentation model.

[0025] Optionally, as a third possible implementation of the second aspect, the first determination module is configured to: for any one of the first pixel units in the first target area, determine a second pixel unit in the second target area that matches the position of the first pixel unit; based on the first depth of the first pixel unit and the second depth of the matched second pixel unit, determine the coal height corresponding to the first pixel unit; and determine the coal volume of the monitoring area corresponding to the vision detection device at the first moment according to the coal area and the coal height corresponding to each of the first pixel units in the first target area.

[0026] Optionally, as a fourth possible implementation of the second aspect, the second determination module is configured to: determine a target coefficient according to the cumulative sum of the coal volumes of the multiple vision detection devices; obtain a correction coefficient, where the correction coefficient is determined according to the field of view overlap range of adjacent vision detection devices; and correct the target coefficient based on the correction coefficient to obtain the total coal volume conveyed by the scraper conveyor at the first moment.

[0027] Optionally, as a fifth possible implementation of the second aspect, the vision detection device is a binocular camera device, and the first input module is configured to: input the first image into the depth estimation model to obtain a first disparity map corresponding to the first image; where the first image is a left-eye image or a right-eye image collected by the binocular camera device; and obtain a first depth map corresponding to the first image according to the first disparity map, and according to the baseline and focal length corresponding to the binocular camera device that collected the first image.

[0028] Optionally, as a sixth possible implementation of the second aspect, the depth estimation model is trained using the following modules:

[0029] A second monitoring module, configured to monitor a sample scraper conveyor using a sample binocular camera device to obtain a training image pair; where the training image pair includes a left-eye training image and a right-eye training image collected by the sample binocular camera device;

[0030] A second input module, configured to input a first reference training image into an initial depth estimation model to obtain a first sample disparity map corresponding to the first reference training image; where the first reference training image is the left-eye training image or the right-eye training image;

[0031] A back-projection module, configured to perform back-projection on a second reference training image other than the first reference training image in the training image pair based on the first sample disparity map to obtain a back-projected image;

[0032] A training module, configured to train the initial depth estimation model according to the difference between the first reference training image and the back-projected image, so as to obtain the trained depth estimation model.

[0033] In the control device of the scraper conveyor according to the embodiment of the present application, by using a plurality of visual detection devices to monitor the scraper conveyor, a first image collected by the plurality of visual detection devices at a first moment and a second image collected at a second moment are obtained; for any one of the visual detection devices, the first image and the second image collected by the visual detection device are respectively input into the depth estimation model to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image; according to the first depth map and the second depth map, the coal volume of the monitoring area corresponding to the visual detection device at the first moment is determined; according to the coal volumes of the plurality of visual detection devices, the total coal volume transported by the scraper conveyor at the first moment is determined, and the transportation speed of the scraper conveyor is controlled according to the total coal volume. Thus, based on the deep learning technology, according to the first image and the second image obtained by monitoring the scraper conveyor by a plurality of visual detection devices, the first depth map corresponding to the first image and the second depth map corresponding to the second image can be obtained. Further, according to each first depth map and each second depth map, the total coal volume transported by the scraper conveyor at the first moment can be automatically predicted, so that the transportation speed of the scraper conveyor can be controlled based on the total coal volume.

[0034] An embodiment of the third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, the control method of the scraper conveyor as described in the first aspect is implemented.

[0035] An embodiment of the fourth aspect of the present application provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the control method of the scraper conveyor as described in the first aspect is implemented.

[0036] The additional aspects and advantages of the present application will be partly given in the following description, partly will become obvious from the following description, or be understood through the practice of the present application. Description of the Drawings

[0037] The above-mentioned and / or additional aspects and advantages of the present application will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0038] Figure 1 It is a schematic flowchart of the control method of the scraper conveyor provided by the first embodiment of the present application;

[0039] Figure 2Schematic diagram of the working process of a single vision detection device provided by the present disclosure;

[0040] Figure 3 Schematic diagram of the working process of multiple vision detection devices provided by the present disclosure;

[0041] Figure 4 Schematic flowchart of the control method for a scraper conveyor provided in the second embodiment of the present application;

[0042] Figure 5 Schematic flowchart of the control method for a scraper conveyor provided in the third embodiment of the present application;

[0043] Figure 6 Schematic structural diagram of a control device for a scraper conveyor provided in the fourth embodiment of the present application;

[0044] Figure 7 Schematic structural diagram of an electronic device provided in the fifth embodiment of the present application. Detailed implementation manners

[0045] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described by referring to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as limiting the present application.

[0046] In the related art, the following two methods can be used to measure or predict the volume of coal:

[0047] 1. Methods such as lidar and machine vision are used to obtain the coal volume information on the transfer machine or belt conveyor to predict the coal quantity on the scraper conveyor.

[0048] This method is relatively accurate when the coal falling volume at the working face is stable. However, when the coal output of the shearer changes, it is difficult to reflect the real coal quantity on the scraper conveyor. Moreover, predicting the real-time coal quantity on the scraper conveyor by measuring the coal volume information of the subsequent transportation equipment of the scraper conveyor has a certain lag and cannot reflect the real data of the real-time coal quantity on the scraper conveyor. The main basis for the frequency conversion adjustment of the scraper conveyor is the real-time coal quantity on the scraper conveyor.

[0049] 2. The coal quantity on the scraper conveyor is calculated by using the coal quantity calculation model of the scraper conveyor through data such as the current, voltage, and running speed of the shearer or scraper conveyor.

[0050] The accuracy of the coal quantity result calculated by this method strongly depends on the rationality of the coal quantity calculation model established for the scraper conveyor, the errors of various sensors, etc., and it is impossible to intuitively display the coal quantity on the scraper conveyor. For example, for the technology of constructing a coal quantity calculation model for the scraper conveyor based on coal mining related parameters such as the cutting height, cutting depth, and traveling speed of the shearer, accurate measurement and precise control of the above parameters are required, and it strongly depends on the degree of conformity between the model establishment and the mining process.

[0051] In view of the above problems, an embodiment of the present application proposes a control method, device, electronic device, and medium for a scraper conveyor.

[0052] The following combines Figure 1 , and details the control method for the scraper conveyor provided by the present application.

[0053] In the embodiment of the present application, it is exemplified that the control method for the scraper conveyor is configured in a control device for the scraper conveyor. The control device for the scraper conveyor can be applied to any electronic device so that the electronic device can execute the control function of the scraper conveyor.

[0054] Among them, the electronic device can be any device with computing power. For example, it can be a PC (Personal Computer), a mobile terminal, a server, etc. The mobile terminal can be a hardware device such as a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc. with various operating systems, touch screens, and / or display screens.

[0055] Figure 1 FIG. is a schematic flowchart of the control method for the scraper conveyor provided by Embodiment 1 of the present application.

[0056] As Figure 1 shown, the control method for the scraper conveyor includes the following steps:

[0057] Step 101, monitor the scraper conveyor by using a plurality of visual detection devices to obtain a first image collected by the plurality of visual detection devices at a first moment and a second image collected at a second moment, where the scraper conveyor is in a coal-containing state at the first moment and the scraper conveyor is in a coal-free state at the second moment.

[0058] In the embodiment of the present application, the visual detection device can be, for example, a camera or a camera, etc., for collecting images; and the plurality of visual detection devices can be distributed in a distributed manner.

[0059] In the embodiment of the present application, the first moment can be the moment when the scraper conveyor is in a coal-containing state, and the second moment can be the moment when the scraper conveyor is in a coal-free state.

[0060] It should be noted that for any visual detection device, the position of the trough of the scraper conveyor in the first image collected at the first moment may be the same as the position of the trough of the scraper conveyor in the second image collected at the second moment.

[0061] In the embodiments of the present application, multiple visual detection devices may be used to monitor the scraper conveyor to obtain the first images collected by the multiple visual detection devices at the first moment and the second images collected at the second moment. Among them, the scraper conveyor is in a state with coal at the first moment, and the scraper conveyor is in a state without coal at the second moment.

[0062] It should be noted that in order to ensure that the images collected by the multiple visual detection devices at the same moment can cover the conveying area of the scraper conveyor, the minimum fields of view of adjacent visual detection devices may have a field of view overlapping area or adjacent fields of view.

[0063] As an example, the visual detection device is a camera, and the camera can be installed at the position of the top beam of the hydraulic support, and the camera lens faces the corresponding conveying area of the scraper conveyor. The working schematic diagram of a single visual detection device is as Figure 2 shown, where 21 is the top beam of the hydraulic support, 22 is the camera, 23 is the light source, and 24 is the scraper conveyor; the working schematic diagram of multiple visual detection devices is as Figure 3 shown, where 31 is the synchronous acquisition control and calculation unit, 32 is the scraper conveyor controller, 33 is the hydraulic support, 34 is the camera, 35 is the scraper conveyor, and 36 is the shearer. In order to ensure that the images collected by the multiple cameras at the same moment can cover the conveying area of the scraper conveyor, assuming that the minimum fields of view of adjacent cameras are adjacent, the maximum distance between adjacent cameras can be determined. For example, if the camera field of view angle is θ, the installation height of the camera is h, and the center distance of the hydraulic supports is D, the maximum distance W between adjacent cameras is:

[0064]

[0065] The maximum value S of the number of hydraulic supports spaced between adjacent cameras can be:

[0066]

[0067] where Floor() is the floor function.

[0068] Step 102, for any visual detection device, input the first image and the second image collected by the visual detection device into the depth estimation model respectively to obtain the first depth map corresponding to the first image and the second depth map corresponding to the second image.

[0069] In the embodiments of the present application, the pixel value of any pixel unit in the first depth map may indicate the depth of the corresponding pixel unit in the first image.

[0070] In an embodiment of the present application, the pixel value of any pixel unit in the second depth map may indicate the depth of the corresponding pixel unit in the second image.

[0071] In an embodiment of the present application, for any visual detection device, the first image and the second image collected by the visual detection device may be respectively input into a depth estimation model to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image.

[0072] Step 103: Determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth map and the second depth map.

[0073] In an embodiment of the present application, the coal volume of the monitoring area corresponding to the visual detection device at the first moment may be determined according to the first depth map and the second depth map.

[0074] Step 104: Determine the total coal volume conveyed by the scraper conveyor at the first moment according to the coal volumes of multiple visual detection devices, and control the transportation speed of the scraper conveyor according to the total coal volume.

[0075] In an embodiment of the present application, the total coal volume conveyed by the scraper conveyor at the first moment may be determined according to the coal volumes of multiple visual detection devices.

[0076] As a possible implementation manner, a target coefficient may be determined according to the sum of the coal volumes of multiple visual detection devices; and a correction coefficient may be obtained, where the correction coefficient may be determined according to the field of view overlap range of adjacent visual detection devices; the target coefficient may be corrected based on the correction coefficient to obtain the total coal volume conveyed by the scraper conveyor at the first moment.

[0077] As an example, assume that there are m visual detection devices, and the coal volume of the j-th visual detection device is V j , j ∈ [1, m], and j is an integer. According to the sum of the coal volumes of the m visual detection devices, the target coefficient H is determined as:

[0078]

[0079] The correction coefficient δ may be obtained, and the target coefficient may be corrected based on the correction coefficient to obtain the total coal volume V s :

[0080] V s = δH; (4)

[0081] The control method of the scraper conveyor according to the embodiment of the present application monitors the scraper conveyor by using a plurality of visual detection devices to obtain a first image collected by the plurality of visual detection devices at a first moment and a second image collected at a second moment. Among them, the scraper conveyor is in a coal-bearing state at the first moment, and the scraper conveyor is in a coal-free state at the second moment; for any visual detection device, the first image and the second image collected by the visual detection device are respectively input into the depth estimation model to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image; according to the first depth map and the second depth map, determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment; according to the coal volumes of the plurality of visual detection devices, determine the total coal volume conveyed by the scraper conveyor at the first moment, and control the transportation speed of the scraper conveyor according to the total coal volume. Thus, based on deep learning technology, according to the first image and the second image obtained by monitoring the scraper conveyor by a plurality of visual detection devices, the first depth map corresponding to the first image and the second depth map corresponding to the second image can be obtained. Further, according to each first depth map and each second depth map, the total coal volume conveyed by the scraper conveyor at the first moment can be automatically predicted, so that the transportation speed of the scraper conveyor can be controlled based on the total coal volume.

[0082] In order to clearly illustrate how to determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth map and the second depth map in the above embodiments of the present application, the present application also proposes a control method for a scraper conveyor.

[0083] Figure 4 It is a schematic flowchart of the control method of the scraper conveyor provided in the second embodiment of the present application.

[0084] As Figure 4 shown, the control method of the scraper conveyor may include the following steps:

[0085] Step 401, monitor the scraper conveyor by using a plurality of visual detection devices to obtain a first image collected by the plurality of visual detection devices at a first moment and a second image collected at a second moment. Among them, the scraper conveyor is in a coal-bearing state at the first moment, and the scraper conveyor is in a coal-free state at the second moment.

[0086] Step 402, for any visual detection device, input the first image and the second image collected by the visual detection device into the depth estimation model respectively to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image.

[0087] The execution processes of steps 401 to 402 can refer to the execution processes of any embodiment of the present application, and will not be elaborated here.

[0088] Step 403: Perform semantic segmentation on the first image to obtain the first coal region in the first image.

[0089] In an embodiment of the present application, the first coal region may be the coal region in the first image.

[0090] In an embodiment of the present application, semantic segmentation may be performed on the first image. For example, a semantic segmentation model based on deep learning (such as an FCN (Fully Convolutional Networks) model, a U-Net model, an OCRNet (Object-Contextual Representations for Semantic Segmentation) model, etc.) may be used to perform semantic segmentation on the first image, so as to obtain the first coal region in the first image.

[0091] As a possible implementation, a trained target semantic segmentation model may be used to perform semantic segmentation on the first image to obtain the first coal region.

[0092] In an embodiment of the present application, the target semantic segmentation model may be, for example, an FCN model, a U-Net model, an OCRNet model, etc., and the present application does not limit this.

[0093] In order to obtain a trained target semantic segmentation model, in a possible implementation of an embodiment of the present application, the target semantic segmentation model may be trained using the following steps:

[0094] 1. Obtain at least one first training image, where each first training image is labeled with a second coal region.

[0095] In an embodiment of the present application, the first training image may have a second coal region, and the second coal region in the first training image may be labeled.

[0096] In an embodiment of the present application, the first training image may be obtained, where the number of first training images may be one or multiple, and the present application does not limit this.

[0097] 2. For any first training image, input the first training image into the initial semantic segmentation model for semantic segmentation to obtain a coal segmentation region.

[0098] In an embodiment of the present application, the initial semantic segmentation model may be, for example, an FCN model, a U-Net model, an OCRNet model, etc., and the present application does not limit this.

[0099] In the embodiments of the present application, for any first training image, the first training image can be input into an initial semantic segmentation model for semantic segmentation to obtain a coal segmentation region.

[0100] 3. Train the initial semantic segmentation model based on the difference between the second position of the second coal region in the second training image and the third position of the coal segmentation region in the second training image to obtain a target semantic segmentation model.

[0101] As a possible implementation, a first loss value can be generated based on the difference between the second position of the second coal region in the second training image and the third position of the coal segmentation region in the second training image; and the initial semantic segmentation model can be trained according to the first loss value to obtain a target semantic segmentation model.

[0102] In the embodiments of the present application, the first loss value can have a positive relationship (i.e., a positive correlation) with the difference between the second position of the second coal region in the second training image and the third position of the coal segmentation region in the second training image, that is, the smaller the difference, the smaller the value of the first loss value, and the larger the difference, the larger the value of the first loss value.

[0103] Therefore, in the present application, the initial semantic segmentation model can be trained according to the first loss value to obtain a target semantic segmentation model. For example, the initial semantic segmentation model can be trained according to the first loss value to minimize the value of the first loss value.

[0104] It should be noted that the above only takes the termination condition of the training of the initial semantic segmentation model as the minimization of the first loss value as an example. In actual applications, other termination conditions can also be set. For example, the number of training times reaches a set number, the training duration reaches a set duration, the first loss value converges, etc. The present application does not limit this.

[0105] Thus, by training the initial semantic segmentation model, a trained target semantic segmentation model can be effectively obtained.

[0106] Step 404, according to the first position of the first coal region in the first image, determine a first target region matching the first position from the first depth map, and determine a second target region matching the first position from the second depth map.

[0107] In the embodiments of the present application, a first target region matching (or the same as) the first position can be determined from the first depth map according to the first position of the first coal region in the first image, and a second target region matching (or the same as) the first position can be determined from the second depth map.

[0108] Step 405: Determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth of each first pixel unit in the first target area and the second depth of each second pixel unit in the second target area.

[0109] As a possible implementation, for any first pixel unit in the first target area, a second pixel unit that matches (or is the same as) the position of the first pixel unit can be determined from the second target area; based on the first depth of the first pixel unit and the second depth of the matched second pixel unit, determine the coal height corresponding to the first pixel unit; according to the coal area and coal height corresponding to each first pixel unit in the first target area, the coal volume of the monitoring area corresponding to the visual detection device at the first moment can be determined.

[0110] It should be noted that the coal area corresponding to each first pixel unit can be determined based on the calibration process of the visual detection device and the actual installation height of the visual detection device, and the coal areas corresponding to each first pixel unit can be the same.

[0111] As an example, assume that there are n first pixel units in the first target area. For the i-th first pixel unit in the first target area, a second pixel unit that matches (or is the same as) the position of the first pixel unit can be determined from the second target area; based on the first depth H i of the i-th first pixel unit and the second depth H i ′ of the matched second pixel unit, the coal height ΔH corresponding to the first pixel unit can be determined according to the following formula i :

[0112] ΔH i = H i - H i ′; (5)

[0113] where i ∈ [1, n] and i is a positive integer.

[0114] Assume that the coal area corresponding to each first pixel unit is S. According to the coal area and coal height corresponding to each first pixel unit in each first target area, the coal volume V of the monitoring area corresponding to the visual detection device at the first moment can be determined according to the following formula:

[0115]

[0116] Thus, the coal volume of the monitoring area corresponding to the visual detection device at the first moment can be effectively determined.

[0117] Step 406: Determine the total coal volume transported by the scraper conveyor at the first moment according to the coal volumes of multiple visual detection devices, and control the transportation speed of the scraper conveyor according to the total coal volume.

[0118] The execution process of step 406 can refer to the execution process of any embodiment of this application, which will not be elaborated here.

[0119] In the control method of the scraper conveyor according to the embodiment of this application, the first image is semantically segmented to obtain the first coal area in the first image; according to the first position of the first coal area in the first image, the first target area matching the first position is determined from the first depth map, and the second target area matching the first position is determined from the second depth map; according to the first depth of each first pixel unit in the first target area and the second depth of each second pixel unit in the second target area, the coal volume of the monitoring area corresponding to the visual detection device at the first moment is determined. Thus, based on deep learning technology, the first coal area belonging to coal can be determined from the first image. Further, the first target area matching the first position of the first coal area can be determined from the first depth map, and the second target area matching the first position can be determined from the second depth map, so that the coal volume of the monitoring area corresponding to the visual detection device at the first moment can be automatically determined according to the first depth of each first pixel unit in the first target area and the second depth of each second pixel unit in the second target area.

[0120] To clearly illustrate how the first image collected by the visual detection device is input into the depth estimation model to obtain the first depth map corresponding to the first image in the embodiment of this application, this application also proposes a control method for the scraper conveyor.

[0121] Figure 5 It is a schematic flowchart of the control method for the scraper conveyor provided in the third embodiment of this application.

[0122] As Figure 5 shown, the control method of the scraper conveyor may include the following steps:

[0123] Step 501, multiple visual detection devices are used to monitor the scraper conveyor to obtain the first image collected by the multiple visual detection devices at the first moment and the second image collected at the second moment, where the scraper conveyor is in a coal-bearing state at the first moment and the scraper conveyor is in a coal-free state at the second moment.

[0124] The execution process of step 501 can refer to the execution process of any embodiment of this application, which will not be elaborated here.

[0125] In the embodiment of this application, the visual detection device may be a binocular camera device.

[0126] That is to say, multiple binocular camera devices can be used to monitor the scraper conveyor to obtain the first image collected by the multiple binocular camera devices at the first moment and the second image collected at the second moment.

[0127] It should be noted that both the first image and the second image can include a left-eye image and a right-eye image. In this application, the left-eye image in the first image is marked as the first left-eye image, and the right-eye image in the first image is marked as the first right-eye image.

[0128] Step 502, for any visual detection device, input the first image into the depth estimation model to obtain the first disparity map corresponding to the first image; wherein, the first image is the left-eye image or the right-eye image collected by the binocular camera device.

[0129] In the embodiments of this application, the depth estimation model can be a monocular depth estimation model, such as the MonoDepthv2 (Monocular Depth Estimation Version 2) model, the PLADE-Net (Pixel-Level Accuracy for Depth Estimation Net) model, etc. This application does not limit this.

[0130] In the embodiments of this application, the first image can be the left-eye image or the right-eye image collected by the binocular camera device.

[0131] In the embodiments of this application, the pixel value corresponding to any pixel unit in the first disparity map can indicate the disparity value of the corresponding pixel unit in the first image.

[0132] In the embodiments of this application, for any visual detection device, the first image collected by the visual detection device at the first moment can be input into the depth estimation model to obtain the first disparity map corresponding to the first image.

[0133] As an example, for any visual detection device, the first left-eye image collected by the visual detection device at the first moment can be input into the depth estimation model to obtain the first disparity map corresponding to the first left-eye image.

[0134] As another example, for any visual detection device, the first right-eye image collected by the visual detection device at the first moment can be input into the depth estimation model to obtain the first disparity map corresponding to the first right-eye image.

[0135] Step 503, according to the first disparity map, and according to the baseline and focal length corresponding to the binocular camera device that collects the first image, obtain the first depth map corresponding to the first image.

[0136] In an embodiment of the present application, a first depth map corresponding to the first image can be obtained based on the first disparity map, as well as the baseline and focal length corresponding to the binocular camera device that captures the first image.

[0137] For example, based on the first disparity map, if it is determined that the disparity value corresponding to any pixel unit in the first image is d, the baseline corresponding to the binocular camera device that captures the first image is b, and the focal length of the binocular camera device is f, the depth z corresponding to any pixel unit in the first image can be determined according to the following formula:

[0138] z = f * b / d; (7)

[0139] Thus, the depth corresponding to any pixel unit in the first image can be determined. Further, the first depth map corresponding to the first image can be obtained.

[0140] In order to obtain a trained depth estimation model, in a possible implementation manner of an embodiment of the present application, the depth estimation model can be trained by the following steps:

[0141] 1. Use a sample binocular camera device to monitor a sample scraper conveyor to obtain a training image pair; wherein, the training image pair includes a left-eye training image and a right-eye training image captured by the sample binocular camera device.

[0142] In an embodiment of the present application, a sample binocular camera device can be used to monitor a sample scraper conveyor to obtain a training image pair; wherein, the training image pair can include a left-eye training image and a right-eye training image captured by the sample binocular camera device.

[0143] It should be noted that the number of training image pairs can be one pair, but is not limited to one pair, and the present application does not limit this.

[0144] 2. Input the first reference training image into the initial depth estimation model to obtain a first sample disparity map corresponding to the first reference training image; wherein, the first reference training image is a left-eye training image or a right-eye training image.

[0145] In an embodiment of the present application, the first reference training image can be the left-eye training image in the training image pair, or it can also be the right-eye training image in the training image pair, and the present application does not limit this.

[0146] In an embodiment of the present application, the first sample disparity map can indicate the disparity map corresponding to the first reference training image.

[0147] In an embodiment of the present application, the first reference training image is input into the initial depth estimation model to obtain a first sample disparity map corresponding to the first reference training image.

[0148] As an example, the left-eye training image can be input into the initial depth estimation model to obtain the first sample disparity map corresponding to the left-eye training image.

[0149] As another example, the right-eye training image can be input into the initial depth estimation model to obtain the first sample disparity map corresponding to the right-eye training image.

[0150] 3. Based on the first sample disparity map, back-project the second reference training image in the training image pair except the first reference training image to obtain the back-projected image.

[0151] In the embodiments of the present application, the second reference training image can be the image in the training image pair except the first reference training image. For example, when the first reference training image is the left-eye training image, the second reference image is the right-eye training image; when the first reference training image is the right-eye training image, the second reference image is the left-eye training image.

[0152] In the embodiments of the present application, the second reference training image in the training image pair except the first reference training image can be back-projected based on the first sample disparity map to obtain the back-projected image.

[0153] As an example, input the left-eye training image I t into the initial depth estimation model, and the first sample disparity map D t corresponding to the left-eye training image can be obtained. Based on the first sample disparity map D t and the calibration process of the sample binocular camera device, back-project the right-eye training image I t′ to obtain the back-projected image I t′→t :

[0154] I t′→t = I t′ <proj(D t , T t′→t , K)>; (8)

[0155] Wherein, T t′→t is the relative pose transformation matrix between the right-eye training image and the left-eye training image; K is the normalized internal parameter matrix of the binocular camera device; the proj() function is the projection calculation formula, and this function can calculate the reprojected two-dimensional coordinates based on D t , T t′→t and K; <> is the sampling operator.

[0156] As an example, input the right-eye training image I t′ into the initial depth estimation model, and the first sample disparity map D t′ corresponding to the right-eye training image can be obtained. Based on the first sample disparity map Dt′ For the calibration process of the sample binocular camera device, perform back-projection on the left-eye training image I t to obtain the back-projected image I t→t′ :

[0157] I t→t′ = I t <proj(D t′ , T t→t′ , K)>; (9)

[0158] where T t→t′ is the relative pose transformation matrix between the left-eye training image and the right-eye training image; K is the normalized internal parameter matrix of the binocular camera device; proj() is the projection calculation function, which can calculate the reprojected two-dimensional coordinates based on D t′ , D t′ and K; <> is the sampling operator.

[0159] 4. Train the initial depth estimation model according to the difference between the first reference training image and the back-projected image to obtain the trained depth estimation model.

[0160] In the embodiments of the present application, the initial depth estimation model can be trained according to the difference between the first reference training image and the back-projected image to obtain the trained depth estimation model.

[0161] As a possible implementation, a second loss value can be generated according to the difference between the first reference training image and the back-projected image; and the initial depth estimation model can be trained according to the second loss value to obtain the trained depth estimation model.

[0162] In the embodiments of the present application, a second loss value can be generated according to the difference between the first reference training image and the back-projected image. For example, assume that the first reference training image is I t , and the back-projected image is I t′→t . The difference between the above first reference training image and the back-projected image is ||I t - I t′→t ||1, and the second loss value is L p :

[0163]

[0164]

[0165] Among them, the pe() function can calculate the photometric error between the first reference training image and the back-projected image; the SSIM() function can calculate the structural similarity between the first reference training image and the back-projected image; α is a preset coefficient, such as 0.85, 0.8, etc., and this application does not limit this.

[0166] Therefore, in this application, the initial depth estimation model can be trained according to the second loss value to obtain a trained depth estimation model. For example, the initial depth estimation model can be trained according to the second loss value to minimize the value of the second loss value.

[0167] It should be noted that the above only takes the termination condition of the training of the initial depth estimation model as the minimization of the second loss value as an example. In actual application, other termination conditions can also be set. For example, the number of training times reaches the set number of times, the training duration reaches the set duration, the second loss value converges, etc. This application does not limit this.

[0168] Thus, a trained depth estimation model can be effectively obtained.

[0169] Step 504, input the second image into the depth estimation model to obtain a second depth map corresponding to the second image.

[0170] The execution process of step 504 can refer to the execution process of any embodiment of this application and will not be elaborated here.

[0171] It should be noted that the method of inputting the second image into the depth estimation model to obtain a second depth map corresponding to the second image is similar to the method of inputting the first image into the depth estimation model to obtain a first depth map corresponding to the first image, and will not be elaborated here.

[0172] Step 505, determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth map and the second depth map.

[0173] Step 506, determine the total coal volume transported by the scraper conveyor at the first moment according to the coal volumes of multiple visual detection devices, and control the transportation speed of the scraper conveyor according to the total coal volume.

[0174] In the embodiment of this application, the execution processes of steps 505 to 506 can refer to the execution processes of any embodiment of this application and will not be elaborated here.

[0175] The control method of the scraper conveyor according to the embodiment of the present application obtains a first disparity map corresponding to the first image by inputting the first image into a depth estimation model; according to the first disparity map, and according to the baseline and focal length corresponding to the binocular camera device that acquires the first image, a first depth map corresponding to the first image is obtained. Thus, the first depth map corresponding to the first image can be effectively obtained.

[0176] As an application scenario, an exemplary description is given of the control method of the scraper conveyor of the present application applied to the control system of the scraper conveyor. The control system of the scraper conveyor may include a distributed vision detection device, a synchronous acquisition trigger device, a lighting device, a shearer drum detection module, a calculation unit, and a communication unit. The vision detection device (i.e., the binocular camera device) and the lighting device are distributed and installed at the position of the hydraulic support roof beam. The vision detection device is triggered by the synchronous acquisition trigger device to collect image data. The calculation unit can, based on vision measurement technology, realize the prediction of the coal height on the scraper conveyor, and can obtain the coal volume on the scraper conveyor in the monitoring area corresponding to the first moment according to the coal height; the shearer drum detection module detects the position of the shearer, and the calculation unit statistically sums the coal volumes in the monitoring areas corresponding to the respective vision detection devices behind the traveling direction of the shearer as the total coal volume transported by the scraper conveyor; the communication unit can send the total coal volume to the control system of the scraper conveyor to realize the control of the transportation speed of the scraper conveyor according to the total coal volume transported by the scraper conveyor.

[0177] It should be noted that, in order to ensure that the images collected by multiple vision detection devices at the same moment can cover the conveying area of the scraper conveyor, the minimum fields of view of adjacent vision detection devices may have a field of view overlap area or adjacent fields of view.

[0178] Among them, when determining the total coal volume transported by the scraper conveyor:

[0179] 1. Pre-build and train a coal semantic segmentation model (denoted as the target semantic segmentation model in the present application)

[0180] Build an initial coal semantic segmentation model, and use a binocular camera device to obtain at least one first training image, where each first training image is marked with a second coal area; for any first training image, input the first training image into the initial coal semantic segmentation model for semantic segmentation to obtain a coal segmentation area; based on the difference between the second position of the second coal area in the second training image and the third position of the coal segmentation area in the second training image, train the initial coal semantic segmentation model to obtain a trained coal semantic segmentation model.

[0181] 2. Pre-build and train a coal depth estimation model (denoted as the depth estimation model in the present application)

[0182] Construct an initial coal depth estimation model, and use a sample binocular camera device to monitor a sample scraper conveyor to obtain a pair of training images; wherein, the pair of training images includes a left-eye training image and a right-eye training image collected by the sample binocular camera device; input the first reference training image into the initial coal depth estimation model to obtain a first sample disparity map corresponding to the first reference training image; wherein, the first reference training image is a left-eye training image or a right-eye training image; based on the first sample disparity map, perform back-projection on the second reference training image other than the first reference training image in the pair of training images to obtain a back-projected image; according to the difference between the first reference training image and the back-projected image, train the initial coal depth estimation model to obtain a trained coal depth estimation model.

[0183] As an example, after using a sample binocular camera device to monitor a sample scraper conveyor to obtain a pair of training images, input the left-eye training image I t into the initial coal depth estimation model, and a first sample disparity map D corresponding to the left-eye training image can be obtained t , and based on the first sample disparity map D t and the calibration process of the sample binocular camera device, perform back-projection on the right-eye training image I t′ . According to formula (8), a back-projected image I t′→t can be obtained. In formula (8), T t′→t is the relative pose transformation matrix between the right-eye training image and the left-eye training image; K is the normalized internal parameter matrix of the binocular camera device; the proj() function is a projection calculation formula, and this function can calculate the reprojected two-dimensional coordinates based on D t , T t′→t and K; <> is a sampling operator.

[0184] Use photometric error as the loss function of the depth estimation model, that is, according to the difference ||I t -I t′→t ||1 between the left-eye training image I t and the back-projected image I t′→t , generate a second loss value L p according to formulas (10) and (11). In formulas (10) and (11), the pe() function can calculate the photometric error between the left-eye training image and the back-projected image; the SSIM() function can calculate the structural similarity between the left-eye training image and the back-projected image; α can be 0.85.

[0185] Thus, according to the second loss value L p , train the initial coal depth estimation model to minimize the value of the second loss value.

[0186] Thus, a trained coal depth estimation model can be effectively obtained.

[0187] 3. Application of Coal Semantic Segmentation Model and Coal Depth Estimation Model

[0188] The shearer drum detection unit can use an infrared pair-switch or visual target detection method installed on the shearer and hydraulic support to determine the position of the shearer drum, detect the hydraulic support corresponding to the drum behind the advancing direction of the shearer, and determine the number of binocular camera devices to be used according to the sorting of the hydraulic supports corresponding to the drum.

[0189] Suppose m binocular camera devices are used to monitor the scraper conveyor to obtain the first images collected by the m binocular camera devices at the first moment and the second images collected at the second moment. Among them, the scraper conveyor is in a coal state at the first moment and in a coal-free state at the second moment. j ∈ [1, m] and j is an integer, and m is a positive integer; for the j-th binocular camera device among the m binocular camera devices, input the first image into the coal depth estimation model to obtain the first disparity map corresponding to the first image; among them, the first image is the left-eye image or right-eye image collected by the j-th binocular camera device; according to the first disparity map, and according to the baseline and focal length corresponding to the j-th binocular camera device that collected the first image, obtain the first depth map corresponding to the first image; similarly, input the second image collected by the j-th binocular camera device into the coal depth estimation model respectively to obtain the second depth map corresponding to the second image; use the trained coal semantic segmentation model to perform semantic segmentation on the first image to obtain the first coal region; according to the first position of the first coal region in the first image, determine the first target region matching the first position from the first depth map, and determine the second target region matching the first position from the second depth map; for the i-th first pixel unit in the first target region, determine the second pixel unit matching the position of the first pixel unit from the second target region; based on the first depth H i of the first pixel unit and the second depth H' i of the matching second pixel unit, determine the coal height ΔH corresponding to the i-th first pixel unit according to formula (5). i Among them, i ∈ [1, n] and i is a positive integer, n is the number of first pixel units in the first target region, and n is a positive integer; according to the coal area S and coal height corresponding to each first pixel unit in the first target region, the coal volume of the monitoring region corresponding to the j-th binocular camera device at the first moment can be determined according to formula (6).

[0190] It should be noted that the value of the coal area S corresponding to the first pixel unit is related to the internal parameters of the binocular camera device and the distance between the binocular camera device and the coal surface. Therefore, S can be determined as a constant according to the calibration result of the binocular camera device and the actual installation height.

[0191] Finally, according to the sum of the coal volumes of the m binocular camera devices, the target coefficient H is determined according to formula (3); the correction coefficient δ is obtained, where the correction coefficient δ is determined according to the field of view overlap range of adjacent binocular camera devices; the target coefficient is corrected based on the correction coefficient, and according to formula (4), the total coal volume V conveyed by the scraper conveyor at the first moment can be obtained. s 。

[0192] In summary, the control method of the scraper conveyor of the present application is based on a distributed vision detection device and uses a depth estimation model to predict the total coal volume on the scraper conveyor, so that during the coal mining and transportation process, the intelligent speed regulation of the scraper conveyor can be realized according to the total coal volume, so that the variable-frequency scraper conveyor can intelligently adjust its working state according to the real-time coal volume (i.e., the total coal volume) on the scraper conveyor, that is, the transportation speed of the scraper conveyor can be controlled to achieve the purpose of energy conservation and consumption reduction.

[0193] Corresponding to the control method of the scraper conveyor provided in the above several embodiments, an embodiment of the present application also provides a control device for a scraper conveyor. Since the control device for the scraper conveyor provided in the embodiment of the present application corresponds to the control method of the scraper conveyor provided in the above several embodiments, the implementation manners of the control method of the scraper conveyor are also applicable to the control device for the scraper conveyor provided in this embodiment and will not be described in detail in this embodiment.

[0194] Figure 6 FIG. is a schematic structural diagram of a control device for a scraper conveyor provided in Embodiment 4 of the present application.

[0195] As Figure 6 shown, the control device 600 for the scraper conveyor may include: a first monitoring module 601, a first input module 602, a first determination module 603, a second determination module 604, and a control module 605...

[0196] Among them, the first monitoring module 601 is used to monitor the scraper conveyor by using a plurality of vision detection devices to obtain a first image collected by the plurality of vision detection devices at the first moment and a second image collected at the second moment, where the scraper conveyor is in a coal-bearing state at the first moment and the scraper conveyor is in a coal-free state at the second moment.

[0197] The first input module 602 is configured to input, for any visual detection device, the first image and the second image collected by the visual detection device into a depth estimation model respectively, so as to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image.

[0198] The first determination module 603 is configured to determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth map and the second depth map.

[0199] The second determination module 604 is configured to determine the total coal volume conveyed by the scraper conveyor at the first moment according to the coal volumes of multiple visual detection devices.

[0200] The control module 605 is configured to control the transportation speed of the scraper conveyor according to the total coal volume.

[0201] In a possible implementation manner of the embodiment of the present application, the first determination module 603 is configured to: perform semantic segmentation on the first image to obtain a first coal area in the first image; according to the first position of the first coal area in the first image, determine a first target area matching the first position from the first depth map, and determine a second target area matching the first position from the second depth map; determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth of each first pixel unit in the first target area and the second depth of each second pixel unit in the second target area.

[0202] In a possible implementation manner of the embodiment of the present application, the first determination module 603 is configured to: use a trained target semantic segmentation model to perform semantic segmentation on the first image to obtain a first coal area; wherein, the target semantic segmentation model is trained by the following steps: obtain at least one first training image, wherein each first training image is labeled with a second coal area; for any first training image, input the first training image into an initial semantic segmentation model for semantic segmentation to obtain a coal segmentation area; based on the difference between the second position of the second coal area in the second training image and the third position of the coal segmentation area in the second training image, train the initial semantic segmentation model to obtain the target semantic segmentation model.

[0203] In a possible implementation manner of the embodiment of the present application, the first determination module 603 is configured to: for any first pixel unit in the first target area, determine a second pixel unit matching the position of the first pixel unit from the second target area; based on the first depth of the first pixel unit and the second depth of the matching second pixel unit, determine the coal height corresponding to the first pixel unit; determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the coal area and the coal height corresponding to each first pixel unit in the first target area.

[0204] In a possible implementation manner of the embodiment of the present application, the second determination module 604 is configured to: determine a target coefficient according to the cumulative sum of the coal volumes of multiple visual detection devices; obtain a correction coefficient, where the correction coefficient is determined according to the field of view overlap range of adjacent visual detection devices; and correct the target coefficient based on the correction coefficient to obtain the total coal volume conveyed by the scraper conveyor at the first moment.

[0205] In a possible implementation manner of the embodiment of the present application, the visual detection device is a binocular camera device, and the first input module is configured to: input a first image into a depth estimation model to obtain a first disparity map corresponding to the first image; where the first image is a left-eye image or a right-eye image collected by the binocular camera device; and obtain a first depth map corresponding to the first image according to the first disparity map, as well as according to the baseline and focal length corresponding to the binocular camera device that collects the first image.

[0206] In a possible implementation manner of the embodiment of the present application, the depth estimation model is trained using the following modules:

[0207] The second monitoring module is configured to monitor a sample scraper conveyor using a sample binocular camera device to obtain a training image pair; where the training image pair includes a left-eye training image and a right-eye training image collected by the sample binocular camera device.

[0208] The second input module is configured to input a first reference training image into an initial depth estimation model to obtain a first sample disparity map corresponding to the first reference training image; where the first reference training image is a left-eye training image or a right-eye training image.

[0209] The back-projection module is configured to perform back-projection on a second reference training image other than the first reference training image in the training image pair based on the first sample disparity map to obtain a back-projected image.

[0210] The training module is configured to train the initial depth estimation model according to the difference between the first reference training image and the back-projected image to obtain a trained depth estimation model.

[0211] The control device of the scraper conveyor according to the embodiment of the present application monitors the scraper conveyor by using a plurality of visual detection devices to obtain a first image collected by the plurality of visual detection devices at a first moment and a second image collected at a second moment; for any one of the visual detection devices, the first image and the second image collected by the visual detection device are respectively input into a depth estimation model to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image; according to the first depth map and the second depth map, determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment; according to the coal volumes of the plurality of visual detection devices, determine the total coal volume conveyed by the scraper conveyor at the first moment, and control the transportation speed of the scraper conveyor according to the total coal volume. Thus, based on deep learning technology, according to the first image and the second image obtained by monitoring the scraper conveyor by a plurality of visual detection devices, the first depth map corresponding to the first image and the second depth map corresponding to the second image can be obtained. Further, according to each first depth map and each second depth map, the total coal volume conveyed by the scraper conveyor at the first moment can be automatically predicted, so that the transportation speed of the scraper conveyor can be controlled based on the total coal volume.

[0212] To implement the above embodiment, the present application also proposes an electronic device. Figure 7 The following is a schematic structural diagram of the electronic device provided in the fifth embodiment of the present application. The electronic device includes:

[0213] A memory 701, a processor 702, and a computer program stored on the memory 701 and executable on the processor 702.

[0214] When the processor 702 executes the program, it implements the control method of the scraper conveyor provided in the above embodiment.

[0215] Further, the electronic device further includes:

[0216] A communication interface 703 for communication between the memory 701 and the processor 702.

[0217] The memory 701 is used to store a computer program executable on the processor 702.

[0218] The memory 701 may include a high-speed RAM memory and may also include a non-volatile memory, such as at least one disk memory.

[0219] The processor 702 is configured to implement the control method of the scraper conveyor described in the above embodiment when executing the program.

[0220] If the memory 701, the processor 702, and the communication interface 703 are implemented independently, the communication interface 703, the memory 701, and the processor 702 can be interconnected via a bus to complete communication with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 only a thick line is used to represent it in Figure 7 , but it does not mean that there is only one bus or one type of bus.

[0221] Optionally, in a specific implementation, if the memory 701, the processor 702, and the communication interface 703 are integrated on a single chip, the memory 701, the processor 702, and the communication interface 703 can complete communication with each other through an internal interface.

[0222] The processor 702 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0223] To implement the above embodiments, the embodiments of the present application also propose a non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the control method of the scraper conveyor provided in the above embodiments.

[0224] To implement the above embodiments, the embodiments of the present application also propose a computer program product, and when the instructions in the computer program product are executed by a processor, it implements the control method of the scraper conveyor provided in the above embodiments.

[0225] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0226] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically and clearly defined.

[0227] Any process or method description shown in a flowchart or described in other ways herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logic function or process. And the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of this application pertain.

[0228] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definitional sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection parts with one or more wirings (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber devices, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.

[0229] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or combinations thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0230] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0231] In addition, each functional unit in various embodiments of the present application may be integrated into one processing module, may exist separately as individual physical units, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0232] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A control method for a scraper conveyor, characterized in that, The method includes: Using multiple visual detection devices to monitor the scraper conveyor to obtain a first image collected by the multiple visual detection devices at a first moment and a second image collected at a second moment, wherein the scraper conveyor is in a coal-bearing state at the first moment and the scraper conveyor is in a coal-free state at the second moment; For any one of the visual detection devices, inputting the first image and the second image collected by the visual detection device into a depth estimation model respectively to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image; According to the first depth map and the second depth map, determining the coal volume of the monitoring area corresponding to the visual detection device at the first moment; According to the coal volumes of the multiple visual detection devices, determining the total coal volume conveyed by the scraper conveyor at the first moment, and controlling the transportation speed of the scraper conveyor according to the total coal volume; The determining the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth map and the second depth map includes: Performing semantic segmentation on the first image to obtain a first coal area in the first image; According to the first position of the first coal area in the first image, determining a first target area matching the first position from the first depth map and determining a second target area matching the first position from the second depth map; According to the first depth of each first pixel unit in the first target area and the second depth of each second pixel unit in the second target area, determining the coal volume of the monitoring area corresponding to the visual detection device at the first moment; The performing semantic segmentation on the first image to obtain a first coal area in the first image includes: Using a trained target semantic segmentation model to perform semantic segmentation on the first image to obtain the first coal area; Wherein, the target semantic segmentation model is trained by the following steps: Obtaining at least one first training image, wherein each first training image is labeled with a second coal area; For any one of the first training images, inputting the first training image into an initial semantic segmentation model for semantic segmentation to obtain a coal segmentation area; Based on the difference between the second position of the second coal area in the second training image and the third position of the coal segmentation area in the second training image, training the initial semantic segmentation model to obtain a target semantic segmentation model.

2. The method according to claim 1, wherein The determining the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the first depth of each first pixel unit in the first target area and the second depth of each second pixel unit in the second target area includes: For any one of the first pixel units in the first target area, determining a second pixel unit matching the position of the first pixel unit from the second target area; Determine the coal height corresponding to the first pixel unit based on the first depth of the first pixel unit and the second depth of the matched second pixel unit; Determine the coal volume of the monitoring area corresponding to the visual detection device at the first moment according to the coal area and the coal height corresponding to each first pixel unit in the first target area.

3. The method according to claim 1, wherein The determining the total coal volume conveyed by the scraper conveyor at the first moment according to the coal volumes of the multiple visual detection devices includes: Determine a target coefficient according to the sum of the coal volumes of the multiple visual detection devices; Obtain a correction coefficient, where the correction coefficient is determined according to the field of view overlap range of adjacent visual detection devices; Correct the target coefficient based on the correction coefficient to obtain the total coal volume conveyed by the scraper conveyor at the first moment.

4. The method according to claim 1, wherein The visual detection device is a binocular camera device, The inputting the first image collected by the visual detection device into a depth estimation model to obtain a first depth map corresponding to the first image includes: Input the first image into the depth estimation model to obtain a first disparity map corresponding to the first image; wherein, the first image is a left-eye image or a right-eye image collected by the binocular camera device; Obtain a first depth map corresponding to the first image according to the first disparity map, and according to the baseline and focal length corresponding to the binocular camera device that collected the first image.

5. The method according to claim 4, wherein The depth estimation model is trained by the following steps: Monitor a sample scraper conveyor using a sample binocular camera device to obtain a training image pair; wherein, the training image pair includes a left-eye training image and a right-eye training image collected by the sample binocular camera device; Input a first reference training image into an initial depth estimation model to obtain a first sample disparity map corresponding to the first reference training image; wherein, the first reference training image is the left-eye training image or the right-eye training image; Based on the first sample disparity map, perform back-projection on a second reference training image other than the first reference training image in the training image pair to obtain a back-projected image; Train the initial depth estimation model according to the difference between the first reference training image and the back-projected image to obtain the trained depth estimation model.

6. A control device for a scraper conveyor, characterized in that, The device includes: A first monitoring module, configured to monitor the scraper conveyor using a plurality of visual detection devices to obtain a first image collected by the plurality of visual detection devices at a first moment and a second image collected at a second moment, wherein the scraper conveyor is in a coal-bearing state at the first moment, and the scraper conveyor is in a coal-free state at the second moment; A first input module, configured to, for any one of the visual detection devices, input the first image and the second image collected by the visual detection device into a depth estimation model respectively to obtain a first depth map corresponding to the first image and a second depth map corresponding to the second image; A first determination module, configured to determine the coal volume of the monitoring area corresponding to the vision detection device at the first moment according to the first depth map and the second depth map; A second determination module, configured to determine the total coal volume conveyed by the scraper conveyor at the first moment according to the coal volumes of the plurality of vision detection devices; A control module, configured to control the transportation speed of the scraper conveyor according to the total coal volume; The first determination module is configured to perform semantic segmentation on the first image to obtain a first coal area in the first image; According to the first position of the first coal area in the first image, determine a first target area matching the first position from the first depth map, and determine a second target area matching the first position from the second depth map; According to the first depth of each first pixel unit in the first target area and the second depth of each second pixel unit in the second target area, determine the coal volume of the monitoring area corresponding to the vision detection device at the first moment; The performing semantic segmentation on the first image to obtain a first coal area in the first image includes: Using a trained target semantic segmentation model to perform semantic segmentation on the first image to obtain the first coal area; Wherein, the target semantic segmentation model is trained by the following steps: Obtain at least one first training image, wherein each first training image is labeled with a second coal area; For any one of the first training images, input the first training image into an initial semantic segmentation model for semantic segmentation to obtain a coal segmentation area; Based on the difference between the second position of the second coal area in the second training image and the third position of the coal segmentation area in the second training image, train the initial semantic segmentation model to obtain a target semantic segmentation model.

7. An electronic device, characterized in that, Including: A memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the control method of the scraper conveyor according to any one of claims 1-5 when executing the program.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the control method of the scraper conveyor according to any one of claims 1-5.

Images