Scraper conveyer large coal identification method based on panoramic perception and scale estimation

Abstract

The invention provides a scraper conveyor large coal identification method based on panoramic perception and scale estimation, and the method comprises the steps: carrying out the panoramic segmentation of a to-be-identified image based on a pre-trained panoramic segmentation model, and obtaining the annotation data of pixels in the to-be-identified image; wherein the image to be recognized comprises a conveying belt, and a coal body is conveyed on the conveying belt; performing stream matching on the to-be-identified image to obtain depth data corresponding to each pixel in the to-be-identified image; classifying and aggregating pixels in the to-be-identified image based on the annotation data, and obtaining at least one target pixel set corresponding to a target coal body on the conveying belt; generating point cloud coordinate data of each pixel in the target pixel set based on the depth data; and identifying the target coal body based on the point cloud coordinate data. Through the technical scheme of the invention, the accuracy of coal body identification can be improved.

Description

Technical Field

[0001] The present application relates to the field of coal mining technology, and in particular to a method, device, equipment and storage medium for identifying large pieces of coal in a scraper conveyor based on panoramic perception and scale estimation. Background Art

[0002] In the related technology, the recognition of large pieces of coal mainly adopts the target detection algorithm in computer vision. This method has the following main problems: 1. Visual data is two-dimensional data. It is impossible to directly transfer the parameter configuration of large pieces of coal judged in one scene to another scene only by relying on the target detection result; 2. It is impossible to distinguish whether the large pieces of coal are on the scraper conveyor; 3. The robustness of the recognition result is poor. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] In the first aspect, the present application proposes a method for identifying large pieces of coal in a scraper conveyor based on panoramic perception and scale estimation, the method comprising: performing panoramic segmentation on the image to be identified based on a pre-trained panoramic segmentation model, and obtaining annotation data of pixels in the image to be identified; wherein the image to be identified includes a conveyor belt, on which coal is conveyed; performing flow matching on the image to be identified, and obtaining depth data corresponding to each pixel in the image to be identified; classifying and aggregating the pixels in the image to be identified based on the annotation data, and obtaining at least one target pixel set corresponding to the target coal on the conveyor belt; generating point cloud coordinate data for each pixel in the target pixel set based on the depth data; and identifying the target coal based on the point cloud coordinate data.

[0005] In one implementation, the panoramic segmentation model is pre-trained through the following steps: obtaining training data and sample annotation data corresponding to the training data; wherein the sample annotation data of each training data includes at least one of a sample semantic type, a sample instance ID, and an empty label; inputting the training data into an initial panoramic segmentation model to obtain predicted annotation data output by the initial panoramic segmentation model; based on a preset loss function, performing loss calculation in combination with the predicted annotation data and the sample annotation data to obtain a loss value; adjusting model parameters of the initial panoramic segmentation model based on the loss value to obtain the panoramic segmentation model.

[0006] In an optional implementation, the loss function of the panoptic segmentation model can be expressed as follows:

[0007]

[0008]

[0009] in, is the loss function value, is the segmentation loss, To identify the loss, and is a monotonically increasing function, To predict pixel regions, is the true pixel area, is the intersection-over-union ratio of the predicted pixel region and the true value pixel region, is the number of true positive examples, is the number of false positive examples, is the number of false negative examples.

[0010] In one implementation, the flow matching of the image to be identified is performed by a pre-trained second model, and the second model is trained by a pre-trained image diffusion model.

[0011] In one implementation, generating point cloud coordinate data of each pixel in the target pixel set based on the depth data includes: obtaining a camera intrinsic parameter matrix corresponding to the image to be identified; obtaining normalized coordinates of pixels in the image to be identified based on the camera intrinsic parameter matrix; and fusing the normalized coordinates with the depth data to obtain point cloud coordinate data of pixels in the image to be identified.

[0012] In one implementation, the annotation data includes semantic categories and instance IDs, the semantic categories include at least one of coal body, conveyor belt area and other categories, and the method of obtaining at least one target pixel set corresponding to the target coal body on the conveyor belt based on the pixel set corresponding to each instance includes: classifying the pixels based on the annotation data, obtaining a first pixel set corresponding to each coal body in the image to be identified in the pixel set, and a second pixel set corresponding to the conveyor belt; performing IOU calculation based on the first pixel set corresponding to each coal body and the second pixel set corresponding to the conveyor belt, respectively, to obtain an IOU value corresponding to each coal body; obtaining the coal body whose IOU value is greater than or equal to a preset threshold as the target coal body on the conveyor belt; and obtaining the target pixel set corresponding to each target coal body in the first pixel set.

[0013] In the second aspect, the present application proposes a device for identifying large pieces of coal in a scraper conveyor based on panoramic perception and scale estimation, and the device includes: a first processing module, which is used to perform panoramic segmentation on the image to be identified based on a pre-trained panoramic segmentation model, and obtain annotation data of pixels in the image to be identified; wherein the image to be identified includes a conveyor belt, and coal is conveyed on the conveyor belt; a second processing module, which is used to perform flow matching on the image to be identified, and obtain depth data corresponding to each pixel in the image to be identified; a third processing module, which is used to classify and aggregate pixels in the image to be identified based on the annotation data, and obtain at least one target pixel set corresponding to the target coal on the conveyor belt; a fourth processing module, which is used to generate point cloud coordinate data of each pixel in the target pixel set based on the depth data; and a fifth processing module, which is used to identify the target coal based on the point cloud coordinate data.

[0014] In one implementation, the panoramic segmentation model is pre-trained through the following steps: obtaining training data and sample annotation data corresponding to the training data; wherein the sample annotation data of each training data includes at least one of a sample semantic type, a sample instance ID, and an empty label; inputting the training data into an initial panoramic segmentation model to obtain predicted annotation data output by the initial panoramic segmentation model; based on a preset loss function, performing loss calculation in combination with the predicted annotation data and the sample annotation data to obtain a loss value; adjusting model parameters of the initial panoramic segmentation model based on the loss value to obtain the panoramic segmentation model.

[0015] In an optional implementation, the loss function of the panoptic segmentation model can be expressed as follows:

[0016]

[0017]

[0018] in, is the loss function value, is the segmentation loss, To identify the loss, and is a monotonically increasing function, To predict pixel regions, is the true pixel area, is the intersection-over-union ratio of the predicted pixel region and the true value pixel region, is the number of true positive examples, is the number of false positive examples, is the number of false negative examples.

[0019] In one implementation, the flow matching of the image to be identified is performed by a pre-trained second model, and the second model is trained by a pre-trained image diffusion model.

[0020] In one implementation, the fifth processing module can be used to: obtain a camera intrinsic parameter matrix corresponding to the image to be identified; obtain normalized coordinates of pixels in the image to be identified based on the camera intrinsic parameter matrix; and fuse the normalized coordinates with the depth data to obtain point cloud coordinate data of pixels in the image to be identified.

[0021] In one implementation, the annotation data includes semantic categories and instance IDs, and the semantic categories include at least one of coal body, conveyor belt area and other categories. The fourth processing module can be used to: classify the pixels based on the annotation data, and obtain the first pixel set corresponding to each coal body in the image to be identified in the pixel set, and the second pixel set corresponding to the conveyor belt; perform IOU calculation based on the first pixel set corresponding to each coal body and the second pixel set corresponding to the conveyor belt, respectively, to obtain the IOU value corresponding to each coal body; obtain the coal body whose IOU value is greater than or equal to a preset threshold as the target coal body on the conveyor belt; obtain the target pixel set corresponding to each target coal body in the first pixel set.

[0022] In the third aspect, the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for identifying large pieces of coal in a scraper conveyor based on panoramic perception and scale estimation as described in the first aspect.

[0023] In a fourth aspect, the present application proposes a computer-readable storage medium for storing instructions, which, when executed, enables the method described in the first aspect to be implemented.

[0024] In a fifth aspect, the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for identifying large coal lumps in a scraper conveyor based on panoramic perception and scale estimation as described in the first aspect.

[0025] The method, device, equipment and storage medium for identifying large pieces of coal in a scraper conveyor based on panoramic perception and scale estimation provided by the present application can perform panoramic segmentation on the image to be identified, obtain the annotation data of the pixels in the image to be identified, thereby obtaining at least one target pixel set corresponding to the target coal body on the conveyor belt based on the annotation data, and perform flow matching on the image to be identified to obtain depth data, so as to obtain the point cloud coordinate data of the pixels based on the depth data, thereby performing coal body identification based on the point cloud coordinate data of the pixels in at least one target pixel set corresponding to the target coal body. The accuracy of coal body identification can be improved.

[0026] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 It is a flow chart of a method for identifying large coal lumps in a scraper conveyor based on panoramic perception and scale estimation provided in an embodiment of the present application; Figure 2 is an example diagram of an image to be recognized provided in an embodiment of the present application; Figure 3 It is a flow chart of another method for identifying large coal lumps in a scraper conveyor based on panoramic perception and scale estimation provided in an embodiment of the present application; Figure 4 This is an example diagram of a flow matching model training process provided by an embodiment of the present application; Figure 5 It is a flow chart of another method for identifying large coal lumps in a scraper conveyor based on panoramic perception and scale estimation provided in an embodiment of the present application; Figure 6 It is a schematic diagram of a large-lump coal identification process of a scraper conveyor based on panoramic perception and scale estimation provided in an embodiment of the present application; Figure 7 It is a schematic diagram of a large-lump coal identification process of a scraper conveyor based on panoramic perception and scale estimation provided in an embodiment of the present application; Figure 8 It is a structural schematic diagram of a large-lump coal identification device for a scraper conveyor based on panoramic perception and scale estimation provided in an embodiment of the present application; Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] The following describes a method and device for identifying large lumps of coal in a scraper conveyor based on panoramic perception and scale estimation according to an embodiment of the present application with reference to the accompanying drawings.

[0030] Figure 1 1 is a flow chart of a method for identifying large coal lumps in a scraper conveyor based on panoramic perception and scale estimation provided in an embodiment of the present application. Figure 1 As shown, the method may include but is not limited to the following steps: Step S101: Perform panoptic segmentation on the image to be identified based on a pre-trained panoptic segmentation model to obtain annotation data of pixels in the image to be identified.

[0031] The image to be identified includes a conveyor belt on which coal is transported. The conveyor belt may be a conveyor belt of a scraper conveyor.

[0032] Exemplarily, an image to be identified including a conveyor belt is acquired by a pre-set image acquisition device, and panoramic segmentation is performed on the image to be identified based on a pre-trained panoramic segmentation model to obtain annotation data corresponding to each pixel in the image to be identified.

[0033] The above-mentioned annotation data may include but is not limited to semantic categories and instance IDs.

[0034] It should be noted that the above semantic categories include coal body, conveyor belt area of ​​scraper conveyor and other categories. As an example, see Figure 2 , Figure 2 is an example of an image to be identified provided by an embodiment of the present application. Figure 2 As shown, the above three semantic categories can be set in the image to be recognized.

[0035] It is understandable that due to the presence of a large number of large lumps of coal in the underground scene, some large lumps of coal are distributed on the scraper conveyor and move with the rotation of the scraper conveyor, while others are located in the cut coal falling area and are not on the scraper conveyor. At the same time, there may be many coal bodies of different sizes on the scraper conveyor. Therefore, it is necessary not only to distinguish which ones belong to the coal body category, but also to distinguish the specific coal body areas, so as to realize parallel analysis of different coal blocks to determine whether they belong to large lumps of coal or narrow and long coal on the conveyor belt.

[0036] Step S102: Perform flow matching on the image to be identified to obtain depth data corresponding to each pixel in the image to be identified.

[0037] Exemplarily, flow matching is performed on the image to be identified to estimate the depth of pixels in the image to be identified, and depth data corresponding to each pixel in the image to be identified is obtained.

[0038] Step S103: Classify and aggregate the pixels in the image to be identified based on the labeled data to obtain at least one target pixel set corresponding to the target coal body on the conveyor belt.

[0039] Exemplarily, pixels in the image to be identified are classified based on the annotation data, and pixels belonging to the same instance are taken as a pixel set, so as to obtain a pixel set corresponding to each instance in the image to be identified, and determine the first position of each instance based on the pixel set corresponding to each instance, and obtain the second position of the conveyor belt in the image to be identified, so as to determine whether the coal body corresponding to each instance is located on the conveyor belt based on the positional relationship between the first position and the second position, so as to obtain at least one target pixel set corresponding to the target coal body on the conveyor belt.

[0040] Step S104: Generate point cloud coordinate data of each pixel in the target pixel set based on the depth data.

[0041] Exemplarily, two-dimensional coordinate information corresponding to each pixel is generated based on the image to be identified, and the two-dimensional coordinate information is combined with the corresponding depth data to generate three-dimensional point cloud coordinate data of each pixel.

[0042] Step S105: Identify the target coal body based on the point cloud coordinate data.

[0043] Exemplarily, the point cloud coordinate data corresponding to the pixels of each target coal body is input into the point cloud prediction network to identify the volume size and shape of each target coal body and determine whether there are large coal bodies and / or narrow and long coal bodies with a volume greater than a preset volume threshold in the target coal body.

[0044] By implementing the embodiments of the present application, the image to be identified can be panorama segmented, and the annotation data of the pixels in the image to be identified can be obtained, so as to obtain at least one target pixel set corresponding to the target coal body on the conveyor belt based on the annotation data, and the image to be identified can be stream matched to obtain depth data, so as to obtain the point cloud coordinate data of the pixels based on the depth data, so as to perform coal body identification based on the point cloud coordinate data of the pixels in at least one target pixel set corresponding to the target coal body. The accuracy of coal body identification can be improved.

[0045] In some embodiments, the panoptic segmentation model is pre-trained by the following steps: A1: Obtain training data and sample annotation data corresponding to the training data.

[0046] The sample annotation data of each training data includes at least one of a sample semantic category, a sample instance ID and an empty label.

[0047] It should be noted that out-of-class pixels can be assigned a null label; that is, not all pixels must have a semantic label.

[0048] It should be noted that semantic tags contain sub-sets and , and meet , sub-collection The labels in the corresponding stuff, sub-collection The semantic label in corresponds to thing. , then the corresponding instance ID is irrelevant, so all stuff categories will belong to the same instance. On the contrary, if the labeled data corresponding to the pixel ,and , then have the same The pixels of the same instance have the same .

[0049] A2: Input the training data into the initial panoramic segmentation model to obtain the predicted annotation data output by the initial panoramic segmentation model.

[0050] A3: Based on the preset loss function, the loss is calculated by combining the predicted labeled data and the sample labeled data to obtain the loss value.

[0051] In some embodiments, a generalized panoptic quality evaluation function (Generalized PanopticQuality) may be used as the above loss function, which may be specifically expressed as follows:

[0052]

[0053]

[0054] in, is the loss function value, is the segmentation loss, To identify the loss, and is a monotonically increasing function, To predict pixel regions, is the true pixel area, is the intersection-over-union ratio of the predicted pixel area and the true value pixel area, (True Positives) is the number of true positives, that is, the number of samples correctly identified as positive classes by the model. (False Positives) is the number of false positive examples, that is, the number of samples where the model mistakenly identifies negative samples as positive samples. (False Negatives) is the number of false negative examples, that is, the number of samples where the model mistakenly identifies positive samples as negative samples.

[0055] Optionally, the Huber Loss function can be used as the above and .

[0056] A4: Adjust the model parameters of the initial panoramic segmentation model based on the loss value to obtain a panoramic segmentation model.

[0057] In some embodiments, the point cloud coordinate data of each pixel can be generated based on the relevant information of the device that obtains the image to be recognized and combined with the depth data. As an example, see Figure 3 , Figure 3 FIG. 1 is a flow chart of another method for identifying large coal lumps in a scraper conveyor based on panoramic perception and scale estimation provided in an embodiment of the present application. Figure 3 As shown, the method may include but is not limited to the following steps: Step S301: Perform panoptic segmentation on the image to be identified based on a pre-trained panoptic segmentation model to obtain annotation data of pixels in the image to be identified.

[0058] In the embodiments of the present application, step S301 can be implemented in any of the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.

[0059] Step S302: Perform flow matching on the image to be identified to obtain depth data corresponding to each pixel in the image to be identified.

[0060] In the embodiments of the present application, step S302 can be implemented by any of the methods in the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.

[0061] Step S303: Classify the pixels in the image to be identified based on the labeled data to obtain at least one pixel set corresponding to the coal body on the conveyor belt.

[0062] In the embodiments of the present application, step S303 can be implemented by any of the methods in the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.

[0063] Step S304: Obtain the camera intrinsic parameter matrix corresponding to the image to be recognized.

[0064] Exemplarily, a camera intrinsic parameter matrix of an image acquisition device corresponding to the image to be recognized is obtained, wherein the image acquisition device is a device that acquires the image to be recognized.

[0065] Step S305: obtaining normalized coordinates of pixels in the image to be identified based on the camera intrinsic parameter matrix.

[0066] Exemplarily, based on the camera intrinsic parameter matrix, the coordinates of each pixel in the image to be identified in the camera coordinate system are converted into coordinates in the image coordinate system, and then the coordinates in the image coordinate system are converted into coordinates in the pixel coordinate system through scaling and translation.

[0067]

[0068] in,( , , ) is the coordinate of the pixel in the camera coordinate system, The image acquisition device is The focal length in the direction, The image acquisition device is The focal length in the direction, and is the coordinate of the principal point of the image to be identified, that is, the position of the center point of the image in the pixel coordinate system.

[0069] Step S306: Fusing the normalized coordinates with the depth data to obtain point cloud coordinate data of pixels in the image to be identified.

[0070] Exemplarily, the coordinates of each pixel in the pixel coordinate system are fused with the corresponding depth data to obtain the point cloud coordinates of each pixel. Specifically, it can be expressed as follows:

[0071] in, is the normalized coordinate, is the depth estimation value. Then the three-dimensional coordinates of the real scale can be obtained, and then the point cloud coordinates corresponding to the coal body under a single perspective can be obtained.

[0072] Step S307: Identify the target coal body based on the point cloud coordinate data.

[0073] By implementing the embodiments of the present application, the normalized coordinates of each pixel in the image to be identified can be obtained based on the camera intrinsic parameter matrix corresponding to the image to be identified, and then the point cloud coordinate data of each pixel can be obtained. Coal body detection can be performed based on the point cloud coordinates. The accuracy of identifying large pieces of coal can be improved and the accuracy of detection can be guaranteed when the device state of the image acquisition device changes.

[0074] In some embodiments, flow matching of the image to be recognized is performed using a pre-trained flow matching model, and the flow matching model is trained using a pre-trained image diffusion model.

[0075] As an example, see Figure 4 , Figure 4 is an example diagram of a flow matching model training process provided by an embodiment of the present application. Figure 4 As shown, the pre-trained image diffusion model can be used as a sufficient prior for the flow matching model, so that the initial flow matching model is trained using only synthetic data, and the auxiliary surface normal loss is used as an auxiliary loss function during the training process. The trained flow matching model is used to perform flow matching on the image to be recognized.

[0076] It is understandable that due to the lack of scale information in the scraper conveyor image under a single perspective, it is impossible to obtain the real coal block size information only by relying on the results of panoramic segmentation, and the scale will also change with the influence of zoom of the image acquisition device, movement of the camera perspective, etc. Therefore, this application realizes the perception of the real scale of the coal body by perceiving the scene depth information to improve the recognition accuracy.

[0077] In some embodiments, the annotation data includes semantic data and instance ID, so that the pixels can be classified based on the annotation data corresponding to each pixel to obtain a pixel set corresponding to the coal body on the conveyor belt. As an example, see Figure 5 , Figure 5 FIG. 1 is a flow chart of another method for identifying large coal lumps in a scraper conveyor based on panoramic perception and scale estimation provided in an embodiment of the present application. Figure 5 As shown, the method may include but is not limited to the following steps: Step S501: Perform panoptic segmentation on the image to be identified based on a pre-trained panoptic segmentation model to obtain annotation data of pixels in the image to be identified.

[0078] In the embodiments of the present application, step S501 can be implemented in any of the ways in the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.

[0079] Step S502: Perform flow matching on the image to be identified to obtain depth data corresponding to each pixel in the image to be identified.

[0080] In the embodiments of the present application, step S502 can be implemented by any of the methods in the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.

[0081] Step S503: Classify and aggregate the pixels based on the labeled data to obtain a first pixel set corresponding to each coal body in the image to be identified in the pixel set, and a second pixel set corresponding to the conveyor belt.

[0082] Exemplarily, based on the annotation data corresponding to the pixels in each pixel set, a first target pixel set corresponding to the coal body in the image to be identified and a second pixel set corresponding to the conveyor belt in the image to be identified are obtained.

[0083] Step S504: performing IOU calculation based on the first pixel set corresponding to each coal body and the second pixel set corresponding to the conveyor belt, respectively, to obtain the IOU value corresponding to each coal body.

[0084] Exemplarily, the second area range of each coal body in the image to be identified is determined based on the first pixel set corresponding to each coal body, and the second area range corresponding to each coal body and the first area range corresponding to the conveyor belt are calculated IOU to obtain the IOU value corresponding to each coal body.

[0085] Step S505: Acquire the coal body whose IOU value is greater than or equal to the preset threshold as the target coal body on the conveyor belt.

[0086] Exemplarily, the coal body with an IOU value greater than or equal to 0.5 is taken as the target coal body on the conveyor belt.

[0087] Step S506: Acquire a target pixel set corresponding to each target coal body in the first target pixel set.

[0088] Exemplarily, a target pixel set corresponding to the target coal body on the conveyor belt in the first target pixel set is obtained.

[0089] Step S507: Generate point cloud coordinate data of each pixel in the target pixel set based on the depth data.

[0090] In the embodiments of the present application, step S507 can be implemented by any of the methods in the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.

[0091] Step S508: Identify the target coal body based on the point cloud coordinate data.

[0092] In the embodiments of the present application, step S508 can be implemented in any of the ways in the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.

[0093] By implementing the embodiments of the present application, the target coal body on the conveyor belt can be determined based on the IOU value between the coal body and the conveyor belt, thereby obtaining the point cloud coordinate data of the pixels corresponding to the target coal body based on the depth data of the pixels in the target pixel set corresponding to the target coal body, so as to identify the coal body on the conveyor belt based on the point cloud coordinate data.

[0094] See also Figure 6 , Figure 6 Schematic diagram of a large coal block identification process for a scraper conveyor based on panoramic perception and scale estimation provided by an embodiment of the present application. Figure 6 As shown, the depth-first search algorithm is first used to select a pixel as the current pixel, and then diffuse from the current pixel to the surrounding areas. When a pixel with the same semantic category and the same instance is encountered, the pixel coordinate tuple is put into the set, otherwise it is exited. Thus, at least one pixel set is obtained, and then the pixel set corresponding to the target coal body on the conveyor belt of the scraper conveyor is obtained from the pixel set.

[0095] Then, the normalized coordinates of the pixels are calculated based on the camera intrinsic parameter matrix. The z coordinates of the calculated normalized coordinates are all 1.0. Therefore, the depth data obtained in the previous steps can be further combined to obtain the real coordinates of each pixel in the camera coordinate system, thereby obtaining the point cloud coordinate data of the target coal body on the conveyor belt under a single perspective.

[0096] Finally, the point cloud coordinate data of the target coal body is input into the pre-trained point cloud prediction network PointNet to identify the volume of the target coal body and determine whether there are large pieces of coal and / or narrow and long coal with a volume greater than or equal to a preset threshold in the target coal body.

[0097] See also Figure 7 , Figure 7 Schematic diagram of a large coal block identification process for a scraper conveyor based on panoramic perception and scale estimation provided by an embodiment of the present application. Figure 7 As shown, firstly, panoramic semantic perception is performed on the image containing the scraper conveyor to obtain the semantic category information and instance information corresponding to each pixel in the image, so as to determine the pixel set corresponding to each coal body based on the semantic category information and instance information, and perform IOU calculation based on the pixel set corresponding to each coal body and the area where the conveyor belt of the scraper conveyor in the image is located, so as to obtain the target coal body on the scraper conveyor; at the same time, flow matching is performed on the image to obtain the depth information of each pixel, so as to fuse the depth information of each pixel with the image coordinates of the pixel corresponding to the target coal body to obtain the point cloud coordinate data of the target coal body, so as to identify based on the point cloud coordinate data of the target coal body to determine whether there are large pieces of coal or narrow and long coal bodies on the conveyor belt of the scraper conveyor.

[0098] See also Figure 8 , Figure 8 Schematic diagram of the structure of a large-lump coal identification device for a scraper conveyor based on panoramic perception and scale estimation provided in an embodiment of the present application. Figure 8 As shown, the device 800 includes: a first processing module 801, which is used to perform panoramic segmentation on the image to be identified based on a pre-trained panoramic segmentation model, and obtain annotation data of pixels in the image to be identified; wherein the image to be identified includes a conveyor belt, and coal is conveyed on the conveyor belt; a second processing module 802, which is used to perform flow matching on the image to be identified, and obtain depth data corresponding to each pixel in the image to be identified; a third processing module 803, which is used to classify and aggregate pixels in the image to be identified based on the annotation data, and obtain at least one target pixel set corresponding to the target coal on the conveyor belt; a fourth processing module 804, which is used to generate point cloud coordinate data of each pixel in the target pixel set based on the depth data; and a fifth processing module 805, which is used to identify the target coal based on the point cloud coordinate data.

[0099] In one implementation, a panoramic segmentation model is pre-trained through the following steps: obtaining training data and sample annotation data corresponding to the training data; wherein the sample annotation data of each training data includes at least one of a sample semantic type, a sample instance ID, and an empty label; inputting the training data into an initial panoramic segmentation model to obtain predicted annotation data output by the initial panoramic segmentation model; based on a preset loss function, performing loss calculation on the predicted annotation data and the sample annotation data to obtain a loss value; and adjusting model parameters of the initial panoramic segmentation model based on the loss value to obtain a panoramic segmentation model.

[0100] In an optional implementation, the loss function of the panoptic segmentation model can be expressed as follows:

[0101]

[0102]

[0103] in, is the loss function value, is the segmentation loss, To identify the loss, and is a monotonically increasing function, To predict pixel regions, is the true pixel area, is the intersection-over-union ratio of the predicted pixel area and the true value pixel area, is the number of true positive examples, is the number of false positive examples, is the number of false negative examples.

[0104] In one implementation, flow matching of the image to be recognized is performed using a pre-trained second model, and the second model is trained using a pre-trained image diffusion model.

[0105] In one implementation, the fifth processing module 805 can be used to: obtain a camera intrinsic parameter matrix corresponding to the image to be identified; obtain normalized coordinates of pixels in the image to be identified based on the camera intrinsic parameter matrix; and fuse the normalized coordinates with the depth data to obtain point cloud coordinate data of pixels in the image to be identified.

[0106] In one implementation, the annotation data includes a semantic category and an instance ID, and the semantic category includes at least one of coal body, conveyor belt area and other categories. The fourth processing module 804 can be used to: classify pixels based on the annotation data, and obtain a first pixel set corresponding to each coal body in the image to be identified in the pixel set, and a second pixel set corresponding to the conveyor belt; perform IOU calculation based on the first pixel set corresponding to each coal body and the second pixel set corresponding to the conveyor belt, respectively, to obtain an IOU value corresponding to each coal body; obtain a coal body with an IOU value greater than or equal to a preset threshold as a target coal body on the conveyor belt; and obtain a target pixel set corresponding to each target coal body in the first target pixel set.

[0107] Through the device of the embodiment of the present application, the image to be identified can be panorama segmented, and the annotation data of the pixels in the image to be identified can be obtained, so as to obtain at least one target pixel set corresponding to the target coal body on the conveyor belt based on the annotation data, and the image to be identified can be stream matched to obtain depth data, so as to obtain the point cloud coordinate data of the pixels based on the depth data, so as to perform coal body identification based on the point cloud coordinate data of the pixels in at least one target pixel set corresponding to the target coal body. The accuracy of coal body identification can be improved.

[0108] It should be noted that the aforementioned explanation of the embodiment of the method for identifying large lumps of coal on a scraper conveyor based on panoramic perception and scale estimation is also applicable to the device for identifying large lumps of coal on a scraper conveyor based on panoramic perception and scale estimation of this embodiment, and will not be repeated here.

[0109] In order to implement the above embodiment, the present application also proposes an electronic device. Fig. 9 , Fig. 9 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Fig. 9 As shown, the electronic device 900 includes: a processor 901, and a memory 902 communicatively connected to the processor 901; the memory 902 stores computer-executable instructions; the processor 901 executes the computer-executable instructions stored in the memory to implement the method provided in the aforementioned embodiment.

[0110] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0111] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0112] In the description of this application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in this article is merely a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0113] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0114] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0115] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0116] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0117] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0118] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0119] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0120] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for identifying large coal lumps in a scraper conveyor based on panoramic perception and scale estimation, characterized in that: include: Performing panoramic segmentation on the image to be identified based on a pre-trained panoramic segmentation model to obtain annotation data of pixels in the image to be identified; wherein the image to be identified includes a conveyor belt on which coal is transported; Performing stream matching on the image to be identified to obtain depth data corresponding to each pixel in the image to be identified; Classify and aggregate pixels in the image to be identified based on the labeled data to obtain at least one target pixel set corresponding to the target coal body on the conveyor belt; Generate point cloud coordinate data of each pixel in the target pixel set based on the depth data; The target coal body is identified based on the point cloud coordinate data.

2. The method according to claim 1, characterized in that The panoptic segmentation model is pre-trained by the following steps: Acquire training data and sample annotation data corresponding to the training data; wherein the sample annotation data of each training data includes at least one of a sample semantic type, a sample instance ID, and an empty label; Inputting the training data into an initial panoramic segmentation model to obtain predicted annotation data output by the initial panoramic segmentation model; Based on a preset loss function, the predicted labeled data and the sample labeled data are combined to perform loss calculation to obtain a loss value; The model parameters of the initial panoramic segmentation model are adjusted based on the loss value to obtain the panoramic segmentation model.

3. The method according to claim 2, characterized in that The loss function of the panoptic segmentation model can be expressed as follows: in, is the loss function value, is the segmentation loss, To identify the loss, and is a monotonically increasing function, To predict pixel area, is the true pixel area, is the intersection-over-union ratio of the predicted pixel region and the true value pixel region, is the number of true positive examples, is the number of false positive examples, is the number of false negative examples.

4. The method according to claim 1, characterized in that The flow matching of the image to be identified is performed by a pre-trained second model, and the second model is trained by a pre-trained image diffusion model.

5. The method according to claim 1, characterized in that The step of generating point cloud coordinate data of each pixel in the target pixel set based on the depth data comprises: Obtaining a camera intrinsic parameter matrix corresponding to the image to be recognized; Acquire normalized coordinates of pixels in the image to be identified based on the camera intrinsic parameter matrix; The normalized coordinates are fused with the depth data to obtain point cloud coordinate data of pixels in the image to be identified.

6. The method according to claim 1, characterized in that The annotated data includes a semantic category and an instance ID, the semantic category includes at least one of a coal body, a conveyor belt area, and other categories, and the pixels in the image to be identified are classified and aggregated based on the annotated data to obtain at least one target pixel set corresponding to the target coal body on the conveyor belt, including: Classify the pixels based on the labeled data, and obtain a first pixel set corresponding to each coal body in the image to be identified in the pixel set, and a second pixel set corresponding to the conveyor belt; Performing intersection-over-union (IOU) calculation based on the first pixel set corresponding to each coal body and the second pixel set corresponding to the conveyor belt, respectively, to obtain an IOU value corresponding to each coal body; Acquire the coal body whose IOU value is greater than or equal to a preset threshold as the target coal body on the conveyor belt; The target pixel set corresponding to each target coal body in the first pixel set is obtained.

7. A large-lump coal identification device for a scraper conveyor based on panoramic perception and scale estimation, characterized in that: include: The first processing module is used to perform panoramic segmentation on the image to be identified based on a pre-trained panoramic segmentation model, and obtain the annotation data of the pixels in the image to be identified; wherein the image to be identified includes a conveyor belt, and the conveyor belt conveys coal; A second processing module is used to perform flow matching on the image to be identified, and obtain depth data corresponding to each pixel in the image to be identified; A third processing module is used to classify and aggregate the pixels in the image to be identified based on the annotation data to obtain at least one target pixel set corresponding to the target coal body on the conveyor belt; A fourth processing module, configured to generate point cloud coordinate data of each pixel in the target pixel set based on the depth data; The fifth processing module is used to identify the target coal body based on the point cloud coordinate data.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The method comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images