Aggregate particle size monitoring method, apparatus, device and engineering machine

Abstract

The application relates to the field of artificial intelligence identification technology, and provides an aggregate particle size monitoring method, device, equipment and storage medium, an original image frame is acquired; the original image frame is preprocessed to obtain multiple sub-images after cutting; the multiple sub-images are input into a segmentation network model as input, a segmentation mask picture after removing the blocked aggregate is acquired from the segmentation network model output, the particle size of the aggregate corresponding to each segmentation mask picture is calculated based on the segmentation mask picture; the above steps are repeated to statistically obtain the particle size of the aggregate in a preset time period, and the overall particle size distribution of the aggregate is obtained according to the particle size of the aggregate in the preset time period. Through the above steps, the real-time statistics of the aggregate particle size distribution are realized, the blocked aggregate and the aggregate at the edge are removed, the subsequent particle size statistics of the blocked aggregate and the aggregate at the edge are avoided, and the accuracy of the statistics of the overall particle size distribution of the aggregate is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence recognition technology, and in particular to a method, apparatus, equipment and engineering machinery for monitoring aggregate particle size. Background Technology

[0002] During the aggregate production process, staff need to determine whether the aggregate meets the production requirements based on the aggregate size distribution information and make real-time adjustments to the aggregate mix ratio.

[0003] Currently, the main method for monitoring aggregate particle size is to use segmentation algorithms based on deep learning. However, due to the significant occlusion caused by stacked aggregates, the predicted aggregate particle size deviates considerably from the actual aggregate particle size. If such occluded aggregates are also included in the overall aggregate distribution, the predicted overall aggregate distribution will also deviate significantly from the actual distribution. Therefore, a monitoring method is needed that can remove occluded aggregates and improve the accuracy of statistical analysis of the overall aggregate distribution. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and storage medium for monitoring aggregate particle size, which addresses the shortcomings of existing aggregate particle size monitoring methods that count the particle size of obscured aggregates, resulting in a large deviation from the actual particle size and inaccurate overall aggregate distribution. The invention removes obscured aggregates, thereby improving the accuracy of the overall aggregate distribution statistics.

[0005] This invention provides a method for monitoring aggregate particle size, comprising:

[0006] Obtain the original image frame;

[0007] The original image frame is preprocessed to obtain multiple sub-images after segmentation;

[0008] The multiple sub-images are fed into the segmentation network model as input to obtain the segmentation mask image after removing the occluded aggregates output by the segmentation network model.

[0009] Based on the segmentation mask, calculate the particle size of the aggregate corresponding to each segmentation mask;

[0010] Repeat the above steps to count the aggregate particle size at a preset time period, and obtain the overall particle size distribution of the aggregate based on the aggregate particle size within the preset time period.

[0011] The segmentation network model is built based on a real-time segmentation network and trained by deep learning on a dataset. Each set of data in the dataset includes sample images containing aggregates, as well as corresponding annotation information for labeling complete aggregates and occluded aggregates in the sample images.

[0012] According to the present invention, an aggregate particle size monitoring method is provided, which calculates the aggregate particle size corresponding to each segmentation mask image, including:

[0013] The boundary coordinate set of the segmentation mask is fed into the cv2.minAreaRect function in the OpenCV library to calculate the rectangular box corresponding to the segmentation mask.

[0014] The length of the shortest side of the rectangle is used as the particle size of the aggregate corresponding to the segmentation mask image.

[0015] According to the present invention, an aggregate particle size monitoring method preprocesses the original image frame to obtain multiple segmented sub-images, including:

[0016] The original image is segmented into multiple sub-images according to a first preset size, so that the sub-images meet the input size requirements of the segmentation network model.

[0017] According to the present invention, an aggregate particle size monitoring method is provided to obtain the overall particle size distribution of the aggregate based on the particle size of the aggregate within a preset time period, including:

[0018] The aggregate particle size is compared with the preset particle size classification threshold, and the aggregate particle size is classified into the corresponding particle size classification.

[0019] Based on the total aggregate particle size within the preset time period and the number of aggregate particles in each particle size category, the proportion of aggregate in each particle size category is calculated.

[0020] According to the present invention, an aggregate particle size monitoring method further includes, after obtaining the segmentation mask image output by the segmentation network model after removing the obscured aggregate, and before calculating the particle size of the aggregate corresponding to each segmentation mask image, the method further includes:

[0021] Based on the boundary information of the segmentation mask, the aggregate at the edge corresponding to the segmentation mask in the sub-image is removed.

[0022] According to the present invention, an aggregate particle size monitoring method is provided, which removes aggregate at the edge corresponding to the segmentation mask in the sub-image based on the boundary information of the segmentation mask, including:

[0023] The boundary coordinates of the segmentation mask are compared with the boundary coordinates of the sub-image. If the boundary coordinates of the segmentation mask overlap with the boundary coordinates of the sub-image, the aggregate corresponding to the segmentation mask in the sub-image is removed.

[0024] According to the aggregate particle size monitoring method provided by the present invention, after acquiring the original image frame and before preprocessing the original image frame, the method further includes:

[0025] Obtain a classified image frame, wherein the classified image frame is an image frame obtained by adjusting the original image frame to a second preset size;

[0026] The classified image frame is fed into the classification network to obtain the classification result output by the classification network;

[0027] If the classification result is not aggregate, proceed to the step of acquiring the original image frame.

[0028] The present invention also provides an aggregate particle size monitoring device, comprising:

[0029] The image acquisition module is used to acquire raw image frames;

[0030] The preprocessing module is used to preprocess the original image frame to obtain multiple sub-images after segmentation;

[0031] The segmentation module is used to feed the multiple sub-images as input into the segmentation network model and obtain the segmentation mask image after removing the occluded aggregates output by the segmentation network model. The segmentation network model is built based on a real-time segmentation network and trained by deep learning based on a dataset. Each set of data in the dataset includes sample images containing aggregates and corresponding annotation information for labeling the complete aggregates and occluded aggregates in the sample images.

[0032] The particle size calculation module is used to calculate the particle size of the aggregate corresponding to each segmentation mask based on the segmentation mask.

[0033] The statistics module is used to count the aggregate particle size at a preset time period, and to obtain the overall particle size distribution of the aggregate based on the aggregate particle size within the preset time period.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aggregate particle size monitoring method as described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aggregate particle size monitoring method as described above.

[0036] The aggregate particle size monitoring method, apparatus, device, and storage medium provided by this invention acquires original image frames; preprocesses the original image frames to obtain multiple sub-images; inputs these sub-images into a segmentation network model to obtain a segmentation mask image output by the model after removing obscured aggregate; calculates the aggregate particle size corresponding to each segmentation mask image; repeats the above steps to statistically analyze the aggregate particle size at a preset time period; and obtains the overall aggregate particle size distribution based on the aggregate particle size within the preset time period. Through these steps, real-time statistical analysis of the aggregate particle size distribution is achieved, and by removing obscured aggregate, subsequent particle size analysis of obscured aggregate is avoided, thus improving the accuracy of the statistical analysis of the overall aggregate particle size distribution. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the aggregate particle size monitoring method provided by the present invention;

[0039] Figure 2 This is a schematic diagram of the aggregate particle size monitoring device provided by the present invention;

[0040] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0041] Figure label:

[0042] 21: Image acquisition module; 22: Preprocessing module; 23: Segmentation module; 24: Particle size calculation module; 25: Statistics module. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] The following is combined with Figure 1 The present invention describes the aggregate particle size monitoring method.

[0045] like Figure 1As shown, the aggregate particle size monitoring method provided by the present invention includes the following steps:

[0046] S1. Obtain the original image frame.

[0047] In this step, it is necessary to acquire the original image frame of the object under test. In an optional embodiment of the present invention, the original image frame can be obtained by real-time imaging of the conveyor belt using an industrial camera deployed above the conveyor belt.

[0048] S2. Preprocess the original image frame to obtain multiple sub-images after segmentation.

[0049] In this step, the original image frame needs to be segmented. Preferably, the original image needs to be segmented into multiple sub-images according to a first preset size so that the sub-images meet the input size requirements of the segmentation network model. For example, assuming the input size requirement (resolution) of the segmentation network model is 512*512, and the resolution of the original image frame is 2048*2048, then the original image frame needs to be segmented into 16 sub-images of 512*512.

[0050] S3. Input multiple sub-images into the segmentation network model and obtain the segmentation mask image output by the segmentation network model after removing the occluded aggregate.

[0051] The segmentation network model in this invention is based on a real-time segmentation network. Specifically, it is built upon the YOLO v8 network, with an input size of 512*512. The backbone network uses the YOLO v8s network, and the head uses the YOLOACT head. The loss function comprises object category loss, bounding box regression loss, and mask classification loss. Specifically, the object category loss uses binary cross-entropy (BCE) to determine the object category; the bounding box regression loss uses IOU (Intersection over Union) loss; and the mask classification loss is calculated using binary cross-entropy loss. The segmentation network model is trained using deep learning on a dataset. Each set of data in the dataset includes sample images containing aggregate, as well as corresponding annotation information for labeling complete and occluded aggregate in the sample images. In actual training, the dataset was divided into a test dataset and a training dataset in a 2:8 ratio, with 50 epochs (50 training epochs total) and a learning rate of 0.005. The resulting segmentation network model can identify all intact aggregates and occluded aggregates in an image. It can also remove occluded aggregates as needed to prevent them from participating in subsequent particle size analysis, outputting a segmentation mask map to represent individual intact aggregates.

[0052] In an optional embodiment of the present invention, the segmentation mask image is a binary image with pixels of 0 or 1 that is the same size as the sub-image, wherein the pixels of the aggregate portion are 0 and the pixels of the other portions are 1.

[0053] S4. Based on the segmentation mask, calculate the aggregate particle size corresponding to each segmentation mask.

[0054] Specifically, in this step, the boundary coordinate set (xy coordinate set) of the segmentation mask is fed into the `cv2.minAreaRect` function to obtain a bounding box. The `cv2.minAreaRect` function is a function in the OpenCV library designed to find the smallest area of ​​a rotated rectangle. It takes a 2D point coordinate set as input and returns a 2D box structure containing the following details: center coordinates (x, y), width, height, and rotation angle. After obtaining the bounding box, the shortest side of the rectangle is used as the aggregate size corresponding to that segmentation mask.

[0055] Repeat this step to calculate the aggregate particle size corresponding to all segmentation mask images.

[0056] S5. Repeat steps S1-S4 to count the aggregate particle size at a preset time period. Based on the aggregate particle size within the preset time period, obtain the overall particle size distribution of the aggregate.

[0057] In this step, raw image frames need to be continuously acquired, and the aggregate particle size information within a preset time period needs to be statistically obtained according to the methods in steps S1-S4. It should be understood that the present invention does not limit the preset time period; the time period can be set according to actual conditions, for example, set to 2 seconds. In an optional embodiment of the present invention, the overall particle size distribution of the aggregate is obtained based on the aggregate particle size within the preset time period, specifically including:

[0058] The aggregate particle size is compared with the preset particle size classification threshold, and the aggregate particle size is classified into the corresponding particle size classification.

[0059] Based on the total aggregate size within a preset time period and the number of aggregate sizes in each size category, the proportion of aggregate in each size category is calculated.

[0060] For example, particle size can be classified into three categories: the first category has a particle size threshold of 5-10 mm, the second category has a threshold of 10-20 mm, and the third category has a threshold of 20-31.5 mm. Assuming that 20 aggregate particle sizes are collected within a preset time period, with 5 aggregates falling within the 5-10 mm range, 10 aggregates falling within the 10-20 mm range, and 5 aggregates falling within the 20-31.5 mm range, then the proportion of aggregates in the first category is 25%, the proportion in the second category is 50%, and the proportion in the third category is 25%.

[0061] Since the aggregate at the image edge is usually incomplete, the predicted particle size of such aggregate differs significantly from the actual result. To further improve the accuracy of monitoring the overall distribution of aggregate particle size, it is necessary to remove the aggregate at the image edge. In an optional embodiment of the present invention, after obtaining the segmentation mask image output by the segmentation network model in step S3 after removing the occluded aggregate, and before calculating the particle size of the aggregate corresponding to each segmentation mask image in step S4, the following steps are also included:

[0062] Based on the boundary information of the segmentation mask, remove the aggregate at the corresponding edge of the segmentation mask in the sub-image.

[0063] Specifically, in the segmentation mask image corresponding to each sub-image, the boundary coordinates of the segmentation mask image are compared with the boundary coordinates of the sub-image. If the boundary coordinates of the segmentation mask image overlap with the boundary coordinates of the sub-image, the aggregate corresponding to that segmentation mask image in the sub-image is removed to prevent the aggregate at the image edge from participating in the subsequent particle size statistics.

[0064] In an optional embodiment of the present invention, in order to ensure that subsequent steps are only performed when the original image frame contains aggregate, thus avoiding wasting computational resources, after obtaining the original image frame in step S1 and before preprocessing the original image frame in step S2, the method further includes identifying the original image frame, specifically including the following steps:

[0065] Obtain the classification image frame. The classification image frame is the image frame obtained by adjusting the original image frame to a second preset size. It should be understood that the second preset size should match the input size of the classification network. Assuming that the input resolution of the classification network is 800*800 and the resolution of the original image frame is 2048*2048, the original image frame needs to be resized to 800*800.

[0066] The classified image frames are fed into the classification network to obtain the classification results output by the classification network. Preferably, the EfficientNet network can be used as the backbone network, with an input of 800*800 and an output layer size of 3, namely sand, aggregate, and belt. If the classification result is aggregate, proceed to step S2; if the classification result is not aggregate, proceed to step S1 to continue obtaining the original image frames.

[0067] In summary, the aggregate particle size monitoring method provided by this invention involves: acquiring original image frames; preprocessing the original image frames to obtain multiple sub-images; inputting these sub-images into a segmentation network model to obtain a segmentation mask image output by the segmentation network model after removing obscured aggregates; calculating the aggregate particle size corresponding to each segmentation mask image based on the segmentation mask image; repeating the above steps to statistically analyze the aggregate particle size at a preset time period; and obtaining the overall aggregate particle size distribution based on the aggregate particle size within the preset time period. Through these steps, real-time statistical analysis of the aggregate particle size distribution is achieved. Furthermore, by removing obscured aggregates and aggregates at edges, subsequent particle size analysis of obscured aggregates and aggregates at edges is avoided, thus improving the accuracy of the statistical analysis of the overall aggregate particle size distribution.

[0068] Based on the same inventive concept, the present invention also provides an aggregate particle size monitoring device. The aggregate particle size monitoring device provided by the present invention is described below. The aggregate particle size monitoring device described below can be referred to in correspondence with the aggregate particle size monitoring method described above.

[0069] like Figure 2 As shown, the aggregate particle size monitoring device provided by the present invention includes an image acquisition module 21, a preprocessing module 22, a segmentation module 23, a particle size calculation module 24, and a statistics module 25.

[0070] Image acquisition module 21 is used to acquire raw image frames;

[0071] Preprocessing module 22 is used to preprocess the original image frame to obtain multiple sub-images after segmentation;

[0072] The segmentation module 23 is used to feed multiple sub-images as input into the segmentation network model and obtain the segmentation mask image after removing the occluded aggregates from the output of the segmentation network model. The segmentation network model is built based on a real-time segmentation network and trained by deep learning based on a dataset. Each set of data in the dataset includes sample images containing aggregates, as well as corresponding annotation information for labeling the complete aggregates and occluded aggregates in the sample images.

[0073] The particle size calculation module 24 is used to calculate the particle size of the aggregate corresponding to each segmentation mask based on the segmentation mask image;

[0074] The statistics module 25 is used to count the aggregate particle size at a preset time period and obtain the overall particle size distribution of the aggregate based on the aggregate particle size within the preset time period.

[0075] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute the aggregate particle size monitoring method provided by the above methods, which includes:

[0076] Obtain the original image frame.

[0077] The original image frame is preprocessed to obtain multiple sub-images after segmentation.

[0078] The multiple sub-images are fed into the segmentation network model as input to obtain the segmentation mask image output by the segmentation network model after removing the occluded aggregate.

[0079] Based on the segmentation mask, the particle size of the aggregate corresponding to each segmentation mask is calculated.

[0080] Repeat the above steps to count the aggregate particle size at a preset time period, and obtain the overall particle size distribution of the aggregate based on the aggregate particle size within the preset time period.

[0081] The segmentation network model is built based on a real-time segmentation network and trained by deep learning on a dataset. Each set of data in the dataset includes sample images containing aggregates, as well as corresponding annotation information for labeling complete aggregates and occluded aggregates in the sample images.

[0082] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] On the other hand, the present invention also provides an engineering machine that includes the aforementioned electronic equipment. It should be understood that the present invention does not limit the specific type of the engineering machine; for example, the engineering machine may be an aggregate conveyor belt with aggregate monitoring function, or a monitoring device independently installed above the conveyor belt.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring aggregate particle size, characterized in that, include: Obtain the original image frame; The original image frame is preprocessed to obtain multiple sub-images after segmentation; The multiple sub-images are fed into the segmentation network model as input to obtain the segmentation mask image after removing the occluded aggregates output by the segmentation network model. Based on the boundary information of the segmentation mask, the aggregate at the edge corresponding to the segmentation mask in the sub-image is removed, including: comparing the boundary coordinates of the segmentation mask with the boundary coordinates of the sub-image; if the boundary coordinates of the segmentation mask and the boundary coordinates of the sub-image overlap, the aggregate corresponding to the segmentation mask in the sub-image is removed. Based on the segmentation mask, calculate the particle size of the aggregate corresponding to each segmentation mask; Repeat the above steps to count the aggregate particle size at a preset time period, and obtain the overall particle size distribution of the aggregate based on the aggregate particle size within the preset time period. The segmentation network model is built based on a real-time segmentation network and trained by deep learning on a dataset. Each set of data in the dataset includes sample images containing aggregates, as well as corresponding annotation information for labeling complete aggregates and occluded aggregates in the sample images.

2. The aggregate particle size monitoring method according to claim 1, characterized in that, Calculating the aggregate particle size corresponding to each of the segmentation mask images includes: The boundary coordinate set of the segmentation mask is fed into the cv2.minAreaRect function in the OpenCV library to calculate the rectangular box corresponding to the segmentation mask. The length of the shortest side of the rectangle is used as the particle size of the aggregate corresponding to the segmentation mask image.

3. The aggregate particle size monitoring method according to claim 1, characterized in that, The original image frame is preprocessed to obtain multiple segmented sub-images, including: The original image is segmented into multiple sub-images according to a first preset size, so that the sub-images meet the input size requirements of the segmentation network model.

4. The aggregate particle size monitoring method according to claim 1, characterized in that, Based on the aggregate particle size within the preset time period, the overall particle size distribution of the aggregate is obtained, including: The aggregate particle size is compared with the preset particle size classification threshold, and the aggregate particle size is classified into the corresponding particle size classification. Based on the total aggregate particle size within the preset time period and the number of aggregate particles in each particle size category, the proportion of aggregate in each particle size category is calculated.

5. The aggregate particle size monitoring method according to any one of claims 1-4, characterized in that, After acquiring the original image frame but before preprocessing the original image frame, the process further includes: Obtain a classified image frame, wherein the classified image frame is an image frame obtained by adjusting the original image frame to a second preset size; The classified image frame is fed into the classification network to obtain the classification result output by the classification network; If the classification result is not aggregate, proceed to the step of obtaining the original image frame.

6. An aggregate particle size monitoring device, characterized in that, include: The image acquisition module is used to acquire raw image frames; The preprocessing module is used to preprocess the original image frame to obtain multiple sub-images after segmentation; The segmentation module is used to feed the multiple sub-images as input into the segmentation network model and obtain the segmentation mask image after removing the occluded aggregates output by the segmentation network model. Based on the boundary information of the segmentation mask, the aggregate at the edge corresponding to the segmentation mask in the sub-image is removed, including: comparing the boundary coordinates of the segmentation mask with the boundary coordinates of the sub-image; if the boundary coordinates of the segmentation mask and the boundary coordinates of the sub-image overlap, the aggregate corresponding to the segmentation mask in the sub-image is removed; the segmentation network model is built based on a real-time segmentation network and trained by deep learning based on a dataset. Each set of data in the dataset includes sample images containing aggregate, and also includes corresponding annotation information for labeling complete aggregate and occluded aggregate in the sample images; The particle size calculation module is used to calculate the particle size of the aggregate corresponding to each segmentation mask based on the segmentation mask. The statistics module is used to count the aggregate particle size at a preset time period, and to obtain the overall particle size distribution of the aggregate based on the aggregate particle size within the preset time period.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the aggregate particle size monitoring method as described in any one of claims 1 to 5.

8. An engineering machinery equipment, characterized in that, The engineering machinery equipment includes the electronic equipment as described in claim 7.

Images