Real-time dumping operation state monitoring method based on image recognition

Through the image recognition method, the real-time depth image of the drain machine is compared with the neural network, and the motion state of the drain arm is automatically judged, solving the problems of low efficiency and irregularity caused by subjective judgment in the prior art, and realizing the problem of low efficiency and irregularity caused by relying on subjective judgment. Efficient monitoring and automated control of soil discharge operations.

CN119048976APending Publication Date: 2024-11-29北京瓦特曼智能科技有限公司 +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411042238.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing soil discharge operation monitoring methods are highly dependent on the operator's subjective judgment, resulting in irregular and inefficient material discharge operations and lack of effective automated monitoring methods.

Method used

Using an image recognition method, by pre-acquisitioning machine area photos as matching templates, a neural network is used to compare real-time depth images, the presence of material flow and material piles is judged, and the depth information of pixel points is based on Calculate the length of the material flow and the distance of the material pile to automatically determine whether the discharge arm needs to be moved.

Benefits of technology

Real-time monitoring of soil discharge operation status is achieved, the basis for moving material discharge arms is provided, monitoring efficiency and accuracy is improved, and it is suitable for large-scale promotion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048976A_ABST
    Figure CN119048976A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image similarity comparison, in particular to a dumping operation state real-time monitoring method based on image recognition. Comprising the following steps: carrying out image shooting on a dumping area of a dumping plough in advance, obtaining a picture in which a material flow and a material pile exist at the same time as a first matching template, obtaining a picture only containing the material flow as a second matching template, and obtaining a picture only containing the material pile as a third matching template; when the dumping plough works, a real-time depth image of a dumping area of the dumping plough is obtained, the real-time depth image is sequentially compared with the first matching template, the second matching template and the third matching template, and whether the dumping arm works normally or not is judged. And whether the discharging arm needs to move or not is judged according to the length of the material flow and the distance between the discharging opening and the material pile. According to the method, the state of the dumping operation can be monitored in real time, and a basis can be provided for whether the dumping arm needs to be moved or not.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image similarity comparison, and in particular to a method for real-time monitoring of soil dumping operation status based on image recognition. Background Art

[0002] The dumper is part of the continuous or semi-continuous mining equipment of open-pit coal mines. It is matched with the steel rope traction belt conveyor and is suitable for the dumping and stacking of loose materials in the dump and material yard of open-pit mines. The dumper is mainly composed of a dumping arm, a driver's cab, a slewing device, a lower steel structure, a main machine running device, a maintenance room, a support vehicle running device, a receiving arm, a transfer arm, a counterweight arm, etc. The dumper arm is mainly used to discharge materials, and when discharging, a material flow and a material pile will be formed.

[0003] The state of the discharge material flow and material pile of the discharge arm depends on the operator's naked eye observation, and then based on experience, it is judged whether there is any abnormality and whether the discharge arm needs to be moved.

[0004] However, the existing method highly relies on the subjective judgment of the operator to monitor the discharging operation and control the movement of the discharging arm according to the discharging status, which easily leads to problems such as irregular discharging operation and low efficiency. Summary of the invention

[0005] The purpose of the present invention is to provide a method for real-time monitoring of the status of soil discharge operation based on image recognition, which can monitor the status of soil discharge operation in real time and provide a basis for whether the discharge arm needs to be moved.

[0006] The technical solution of the present invention is to provide a method for real-time monitoring of soil dumping operation status based on image recognition, comprising the following steps:

[0007] S1: Pre-shoot an image of the soil discharge area of ​​the soil discharger, obtain a photo of both the material flow and the material pile as a first matching template, obtain a photo of only the material flow as a second matching template, and obtain a photo of only the material pile as a third matching template;

[0008] S2: when the soil dumper is working, a real-time depth image of the soil dumping area of ​​the soil dumper is obtained, and the real-time depth image is compared with the first matching template to determine whether the real-time depth image is similar to the first matching template. Based on the similarity between the two, it is determined whether the acquired real-time depth image contains both material flow and material pile. If so, step S3 is executed; otherwise, step S4 is executed;

[0009] S3: The position of the end of the material flow in the camera coordinate system is restored through the depth information of the pixel points in the real-time depth image, and the relative positions of the starting point and the end point of the material flow are calculated to represent the length of the material flow. It is also confirmed that the working state of the soil discharge operation is in working state, and it is determined whether the discharge arm needs to be moved according to the length of the material flow;

[0010] S4: performing image comparison between the real-time depth image and the second matching template to determine whether the real-time depth image and the second matching template are similar, and determining whether the acquired real-time depth image only has material flow based on the similarity between the two. If so, confirming that the working state of the soil discharge operation is in working state, and there is no need to move the discharge arm, otherwise, executing step S5;

[0011] S5: performing image comparison between the real-time depth image and the third matching template to determine whether the real-time depth image and the third matching template are similar, and determining whether the acquired real-time depth image only contains a pile of materials based on the similarity between the two. If not, confirming that the working state of the soil dumping operation is a non-working state, otherwise, executing step S6;

[0012] S6: Take the highest point of the pile in the real-time depth image, calculate the relative position between the highest point of the pile and the discharge port, and confirm that the working state of the soil discharge operation is not working. According to the relative position relationship, obtain the distance between the discharge port and the pile.

[0013] Preferably, the depth photo is acquired by installing a depth camera at a position of the discharge arm close to the discharge port, and the field of view of the depth camera at least includes the discharge port and a range of 10 m below the discharge port.

[0014] Preferably, in step S2, the method for determining whether the real-time depth image is similar to the first matching template is:

[0015] Using a neural network to train the first matching template;

[0016] Several photos of material flow and material pile at the same time are obtained as training sets and input into the neural network for training. After the training is completed, the real-time depth image of the soil discharge area of ​​the soil discharger is obtained as a test set and input into the neural network to obtain the similarity matching result. If the similarity is higher than the preset threshold, it is judged that the real-time depth image and the photo in the first matching template are similar, otherwise, they are not similar.

[0017] Preferably, in step S4, the method for determining whether the real-time depth image is similar to the second matching template is:

[0018] Using a neural network to train the second matching template;

[0019] Several photos of only material flow are obtained as training sets and input into the neural network for training. After the training is completed, the real-time depth image of the soil discharge area of ​​the soil discharger is obtained as a test set and input into the neural network to obtain the similarity matching result. If the similarity is higher than the preset threshold, the real-time depth image and the photo in the second matching template are judged to be similar, otherwise, they are not similar.

[0020] Preferably, in step S5, the method for determining whether the real-time depth image is similar to the third matching template is:

[0021] Using a neural network to train the third matching template;

[0022] Several photos of only the material pile are obtained as training sets and input into the neural network for training. After the training is completed, the real-time depth image of the soil discharge area of ​​the soil discharger is obtained as a test set and input into the neural network to obtain the similarity matching result. If the similarity is higher than the preset threshold, the real-time depth image and the photo in the second matching template are judged to be similar, otherwise, they are not similar.

[0023] Furthermore, in step S3, the method for determining whether the discharging arm needs to be moved according to the length of the material flow is:

[0024] When the discharge arm needs to be moved, the length of the material flow is preset to be X1, and the length of the material flow calculated in step S3 is X. If X>X1, the discharge arm does not need to be moved; if X≤X1, the discharge arm needs to be moved.

[0025] Furthermore, in step S6, after obtaining the distance between the discharge port and the material pile, it is determined whether the discharge arm needs to be moved according to the distance.

[0026] Preferably, the method for judging whether the discharging arm needs to be moved according to the distance is:

[0027] When the discharge arm needs to be moved, the distance between the discharge port and the material pile is Y1, and the distance between the discharge port and the material pile obtained in step S6 is Y. If Y<Y1, the discharge arm does not need to be moved. If Y≥Y1, the discharge arm needs to be moved.

[0028] The beneficial effects of the present invention are:

[0029] 1. The present invention provides a real-time monitoring method for a soil discharge operation device based on image recognition. This method can not only monitor the status of the soil discharge operation in real time, but also provide a basis for whether the discharge arm needs to be moved.

[0030] 2. The real-time monitoring method provided by the present invention has low cost, high efficiency and is suitable for large-scale promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The advantages of the above and additional aspects of the present invention will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0032] Figure 1 This is a flow chart of a method for real-time monitoring of soil discharge operation status based on image recognition according to Example 1 of the present invention. DETAILED DESCRIPTION

[0033] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0035] Example 1

[0036] like Figure 1 As shown, the real-time monitoring method of soil dumping operation status based on image recognition provided by the present invention comprises the following steps:

[0037] S1: Pre-shoot an image of the soil discharge area of ​​the soil discharger, obtain a photo of both the material flow and the material pile as a first matching template, obtain a photo of only the material flow as a second matching template, and obtain a photo of only the material pile as a third matching template;

[0038] S2: When the soil discharging machine is working, obtain a real-time depth image of the soil discharging area of ​​the soil discharging machine, compare the real-time depth image with the first matching template, and judge whether the real-time depth image is similar to the first matching template. The judgment method is: use a neural network to train the first matching template; obtain several photos of material flow and material pile at the same time as the training set, input them into the neural network for training, after the training is completed, obtain the real-time depth image of the soil discharging area of ​​the soil discharging machine as the test set, input it into the neural network, and obtain the similarity matching result. If the similarity is higher than the preset threshold, it is judged that the real-time depth image and the photo in the first matching template are similar, otherwise, they are not similar. Based on the similarity between the two, determine whether the acquired real-time depth image contains material flow and material pile at the same time. If so, execute step S3, otherwise, execute step S4;

[0039] S3: The position of the end of the material flow in the camera coordinate system is restored through the depth information of the pixel points in the real-time depth image, and the relative positions of the starting point and the end point of the material flow are calculated to represent the length of the material flow. It is also confirmed that the working state of the soil discharge operation is in working state, and it is determined whether the discharge arm needs to be moved according to the length of the material flow;

[0040] S4: Perform image comparison between the real-time depth image and the second matching template to determine whether the real-time depth image is similar to the second matching template. The determination method is: use a neural network to train the second matching template; obtain a number of photos with only material flow as a training set, and input them into the neural network for training. After the training is completed, obtain a real-time depth image of the soil discharge area of ​​the soil discharger as a test set, input it into the neural network, and obtain a similarity matching result. If the similarity is higher than a preset threshold, it is determined that the real-time depth image and the photos in the second matching template are similar, otherwise, they are not similar. Based on the similarity between the two, determine whether the acquired real-time depth image only has material flow. If so, confirm that the working state of the soil discharge operation is in working state, and there is no need to move the discharge arm. Otherwise, execute step S5;

[0041] S5: Perform image comparison between the real-time depth image and the third matching template to determine whether the real-time depth image and the third matching template are similar. The determination method is: use a neural network to train the third matching template; obtain a number of photos with only material piles as training sets, input them into the neural network for training, and after the training is completed, obtain the real-time depth image of the soil discharge area of ​​the soil discharger as a test set, input it into the neural network, and obtain the similarity matching result. If the similarity is higher than the preset threshold, it is determined that the real-time depth image and the photo in the second matching template are similar, otherwise, they are not similar. Based on the similarity between the two, determine whether the acquired real-time depth image only contains material piles. If not, confirm that the working state of the soil discharge operation is not working, otherwise, execute step S6;

[0042] S6: Take the highest point of the pile in the real-time depth image, calculate the relative position between the highest point of the pile and the discharge port, and confirm that the working state of the soil discharge operation is not working. According to the relative position relationship, obtain the distance between the discharge port and the pile.

[0043] In step S3, the method for determining whether the discharging arm needs to be moved according to the length of the material flow is:

[0044] When the discharge arm needs to be moved, the length of the material flow is preset to be X1, and the length of the material flow calculated in step S3 is X. If X>X1, the discharge arm does not need to be moved; if X≤X1, the discharge arm needs to be moved.

[0045] In step S6, after obtaining the distance between the discharge port and the material pile, it is determined whether the discharge arm needs to be moved according to the distance. The method for determining whether the discharge arm needs to be moved according to the distance is as follows: when the discharge arm needs to be moved, the distance between the discharge port and the material pile is Y1, and the distance between the discharge port and the material pile obtained in step S6 is Y. If Y<Y1, the discharge arm does not need to be moved, and if Y≥Y1, the discharge arm needs to be moved.

[0046] Example 2

[0047] The method of Example 1 is used to monitor the discharging state of a soil discharging machine in real time. The specific method is as follows:

[0048] 1. Install a depth camera on the discharge arm of the soil discharging machine near the discharge port. The field of view of the depth camera includes the discharge port and the area 10m below the discharge port.

[0049] 2. When the soil dumper is working, obtain the real-time depth image of the soil dumping area of ​​the soil dumper taken by the depth camera, transfer it to the first matching template, and analyze whether the real-time depth image and the photo in the first matching template are similar. The possible situations are:

[0050] (1) If the real-time depth image taken by the depth camera is similar to the photo in the first matching template, the working state of the discharging arm is judged to be in working state, and then the position of the end of the material flow in the camera coordinate system is restored through the depth information of the pixel point, and the relative position of the starting point and the end point of the material flow is calculated to represent the length of the material flow. The length X of the material flow is 8m, and the preset material flow length X1 is 2m. X>X1, and the discharging arm does not need to move.

[0051] (2) If the real-time depth image taken by the depth camera is similar to the photo in the first matching template, the working state of the discharge arm is judged to be in working state, and then the position of the end of the material flow in the camera coordinate system is restored through the depth information of the pixel point, and the relative position of the starting point and the end point of the material flow is calculated to represent the length of the material flow. The length X of the material flow is 2m, and the preset material flow length X1 is 2m, X=X1, and the discharge arm needs to be moved.

[0052] (3) It is found that the real-time depth image taken by the depth camera is not similar to the photo in the first matching template, and then the real-time depth image is transmitted to the second matching template, and whether the real-time depth image and the photo in the second matching template are similar are analyzed. It is found that the real-time depth image taken by the depth camera and the photo in the second matching template are similar, and it is judged that the working state of the discharge arm is in working state, and there is no need to move the discharge arm.

[0053] (4) It is found that the real-time depth image taken by the depth camera is not similar to the photo in the first matching template, and then the real-time depth image is transmitted to the second matching template, and the real-time depth image and the photo in the second matching template are analyzed to see whether they are similar. It is found that the real-time depth image taken by the depth camera is not similar to the photo in the second matching template, and then the real-time depth image is transmitted to the third matching template, and the real-time depth image and the photo in the third matching template are analyzed to see whether they are similar. It is found that the real-time depth image and the photo in the third matching template are not similar, and it is determined that the discharge arm is in a non-working state.

[0054] (5) It is found that the real-time depth image taken by the depth camera is not similar to the photo in the first matching template, and then the real-time depth image is transmitted to the second matching template, and the real-time depth image and the photo in the second matching template are analyzed to see whether they are similar. It is found that the real-time depth image taken by the depth camera is not similar to the photo in the second matching template, and then the real-time depth image is transmitted to the third matching template, and the real-time depth image and the photo in the third matching template are analyzed to see whether they are similar. It is found that the real-time depth image and the photo in the third matching template are similar, and the discharge arm is judged to be in an off-working state. The highest point of the material pile in the depth photo is removed, and the relative position between the highest point of the material pile and the discharge port is calculated. According to the relative position relationship, the distance Y between the discharge port and the material pile is obtained to be 2m. When the discharge arm needs to be moved, the distance Y1 between the discharge port and the material pile is 8m, Y<Y1, and the discharge arm does not need to be moved.

[0055] (6) It is found that the real-time depth image taken by the depth camera is not similar to the photo in the first matching template, and then the real-time depth image is transmitted to the second matching template, and the real-time depth image and the photo in the second matching template are analyzed to see whether they are similar. It is found that the real-time depth image taken by the depth camera is not similar to the photo in the second matching template, and then the real-time depth image is transmitted to the third matching template, and the real-time depth image and the photo in the third matching template are analyzed to see whether they are similar. It is found that the real-time depth image and the photo in the third matching template are similar, and the discharge arm is judged to be in an off-working state. The highest point of the material pile in the depth photo is removed, and the relative position between the highest point of the material pile and the discharge port is calculated. According to the relative position relationship, the distance Y between the discharge port and the material pile is obtained to be 8m. When the discharge arm needs to be moved, the distance Y1 between the discharge port and the material pile is 8m, and Y=Y1. The discharge arm needs to be moved.

[0056] Although the present invention is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present invention. The scope of protection of the present invention is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present invention.

Claims

1. A real-time monitoring method for soil dumping operation status based on image recognition, characterized in that: The following steps are involved: S1: Pre-shoot an image of the soil discharge area of ​​the soil discharger, obtain a photo of both the material flow and the material pile as a first matching template, obtain a photo of only the material flow as a second matching template, and obtain a photo of only the material pile as a third matching template; S2: when the soil dumper is working, a real-time depth image of the soil dumping area of ​​the soil dumper is obtained, and the real-time depth image is compared with the first matching template to determine whether the real-time depth image is similar to the first matching template. Based on the similarity between the two, it is determined whether the acquired real-time depth image contains both material flow and material pile. If so, step S3 is executed; otherwise, step S4 is executed; S3: The position of the end of the material flow in the camera coordinate system is restored through the depth information of the pixel points in the real-time depth image, and the relative positions of the starting point and the end point of the material flow are calculated to represent the length of the material flow. It is also confirmed that the working state of the soil discharge operation is in working state, and it is determined whether the discharge arm needs to be moved according to the length of the material flow; S4: performing image comparison between the real-time depth image and the second matching template to determine whether the real-time depth image and the second matching template are similar, and determining whether the acquired real-time depth image only has material flow based on the similarity between the two. If so, confirming that the working state of the soil discharge operation is in working state, and there is no need to move the discharge arm, otherwise, executing step S5; S5: performing image comparison between the real-time depth image and the third matching template to determine whether the real-time depth image and the third matching template are similar, and determining whether the acquired real-time depth image only contains a pile of materials based on the similarity between the two. If not, confirming that the working state of the soil dumping operation is a non-working state, otherwise, executing step S6; S6: Take the highest point of the pile in the real-time depth image, calculate the relative position between the highest point of the pile and the discharge port, and confirm that the working state of the soil discharge operation is not working. According to the relative position relationship, obtain the distance between the discharge port and the pile.

2. The method for real-time monitoring of soil dumping operation status based on image recognition according to claim 1 is characterized in that: The real-time depth image is acquired in the following manner: a depth camera is installed at a position of the discharge arm close to the discharge port, and the field of view of the depth camera at least includes the discharge port and a range of 10m below the discharge port.

3. The method for real-time monitoring of soil dumping operation status based on image recognition according to claim 1, characterized in that: In step S2, the method for determining whether the real-time depth image is similar to the first matching template is: Using a neural network to train the first matching template; Several photos of material flow and material pile at the same time are obtained as training sets and input into the neural network for training. After the training is completed, the real-time depth image of the soil discharge area of ​​the soil discharger is obtained as a test set and input into the neural network to obtain the similarity matching result. If the similarity is higher than the preset threshold, it is judged that the real-time depth image and the photo in the first matching template are similar, otherwise, they are not similar.

4. The method for real-time monitoring of soil dumping operation status based on image recognition according to claim 1, characterized in that: In step S4, the method for determining whether the real-time depth image is similar to the second matching template is: Using a neural network to train the second matching template; Several photos of only material flow are obtained as training sets and input into the neural network for training. After the training is completed, the real-time depth image of the soil discharge area of ​​the soil discharger is obtained as a test set and input into the neural network to obtain the similarity matching result. If the similarity is higher than the preset threshold, the real-time depth image and the photo in the second matching template are judged to be similar, otherwise, they are not similar.

5. The method for real-time monitoring of soil dumping operation status based on image recognition according to claim 1, characterized in that: In step S5, the method for determining whether the real-time depth image is similar to the third matching template is: Using a neural network to train the third matching template; Several photos of only the material pile are obtained as training sets and input into the neural network for training. After the training is completed, the real-time depth image of the soil discharge area of ​​the soil discharger is obtained as a test set and input into the neural network to obtain the similarity matching result. If the similarity is higher than the preset threshold, the real-time depth image and the photo in the second matching template are judged to be similar, otherwise, they are not similar.

6. The method for real-time monitoring of soil dumping operation status based on image recognition according to claim 1, characterized in that: In step S3, the method for determining whether the discharge arm needs to be moved according to the length of the material flow is: When the discharge arm needs to be moved, the length of the material flow is preset to be X1, and the length of the material flow calculated in step S3 is X. If X>X1, the discharge arm does not need to be moved; if X≤X1, the discharge arm needs to be moved.

7. The method for real-time monitoring of soil dumping operation status based on image recognition according to claim 1, characterized in that: In step S6, after the distance between the discharge port and the material pile is obtained, it is determined whether the discharge arm needs to be moved according to the distance.

8. The method for real-time monitoring of soil dumping operation status based on image recognition according to claim 7 is characterized in that: The method to determine whether the discharge arm needs to be moved according to the distance is: When the discharge arm needs to be moved, the distance between the discharge port and the material pile is Y1, and the distance between the discharge port and the material pile obtained in step S6 is Y. If Y<Y1, the discharge arm does not need to be moved. If Y≥Y1, the discharge arm needs to be moved.