A method for obtaining evidence image based on working behavior of engineering machinery

Abstract

The application discloses a kind of based on engineering machinery work behavior's evidence image acquisition method, including obtaining initial image set before work, obtaining evidence image set after work and obtaining feature image set in key work area.The application forms evidence information by work point position, azimuth, video stream signal, by video shooting angle, field of view range to video stream is intercepted, screening, back, realizes independent acquisition photo / video, does not need manual intervention, realizes the full coverage of evidence photo / video to work area in the case where flow is as far as possible saved.Thereby also effectively reduce background photo auditing workload.

Description

Technical Field

[0001] This invention relates to the field of evidence collection technology for natural resource monitoring and supervision, specifically to a method for obtaining evidence images based on the operational behavior of engineering machinery. Background Technology

[0002] With the industry's increasingly refined management requirements for various natural resources and assets, the investigation, monitoring, and supervision of natural resources are becoming more and more frequent. At present, natural resource investigation and monitoring mainly adopts the "post-event" monitoring method to monitor changes in land parcels, which includes the following methods: (1) Manual investigation and monitoring. This method requires the use of remote sensing images, manual provision of investigation clues, and then sending special personnel to the site to take photos for verification. This method has high labor costs and low efficiency. (2) Remote sensing monitoring. This method requires the use of remote sensing image interpretation to monitor natural resources. Its drawback is that low-resolution images cannot meet the monitoring needs, and high-resolution images are not only costly but also have long update cycles, making it difficult to achieve real-time monitoring. (3) Aerial monitoring. This method is mainly used for key areas, with a very limited monitoring range and high cost, making it difficult to meet the needs of large-scale monitoring of natural resources. (4) High-altitude electronic eye (tower video) monitoring. This method not only requires the construction of ultra-high-speed towers but also relies on ultra-high-performance cameras. The installation, operation, and maintenance costs of the equipment are high, and this method has a large error in identifying the boundaries of land parcels at a long distance, making it difficult to meet the requirements of refined investigation and monitoring. In summary, current monitoring methods for the development, utilization, and protection of national land space suffer from problems such as high cost, long cycle, monitoring lag, and inability to supervise the actors involved.

[0003] With the continuous development of economy and technology, the construction of various land development, utilization and protection projects has basically achieved mechanization. Construction machinery has become the most important tool for land development, utilization and ecological protection and restoration operations, which provides a new perspective for natural resource monitoring technology. Conducting operational posture monitoring of construction machinery is an effective technical method to achieve real-time monitoring and timely early warning of land space development, utilization and protection. It is also an effective way to transform from ex-post supervision to ex-ante and in-process supervision, and promote the modernization of governance system and governance capacity.

[0004] Invention patent application 202110792771.9 discloses a method, system, terminal, and readable storage medium for monitoring natural resources based on construction machinery. It describes how to aggregate the locations of construction machinery operation points into work surfaces, and then, based on the distribution characteristics of these work points and in conjunction with a geographic information database platform, identify natural resource development and utilization activities. This method can identify a large number of natural resource development and utilization activities; however, for some difficult-to-identify activities, photographic evidence is still required for verification.

[0005] Invention patent application 202211170547.7 discloses a vision-based method for monitoring the working posture of excavators. This application introduces a deep learning-based visual recognition method for the excavator boom. Based on the ratio of the boom's size in the image to its actual size, the distance between the boom's end (bucket end) and the camera (cab) is estimated. The coordinates of the boom's end (bucket end) are then calculated in real time using positioning, azimuth, and horizontal tilt angles. While this method solves the problem of determining the location of the work point, it still requires photographic evidence for some natural resource development and utilization activities that are difficult to identify based on the distribution of work points.

[0006] Invention patent application 202210248207.5 provides a method and system for capturing evidence images. This method converts the camera's shooting range into a first graphic and maps it onto a GIS map. The overlap area between the first graphic and the image patch to be presented as evidence is calculated. By determining the overlap area, the spatial relationship between the camera's shooting range and the image patch is determined, thereby deciding whether to perform a shooting action. However, this method requires manual on-site photography for evidence collection, consuming significant manpower and resources.

[0007] In summary, the above methods provide information collection and communication methods for construction machinery and equipment, as well as real-time video stream information of construction machinery and equipment construction. Based on the real-time video stream information of construction machinery and equipment and its working posture distribution characteristics, video / photo information is extracted, analyzed, and transmitted back, which can realize real-time evidence monitoring of natural resource development and utilization. However, there are still problems in how to obtain as few evidence photos as possible and reflect development and utilization behavior. Summary of the Invention

[0008] To address the existing technologies, particularly the methods for acquiring photos or video streams involved in 202210248207.5, 202110792771.9, and 202211170547.7, including the automatic segmentation, acquisition, processing, and transmission of real-time video streams, this approach aims to ensure that the transmitted photos or video streams can effectively identify the development and utilization of natural resources, minimize the redundancy of the number of photos to save network traffic and backend storage space, and simultaneously reduce the workload of backend photo review.

[0009] This invention provides an intelligent evidence collection method based on the operational behavior of construction machinery. The invention uses the location, azimuth, and video stream signal of the work site, along with the video shooting angle and field of view, to extract, filter, and transmit the video stream, forming evidence information. In the process of filtering and transmitting evidence photos, this invention achieves autonomous acquisition of photos / videos without manual intervention, and achieves full coverage of the work area with evidence photos / videos while minimizing data usage.

[0010] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows:

[0011] A method for acquiring evidentiary images based on the operational behavior of construction machinery includes acquiring an initial set of images before the operation and acquiring an evidentiary image set after the operation.

[0012] Obtaining the initial image set before the operation includes the following steps:

[0013] Step A1: Before the construction machinery is ready to start operation, take pictures of the location of the construction machinery using the camera on the construction machinery, and save the pictures and image parameters together in the initial image set before operation. At the same time, map the captured pictures onto the GIS map to form the operation area and save it.

[0014] Step A2: After the construction machinery starts working, the images captured by the camera are continuously mapped onto the GIS map. When the overlap area between the mapped area and the saved work area is less than a preset value, the image and image parameters at this time are saved to the initial image set before the operation. At the same time, the mapped area at this time is merged with the saved work area to form a new work area until the construction machinery completes all the work. In this way, the initial image set before the operation is obtained throughout the entire operation process, and a complete work area is formed and saved.

[0015] The process of obtaining the evidence image set after the job is completed includes the following steps:

[0016] Step B1: When the construction machinery starts working, the location of the construction machinery is photographed and the movement of the construction machinery is monitored by the camera on the construction machinery. When the construction machinery is detected to be performing a work action, the location of the current work action is taken as the work point, and the image and image parameters are saved together in the post-work evidence image set. At the same time, the photographed image is mapped onto the GIS map to form the work point evidence range and is saved.

[0017] Step B2: During the continuous operation of the construction machinery, the images captured by the camera are continuously mapped onto the GIS map, and the movement of the construction machinery is monitored. Whenever the position of the construction machinery changes and it performs an operation, it is recorded as a new working point. When the overlapping area between the mapped range and the working point evidence range is less than a set value, and the new working point in the mapped range exceeds a predetermined value, the image and image parameters at this time are saved to the post-operation evidence image set. At the same time, the mapped range at this time is merged with the saved working point evidence range to form a new working point evidence range, until the construction machinery completes all operations. Thus, the post-operation evidence image set of the entire operation process is obtained, and a complete working point evidence range is formed and saved.

[0018] The method further includes acquiring a set of feature images of key work areas:

[0019] If the number of work points within the mapping range formed by the currently captured image reaches a predetermined value, the current image and image parameters are saved, and the mapping range is saved, thereby forming a set of evidence images for the key work area.

[0020] The method also includes updating the feature image set of key work areas:

[0021] After acquiring the feature image set of the key work area, if the number of work points in the mapping range formed by the currently captured image exceeds the number of work points in the saved evidence image set of the key work area, then the current image, image parameters, and mapping range will be overwritten with the saved evidence image set of the key work area, thereby updating the feature image set of the key work area.

[0022] The method involves a preset number of evidence images for key work areas, which is greater than 1. Before the preset number of evidence images for key work areas is reached, all evidence images for key work areas are independently saved based on a predetermined value of work points within the mapping range. After the preset number is reached, the number of work points within the mapping range formed by the currently captured image is compared in real time with all saved evidence images for key work areas. When the number of work points exceeds the number of saved evidence images for key work areas, an overwrite is performed to achieve an update.

[0023] The method described above, if the number of work points in the mapping range formed by the currently captured image exceeds multiple saved evidence image sets of key work areas, then randomly overwrites one saved evidence image set of key work areas.

[0024] The image parameters described in the method include the coordinate position and azimuth angle at the time of image capture.

[0025] The method described herein, in step B2, further includes the step of updating the post-operation evidence image set:

[0026] When the overlapping area between the mapping range of the continuously captured images and the evidence range of the work point is greater than the set value, and the new work point in the mapping range exceeds the predetermined value, the current image and image parameters will overwrite the original image and image parameters. At the same time, the mapping range at this time will be merged with the saved evidence range of the work point to form a new evidence range of the work point.

[0027] The method further includes the step of sending the saved data back to the background:

[0028] The saved data is sent back to the backend according to preset time nodes, which are the 5th minute, 10th minute, 30th minute and 1st hour after the start of the job, and then sent back once every hour.

[0029] The method is characterized in that, when more than one piece of construction machinery is operating at the work site, the work site information is shared among the various pieces of construction machinery via a network.

[0030] The technical advantage of this invention lies in its ability to capture, filter, and transmit video streams based on the work site location, azimuth, and video stream signal, as well as the video shooting angle and field of view, to form evidence information. During the filtering and transmission of evidence photos, the invention achieves autonomous acquisition of photos / videos without manual intervention, and achieves full coverage of the work area with evidence photos / videos while minimizing data usage. This also effectively reduces the workload of backend photo review. Attached Figure Description

[0031] Figure 1 This is a flowchart of the method of the present invention;

[0032] Figure 2 This is a diagram showing the relationship between camera parameters and shooting range in this invention;

[0033] Figure 3 This is a schematic diagram of the initial photo image mapping range before the operation in this invention;

[0034] Figure 4 This is a schematic diagram of the image mapping range and the fusion of image mapping ranges for evidence photos after the operation in this invention, wherein (a) is a schematic diagram of the image mapping range for evidence photos after the operation, and (b) is a schematic diagram of the fusion of image mapping ranges.

[0035] Figure 3 In the diagram, a1, a2, and a3 are the initial photo image mapping range fusion diagrams before the operation;

[0036] Figure 4 b1, b2, and b3 are the graphic mapping range diagrams of the evidence photos after the operation. There are no operation points in b1, so it is not included in the fusion. The non-overlapping operation points in the ranges of b2 and b3 each exceed 10, and the overlapping area of ​​the graphic mapping range is less than 80%. The two are merged to obtain the graphic mapping range fusion and merging diagram. Detailed Implementation

[0037] The method for obtaining evidentiary images based on the operational behavior of construction machinery provided by this invention includes the following steps:

[0038] Step S1, Obtaining the initial photos before the operation:

[0039] S1-1: When the engineering machinery starts to work at the first work point, record the initial coordinate position and the corresponding initial azimuth angle. According to the method in Chinese patent application publication CN114650353A, convert the camera shooting range into a first graphic mapping on the GIS map to form the first graphic mapping range.

[0040] S1-2: As the construction machinery moves and rotates during operation, the area captured by the next camera is converted into a graphic mapping on the GIS map. When the overlap between the area of ​​this graphic mapping and the previously captured area on the GIS map is less than a preset percentage, a video stream photo of that location and its corresponding azimuth angle is captured. Simultaneously, the graphic corresponding to this photo is mapped onto the GIS map and merged with the previous graphic mapping area to form a new graphic mapping area. The corresponding image parameters, including the coordinates and azimuth angle at the time of image capture, are obtained through the GPS positioning module and electronic compass module on the construction machinery and recorded in the corresponding photo.

[0041] S1-3: Repeat step S1-2 to obtain an initial set of evidence photos / videos covering all operations within the scope of the operation.

[0042] Step S2, Obtaining Evidence Photos After the Task:

[0043] S2-1: When the construction machinery starts working, the excavation point and / or unloading point (work point) are projected onto the GIS map, and the camera shooting range is converted into a graphic mapping on the GIS map in real time. When the excavation point and / or unloading point (work point) is within the graphic mapping range, its graphic mapping range is recorded, and the photo / video (P0) is captured and its coordinate position and azimuth data are stored in the local edge device.

[0044] S2-2: As the construction machinery operates, the camera's shooting range is converted into a graphic mapping on the GIS map in real time. When the graphic mapping range captured by the current camera covers the preset value of the number / area of ​​newly added work points (excavation points and / or unloading points), the photo / video (P0) at the previous location is replaced and updated to the photo / video (P1) at the current location, and its coordinate position and azimuth data are updated synchronously. At the same time, its graphic mapping range is replaced and updated, and then merged with the previously formed graphic range.

[0045] S2-3: Repeat step S2-2 to obtain a set of post-job evidence photos / videos covering the entire job scope.

[0046] Step S3: Obtain feature photos of key work areas:

[0047] As the construction machinery operates, the camera's shooting range is converted into a graphic mapping on the GIS map. When the number of previous work points (excavation points and unloading points) covered by the graphic mapping range of the current camera reaches a certain preset value / rank, the current location of the photo / video (P2), coordinate position, azimuth data, and its graphic mapping range are recorded.

[0048] When the number of work points covered by the graphic mapping range captured by the next camera exceeds the number of work points covered by the graphic mapping range corresponding to the previous photos / videos, the evidence photos / videos (P2) of the key work areas are replaced and updated, and their coordinate positions, azimuth data, and graphic mapping range are updated synchronously.

[0049] Step S4, return of evidence photos:

[0050] Photos of the work site, along with their corresponding locations and azimuth angles, are returned via the network module according to time nodes. Evidence photos / videos and other information are transmitted back to the backend according to preset time nodes. For construction machinery and equipment that has been stopped and restarted midway, the previous data is saved and processed and analyzed again.

[0051] Furthermore, the preset area percentage mentioned in steps S1-2 is 5%-20%.

[0052] Furthermore, in step S2-2, the preset values ​​for the number / area of ​​newly added work points (excavation points and / or unloading points) are 10-20, and the preset area value is an overlap area exceeding 80%-90%.

[0053] Furthermore, the number of work points (excavation points and / or unloading points) mentioned in step S3 reaches a certain preset value. The preset value is the top 3-5 locations with the highest density of work points since the start of the construction machinery operation, or the preset value is that the number of work points within the current camera's image mapping range exceeds 50-100 since the start of the construction machinery operation.

[0054] Furthermore, the time nodes mentioned in step S4 are 5 minutes, 10 minutes, 30 minutes, and 1 hour, and then the data is transmitted back every hour.

[0055] Furthermore, when multiple construction machines are operating, the location information of the work site is shared among the various construction machines via the Internet of Things, Bluetooth, or local area network.

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0057] In this embodiment, an 8-megapixel, 3.50mm focal length distortion-free camera is selected for image data acquisition. Based on its product performance, its shooting angle is 123°, and its shooting distance is 20m, corresponding to a maximum monitoring width of 72m. Considering the need for image distortion at the boundary and the stitching of different photos, a monitoring angle of 100° is set as the effective monitoring angle, and the corresponding maximum monitoring width is calculated to be approximately 45m. Figure 2 As shown.

[0058] like Figure 1 As shown in the figure, this embodiment provides an intelligent evidence collection method based on the operational behavior of construction machinery, which includes the following steps:

[0059] Step S1, Obtaining the initial photos before the operation:

[0060] S1-1: When the engineering machinery starts to work at the first work point, record the initial coordinate position and the corresponding initial azimuth angle, and convert the camera shooting range into the first graphic mapping on the GIS map to form an isosceles triangle with the shooting point as the vertex angle of 100° and the direction of the azimuth angle as the angle bisector, with a base width of 45m and a height of 20m.

[0061] S1-2: As the engineering machinery moves and rotates during operation, the camera's shooting range, as in step S1-1, is converted into a graphic mapping on the GIS map in real time. This mapped area is then overlaid and analyzed in real time with the previously photographed area on the GIS map.

[0062] When the overlapping area is greater than or equal to 10% of the preset area, that is, when the overlapping area is greater than or equal to 45m² 2 At that time, the video stream corresponding to the mapped area will not be captured, and the corresponding azimuth and positioning information will not be saved.

[0063] When the overlapping area is less than 10% of the preset area, that is, the overlapping area is less than 45m² 2 At that time, capture the video stream image at that location and its corresponding azimuth angle, simultaneously map the corresponding graphic onto the GIS map, and merge it with the previous graphic mapping range to form a new graphic mapping range, as shown in the attached figure. Figure 2 As shown.

[0064] S1-3: Repeat step S1-2 to obtain an initial set of evidence photos / videos covering the entire scope of the operation.

[0065] Step S2, Obtaining Evidence Photos After the Task:

[0066] S2-1: After the construction machinery starts working, the excavation point and / or unloading point (work point) will be projected onto the GIS map, and the camera shooting range will be converted into the graphic coverage range in S1-1 and mapped onto the GIS map in real time.

[0067] If the excavation point and / or unloading point (work point) are located outside the map mapping range, the video stream corresponding to the map area will not be captured, and the corresponding azimuth and positioning information will not be saved.

[0068] If the excavation point and / or unloading point (work point) are within the graphic mapping range, record its graphic mapping range, capture the photo / video (P0) at this time, and store its coordinate position and azimuth data in the local edge device to form the initial work point evidence range.

[0069] S2-2: As the construction machinery continues to operate, the camera's shooting range is converted into a graphic mapping on the GIS map in real time.

[0070] If the current camera's image mapping range covers more than 10 newly added work points (excavation points and / or unloading points), and the overlapping area of ​​the current camera's image mapping range exceeds 80%, then the previous photo / video (P0) corresponding to that location will be replaced and updated to the current location corresponding to that photo / video (P1), and its coordinate position and azimuth data will be updated synchronously. Its image mapping range will also be replaced and updated, and then merged with the previously formed image range.

[0071] If the current camera's image mapping range covers more than 10 newly added work points (excavation points and / or unloading points), and the overlapping area of ​​the current camera's image mapping range is less than 80%, then the current position is matched with the corresponding photo / video (P1), along with the corresponding coordinate position and azimuth data, and its image mapping range is merged with the previously formed image range.

[0072] If the number of newly added work points (excavation points and / or unloading points) covered by the current camera's image mapping area is less than 10, the video stream corresponding to that mapping area will not be captured, and the corresponding azimuth and positioning information will not be saved.

[0073] S2-3: Repeat step S2-2 to obtain a set of post-job evidence photos / videos covering the entire scope of the job.

[0074] S3, acquire feature photos of key work areas:

[0075] As the construction machinery operates, the camera's shooting range is converted into a graphic mapping on the GIS map. When the construction machinery moves to the location where the graphic mapping range captured by the current camera covers the top 3 points in terms of density of all previous work points (excavation points and / or unloading points) or the total number of work points reaches a preset value, the current photo / video (P2), coordinate position, azimuth data, and graphic mapping range are recorded.

[0076] When the number of work points covered by the graphic mapping range captured by the next camera exceeds the number of work points covered by the graphic mapping range corresponding to the previous photos / videos, the evidence photos / videos (P2) of the key work areas are replaced and updated, and their coordinate positions, azimuth data, and graphic mapping range are updated synchronously.

[0077] Step S4, return of evidence photos:

[0078] Based on the network module, evidence photos / videos, along with corresponding location, azimuth, and other information, are transmitted back to the backend management platform every hour at time points of 5 minutes, 10 minutes, 30 minutes, and 1 hour. For construction machinery equipment, if it stops and restarts midway, the previous data is saved and processed and analyzed again.

[0079] It should be emphasized that the examples described in this invention are illustrative rather than limiting. Therefore, this invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solutions of this invention, without departing from the spirit and scope of this invention, whether modifications or substitutions, are also within the protection scope of this invention.

Claims

1. A method for acquiring evidentiary images based on the operational behavior of construction machinery, characterized in that, This includes the initial image set before the operation and the evidence image set after the operation: Obtaining the initial image set before the operation includes the following steps: Step A1: Before the construction machinery is ready to start operation, take pictures of the location of the construction machinery using the camera on the construction machinery, and save the pictures and image parameters together in the initial image set before operation. At the same time, map the captured pictures onto the GIS map to form the operation area and save it. Step A2: After the construction machinery starts working, the images captured by the camera are continuously mapped onto the GIS map. When the overlap area between the mapped area and the saved work area is less than a preset value, the image and image parameters at this time are saved to the initial image set before the operation. At the same time, the mapped area at this time is merged with the saved work area to form a new work area until the construction machinery completes all the work. In this way, the initial image set before the operation is obtained throughout the entire operation process, and a complete work area is formed and saved. The process of obtaining the evidence image set after the job is completed includes the following steps: Step B1: When the construction machinery starts working, the location of the construction machinery is photographed and the movement of the construction machinery is monitored by the camera on the construction machinery. When the construction machinery is detected to be performing a work action, the location of the current work action is taken as the work point, and the image and image parameters are saved together in the post-work evidence image set. At the same time, the photographed image is mapped onto the GIS map to form the work point evidence range and is saved. Step B2: During the continuous operation of the construction machinery, the images captured by the camera are continuously mapped onto the GIS map, and the movement of the construction machinery is monitored. Whenever the position of the construction machinery changes and it performs an operation, it is recorded as a new working point. When the overlapping area between the mapped range and the working point evidence range is less than a set value, and the new working point in the mapped range exceeds a predetermined value, the image and image parameters at this time are saved to the post-operation evidence image set. At the same time, the mapped range at this time is merged with the saved working point evidence range to form a new working point evidence range, until the construction machinery completes all operations. Thus, the post-operation evidence image set of the entire operation process is obtained, and a complete working point evidence range is formed and saved.

2. The method according to claim 1, characterized in that, It also includes acquiring a set of feature images of key work areas: If the number of work points within the mapping range formed by the currently captured image reaches a predetermined value, the current image and image parameters are saved, and the mapping range is saved, thereby forming a set of evidence images for the key work area.

3. The method according to claim 2, characterized in that, This also includes updating the feature image set of key work areas: After acquiring the feature image set of the key work area, if the number of work points in the mapping range formed by the currently captured image exceeds the number of work points in the saved evidence image set of the key work area, then the current image, image parameters, and mapping range will be overwritten with the saved evidence image set of the key work area, thereby updating the feature image set of the key work area.

4. The method according to claim 3, characterized in that, The number of evidence images for key work areas is a preset amount greater than 1. Before the number of evidence images for key work areas reaches the preset amount, all evidence images for key work areas are independently saved according to the preset value of work points in the mapping range. After the preset amount is reached, the number of work points in the mapping range formed by the currently captured image is compared with all the saved evidence images for key work areas in real time, and when the number of work points exceeds the number of saved evidence images for key work areas, it is overwritten to achieve the update.

5. The method according to claim 4, characterized in that, If the number of work points in the mapping range formed by the current captured image exceeds the number of existing evidence image sets for key work areas, then one existing evidence image set for key work areas will be randomly overwritten.

6. The method according to any one of claims 1-3, characterized in that, The image parameters include the coordinate position and azimuth angle at the time the image was captured.

7. The method according to claim 1, characterized in that, Step B2 further includes updating the post-operation evidence image set: When the overlapping area between the mapping range of the continuously captured images and the evidence range of the work point is greater than the set value, and the new work point in the mapping range exceeds the predetermined value, the current image and image parameters will overwrite the original image and image parameters. At the same time, the mapping range at this time will be merged with the saved evidence range of the work point to form a new evidence range of the work point.

8. The method according to claim 1, characterized in that, It also includes the step of sending the saved data back to the backend: The saved data is sent back to the backend according to preset time nodes, which are the 5th minute, 10th minute, 30th minute and 1st hour after the start of the job, and then sent back once every hour.

9. The method according to claim 1, characterized in that, When more than one piece of construction machinery is operating at a work site, the work site information is shared among the various pieces of construction machinery via the network.

Images