Excavation volume real-time monitoring method and device based on double-trunk target detection and product

Through the dual-trunk object detection model, the real-time and cost problems of real-time monitoring of excavation volume in open excavation operations in earthworks were solved, and the rapid and accurate monitoring of excavation volume was achieved, reducing the cost of manpower and data acquisition. Improve monitoring efficiency and accuracy.

CN119068407AActive Publication Date: 2024-12-03CHINA THREE GORGES CORPORATION

Patent Information

Application Number
CN202410994366.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-12-03
Estimated Expiration
2044-07-23

AI Technical Summary

Technical Problem

The existing real-time monitoring methods for excavation volume in open excavation operations in soil and stone have problems such as long data processing time, poor real-time performance and high cost, especially during the construction stage, which requires a lot of manpower and data collection costs.

Method used

Using a method based on dual-backbone object detection, through video stream image analysis, the dual-backbone object detection model is used to target detection of excavation equipment and self-dumping equipment, determine the excavation operation status and quantify the excavation volume, and realize real-time monitoring. The model includes a first backbone network, a feature segmentation network, a second backbone network and a detection head network, and can quickly and accurately identify the target area and job status in the video stream image.

Benefits of technology

Real-time monitoring of the excavation volume of earth and rock is achieved, reducing manpower and data acquisition costs, improving processing efficiency and real-time performance, and improving the accuracy of target detection through multi-level feature fusion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068407A_ABST
    Figure CN119068407A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an excavation volume real-time monitoring method and device based on double-trunk target detection and a product, and the method specifically comprises the steps: carrying out the target detection of a video stream image of a to-be-counted construction region through a double-trunk target detection model, and obtaining the target information of excavation equipment and self-discharging equipment; determining N target areas in the video stream image; determining an excavation operation state of the current target area; when the excavation operation state of the current target area is updated to a non-excavation state or an excavation state from a loading state, determining the excavation volume corresponding to the current loading of the current target area according to the rated loading capacity corresponding to the self-unloading equipment in the current target area, and updating the accumulated excavation volume of the current target area; and obtaining the total real-time excavation volume of the to-be-counted construction area according to the accumulated excavation volume of all the current target areas in the N target areas. According to the embodiment of the invention, real-time monitoring of the earth-rock excavation volume can be realized at a relatively low cost and a relatively high speed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of open excavation of earth and stone, and in particular to a method, device and product for real-time monitoring of excavation volume based on dual-trunk target detection. Background Art

[0002] Open excavation of earth and stone is a very common type of construction operation. Real-time monitoring of excavation volume helps to follow up on-site construction progress in a timely manner, assist in formulating short-term excavation construction plans, and timely dispatch the number of construction machinery entering the site.

[0003] The calculation method of excavation volume in related technologies usually uses drone oblique photography to obtain digital elevation models before and after excavation during the construction stage, or uses three-dimensional laser scanning technology to obtain spatial point cloud data to realize the calculation of excavation volume.

[0004] The above-mentioned on-site measurement method not only requires a certain amount of manpower and data collection costs, but also has the disadvantages of long data processing time and poor real-time performance. Summary of the invention

[0005] The embodiment of the present application provides a real-time monitoring method for excavation volume based on dual-trunk target detection, which can realize real-time monitoring of earth and stone excavation volume at a relatively low cost and relatively fast speed based on the analysis of video stream images.

[0006] Correspondingly, the embodiments of the present application also provide a real-time monitoring device for excavation volume based on dual-trunk target detection, an electronic device and a machine-readable medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the embodiment of the present application discloses a real-time monitoring method of excavation volume based on dual-trunk target detection, comprising:

[0008] Collect video stream images of the construction area to be counted;

[0009] The dual-trunk target detection model is used to perform target detection on the video stream image to obtain first target information corresponding to the mining equipment contained in the video stream image and second target information corresponding to the dump equipment contained in the video stream image; wherein the dual-trunk target detection model includes: a first trunk network, a feature segmentation network, a second trunk network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second trunk network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1;

[0010] Determine N target areas in the video stream image according to the first target information and the second target information; N is a positive integer;

[0011] For a current target area among the N target areas, the excavation operation state of the current target area is determined according to the operation state of the excavation equipment in the current target area, whether the current target area includes a self-unloading equipment, and the positional relationship between the excavation equipment and the self-unloading equipment in the current target area; the excavation operation state includes: one of: an unexcavated state, an excavating state, and a loading state;

[0012] When the excavation operation state of the current target area is updated from the loading state to the unexcavated state or the excavation state, the current target area is considered to have completed the current loading, and the excavation volume corresponding to the current loading of the current target area is determined according to the rated loading capacity corresponding to the self-unloading equipment in the current target area, and the cumulative excavation volume of the current target area is updated;

[0013] According to the accumulated excavation volumes of all current target areas among the N target areas, the total real-time excavation volume corresponding to the construction area to be counted is determined.

[0014] The embodiment of the present application also discloses a real-time monitoring device for excavation volume based on dual-trunk target detection, the device comprising:

[0015] An acquisition module, used to acquire video stream images of the construction area to be counted;

[0016] The target detection module uses a dual-trunk target detection model to perform target detection on the video stream image to obtain first target information corresponding to the mining equipment contained in the video stream image and second target information corresponding to the dump equipment contained in the video stream image; wherein the dual-trunk target detection model includes: a first trunk network, a feature segmentation network, a second trunk network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second trunk network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1;

[0017] A target region determination module, configured to determine N target regions in the video stream image according to the first target information and the second target information; N is a positive integer;

[0018] An excavation operation state determination module is used to determine the excavation operation state of a current target area among the N target areas according to the operation state of the excavation equipment in the current target area, whether the current target area contains a self-unloading device, and the positional relationship between the excavation equipment and the self-unloading device in the current target area; the operation state includes: one of: an unexcavated state, an excavating state, and a loading state;

[0019] The target area excavation volume determination module is used to determine the excavation volume corresponding to the current loading of the current target area when the excavation operation state of the current target area is updated from the loading state to the unexcavated state or the excavation state, and to determine the excavation volume corresponding to the current loading of the current target area according to the rated loading capacity corresponding to the self-unloading equipment in the current target area, and to update the cumulative excavation volume of the current target area;

[0020] The module for determining the excavation volume of the construction area to be counted is used to determine the total real-time excavation volume corresponding to the construction area to be counted according to the cumulative excavation volume of all current target areas among the N target areas.

[0021] Optionally, the first target information includes: a first bounding box, and the second target information includes: a second bounding box;

[0022] The target area determination module comprises:

[0023] A target region number determination module, configured to determine the number N of target regions according to the number N of first bounding boxes;

[0024] A bounding box matching module, used to perform one-to-one matching between the first bounding box and the second bounding box to obtain a corresponding matching result;

[0025] A first determination module is configured to, if there is a second bounding box matching the first bounding box, use a minimum enclosing rectangular area containing both the first bounding box and the second bounding box as a target area;

[0026] The second determining module is configured to use a rectangular area corresponding to the first bounding box as a target area if there is no second bounding box matching the first bounding box.

[0027] Optionally, the bounding box matching module includes:

[0028] A distance intersection-over-union ratio index value determination module, used to determine a distance intersection-over-union ratio index value between a first bounding box and a second bounding box;

[0029] The matching determination module is used to select a second bounding box with the largest distance intersection and union ratio index value from the multiple second bounding boxes according to the distance intersection and union ratio index values ​​between the first bounding box and the multiple second bounding boxes as the second bounding box matching the first bounding box.

[0030] Optionally, the excavation operation status determination module includes:

[0031] A first judgment module is used to judge whether the excavation equipment included in the current target area is in a stationary state to obtain a first judgment result;

[0032] A first determination module, configured to determine that the excavation operation state of the current target area is an unexcavated state if the first determination result is yes;

[0033] A second judgment module is used to judge whether the current target area contains a self-unloading device if the first judgment result is no, so as to obtain a second judgment result;

[0034] A second determination module, configured to determine that the excavation operation state of the current target area is an excavation state if the second determination result is no;

[0035] a third judgment module, for judging whether there is overlap between the self-unloading equipment and the excavating equipment in the current target area if the second judgment result is yes, so as to obtain a third judgment result;

[0036] A third determination module, configured to determine that the excavation operation state of the current target area is a loading state if the third determination result is yes;

[0037] The fourth determination module is used to determine that the excavation operation status of the current target area is an excavation status if the third determination result is no.

[0038] Optionally, the third judgment module includes:

[0039] An intersection-and-union ratio index value determination module is used to determine an intersection-and-union ratio index value between a second bounding box corresponding to the self-unloading equipment in the current target area and a first bounding box corresponding to the excavation equipment in the current target area;

[0040] The overlap determination module is used to determine that there is overlap between the dump equipment and the excavation equipment in the current target area if the intersection-to-union ratio index value is greater than a first threshold.

[0041] Optionally, the first judgment module is specifically used for: in the i-th frame video stream image to the i+α-th frame video stream image, if the coordinate difference of the mining equipment in any two frames of video stream image is less than a second threshold, then determining that the mining equipment contained in the current target area is in a stationary state; wherein i and α are positive integers.

[0042] Optionally, the device further comprises:

[0043] A model determination module, used to determine the target model corresponding to the self-unloading equipment in the current target area by using the self-unloading equipment model classification model;

[0044] The search module is used to search the mapping relationship between the search model and the rated loading capacity according to the target model, so as to determine the rated loading capacity corresponding to the self-unloading equipment in the current target area.

[0045] Optionally, the performing target detection on the video stream image using a dual-trunk target detection model includes:

[0046] The first backbone network determines n first image features corresponding to the video stream image;

[0047] The i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, and the obtained segmentation processing result includes: i seed image features, and outputs the j-th seed image features to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i seed image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers;

[0048] The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result, and the channel-to-pixel module in the second backbone network determines a second image feature according to the first fusion result, and outputs the second image feature to the neck network;

[0049] The neck network performs a second fusion process on the second image feature to obtain a second fusion process result;

[0050] The detection head network determines the first target information and the second target information according to the second fusion processing result.

[0051] Optionally, the i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, including:

[0052] The i-th feature segmentation module in the feature segmentation network performs convolution processing on the i-th first image feature to obtain image features with c channels;

[0053] According to the channel dimension, the image features with c channels are divided into i seed image features with different numbers of channels; the number of channels of the jth seed image feature in the i seed image features is: t*2 (j-1) .

[0054] Optionally, the second backbone network further includes: a first convolution module connected in front of the feature fusion module;

[0055] The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature, including:

[0056] The i-th feature fusion module receives the n-i+1 seed image features and the convolution image features output by the first convolution module; the number of channels corresponding to the n-i+1 seed image features is the first number of channels, and the number of channels corresponding to the convolution image features is the second number of channels;

[0057] The i-th feature fusion module performs interpolation processing on the n-i+1 seed image features, and the number of channels corresponding to the n-i+1 seed image features after the interpolation processing is the second channel number;

[0058] The i-th feature fusion module fuses the n-i+1 seed image features and convolution image features after interpolation.

[0059] Optionally, the first convolution module connected in front of the first feature fusion module is used to perform convolution processing on the video stream image and output corresponding convolution image features to the first feature fusion module.

[0060] Optionally, the n feature segmentation modules include: a first feature segmentation module, a second feature segmentation module, a third feature segmentation module, a fourth feature segmentation module and a fifth feature segmentation module; the n feature fusion modules include: a first feature fusion module, a second feature fusion module, a third feature fusion module, a fourth feature fusion module and a fifth feature fusion module;

[0061] The segmentation processing result of the first feature segmentation module includes: 1 m-channel first sub-image feature, and 1 m-channel first sub-image feature is output to the first feature fusion module;

[0062] The segmentation processing result of the second feature segmentation module includes: one m-channel second sub-image feature and one 2m-channel third sub-image feature; one m-channel second sub-image feature is output to the first feature fusion module; one 2m-channel third sub-image feature is output to the second feature fusion module;

[0063] The segmentation processing result of the third feature segmentation module includes: one fourth sub-image feature of the m channel, one fifth sub-image feature of the 2m channel and one sixth sub-image feature of the 4m channel; one fourth sub-image feature of the m channel is output to the first feature fusion module; one fifth sub-image feature of the 2m channel is output to the second feature fusion module; one sixth sub-image feature of the 4m channel is output to the third feature fusion module;

[0064] The segmentation processing result of the fourth feature segmentation module includes: one seventh sub-image feature of the m channel, one eighth sub-image feature of the 2m channel, one ninth sub-image feature of the 4m channel, and one tenth sub-image feature of the 8m channel; one seventh sub-image feature of the m channel is output to the first feature fusion module; one eighth sub-image feature of the 2m channel is output to the second feature fusion module; one ninth sub-image feature of the 4m channel is output to the third feature fusion module; and one tenth sub-image feature of the 8m channel is output to the fourth feature fusion module;

[0065] The segmentation processing results of the fifth feature segmentation module include: one eleventh sub-image feature of the m channel, one twelfth sub-image feature of the 2m channel, one thirteenth sub-image feature of the 4m channel, one fourteenth sub-image feature of the 8m channel and one fifteenth sub-image feature of the 16m channel; one eleventh sub-image feature of the m channel is output to the first feature fusion module; one twelfth sub-image feature of the 2m channel is output to the second feature fusion module; one thirteenth sub-image feature of the 4m channel is output to the third feature fusion module; one fourteenth sub-image feature of the 8m channel is output to the fourth feature fusion module; and one fifteenth sub-image feature of the 16m channel is output to the fifth feature fusion module.

[0066] The embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the method described in the embodiment of the present application.

[0067] The embodiment of the present application further discloses a machine-readable medium on which executable code is stored. When the executable code is executed, a processor executes the method described in the embodiment of the present application.

[0068] The embodiments of the present application include the following advantages:

[0069] In the technical solution of the embodiment of the present application, first, a video stream image of the construction area to be counted is collected; then, a dual-trunk target detection model is used to perform target detection on the above video stream image to obtain target information corresponding to targets such as excavating equipment and dump equipment in the video stream image; then, based on the above target information, N target areas in the above video stream image are determined; then, based on the objective laws between the excavation operation status of the target area and the operation status of the excavation equipment, whether there is a dump equipment, and the position relationship between the excavation equipment and the dump equipment, the excavation volume corresponding to the current loading of the current target area is determined, and the cumulative excavation volume of the current target area is updated; and then, based on the cumulative excavation volume of all current target areas among the above N target areas, the total real-time excavation volume corresponding to the construction area to be counted is determined.

[0070] Since the embodiment of the present application determines N target areas in the video stream image according to the target information corresponding to the targets such as the excavation equipment and the self-unloading equipment in the video stream image, and determines the excavation amount corresponding to the current loading of the current target area according to the objective law between the excavation operation state of the target area and the operation state of the excavation equipment, whether there is a self-unloading equipment, and the position relationship between the excavation equipment and the self-unloading equipment; in this way, the embodiment of the present application can continuously determine the target area existing in the video stream image and the excavation operation state corresponding to the target area according to the continuously updated video stream image; on this basis, the embodiment of the present application can continuously determine the excavation amount corresponding to the current loading of the current target area in the video stream image, and the cumulative excavation amount of all current target areas in the video stream image. Therefore, the embodiment of the present application can realize the real-time monitoring of the earthwork excavation amount at a lower cost and faster speed based on the analysis of the video stream image. In other words, the embodiment of the present application can save the labor cost, data collection cost, and data processing time cost consumed by using drone oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation amount.

[0071] In addition, the embodiment of the present application utilizes a dual-trunk target detection model to perform target detection on video image frames. Among them, the n feature segmentation modules contained in the feature segmentation network in the above dual-trunk target detection model can play a role in aggregating sub-image features of multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features of multiple levels. In this way, the embodiment of the present application can enhance the expression ability of the second image features output by the second backbone network and the accuracy of the second image features, thereby improving the accuracy of the target detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a schematic diagram of an application environment of a real-time monitoring method of excavation volume based on dual-trunk target detection according to an embodiment of the present application;

[0073] Figure 2 It is a schematic diagram of the steps of a real-time monitoring method of excavation volume based on dual-trunk target detection according to an embodiment of the present application;

[0074] Figure 3 is a flowchart of the steps of a method for determining the excavation operation status of a current target area according to an embodiment of the present application;

[0075] Figure 4 It is a structural diagram of a dual-trunk target detection model according to an embodiment of the present application;

[0076] Figure 5 It is a schematic diagram of the structure of a first backbone network 301, a feature segmentation network 302, and a second backbone network 303 in one embodiment of the present application;

[0077] Figure 6 is a schematic diagram of the structure of a channel-to-pixel module according to an embodiment of the present application;

[0078] Figure 7 is a schematic diagram of the structure of a bottleneck module of an embodiment of the present application;

[0079] Figure 8 is a schematic diagram of the structure of a second backbone network according to an embodiment of the present application;

[0080] Fig. 9 is a schematic diagram of the structure of a first backbone network according to an embodiment of the present application;

[0081] Fig.10 is a schematic diagram of the structure of a neck network according to an embodiment of the present application;

[0082] Fig.11 It is a structural diagram of a spatial pyramid pooling module according to an embodiment of the present application;

[0083] Fig.12 It is a structural schematic diagram of a real-time monitoring device for excavation volume based on dual-trunk target detection according to an embodiment of the present application;

[0084] Fig.13 It is a schematic diagram of the structure of a device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0085] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0086] The embodiments of the present application can be applied to engineering industries such as hydropower and civil engineering, and can be used for open excavation operations in the engineering industry to monitor the corresponding excavation volume in real time.

[0087] Taking the pumped-storage power station project as an example, the construction project presents a situation where excavation operations are carried out simultaneously at multiple construction sites. At the same time, due to the continuous changes in the on-site construction situation, there is a need to continuously carry out real-time monitoring of excavation volume based on dual-trunk target detection for multiple construction sites.

[0088] The traditional method for calculating the excavation volume during the construction phase usually involves first obtaining a digital elevation model of the excavation area using drone oblique photography before and after excavation, or using 3D laser scanning technology to obtain spatial point cloud data of the excavation area; then, the volume of the excavation area is calculated based on the difference between the digital elevation model of the excavation area or the spatial point cloud data; then, the excavation volume is calculated based on the volume of the excavation area. The above-mentioned on-site measurement method not only requires a certain amount of manpower and data collection costs, but also has the disadvantages of long data processing time and poor real-time performance.

[0089] In view of the technical problems of traditional technology, such as labor consumption, data collection cost, long processing time and poor real-time performance, the embodiment of the present application provides a real-time monitoring method of excavation volume based on dual-trunk target detection, which specifically includes the following steps:

[0090] Collect video stream images of the construction area to be counted;

[0091] The dual-trunk target detection model is used to perform target detection on the video stream image to obtain the first target information corresponding to the mining equipment contained in the video stream image and the second target information corresponding to the dump equipment contained in the video stream image; wherein the dual-trunk target detection model includes: a first trunk network, a feature segmentation network, a second trunk network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second trunk network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1;

[0092] According to the first target information and the second target information, determine N target areas in the video stream image; N is a positive integer;

[0093] For a current target area among the N target areas, the excavation operation state of the current target area is determined according to the operation state of the excavation equipment in the current target area, whether the current target area includes a self-unloading equipment, and the positional relationship between the excavation equipment and the self-unloading equipment in the current target area; the excavation operation state includes: one of: an unexcavated state, an excavating state, and a loading state;

[0094] When the excavation operation state of the current target area is updated from the loading state to the unexcavated state or the excavation state, the current target area is considered to have completed the current loading, and the excavation volume corresponding to the current loading of the current target area is determined according to the rated loading capacity corresponding to the self-unloading equipment in the current target area, and the cumulative excavation volume of the current target area is updated;

[0095] According to the accumulated excavation volume of all current target areas among the above N target areas, the total real-time excavation volume corresponding to the construction area to be counted is determined.

[0096] The embodiment of the present application first collects video stream images of the construction area to be counted; then, using the dual-trunk target detection model, target detection is performed on the above video stream images to obtain target information corresponding to targets such as excavating equipment and dump equipment in the video stream images; then, based on the above target information, N target areas in the above video stream images are determined; then, based on the objective laws between the excavation operation status of the target area and the operation status of the excavation equipment, whether there is a dump equipment, and the position relationship between the excavation equipment and the dump equipment, the excavation volume corresponding to the current loading of the current target area is determined, and the cumulative excavation volume of the current target area is updated; and then, based on the cumulative excavation volume of all current target areas among the above N target areas, the total real-time excavation volume corresponding to the construction area to be counted is determined.

[0097] Since the embodiment of the present application determines N target areas in the video stream image according to the target information corresponding to the targets such as the excavation equipment and the self-unloading equipment in the video stream image, and determines the excavation amount corresponding to the current loading of the current target area according to the objective law between the excavation operation state of the target area and the operation state of the excavation equipment, whether there is a self-unloading equipment, and the position relationship between the excavation equipment and the self-unloading equipment; in this way, the embodiment of the present application can continuously determine the target area and the operation state corresponding to the target area in the video stream image according to the continuously updated video stream image; on this basis, the embodiment of the present application can continuously determine the excavation amount corresponding to the current loading of the current target area in the video stream image, and the cumulative excavation amount of all current target areas in the video stream image. Therefore, the embodiment of the present application can realize the real-time monitoring of the earthwork excavation amount at a lower cost and faster speed based on the analysis of the video stream image. In other words, the embodiment of the present application can save the labor cost, data collection cost, and data processing time cost consumed by using drone oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation amount.

[0098] In addition, the embodiment of the present application utilizes a dual-trunk target detection model to perform target detection on video image frames. Among them, the n feature segmentation modules contained in the feature segmentation network in the above dual-trunk target detection model can play a role in aggregating sub-image features of multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features of multiple levels. In this way, the embodiment of the present application can enhance the expression ability of the second image features output by the second backbone network and the accuracy of the second image features, thereby improving the accuracy of the target detection results.

[0099] Reference Figure 1 , showing a schematic diagram of an application environment of a real-time monitoring method for excavation volume based on dual-trunk target detection according to an embodiment of the present invention, wherein the image acquisition end 101 and the service end 102 can perform data exchange based on a wireless network or a wired network.

[0100] In practical applications, the image acquisition terminal 101 may be equipped with an image acquisition device having an image acquisition function, such as an image sensor, etc. The image acquisition device may acquire construction videos and send the construction videos to the server 102 according to a preset time period.

[0101] After receiving the construction video sent by the image acquisition end 101, the server end 102 can extract the video stream image from the construction video and process the video stream image using the method of the embodiment of the present application to determine the excavation volume corresponding to all current target areas in the video stream image of the construction area to be counted.

[0102] It should be noted that, as the video stream image is updated, the embodiment of the present application can continuously determine the target area where the video stream image exists and the excavation operation status corresponding to the target area; on this basis, the embodiment of the present application can continuously determine the excavation volume corresponding to the current load of the current target area in the video stream image, and the cumulative excavation volume of all current target areas in the video stream image. Therefore, the embodiment of the present application can realize real-time monitoring of earthwork excavation volume at a relatively low cost and relatively fast speed based on the analysis of the video stream image.

[0103] Method Example 1

[0104] refer to Figure 2 , shows a schematic flow chart of the steps of a real-time monitoring method for excavation volume based on dual-trunk target detection according to an embodiment of the present application, the method specifically comprises the following steps:

[0105] Step 201, collecting video stream images of the construction area to be counted;

[0106] Step 202: Utilize the dual-trunk target detection model to perform target detection on the video stream image to obtain first target information corresponding to the mining equipment contained in the video stream image and second target information corresponding to the dump equipment contained in the video stream image; wherein the dual-trunk target detection model specifically includes: a first trunk network, a feature segmentation network, a second trunk network, a neck network and a detection head network; the feature segmentation network specifically includes: n feature segmentation modules; the second trunk network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n can be a positive integer greater than 1;

[0107] Step 203: Determine N target areas in the video stream image according to the first target information and the second target information; N is a positive integer;

[0108] Step 204: for a current target area among the N target areas, according to the operating state of the excavation equipment in the current target area, whether the current target area includes a self-unloading equipment, and the positional relationship between the excavation equipment and the self-unloading equipment in the current target area, determine the excavation operating state of the current target area; the excavation operating state includes: one of: an unexcavated state, an excavating state, and a loading state;

[0109] Step 205: When the excavation operation state of the current target area is updated from the loading state to the unexcavated state or the excavation state, it is considered that the current target area has completed the current loading, and the excavation volume corresponding to the current loading of the current target area is determined according to the rated loading capacity corresponding to the self-unloading equipment in the current target area, and the cumulative excavation volume of the current target area is updated;

[0110] Step 206: Determine the total real-time excavation volume corresponding to the construction area to be counted based on the cumulative excavation volume of all current target areas among the N target areas.

[0111] Figure 2 At least one step included in the method shown can be executed by the server. It can be understood that the embodiment of the present application is for Figure 2 The specific execution subject of the method shown is not limited.

[0112] In step 201, the process of collecting video stream images specifically includes: after receiving the construction video sent by the image acquisition end, the server extracts the video stream images from the construction video. The server can receive the video stream images from the image acquisition end according to a preset time period.

[0113] In a hydropower engineering scenario, multiple video sources may be set, and different video sources may correspond to different video stream images. Among them, the video source may continuously provide continuous video stream images. The embodiment of the present application may process the continuous video stream images provided by any video source to obtain the excavation amount corresponding to the video source. The construction area to be counted may be a construction area represented by a certain video source.

[0114] In step 202, a dual-trunk target detection model may be used to perform target detection on the video stream image to obtain first target information corresponding to the mining equipment contained in the video stream image and second target information corresponding to the dump truck contained in the video stream image.

[0115] The dual-trunk target detection model is a deep learning model in the field of computer vision, which aims to identify target information corresponding to a target in an image. Specifically in the embodiment of the present application, the video stream image may include targets such as excavating equipment and dump equipment. Examples of excavating equipment may include excavators, etc. Examples of dump equipment may include dump trucks, etc. In actual applications, the dump truck drives into the excavation site and stops within the working radius of the excavator so that the excavator can easily load the material into the cargo box of the dump truck.

[0116] The first target information and the second target information belong to the category of target information. The target information may correspond to category information. For example, the category corresponding to the first target information is the excavation equipment category, and the category corresponding to the second target information is the dump equipment category.

[0117] The target information specifically includes: the bounding box corresponding to the target. The bounding box is used to indicate the position and range of the target in the video stream image. The bounding box is usually a rectangular box whose four sides are aligned with the outermost edges of the target, thereby surrounding the entire target. The bounding box information may include: the coordinates of the upper left corner of the rectangular box, the width and the height; or, the bounding box information may include: the coordinates of the center point of the rectangular box, the width and the height.

[0118] The specific process of performing target detection on the above video stream image using target detection technology will be described in detail in the second method embodiment.

[0119] In step 203, the target area may be an area in the video stream image that contains an excavating device. Depending on whether the target area contains a dumping device, the target area may be divided into a first target area and a second target area. The first target area contains an excavating device and does not contain a dumping device. The second target area contains an excavating device and a dumping device.

[0120] In one implementation of the present application, the first target information includes: a first bounding box, and the second target information includes: a second bounding box;

[0121] Step 203 is a process of determining N target areas in the video stream image according to the first target information and the second target information, which specifically includes:

[0122] Step A1, determining the number N of target areas according to the number N of first bounding boxes;

[0123] Step A2: performing one-to-one matching on the first bounding box and the second bounding box to obtain a corresponding matching result;

[0124] Step A3: If the first bounding box has a second bounding box that matches it, the minimum enclosing rectangular area that contains both the first bounding box and the second bounding box is used as the target area;

[0125] Step A4: If there is no second bounding box matching the first bounding box, the rectangular area corresponding to the first bounding box is used as the target area.

[0126] In step A1, the number N of target regions may be equal to the number N of first bounding boxes. In other words, the number of target regions included in one video stream image may be equal to the number of first bounding boxes included in one video stream image.

[0127] In step A2, a one-to-one matching is performed between the first bounding box and the second bounding box, which may be a distance matching between the first bounding box and the second bounding box.

[0128] In one implementation, step A2 is a process of performing one-to-one matching of the first bounding box and the second bounding box, specifically comprising: determining a distance intersection-and-union index value between a first bounding box and a second bounding box; and selecting, from the multiple second bounding boxes, one with the largest distance intersection-and-union index value according to the distance intersection-and-union index value between the first bounding box and multiple second bounding boxes as the second bounding box that matches the first bounding box.

[0129] Among them, for the second bounding boxes of the M dump trucks in the video stream image, the distance intersection and union ratio index value is used as a criterion to perform one-to-one matching with the first bounding boxes in the video stream image one by one. mn The determination process of is shown in formula (1).

[0130]

[0131] Among them, A m represents the mth (m∈[1,M], and is a positive integer) second bounding box, B n Represents the nth (n∈[1,N], and is a positive integer) first bounding box. a m , b n Represent the second bounding box A respectively m , first bounding box B n The center point, ρ mn The center point a m , b n The Euclidean distance between them, d is the diagonal distance of the minimum enclosing area that contains both the first bounding box and the second bounding box.

[0132] The distance intersection and union ratio index value reflects the degree of overlap between the first bounding box and the second bounding box and the distance between the center points. Generally speaking, the larger the distance intersection and union ratio index value, the higher the degree of overlap between the two and the closer the distance between the center points.

[0133] For a first bounding box, there are M distance intersection and union index values ​​between it and the M second bounding boxes. Then, a maximum distance intersection and union index value can be selected from the M distance intersection and union index values, and the second bounding box corresponding to the maximum distance intersection and union index value can be used as the second bounding box that matches the first bounding box.

[0134] Step A3 and step A4 may be performed in parallel.

[0135] In practical applications, the N first bounding boxes in the video stream image can be traversed in a first preset order to determine the target area corresponding to the first bounding box one by one. The first preset order can be: from left to right, from bottom to top, or from left to right, from top to bottom.

[0136] The embodiment of the present application may also number the N first boundary boxes in the video stream image according to a first preset order.

[0137] The numbering may refer to a number or a number assigned in sequence using any ordered or unordered symbol. In a specific implementation, the numbering of the N first bounding boxes may be the same as the numbering of the N target areas. For example, the numbering of the N first bounding boxes in the video stream image A may be respectively: A0001, A0002, ... A000N, and the numbering of the N target areas in the video stream image A may also be respectively: A0001, A0002, ... A000N.

[0138] Of course, the numbers of the N first bounding boxes and the numbers of the N target areas may also be different. For example, the numbers of the N first bounding boxes in the video stream image A may be respectively: A0001, A0002, ... A000N, and the numbers of the N target areas in the video stream image A may be respectively: A+0001, A+0002, ... A+000N.

[0139] In the process of traversing the N first bounding boxes in the video stream image, it can be determined whether there is a second bounding box matching a first bounding box. If so, step A3 can be executed to use the minimum enclosed rectangular area containing both the first bounding box and the second bounding box as the target area. If not, step A4 can be executed to use the rectangular area corresponding to the first bounding box as the target area.

[0140] After the target area is determined based on step A3 and step A4, information about the target area can be obtained, which specifically includes: the number of the target area, the position information of the rectangular frame corresponding to the target area, whether the second boundary frame is included, and other information.

[0141] In step 204, the excavation operation status of the current target area among the N target areas can be determined. Among them, the current target area can represent one target area among the N target areas. It can be understood that the embodiment of the present application can execute the processing of multiple current target areas among the N target areas in parallel or in series. The processing of the current target area may include: the processing of step 204 and step 205. The process of serially executing the processing of multiple current target areas among the N target areas specifically includes: traversing the N target areas in a second preset order of the target area numbers from small to large or from large to small to obtain a current target area among the N target areas, and serially executing the processing of multiple current target areas among the N target areas in a second preset order.

[0142] The excavation operation status of the present application specifically includes: one of the unexcavated state, the excavation state and the loading state; wherein, the unexcavated state means that the current target area is not undergoing excavation operation. The excavation state means that the current target area is undergoing excavation operation. The loading state means that the current target area is undergoing loading operation.

[0143] The embodiment of the present application can determine the excavation operation status of the above-mentioned current target area according to the operating status of the excavating equipment in the above-mentioned current target area, whether the above-mentioned current target area contains a dump equipment, and the positional relationship between the excavating equipment and the dump equipment in the current target area.

[0144] Reference Figure 3 , shows a flowchart of the steps of a method for determining the excavation operation status of the current target area in one embodiment of the present application, which may specifically include the following steps:

[0145] Step 301: determine whether the excavation equipment included in the current target area is in a stationary state to obtain a first determination result;

[0146] If the first judgment result is yes, then step 302 is executed: determining that the excavation operation state of the current target area is an unexcavated state;

[0147] If the first judgment result is no, then executing step 303: judging whether the current target area includes a self-unloading device, so as to obtain a second judgment result;

[0148] If the second judgment result is no, then step 304 is executed: determining that the excavation operation state of the current target area is in the excavation state;

[0149] If the second judgment result is yes, then executing step 305: judging whether there is overlap between the self-unloading equipment and the excavating equipment in the current target area, so as to obtain a third judgment result;

[0150] If the third judgment result is yes, then step 306 is executed: determining that the excavation operation state of the current target area is in a loading state;

[0151] If the third judgment result is no, step 307 is executed: determining that the excavation operation state of the current target area is in the excavation state.

[0152] The present application embodiment provides the following determination rules for the excavation operation status:

[0153] The rule for determining the unexcavated state is that the excavation equipment is in a stationary state.

[0154] The judgment rules for the excavation state include: the excavator is in a non-stationary state, and the current target area does not include the dump equipment; or, the current target area includes the dump equipment, and there is no overlap between the dump equipment and the excavation equipment.

[0155] The judgment rules of the loading state include: the excavation equipment is in a non-stationary state, the current target area contains the dump equipment, and there is overlap between the dump equipment and the excavation equipment.

[0156] Step 301 is a process of determining whether the excavation equipment included in the current target area is in a stationary state, specifically including:

[0157] In the i-th frame video stream image to the i+α-th frame video stream image, if the coordinate difference of the mining equipment in any two frames of video stream images is less than the second threshold, it is determined that the mining equipment contained in the current target area is in a stationary state; wherein i and α can both be positive integers, and α is greater than i.

[0158] The embodiment of the present application can select two video stream images to be compared from the α-frame video stream images corresponding to the i-th frame video stream image to the i+α-th frame video stream image, and determine whether the coordinate difference between the two video stream images to be compared is less than a second threshold.

[0159] Two video stream images to be compared are selected from the α frame video stream images, and the selection schemes may be P. If the coordinate differences corresponding to the P selection schemes are all less than the second threshold, it can be determined that the mining equipment included in the current target area is in a stationary state.

[0160] Considering the influence of signal transmission, camera shaking, etc., when the mining device is in a stationary state, the coordinates of the first bounding box corresponding to it may be offset to a certain extent. Assume that the upper left corner pixel coordinates of the first bounding box of the mining device in the i-th frame of the current target area are The pixel coordinates of the lower right corner are Assume that the second threshold corresponding to the coordinate offset is δ and the frame number threshold is α. If formula (2) holds, that is, the coordinate difference between any two frames from the i-th frame video stream image to the i+α-th frame video stream image is less than the second threshold δ, then it can be determined that the excavation equipment is in a stationary state.

[0161] Formula (2) calculates and determines the coordinate difference between the upper left corner pixel coordinates of the i-th frame video stream image and the (i+j)-th frame video stream image. Therefore, Represents the horizontal coordinate of the upper left corner pixel of the (i+j)th frame of the video stream image, Represents the vertical coordinate of the upper left corner pixel of the (i+j)th frame of the video stream image.

[0162]

[0163] It can be understood that formula (2) calculates and determines the coordinate difference between the upper left pixel coordinates of the i-th frame video stream image and the (i+j)-th frame video stream image, and is only an optional embodiment. In fact, the coordinate difference can be calculated and determined for the upper right pixel coordinates, lower left pixel coordinates, or lower right pixel coordinates of the i-th frame video stream image and the (i+j)-th frame video stream image.

[0164] After determining the target area, step 203 obtains the target area number, the position information of the rectangular box corresponding to the target area, whether the second bounding box is included, and other information. Therefore, step 303 can determine whether the current target area contains the self-unloading equipment according to whether the current target area contains the information corresponding to the second bounding box. Of course, target detection technology can be used to perform target detection on the area image corresponding to the current target area to determine whether the current target area contains the self-unloading equipment.

[0165] Step 305 is a process of determining whether there is overlap between the dump equipment and the excavation equipment in the current target area, specifically including:

[0166] Step C1, determining an intersection-and-union ratio index value between a second bounding box corresponding to the dump equipment in the current target area and a first bounding box corresponding to the mining equipment in the current target area;

[0167] Step C2: If the intersection-to-union ratio index value is greater than a first threshold, it is determined that there is overlap between the dump equipment and the excavation equipment in the current target area.

[0168] Formula (3) shows the calculation process of the intersection-over-union index value between the second bounding box and the first bounding box.

[0169]

[0170] Wherein, A represents the second bounding box, B represents the first bounding box, A∩B represents the intersection area between the second bounding box and the first bounding box, and A∪B represents the union area between the second bounding box and the first bounding box.

[0171] In practical applications, when the intersection-and-union ratio index value is greater than 0, it can be considered that there is an overlap between the dump equipment and the excavation equipment in the current target area. Therefore, examples of the first threshold may include: 0 or a real number greater than 0.

[0172] In step 205, the change of the excavation operation state of the current target area can be monitored. If the excavation operation state is updated from the loading state to the unexcavated state or the excavation state, it can be considered that the current target area has completed the current loading, and the excavation volume corresponding to the current loading of the current target area is determined according to the rated loading capacity corresponding to the self-unloading equipment in the current target area, and the cumulative excavation volume of the current target area is updated. The cumulative excavation volume of the current target area can be the sum of the excavation volumes corresponding to multiple loadings of the current target area.

[0173] The process of determining the rated loading capacity corresponding to the self-unloading equipment in the current target area specifically includes:

[0174] Step D1, using the self-unloading equipment model classification model, determining the target model corresponding to the self-unloading equipment in the current target area;

[0175] Step D2: According to the target model, search in the mapping relationship between the search model and the rated loading capacity to determine the rated loading capacity corresponding to the self-unloading equipment in the current target area.

[0176] The dump truck model classification model of the embodiment of the present application may have the ability to classify dump truck models. In other words, it may determine the target model corresponding to the dump truck in the area image based on the area image corresponding to the current target area.

[0177] In a specific implementation, the embodiment of the present application can label the model category for the dump truck image data set, and train the dump truck model classification model based on the dump truck image data set.

[0178] Here, a process for acquiring a dump equipment image dataset is provided. Specifically, based on a video surveillance system, images of dump equipment covering different construction backgrounds, different lighting conditions, different working postures, etc. can be collected. In addition, it is also necessary to collect a combined image of a dump equipment and an excavation equipment in cooperation with each other in excavation operations to reflect the actual situation of mutual occlusion of the two types of construction machinery. Therefore, the dump equipment image dataset may include: dump equipment images and combined images.

[0179] The embodiment of the present application does not limit the specific structure of the dump truck model classification model. For example, the dump truck model classification model may include: a convolutional neural network structure such as VGG (Visual Geometry Group).

[0180] The embodiment of the present application can pre-save the mapping relationship between the model and the rated loading capacity. In this way, according to the target model corresponding to the self-unloading equipment in the current target area, the rated loading capacity corresponding to the self-unloading equipment in the current target area can be searched in the above mapping relationship to obtain.

[0181] In the embodiment of the present application, when the excavation operation state of the current target area is monitored to be updated from the loading state to the unexcavated state or the excavation state, the current target area may be considered to have completed the current loading, and the excavation amount corresponding to the current loading of the current target area may be determined according to the rated loading capacity corresponding to the self-unloading equipment in the current target area. The excavation amount corresponding to the current loading may be equal to the rated loading capacity corresponding to the self-unloading equipment in the current target area.

[0182] In step 206, the accumulated excavation volumes of all current target areas among the N target areas may be summed up to obtain the total real-time excavation volume corresponding to the construction area to be counted.

[0183] In one example of the present application, the total real-time excavation volume corresponding to the construction area to be counted can be set to 0 at the beginning of a natural day. Subsequently, the total real-time excavation volume can be updated according to the completion of the current loading of the current target area to obtain the continuously accumulated total real-time excavation volume. Therefore, the real-time monitoring process of the excavation volume can be a process of continuously updating the total real-time excavation volume according to the completion of the current loading of N current target areas.

[0184] In practical applications, if there are multiple construction areas to be counted in a hydropower project, the total real-time excavation volume corresponding to each construction area to be counted can be monitored separately according to natural days. After a natural day, the total real-time excavation volume corresponding to multiple construction areas to be counted can be merged. It can be understood that the embodiment of the present application does not limit the specific fusion method.

[0185] In summary, the real-time monitoring method of excavation volume based on dual-trunk target detection in the embodiment of the present application determines N target areas in the video stream image according to the target information corresponding to the targets such as the excavation equipment and the self-unloading equipment in the video stream image, and determines the excavation volume corresponding to the current loading of the current target area according to the objective law between the excavation operation state of the target area and the operation state of the excavation equipment, whether there is a self-unloading equipment, and the position of the excavation equipment and the self-unloading equipment; in this way, the embodiment of the present application can continuously determine the target area and the operation state corresponding to the target area in the video stream image according to the continuously updated video stream image; on this basis, the embodiment of the present application can continuously determine the excavation volume corresponding to the current loading of the current target area in the video stream image, and the cumulative excavation volume of all current target areas in the video stream image. Therefore, the embodiment of the present application can realize the real-time monitoring of the earthwork excavation volume at a lower cost and faster speed based on the analysis of the video stream image. In other words, the embodiment of the present application can save the labor cost, data collection cost, and data processing time cost consumed by using drone oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation volume.

[0186] In addition, the embodiment of the present application utilizes a dual-trunk target detection model to perform target detection on video image frames. Among them, the n feature segmentation modules contained in the feature segmentation network in the above dual-trunk target detection model can play a role in aggregating sub-image features of multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features of multiple levels. In this way, the embodiment of the present application can enhance the expression ability of the second image features output by the second backbone network and the accuracy of the second image features, thereby improving the accuracy of the target detection results.

[0187] Method Example 2

[0188] In an embodiment of the present application, a dual-trunk target detection model can be used to perform target detection on the video stream image to obtain first target information corresponding to the mining equipment contained in the video stream image and second target information corresponding to the dump equipment contained in the video stream image. In a specific implementation, the video stream image can be input into the dual-trunk target detection model to obtain the first target information and the second target information output by the dual-trunk target detection model.

[0189] Reference Figure 4 , shows a schematic diagram of the structure of a dual-trunk target detection model of an embodiment of the present application, which specifically includes: a first backbone network 401, a feature segmentation network 402, a second backbone network 403, a neck network 404 and a detection head network 405 connected in sequence.

[0190] The first backbone network 401 is used to extract features from the input image to obtain n first image features. In the training phase, the input image may be a video stream image sample. In the image recognition phase, the input image may be a real-time video stream image.

[0191] The feature segmentation network 402 is used to perform convolution processing and segmentation processing on the n types of first image features, and output the multiple sub-image features obtained by segmentation to the second backbone network 403.

[0192] The second backbone network 403 is used to perform a first fusion process and a channel-to-pixel process on the multiple sub-image features to obtain a second image feature, and output the second image feature to the neck network 404.

[0193] The neck network 404 is used to perform a second fusion process on the second image feature to obtain a second fusion process result.

[0194] The detection head network 405 is used to determine the recognition result according to the above second fusion processing result.

[0195] Reference Figure 5 , shows a schematic diagram of the structure of a first backbone network 401, a feature segmentation network 402 and a second backbone network 403 of an embodiment of the present application, wherein the first backbone network 401 outputs n first image features to n feature segmentation modules of the feature segmentation network 402.

[0196] The feature segmentation network 402 specifically includes: n feature segmentation modules, which are respectively recorded as: a first feature segmentation module 421, a second feature segmentation module 422... and an nth feature segmentation module 42n.

[0197] The i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, and the obtained segmentation processing result includes: i seed image features, and outputs the j-th seed image features to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i seed image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers.

[0198] The second backbone network 403 specifically includes: n feature fusion modules, which are respectively recorded as: a first feature fusion module 431, a second feature fusion module 432... and an nth feature fusion module 43n.

[0199] In a specific implementation, the process of performing convolution processing and segmentation processing on the i-th first image feature by the i-th feature segmentation module in the feature segmentation network specifically includes:

[0200] Step E1, the i-th feature segmentation module in the feature segmentation network performs convolution processing on the i-th first image feature to obtain image features with c channels;

[0201] Step E2: according to the channel dimension, the image features with c channels are divided into i seed image features with different channel numbers; the number of channels of the jth seed image feature in the i seed image features is: t*2 (j-1) The value of t may be determined by those skilled in the art according to actual application requirements, and examples of the value of t may include: 32, 64, and the like.

[0202] The number of channels of the i seed image features is t*2 0 , t*2 1 , t*2 2 ..., t*2 (i-1) Among them, the sum of the number of channels corresponding to the i seed image features can be c, that is: t*2 0 +t*2 1 +t*2 2 ……+t*2 (i-1) =c.

[0203] When n is 5, the i-th feature segmentation module can output the j-th seed image feature to the connected j-th feature fusion module. Specific examples include:

[0204] For example, the segmentation processing result of the first feature segmentation module 421 includes: 1 m-channel first sub-image feature, and the 1 m-channel first sub-image feature is output to the first feature fusion module 431 .

[0205] For another example, the segmentation processing result of the second feature segmentation module 422 includes: one m-channel second sub-image feature and one 2m-channel third sub-image feature. One m-channel second sub-image feature is output to the first feature fusion module 431. One 2m-channel third sub-image feature is output to the second feature fusion module 432.

[0206] For another example, the segmentation processing result of the third feature segmentation module 423 (not shown in the figure) includes: one fourth sub-image feature of the m-channel, one fifth sub-image feature of the 2m-channel, and one sixth sub-image feature of the 4m-channel. One fourth sub-image feature of the m-channel is output to the first feature fusion module 431. One fifth sub-image feature of the 2m-channel is output to the second feature fusion module 432. One sixth sub-image feature of the 4m-channel is output to the third feature fusion module 433.

[0207] Alternatively, the segmentation processing result of the fourth feature segmentation module 424 (not shown in the figure) includes: one seventh sub-image feature of the m channel, one eighth sub-image feature of the 2m channel, one ninth sub-image feature of the 4m channel, and one tenth sub-image feature of the 8m channel. One seventh sub-image feature of the m channel is output to the first feature fusion module 431. One eighth sub-image feature of the 2m channel is output to the second feature fusion module 432. One ninth sub-image feature of the 4m channel is output to the third feature fusion module 433. One tenth sub-image feature of the 8m channel is output to the fourth feature fusion module 434.

[0208] Alternatively, the segmentation processing result of the fifth feature segmentation module 425 (not shown in the figure) includes: one kind of eleventh sub-image feature of the m channel, one kind of twelfth sub-image feature of the 2m channel, one kind of thirteenth sub-image feature of the 4m channel, one kind of fourteenth sub-image feature of the 8m channel, and one kind of fifteenth sub-image feature of the 16m channel. One kind of the eleventh sub-image feature of the m channel is output to the first feature fusion module 431. One kind of the twelfth sub-image feature of the 2m channel is output to the second feature fusion module 432. One kind of the thirteenth sub-image feature of the 4m channel is output to the third feature fusion module 433. One kind of the fourteenth sub-image feature of the 8m channel is output to the fourth feature fusion module 434. One kind of the fifteenth sub-image feature of the 16m channel is output to the fifth feature fusion module 435.

[0209] The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result. The channel-to-pixel module in the second backbone network determines the second image feature based on the first fusion result and outputs the second image feature to the neck network.

[0210] In an optional implementation of the present application, the second backbone network may further include: a first convolution module connected in front of the feature fusion module;

[0211] Then the process of the i-th feature fusion module in the second backbone network performing the first fusion processing on at least one sub-image feature specifically includes:

[0212] Step F1, the i-th feature fusion module receives the n-i+1 seed image features and the convolution image features output by the first convolution module; the number of channels corresponding to the n-i+1 seed image features is the first number of channels, and the number of channels corresponding to the convolution image features is the second number of channels;

[0213] Step F2: the i-th feature fusion module performs interpolation processing on the n-i+1 seed image features, and the number of channels corresponding to the n-i+1 seed image features after the interpolation processing is the second number of channels;

[0214] Step F3: The i-th feature fusion module fuses the n-i+1 seed image features and the convolution image features after the interpolation processing.

[0215] The first convolution module, the second convolution module, or the first convolution processing module of the embodiment of the present application all belong to the category of convolution structure. In one example, the above-mentioned convolution structure specifically includes: at least one convolution layer, at least one batch normalization layer, and at least one activation function. It can be understood that those skilled in the art can adopt the required convolution structure according to the actual application requirements, and the embodiment of the present application does not limit the specific convolution structure.

[0216] For example, the number of first channels corresponding to the n seed image features received by the first feature fusion module is t*2 0 Those skilled in the art may adopt corresponding interpolation processing technology according to actual application requirements, and the embodiments of the present application do not limit the specific interpolation processing technology.

[0217] The process of fusing the n-i+1 seed image features and the convolution image features after the interpolation processing by the i-th feature fusion module specifically includes: the i-th feature fusion module performs addition processing on the values ​​corresponding to the n-i+1 seed image features and the convolution image features after the interpolation processing. The sub-image features and the convolution image features may correspond to a multi-dimensional matrix, and the above-mentioned addition processing may be to add the element values ​​of the multi-dimensional matrix.

[0218] In another optional implementation of the present application, a first convolution module connected in front of the first feature fusion module is used to perform convolution processing on the video stream image and output the corresponding convolution image features to the first feature fusion module.

[0219] At least some of the feature fusion modules are connected to the channel-to-pixel module. Figure 6 , shows a schematic diagram of the structure of a channel-to-pixel module of an embodiment of the present application, the channel-to-pixel module specifically includes: a first convolution processing module 601, a segmentation processing module 602, M bottleneck modules 603, a connection module 604, and a second convolution processing module 605. M can be a positive integer greater than 1.

[0220] Among them, the first convolution processing module 601 is used to perform a first convolution processing on the first fusion result output by the feature fusion module to obtain a first convolution processing result.

[0221] The segmentation processing module 602 is used to segment the first convolution processing result into two parts with the same number of channels. The two parts may include: a first part of features and a second part of features. Assuming that the number of channels before segmentation is the third number of channels, the number of channels after segmentation may be the fourth number of channels, and the third number of channels may be twice the fourth number of channels.

[0222] After the first part of the features are processed by the M bottleneck modules 603, the obtained bottleneck processing features enter the connection module 604. The first convolution processing result and the second part of the features also enter the connection module 604. The connection module 604 is used to connect the first convolution processing result, the second part of the features and the bottleneck processing features to obtain the connection features.

[0223] The connection feature enters the second convolution processing module 605, and the second convolution processing module 605 restores the number of channels for the connection feature, for example, restores the number of channels from the fourth number of channels to the third number of channels.

[0224] Reference Figure 7 , shows a schematic structural diagram of a bottleneck module of an embodiment of the present application, and the bottleneck module specifically includes: a third convolution processing module 701 and a fourth convolution processing module 702.

[0225] The third convolution processing module 701 is used to reduce the number of channels of the input feature to half of the original number to obtain a second convolution processing result.

[0226] The fourth convolution processing module 702 is used to increase the number of channels of the second convolution processing result to twice the original number to obtain a third convolution processing result. The third convolution processing result has the same number of channels as the input feature. The third convolution processing result is fused with the input feature to obtain an output feature.

[0227] Reference Figure 8 , shows a schematic diagram of the structure of the second backbone network of an embodiment of the present application, which specifically includes: a first convolution module A801, a first feature fusion module 802, a first convolution module B803, a second feature fusion module 804, a first channel to pixel module 805, a first convolution module C806, a third feature fusion module 807, a second channel to pixel module 808, a first convolution module D809, a fourth feature fusion module 810, a third channel to pixel module 811, a first convolution module E812, a fifth feature fusion module 813 and a fourth channel to pixel module 814.

[0228] Among them, the first convolution module A801 ​​is used to perform convolution processing on the video stream image and output the corresponding convolution image feature A to the first feature fusion module.

[0229] The first feature fusion module 802 is used to receive the five seed image features A from the first feature segmentation module and the convolution image feature A from the first convolution module A801, perform interpolation processing on the five seed image features A, and the number of channels corresponding to the five seed image features A after the interpolation processing is the second number of channels; and, perform fusion processing on the five seed image features A after the interpolation processing and the convolution image feature A, and assume that the first fusion result A is obtained.

[0230] The first convolution module B803 is used to perform convolution processing on the first fusion result A to obtain a convolution image feature B.

[0231] The second feature fusion module 804 is used to receive the four seed image features B from the second feature segmentation module and the convolution image feature B from the first convolution module B803, perform interpolation processing on the four seed image features B, and the number of channels corresponding to the four seed image features B after the interpolation processing is the second number of channels; and, perform fusion processing on the four seed image features B after the interpolation processing and the convolution image feature B, and assume that the first fusion result B is obtained.

[0232] The first channel to pixel module 805 is used to determine the intermediate image features according to the first fusion result B mentioned above.

[0233] The first convolution module C806 is used to perform convolution processing on the intermediate image features to obtain the convolution image features C.

[0234] The third feature fusion module 807 is used to receive the three seed image features C from the third feature segmentation module and the convolution image feature C from the first convolution module C806, perform interpolation processing on the three seed image features C, and the number of channels corresponding to the three seed image features C after the interpolation processing is the second number of channels; and, perform fusion processing on the three seed image features C after the interpolation processing and the convolution image feature C, and assume that the first fusion result C is obtained.

[0235] The second channel to pixel module 808 is used to determine the second image feature A according to the first fusion result C, and output the second image feature A to the neck network.

[0236] The first convolution module D809 is used to perform convolution processing on the second image feature A to obtain a convolution image feature D.

[0237] The fourth feature fusion module 810 is used to receive the 2-seed image features D from the fourth feature segmentation module and the convolution image features D from the first convolution module D809, perform interpolation processing on the 3-seed image features D, and the number of channels corresponding to the 2-seed image features D after the interpolation processing is the second number of channels; and, perform fusion processing on the 2-seed image features D after the interpolation processing and the convolution image features D, and assume that the first fusion result D is obtained.

[0238] The third channel to pixel module 811 is used to determine the second image feature B according to the first fusion result D, and output the second image feature B to the neck network.

[0239] The first convolution module E812 is used to perform convolution processing on the second image feature B to obtain a convolution image feature E.

[0240] The fifth feature fusion module 813 is used to receive the 1 seed image feature E from the fifth feature segmentation module and the convolution image feature E from the first convolution module E812, perform interpolation processing on the 1 seed image feature E, and the number of channels corresponding to the 1 seed image feature E after the interpolation processing is the second number of channels; and, perform fusion processing on the 1 seed image feature E after the interpolation processing and the convolution image feature E, and assume that the first fusion result E is obtained.

[0241] The fourth channel-to-pixel module 814 is used to determine the second image feature C according to the first fusion result E, and output the second image feature C to the neck network.

[0242] In a specific implementation, the first backbone network may specifically include: a second convolution module and n-1 processing units connected in sequence; the processing unit specifically includes: a third convolution module and a channel to pixel module;

[0243] Among them, the second convolution module is connected to the first feature segmentation module, and is used to output the first first image feature to the first feature segmentation module; the channel-to-pixel modules contained in the n-1 processing units are respectively connected to the corresponding n-1 feature segmentation modules, and are used to output the first image feature to the corresponding feature segmentation module.

[0244] Reference Fig. 9 , shows a schematic diagram of the structure of the first backbone network of an embodiment of the present application, wherein the first backbone network specifically includes: a second convolution module 901, a third convolution module A902, a fifth channel to pixel module 903, a third convolution module B904, a sixth channel to pixel module 905, a third convolution module C906, a seventh channel to pixel module 907, a third convolution module D908 and an eighth channel to pixel module 909 connected in sequence. The structure of the fifth channel to pixel module 903 can be found in Figure 6 , I will not go into details here.

[0245] The second convolution module 901 is used to perform convolution processing on the video stream image to obtain the first image feature A, and send the first image feature A to the first feature segmentation module.

[0246] The third convolution module A902 is used to perform convolution processing on the first image feature A to obtain a first convolution result.

[0247] The fifth channel to pixel module 903 is used to determine the first image feature B according to the first convolution result, and send the first image feature B to the second feature segmentation module.

[0248] The third convolution module B904 is used to perform convolution processing on the first image feature B to obtain a second convolution result.

[0249] The sixth channel to pixel module 905 is used to determine the first image feature C according to the second convolution result, and send the first image feature C to the third feature segmentation module.

[0250] The third convolution module C906 is used to perform convolution processing on the first image feature C to obtain a third convolution result.

[0251] The seventh channel-to-pixel module 907 is used to determine the first image feature D according to the third convolution result, and send the first image feature D to the fourth feature segmentation module.

[0252] The third convolution module D908 is used to perform convolution processing on the first image feature D to obtain a fourth convolution result.

[0253] The seventh channel-to-pixel module 907 is used to determine the first image feature E according to the fourth convolution result, and send the first image feature E to the fifth feature segmentation module.

[0254] The neck network is used to perform a second fusion process on the second image feature to obtain a second fusion process result.

[0255] Reference Fig.10 , shows a schematic diagram of the structure of a neck network of an embodiment of the present application, which specifically includes: a spatial pyramid pooling module 1001, a first upsampling module 1002, a first connection module 1003, a ninth channel to pixel module 1004, a second upsampling module 1005, a second connection module 1006, a tenth channel to pixel module 1007, a fourth convolution module 1008, a third connection module 1009, an eleventh channel to pixel module 1010, a fifth convolution module 1011, a fourth connection module 1012 and a twelfth channel to pixel module 1013.

[0256] The spatial pyramid pooling module 1001 is used to receive the second image feature C and perform spatial pyramid pooling processing on the second image feature C. The spatial pyramid pooling processing may include: a convolution operation and a maximum pooling operation, which can achieve deep fusion of the second image feature C.

[0257] Reference Fig.11, shows a schematic diagram of the structure of a spatial pyramid pooling module of an embodiment of the present application, and the spatial pyramid pooling module specifically includes: a sixth convolution module 1101, p maximum pooling modules 1102, a fifth connection module 1103 and a seventh convolution module 1104. p can be a positive integer greater than 1, and the value of p in the figure is 3.

[0258] The sixth convolution module 1101 is used to perform convolution processing on the input second image feature C to obtain a fifth convolution result.

[0259] The p maximum pooling modules 1102 are respectively used to perform maximum pooling processing on the input image features to obtain corresponding p maximum pooling processing results.

[0260] The fifth convolution result and the p maximum pooling processing results are respectively input into the fifth connection module 1103, and the fifth connection module 1103 performs deep fusion on the fifth convolution result and the p maximum pooling processing results to obtain a corresponding deep fusion result.

[0261] The seventh convolution module 1104 is used to perform convolution processing on the depth fusion result to obtain a sixth convolution result. The sixth convolution result is provided to the first upsampling module 1002.

[0262] The first up-sampling module 1002 is used to perform a first up-sampling process on the sixth convolution result to obtain a first up-sampling process result.

[0263] The first connection module 1003 is used to perform connection processing on the second image feature B and the first upsampling processing result to obtain a first connection result.

[0264] The ninth channel-to-pixel module 1004 is configured to determine a first processing result according to the first connection result.

[0265] The second up-sampling module 1005 is used to perform a second up-sampling process on the first processing result to obtain a second up-sampling processing result.

[0266] The second connection module 1006 is used to connect the second upsampling result with the second image feature A to obtain a second connection result. The connection process may be a concatenation process of two image features.

[0267] The tenth channel-to-pixel module 1007 is used to determine a second fusion processing result A according to the second connection result. The second fusion processing result A is output to the detection head network.

[0268] The fourth convolution module 1008 is used to perform convolution processing on the second fusion processing result A to obtain a sixth convolution result.

[0269] The third connection module 1009 is used to connect the sixth convolution result and the first processing result to obtain a third connection result.

[0270] The eleventh channel-to-pixel module 1010 is used to determine a second fusion processing result B according to the third connection result. The second fusion processing result B is output to the detection head network.

[0271] The fifth convolution module 1011 is used to perform convolution processing on the second fusion result B to obtain a seventh convolution result.

[0272] The fourth connection module 1012 is used to connect the sixth convolution result and the seventh convolution result to obtain a fourth connection result.

[0273] The twelfth channel-to-pixel module 1013 is used to determine a second fusion processing result C according to the fourth connection result. The second fusion processing result C is output to the detection head network.

[0274] The detection head network can determine the recognition result based on the above-mentioned second fusion processing result.

[0275] The detection head network may include: at least one detection module. The detection module may be used to perform classification and regression calculations respectively according to the second fusion processing result using the convolution module and the convolution layer to obtain target information such as the category and position of the target in the video stream image.

[0276] In summary, the n feature segmentation modules of the feature segmentation network of the embodiment of the present application can play the role of aggregating sub-image features of multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features of multiple levels. In this way, the embodiment of the present application can enhance the expression ability and accuracy of the second image features output by the second backbone network, thereby improving the target detection accuracy of the video stream image, and on this basis can improve the accuracy of real-time monitoring of excavation volume.

[0277] The dual-trunk target detection model of the embodiment of the present application was experimentally analyzed using the construction scene target detection benchmark dataset. Among them, the 19404 training set images that have been annotated in the construction scene target detection benchmark dataset are used as the training dataset, and the 4000 validation set images are used as the test dataset, and AP (average precision, Average Precision) 50 and AP75 are defined as evaluation indicators. Among them, AP50 represents the average precision when the intersection-over-union ratio threshold is 0.5. AP75 represents the average precision when the intersection-over-union ratio threshold is 0.75.

[0278] The experimental results show that the detection accuracy of the embodiment of the present application on the construction scene target detection benchmark dataset is better than that of the prior art. Taking the AP50 index as an example, compared with the prior art, the detection accuracy of the first version, second version and third version of the dual-trunk target detection model has increased by 13.6%, 8.7% and 4.3% respectively, verifying the effectiveness of the dual-trunk target detection model proposed in this application. Among them, the first version, second version and third version of the dual-trunk target detection model correspond to different parameter quantities.

[0279] In an embodiment of the present application, the training process of the dual-trunk target detection model may include: forward propagation and back propagation.

[0280] Among them, forward propagation can calculate the prediction information of the target detection result in sequence from the first backbone network to the detection head network according to the parameters of the dual-backbone target detection model. The prediction information is used to determine the loss information

[0281] Backward Propagation can calculate and update the parameters of the dual-backbone target detection model in sequence from the output layer to the input layer according to the loss information. The dual-backbone target detection model usually adopts the structure of a neural network, and the parameters of the dual-backbone target detection model may include: parameters such as the weights of the neural network. Among them, during the backpropagation process, the gradient information of the parameters of the dual-backbone target detection model can be determined, and the parameters of the dual-backbone target detection model can be updated using the gradient information. For example, backpropagation can calculate and store the gradient information of the parameters of the target detection model in sequence from the detection head network to the neck network to the second backbone network to the feature segmentation network and then to the first backbone network according to the chain rule in calculus.

[0282] In one embodiment, the training process of the dual-backbone object detection model includes:

[0283] Step G1, inputting the video stream image sample into the dual-trunk target detection model, and the dual-trunk target detection model outputs the prediction result corresponding to the video stream image sample; the prediction result includes: the prediction frame information corresponding to the construction object category; the real frame information corresponding to the video stream image sample;

[0284] Step G2: determining loss information corresponding to the real frame information and the predicted frame information according to a minimum point distance function based on a horizontal rectangle;

[0285] Step G3 updates the parameters of the dual-trunk target detection model according to the loss information.

[0286] The process of obtaining the video stream image samples specifically includes:

[0287] Step H1, randomly reading four original video stream images from the video stream image set;

[0288] Step H2, performing random enhancement operations on the four original video stream images respectively to obtain four enhanced video stream images;

[0289] Step H3, fusing the four enhanced video stream images into a fused image;

[0290] Step H4: annotate a pair of fused images with real frame information to obtain a video stream image sample.

[0291] The random enhancement operation specifically includes at least one of the following operations: a flip operation, a random scaling operation, a random tone transformation operation, and a random perspective transformation operation.

[0292] Step H3 is a process of fusing four enhanced video stream images into a fused image, specifically including: placing the four enhanced video stream images on an intermediate image according to a preset offset; cutting off the portion of the intermediate image that exceeds the size range, and shrinking the range of the real frame to prevent crossing the boundary, and finally obtaining a fused image. The size range specifically includes: the coordinate range corresponding to [0,0] to [K,K]. K can be a positive integer.

[0293] Of course, the above process of obtaining video stream image samples based on the random enhancement operation of four images is only an example. In fact, those skilled in the art can obtain video stream image samples in other ways according to actual application requirements.

[0294] For example, in another example of the present application, the process of obtaining a video stream image sample specifically includes: randomly reading an original video stream image from a video stream image set; and marking real frame information on an original video stream image to obtain a video stream image sample.

[0295] For example, in another example of the present application, the process of obtaining a video stream image sample specifically includes: randomly reading an original video stream image from a video stream image set; performing a random enhancement operation on an original video stream image; and annotating real frame information for the enhanced image to obtain a video stream image sample.

[0296] Referring to formula (4), a process of determining the loss information corresponding to the real box information and the predicted box information according to the minimum point distance function based on the horizontal rectangle is shown.

[0297]

[0298] Among them, L MPDIoURepresents loss information; A′ represents the real box; B′ represents the predicted box; Indicates the pixel coordinates of the upper left corner of the real box; Indicates the pixel coordinates of the lower right corner of the real box; Indicates the pixel coordinate of the upper left corner of the prediction box, represents the pixel coordinates of the lower right corner of the prediction box; w and h represent the width and height of the video stream image sample respectively; d1 represents the first distance between the pixel coordinates of the upper left corner of the real box and the prediction box; d2 represents the second distance between the pixel coordinates of the lower right corner of the real box and the prediction box.

[0299] The minimum point distance function based on the horizontal rectangle helps ensure that the predicted box is geometrically closer to the true box, especially when the predicted box has the same aspect ratio as the true box but different width and height values. The metric optimization is performed by calculating the distance between the predicted box and the upper left corner and the lower right corner of the true box. Since the minimum point distance function is more sensitive to the position error of the predicted box, it may help improve the detection accuracy of the dual-backbone object detection model.

[0300] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0301] Based on the above embodiment, this embodiment also provides a real-time monitoring device for excavation volume based on dual trunk target detection. Fig.12 The device may specifically include: a collection module 1201, a target detection module 1202, a target area determination module 1203, an excavation operation status determination module 1204, a target area excavation volume determination module 1205 and a construction area excavation volume determination module 1206 to be counted.

[0302] Wherein, the acquisition module 1201 is used to acquire video stream images of the construction area to be counted;

[0303] The target detection module 1202 is used to perform target detection on the video stream image using a dual-trunk target detection model to obtain first target information corresponding to the mining equipment contained in the video stream image and second target information corresponding to the dump equipment contained in the video stream image; wherein the dual-trunk target detection model includes: a first trunk network, a feature segmentation network, a second trunk network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second trunk network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1;

[0304] The target region determination module 1203 is used to determine N target regions in the video stream image according to the first target information and the second target information; N is a positive integer;

[0305] The excavation operation state determination module 1204 is used to determine the excavation operation state of the current target area among the N target areas according to the operation state of the excavation equipment in the current target area, whether the current target area includes a self-unloading device, and the positional relationship between the excavation equipment and the self-unloading device in the current target area; the excavation operation state includes: one of: an unexcavated state, an excavating state, and a loading state;

[0306] The target area excavation amount determination module 1205 is used to determine the excavation amount corresponding to the current loading of the current target area when the excavation operation state of the current target area is updated from the loading state to the unexcavated state or the excavation state, and to determine the excavation amount corresponding to the current loading of the current target area according to the rated loading capacity corresponding to the self-unloading equipment in the current target area, and to update the cumulative excavation amount of the current target area;

[0307] The excavation volume determination module 1206 for the construction area to be counted is used to determine the total real-time excavation volume corresponding to the construction area to be counted according to the cumulative excavation volume of all current target areas among the N target areas.

[0308] Optionally, the first target information specifically includes: a first bounding box, and the second target information specifically includes: a second bounding box;

[0309] The target area determination module comprises:

[0310] A target region number determination module, configured to determine the number N of target regions according to the number N of first bounding boxes;

[0311] A bounding box matching module, used to perform one-to-one matching between the first bounding box and the second bounding box to obtain a corresponding matching result;

[0312] A first determination module is configured to, if there is a second bounding box matching the first bounding box, use a minimum enclosing rectangular area containing both the first bounding box and the second bounding box as a target area;

[0313] The second determining module is configured to use a rectangular area corresponding to the first bounding box as a target area if there is no second bounding box matching the first bounding box.

[0314] Optionally, the bounding box matching module includes:

[0315] A distance intersection-over-union ratio index value determination module, used to determine a distance intersection-over-union ratio index value between a first bounding box and a second bounding box;

[0316] The matching determination module is used to select a second bounding box with the largest distance intersection and union ratio index value from the multiple second bounding boxes according to the distance intersection and union ratio index values ​​between the first bounding box and the multiple second bounding boxes as the second bounding box matching the first bounding box.

[0317] Optionally, the excavation operation status determination module includes:

[0318] A first judgment module is used to judge whether the excavation equipment included in the current target area is in a stationary state to obtain a first judgment result;

[0319] A first determination module, configured to determine that the excavation operation state of the current target area is an unexcavated state if the first determination result is yes;

[0320] A second judgment module is used to judge whether the current target area contains a self-unloading device if the first judgment result is no, so as to obtain a second judgment result;

[0321] A second determination module, configured to determine that the excavation operation state of the current target area is an excavation state if the second determination result is no;

[0322] a third judgment module, for judging whether there is overlap between the self-unloading equipment and the excavating equipment in the current target area if the second judgment result is yes, so as to obtain a third judgment result;

[0323] A third determination module, configured to determine that the excavation operation state of the current target area is a loading state if the third determination result is yes;

[0324] The fourth determination module is used to determine that the excavation operation status of the current target area is an excavation status if the third determination result is no.

[0325] Optionally, the third judgment module includes:

[0326] An intersection-and-union ratio index value determination module is used to determine an intersection-and-union ratio index value between a second bounding box corresponding to the self-unloading equipment in the current target area and a first bounding box corresponding to the excavation equipment in the current target area;

[0327] The overlap determination module is used to determine that there is overlap between the dump equipment and the excavation equipment in the current target area if the intersection-to-union ratio index value is greater than a first threshold.

[0328] Optionally, the first judgment module is specifically used for: in the i-th frame video stream image to the i+α-th frame video stream image, if the coordinate difference of the mining equipment in any two frames of video stream image is less than a second threshold, then determining that the mining equipment contained in the current target area is in a stationary state; wherein i and α are positive integers.

[0329] Optionally, the device further comprises:

[0330] A model determination module, used to determine the target model corresponding to the self-unloading equipment in the current target area by using the self-unloading equipment model classification model;

[0331] The search module is used to search the mapping relationship between the search model and the rated loading capacity according to the target model, so as to determine the rated loading capacity corresponding to the self-unloading equipment in the current target area.

[0332] In summary, the real-time monitoring device for excavation volume based on dual-trunk target detection of the embodiment of the present application determines N target areas in the video stream image according to the target information corresponding to the targets such as the excavation equipment and the self-unloading equipment in the video stream image, and determines the excavation volume corresponding to the current loading of the current target area according to the objective law between the excavation operation state of the target area and the operation state of the excavation equipment, whether there is a self-unloading equipment, and the position relationship between the excavation equipment and the self-unloading equipment; in this way, the embodiment of the present application can continuously determine the target area and the operation state corresponding to the target area in the video stream image according to the continuously updated video stream image; on this basis, the embodiment of the present application can continuously determine the excavation volume corresponding to the current loading of the current target area in the video stream image, and the cumulative excavation volume of all current target areas in the video stream image. Therefore, the embodiment of the present application can realize the real-time monitoring of the earthwork excavation volume at a lower cost and faster speed based on the analysis of the video stream image. In other words, the embodiment of the present application can save the labor cost, data collection cost, and data processing time cost consumed by using drone oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation volume.

[0333] In addition, the embodiment of the present application utilizes a dual-trunk target detection model to perform target detection on video image frames. Among them, the n feature segmentation modules contained in the feature segmentation network in the above dual-trunk target detection model can play a role in aggregating sub-image features of multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features of multiple levels. In this way, the embodiment of the present application can enhance the expression ability of the second image features output by the second backbone network and the accuracy of the second image features, thereby improving the accuracy of the target detection results.

[0334] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0335] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0336] The embodiments of the present disclosure may be implemented as a device that uses any appropriate hardware, firmware, software, or any combination thereof to perform a desired configuration, and the device may include: electronic devices such as terminal devices and servers (clusters). Fig.13 An exemplary apparatus 1300 that can be used to implement various embodiments described in this application is schematically shown.

[0337] For one embodiment, Fig.13 An exemplary apparatus 1300 is shown having one or more processors 1302, a control module (chip set) 1304 coupled to at least one of the (one or more) processors 1302, a memory 1306 coupled to the control module 1304, an NVM (non-volatile memory) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0338] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1300 can be used as a terminal device, server (cluster), and other devices described in the embodiments of the present application.

[0339] In some embodiments, the device 1300 may include one or more computer-readable media (e.g., memory 1306 or non-volatile memory / storage device 1308) having instructions 1314 and one or more processors 1302 configured to execute the instructions 1314 in combination with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.

[0340] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304 .

[0341] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0342] The memory 1306 may be used, for example, to load and store data and / or instructions 1314 for the device 1300. For one embodiment, the memory 1306 may include any suitable volatile memory, such as a suitable DRAM (Dynamic Random Access Memory). In some embodiments, the memory 1306 may include a double data rate type four synchronous dynamic random access memory.

[0343] For one embodiment, the control module 1304 may include one or more input / output controllers to provide an interface to the non-volatile memory / storage device 1308 and the input / output device(s) 1310 .

[0344] For example, non-volatile memory / storage 1308 may be used to store data and / or instructions 1314. Non-volatile memory / storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile optical disk drives).

[0345] The non-volatile memory / storage device 1308 may include storage resources that are physically part of the device on which the apparatus 1300 is installed, or it may be accessible to the device without being part of the device. For example, the non-volatile memory / storage device 1308 may be accessed via (one or more) input / output devices 1310 over a network.

[0346] (One or more) input / output devices 1310 may provide an interface for the apparatus 1300 to communicate with any other appropriate device, and the input / output device 1310 may include a communication component, an audio component, a sensor component, etc. The network interface 1312 may provide an interface for the apparatus 1300 to communicate through one or more networks, and the apparatus 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2nd Generation Wireless Telephone Technology), 3G (3rd Generation Wireless Telephone Technology), 4G (4th Generation Wireless Telephone Technology), 5G (5th Generation Wireless Telephone Technology), etc., or a combination thereof for wireless communication.

[0347] For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 to form a system-in-package. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304 to form a system-on-chip.

[0348] In various embodiments, the device 1300 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touch screen device, a netbook, etc.). In various embodiments, the device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1300 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, a dedicated integrated circuit, and a speaker.

[0349] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or non-volatile memory / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0350] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0351] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0352] The present application embodiment is described with reference to the flowchart and / or block diagram of the method, terminal device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0353] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0354] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0355] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0356] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0357] The above is a detailed introduction to a real-time monitoring method and device for excavation volume based on dual-trunk target detection, an electronic device and a machine-readable medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application sets. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A real-time monitoring method for excavation volume based on dual-trunk target detection, characterized in that: The method comprises: Collect video stream images of the construction area to be counted; The dual-trunk target detection model is used to perform target detection on the video stream image to obtain first target information corresponding to the mining equipment contained in the video stream image and second target information corresponding to the dump equipment contained in the video stream image; wherein the dual-trunk target detection model includes: a first trunk network, a feature segmentation network, a second trunk network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second trunk network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1; Determine N target areas in the video stream image according to the first target information and the second target information; N is a positive integer; For a current target area among the N target areas, the excavation operation state of the current target area is determined according to the operation state of the excavation equipment in the current target area, whether the current target area includes a self-unloading equipment, and the positional relationship between the excavation equipment and the self-unloading equipment in the current target area; the excavation operation state includes: one of: an unexcavated state, an excavating state, and a loading state; When the excavation operation state of the current target area is updated from the loading state to the unexcavated state or the excavation state, the current target area is considered to have completed the current loading, and the excavation volume corresponding to the current loading of the current target area is determined according to the rated loading capacity corresponding to the self-unloading equipment in the current target area, and the cumulative excavation volume of the current target area is updated; According to the accumulated excavation volumes of all current target areas among the N target areas, the total real-time excavation volume corresponding to the construction area to be counted is determined.

2. The method according to claim 1, characterized in that: The first target information includes: a first bounding box, and the second target information includes: a second bounding box; The determining, according to the first target information and the second target information, N target areas in the video stream image includes: Determine the number N of target regions according to the number N of first bounding boxes; Performing one-to-one matching on the first bounding box and the second bounding box to obtain corresponding matching results; If there is a second bounding box that matches the first bounding box, the minimum enclosing rectangular area that contains both the first bounding box and the second bounding box is used as the target area; If there is no second bounding box matching the first bounding box, the rectangular area corresponding to the first bounding box is used as the target area.

3. The method according to claim 2, characterized in that The one-to-one matching of the first bounding box and the second bounding box includes: Determine a distance intersection-over-union ratio index value between a first bounding box and a second bounding box; According to the distance intersection-and-union index values ​​between a first bounding box and multiple second bounding boxes, one with the largest distance intersection-and-union index value is selected from the multiple second bounding boxes as the second bounding box matching the first bounding box.

4. The method according to claim 1, characterized in that: The method of determining the excavation operation state of the current target area among the N target areas according to the operation state of the excavation equipment in the current target area, whether the current target area includes a self-unloading device, and the positional relationship between the excavation equipment and the self-unloading device in the current target area includes: Determining whether the excavation equipment included in the current target area is in a stationary state to obtain a first determination result; If the first judgment result is yes, it is determined that the excavation operation state of the current target area is an unexcavated state; If the first judgment result is no, then judging whether the current target area includes the self-unloading equipment to obtain a second judgment result; If the second judgment result is no, then the excavation operation state of the current target area is determined to be in the excavation state; If the second judgment result is yes, then judging whether there is overlap between the self-unloading equipment and the excavating equipment in the current target area to obtain a third judgment result; If the third judgment result is yes, it is determined that the excavation operation state of the current target area is in a loading state; If the third judgment result is no, then it is determined that the excavation operation status of the current target area is in the excavation state.

5. The method according to claim 4, characterized in that The determining whether there is overlap between the dump equipment and the excavation equipment in the current target area includes: Determine an intersection-and-union ratio index value between a second bounding box corresponding to the dump equipment in the current target area and a first bounding box corresponding to the mining equipment in the current target area; If the intersection-over-union ratio index value is greater than the first threshold, it is determined that there is an overlap between the dump equipment and the excavation equipment in the current target area.

6. The method according to claim 4, characterized in that The determining whether the excavation equipment included in the current target area is in a stationary state includes: From the i-th frame of video stream image to the i+α-th frame of video stream image, if the coordinate difference of the mining equipment in any two frames of video stream image is less than the second threshold, it is determined that the mining equipment contained in the current target area is in a stationary state; wherein i and α are positive integers.

7. The method according to claim 1, characterized in that The method further comprises: Using the dump equipment model classification model, determine the target model corresponding to the dump equipment in the current target area; According to the target model, a search is performed in a mapping relationship between the search model and the rated loading capacity to determine the rated loading capacity corresponding to the self-unloading equipment in the current target area.

8. The method according to any one of claims 1 to 7, characterized in that: The using of the dual-trunk target detection model to perform target detection on the video stream image includes: The first backbone network determines n first image features corresponding to the video stream image; The i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, and the obtained segmentation processing result includes: i seed image features, and outputs the j-th seed image features to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i seed image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers; The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result, and the channel-to-pixel module in the second backbone network determines a second image feature according to the first fusion result, and outputs the second image feature to the neck network; The neck network performs a second fusion process on the second image feature to obtain a second fusion process result; The detection head network determines the first target information and the second target information according to the second fusion processing result.

9. The method according to claim 8, characterized in that The i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, including: The i-th feature segmentation module in the feature segmentation network performs convolution processing on the i-th first image feature to obtain image features with c channels; According to the channel dimension, the image features with c channels are divided into i seed image features with different channel numbers; the number of channels of the jth seed image feature in the i seed image features is: t*2 (j-1) .

10. The method according to claim 8, characterized in that The second backbone network further includes: a first convolution module connected in front of the feature fusion module; The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature, including: The i-th feature fusion module receives the n-i+1 seed image features and the convolution image features output by the first convolution module; the number of channels corresponding to the n-i+1 seed image features is the first number of channels, and the number of channels corresponding to the convolution image features is the second number of channels; The i-th feature fusion module performs interpolation processing on the n-i+1 seed image features, and the number of channels corresponding to the n-i+1 seed image features after the interpolation processing is the second channel number; The i-th feature fusion module fuses the n-i+1 seed image features and convolution image features after interpolation.

11. The method according to claim 10, characterized in that The first convolution module connected in front of the first feature fusion module is used to perform convolution processing on the video stream image and output the corresponding convolution image features to the first feature fusion module.

12. The method according to claim 8, characterized in that The n feature segmentation modules include: a first feature segmentation module, a second feature segmentation module, a third feature segmentation module, a fourth feature segmentation module and a fifth feature segmentation module; the n feature fusion modules include: a first feature fusion module, a second feature fusion module, a third feature fusion module, a fourth feature fusion module and a fifth feature fusion module; The segmentation processing result of the first feature segmentation module includes: 1 m-channel first sub-image feature, and 1 m-channel first sub-image feature is output to the first feature fusion module; The segmentation processing result of the second feature segmentation module includes: one m-channel second sub-image feature and one 2m-channel third sub-image feature; one m-channel second sub-image feature is output to the first feature fusion module; one 2m-channel third sub-image feature is output to the second feature fusion module; The segmentation processing result of the third feature segmentation module includes: one fourth sub-image feature of the m channel, one fifth sub-image feature of the 2m channel and one sixth sub-image feature of the 4m channel; one fourth sub-image feature of the m channel is output to the first feature fusion module; one fifth sub-image feature of the 2m channel is output to the second feature fusion module; one sixth sub-image feature of the 4m channel is output to the third feature fusion module; The segmentation processing result of the fourth feature segmentation module includes: one seventh sub-image feature of the m channel, one eighth sub-image feature of the 2m channel, one ninth sub-image feature of the 4m channel, and one tenth sub-image feature of the 8m channel; one seventh sub-image feature of the m channel is output to the first feature fusion module; one eighth sub-image feature of the 2m channel is output to the second feature fusion module; one ninth sub-image feature of the 4m channel is output to the third feature fusion module; and one tenth sub-image feature of the 8m channel is output to the fourth feature fusion module; The segmentation processing results of the fifth feature segmentation module include: one eleventh sub-image feature of the m channel, one twelfth sub-image feature of the 2m channel, one thirteenth sub-image feature of the 4m channel, one fourteenth sub-image feature of the 8m channel and one fifteenth sub-image feature of the 16m channel; one eleventh sub-image feature of the m channel is output to the first feature fusion module; one twelfth sub-image feature of the 2m channel is output to the second feature fusion module; one thirteenth sub-image feature of the 4m channel is output to the third feature fusion module; one fourteenth sub-image feature of the 8m channel is output to the fourth feature fusion module; and one fifteenth sub-image feature of the 16m channel is output to the fifth feature fusion module.

13. A real-time monitoring device for excavation volume based on dual-trunk target detection, characterized in that: The device comprises: An acquisition module, used to acquire video stream images of the construction area to be counted; The target detection module is used to perform target detection on the video stream image using a dual-trunk target detection model to obtain first target information corresponding to the mining equipment contained in the video stream image and second target information corresponding to the dump equipment contained in the video stream image; wherein the dual-trunk target detection model includes: a first trunk network, a feature segmentation network, a second trunk network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second trunk network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1; A target region determination module, configured to determine N target regions in the video stream image according to the first target information and the second target information; N is a positive integer; An excavation operation state determination module is used to determine the excavation operation state of a current target area among the N target areas according to the operation state of the excavation equipment in the current target area, whether the current target area contains a self-unloading device, and the positional relationship between the excavation equipment and the self-unloading device in the current target area; the excavation operation state includes: one of: an unexcavated state, an excavating state, and a loading state; The target area excavation volume determination module is used to determine the excavation volume corresponding to the current loading of the current target area when the excavation operation state of the current target area is updated from the loading state to the unexcavated state or the excavation state, and to determine the excavation volume corresponding to the current loading of the current target area according to the rated loading capacity corresponding to the self-unloading equipment in the current target area, and to update the cumulative excavation volume of the current target area; The module for determining the excavation volume of the construction area to be counted is used to determine the total real-time excavation volume corresponding to the construction area to be counted according to the cumulative excavation volume of all current target areas among the N target areas.

14. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when the executable codes are executed, enable the processor to perform the method according to any one of claims 1 to 12.

15. A machine-readable medium having executable codes stored thereon, which, when executed, causes a processor to perform the method according to any one of claims 1 to 12.

Citation Information

Patent Citations

  • Transport vehicle operation recording method and device, computer equipment and storage medium

    CN111126320A

  • Solid waste sorting method and system based on RGB-MSI feature fusion

    CN117218446A

  • Concrete dam face operation scene intelligent identification method

    CN117351410A

  • Excavation and transportation matching metering method and system based on high-precision positioning and machine vision

    CN117557989A

  • Transport vehicle scheduling method and system based on loading area identification

    CN118171889A

Cited By

  • A whole-process measurement method for excavation volume during the construction period based on multi-source data fusion

    CN119741361B