Real-time Monitoring Method, Device and Product for Excavation Volume Based on Dual-backbone Object Detection

Abstract

The embodiments of the present application provide a real-time monitoring method, device and product for excavation volume based on dual-backbone object detection. The method specifically includes: using a dual-backbone object detection model to perform object detection on the video stream images of the construction area to be counted, and obtaining the object information of the excavation equipment and the dump truck equipment; determining N target areas in the video stream images; determining the excavation operation status of the current target area; when the excavation operation status of the current target area is updated from the loading state to the non-excavation state or the excavation state, determining the excavation volume corresponding to the current loading of the current target area according to the rated loading capacity corresponding to the dump truck equipment in the current target area, and updating the cumulative excavation volume of the current target area; obtaining the total real-time excavation volume of the construction area to be counted according to the cumulative excavation volumes of all the current target areas in the N target areas. The embodiments of the present application can achieve real-time monitoring of the earthwork excavation volume at a low cost and at a relatively fast speed.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of open cut of earthwork and stonework, and particularly to a real-time monitoring method, device and product for excavation volume based on dual-backbone object detection. Background Art

[0002] The open cut operation of earthwork and stonework is a very common type of construction operation. The real-time monitoring of the excavation volume helps to timely follow up the on-site construction progress, assist in formulating the layout of short-term excavation construction plans, and timely dispatch the number of construction machinery entering the site.

[0003] In the related art, for the calculation method of the excavation volume, during the engineering construction stage, usually before and after excavation, the digital elevation model is obtained by using the drone oblique photography method, or the spatial point cloud data is obtained by using the three-dimensional laser scanning technology to realize the calculation of the excavation volume.

[0004] The above on-site measurement methods not only require certain manpower and data acquisition costs, but also have the deficiencies of long data processing time and poor real-time performance. Summary of the Invention

[0005] The embodiments of the present application provide a real-time monitoring method for excavation volume based on dual-backbone object detection, which can realize the real-time monitoring of the earthwork and stonework excavation volume at a lower cost and faster speed based on the analysis of video stream images.

[0006] Correspondingly, the embodiments of the present application also provide a real-time monitoring device for excavation volume based on dual-backbone object detection, an electronic device and a machine-readable medium to ensure the implementation and application of the above method.

[0007] To solve the above problems, the embodiments of the present application disclose a real-time monitoring method for excavation volume based on dual-backbone object detection, including:

[0008] Collect video stream images of the construction area to be counted;

[0009] Using a dual-backbone object detection model, perform object detection on the video stream images to obtain first target information corresponding to the excavation equipment included in the video stream images, and second target information corresponding to the dump trucks included in the video stream images; wherein, the dual-backbone object detection model includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected with a channel-to-pixel module; n is a positive integer greater than 1;

[0010] Determine N target areas in the video stream images according to the first target information and the second target information; N is a positive integer;

[0011] For the current target area among N target areas, determine the excavation operation status of the current target area according to the operation status of the excavation equipment in the current target area, whether the current target area contains a dump truck, and the positional relationship between the excavation equipment and the dump truck in the current target area; the excavation operation status includes one of: an unexcavated status, an excavating status, and a loading status.

[0012] When the excavation operation status of the current target area is updated from the loading status to the unexcavated status or the excavating status, it is considered that the current target area has completed the current loading. According to the rated loading capacity corresponding to the dump truck in the current target area, determine the excavation volume corresponding to the current loading in the current target area, and update the cumulative excavation volume of the current target area.

[0013] Determine the total real-time excavation volume corresponding to the construction area to be statistically analyzed according to the cumulative excavation volumes of all current target areas among the N target areas.

[0014] The embodiment of the present application also discloses a real-time excavation volume monitoring device based on dual-backbone object detection. The device includes:

[0015] An acquisition module for acquiring video stream images of the construction area to be statistically analyzed.

[0016] An object detection module for performing object detection on the video stream images by using a dual-backbone object detection model to obtain first target information corresponding to the excavation equipment included in the video stream images and second target information corresponding to the dump truck included in the video stream images; wherein, the dual-backbone object detection model includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network, and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected with a channel-to-pixel module; n is a positive integer greater than 1.

[0017] A target area determination module for determining N target areas in the video stream images according to the first target information and the second target information; N is a positive integer.

[0018] An excavation operation status determination module for, for the current target area among the N target areas, determining the excavation operation status of the current target area according to the operation status of the excavation equipment in the current target area, whether the current target area contains a dump truck, and the positional relationship between the excavation equipment and the dump truck in the current target area; the operation status includes one of: an unexcavated status, an excavating status, and a loading status.

[0019] The target area excavation volume determination module is used to consider that the current target area has completed the current loading when the excavation operation status of the current target area is updated from the loading status to the unexcavated status or the excavating status, determine the excavation volume corresponding to the current loading of the current target area according to the rated loading capacity of the dump equipment corresponding to the current target area, and update the cumulative excavation volume of the current target area;

[0020] The excavation volume determination module for the construction area to be counted is used to determine the total real-time excavation volume corresponding to the construction area to be counted according to the cumulative excavation volumes of all current target areas among the N target areas.

[0021] Optionally, the first target information includes: a first bounding box, and the second target information includes: a second bounding box;

[0022] The target area determination module includes:

[0023] The target area quantity determination module is used to determine the quantity N of target areas according to the quantity N of the first bounding boxes;

[0024] The bounding box matching module is used to perform one-to-one matching on the first bounding box and the second bounding box to obtain the corresponding matching result;

[0025] The first determination module is used to, if there is a second bounding box that matches the first bounding box, use the smallest closed rectangle area that contains both the first bounding box and the second bounding box as the target area;

[0026] The second determination module is used to, if there is no second bounding box that matches the first bounding box, use the rectangular area corresponding to the first bounding box as the target area.

[0027] Optionally, the bounding box matching module includes:

[0028] The distance intersection-over-union index value determination module is used to determine the distance intersection-over-union index value between a first bounding box and a second bounding box;

[0029] The matching determination module is used to select, according to the distance intersection-over-union index values between a first bounding box and multiple second bounding boxes, the one with the largest distance intersection-over-union index value from the multiple second bounding boxes as the second bounding box that matches the first bounding box.

[0030] Optionally, the excavation operation status determination module includes:

[0031] The first judgment module is used to judge whether the excavation equipment included in the current target area is in a stationary state to obtain a first judgment result;

[0032] The first determination module is used to determine that the excavation operation status of the current target area is the non-excavated status if the first judgment result is yes;

[0033] The second judgment module is used to determine whether the current target area contains a dump truck if the first judgment result is no, so as to obtain a second judgment result;

[0034] The second determination module is used to determine that the excavation operation status of the current target area is the in-excavation status if the second judgment result is no;

[0035] The third judgment module is used to determine whether there is an overlap between the dump truck and the excavating equipment in the current target area if the second judgment result is yes, so as to obtain a third judgment result;

[0036] The third determination module is used to determine that the excavation operation status of the current target area is the in-loading status if the third judgment result is yes;

[0037] The fourth determination module is used to determine that the excavation operation status of the current target area is the in-excavation status if the third judgment result is no.

[0038] Optionally, the third judgment module includes:

[0039] The intersection-over-union index value determination module is used to determine the intersection-over-union index value between the second bounding box corresponding to the dump truck and the first bounding box corresponding to the excavating equipment in the current target area;

[0040] The overlap determination module is used to determine that there is an overlap between the dump truck and the excavating equipment in the current target area if the intersection-over-union index value is greater than a first threshold.

[0041] Optionally, the first judgment module is specifically used for: in the i-th frame to the (i + α)-th frame of video stream images, if the coordinate difference of the excavating equipment in any two frames of video stream images is less than a second threshold, it is determined that the excavating equipment included in the current target area is in a stationary state; where i and α are positive integers.

[0042] Optionally, the device further includes:

[0043] The model determination module is used to determine the target model corresponding to the dump truck in the current target area by using a dump truck model classification model;

[0044] The search module is used to search in the mapping relationship between the search model and the rated loading capacity according to the target model to determine the rated loading capacity corresponding to the dump truck in the current target area.

[0045] Optionally, using the dual-backbone object detection model to perform object detection on the video stream image includes:

[0046] The first backbone network determines n types of first image features corresponding to the video stream image;

[0047] The i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th type of first image feature. The segmentation processing result obtained includes: i types of sub-image features, and outputs the j-th type of sub-image feature to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i types of sub-image features output by the same i-th feature segmentation module are sub-image features with different numbers of channels; one feature fusion module corresponds to one number of channels, and different feature fusion modules correspond to different numbers of channels;

[0048] The i-th feature fusion module in the second backbone network performs first fusion processing on at least one type of sub-image feature to obtain a first fusion result. The channel-to-pixel module in the second backbone network determines a second image feature according to the first fusion result and outputs the second image feature to the neck network;

[0049] The neck network performs second fusion processing on the second image feature to obtain a second fusion processing result;

[0050] The detection head network determines the first target information and the second target information according to the second fusion processing result.

[0051] Optionally, the i-th feature segmentation module in the feature segmentation network performing convolution processing and segmentation processing on the i-th type of first image feature includes:

[0052] The i-th feature segmentation module in the feature segmentation network performs convolution processing on the i-th type of first image feature to obtain an image feature with c channels;

[0053] According to the channel dimension, the image feature with c channels is segmented into i types of sub-image features with different numbers of channels; the number of channels of the j-th type of sub-image feature among the i types of sub-image features is: t * 2 (j-1) .

[0054] Optionally, the second backbone network further includes: a first convolution module connected in front of the feature fusion module;

[0055] The i-th feature fusion module in the second backbone network performing first fusion processing on at least one type of sub-image feature includes:

[0056] The i-th feature fusion module receives n - i + 1 seed image features and the convolutional image features output by the first convolutional module; the number of channels corresponding to the n - i + 1 seed image features is the first channel number, and the number of channels corresponding to the convolutional image features is the second channel number;

[0057] The i-th feature fusion module performs interpolation processing on the n - i + 1 seed image features, and the number of channels corresponding to the n - i + 1 seed image features after interpolation processing is the second channel number;

[0058] The i-th feature fusion module performs fusion processing on the n - i + 1 seed image features after interpolation processing and the convolutional image features.

[0059] Optionally, the first convolutional module connected in front of the first feature fusion module is used to perform convolutional processing on the video stream image and output the corresponding convolutional image features to the first feature fusion module.

[0060] Optionally, the n feature segmentation modules include: the first feature segmentation module, the second feature segmentation module, the third feature segmentation module, the fourth feature segmentation module, and the fifth feature segmentation module; the n feature fusion modules include: the first feature fusion module, the second feature fusion module, the third feature fusion module, the fourth feature fusion module, and the fifth feature fusion module;

[0061] Among them, the segmentation processing result of the first feature segmentation module includes: 1 type of m-channel first sub-image feature, and 1 type of m-channel first sub-image feature is output to the first feature fusion module;

[0062] The segmentation processing result of the second feature segmentation module includes: 1 type of m-channel second sub-image feature and 1 type of 2m-channel third sub-image feature; 1 type of m-channel second sub-image feature is output to the first feature fusion module; 1 type of 2m-channel third sub-image feature is output to the second feature fusion module;

[0063] The segmentation processing result of the third feature segmentation module includes: 1 type of m-channel fourth sub-image feature, 1 type of 2m-channel fifth sub-image feature, and 1 type of 4m-channel sixth sub-image feature; 1 type of m-channel fourth sub-image feature is output to the first feature fusion module; 1 type of 2m-channel fifth sub-image feature is output to the second feature fusion module; 1 type of 4m-channel sixth sub-image feature is output to the third feature fusion module;

[0064] The segmentation processing results of the fourth feature segmentation module include: one kind of seventh sub-image feature with m channels, one kind of eighth sub-image feature with 2m channels, one kind of ninth sub-image feature with 4m channels, and one kind of tenth sub-image feature with 8m channels; one kind of seventh sub-image feature with m channels is output to the first feature fusion module; one kind of eighth sub-image feature with 2m channels is output to the second feature fusion module; one kind of ninth sub-image feature with 4m channels is output to the third feature fusion module; one kind of tenth sub-image feature with 8m channels is output to the fourth feature fusion module;

[0065] The segmentation processing results of the fifth feature segmentation module include: one kind of eleventh sub-image feature with m channels, one kind of twelfth sub-image feature with 2m channels, one kind of thirteenth sub-image feature with 4m channels, one kind of fourteenth sub-image feature with 8m channels, and one kind of fifteenth sub-image feature with 16m channels; one kind of eleventh sub-image feature with m channels is output to the first feature fusion module; one kind of twelfth sub-image feature with 2m channels is output to the second feature fusion module; one kind of thirteenth sub-image feature with 4m channels is output to the third feature fusion module; one kind of fourteenth sub-image feature with 8m channels is output to the fourth feature fusion module; one kind of fifteenth sub-image feature with 16m channels is output to the fifth feature fusion module.

[0066] The embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored, and when the executable code is executed, the processor is caused to execute the method as described in the embodiment of the present application.

[0067] The embodiment of the present application also discloses a machine-readable medium, on which executable code is stored, and when the executable code is executed, a processor is caused to execute the method as described in the embodiment of the present application.

[0068] The embodiment of the present application has the following advantages:

[0069] In the technical solution of the embodiment of the present application, first, a video stream image of the construction area to be counted is collected; then, using a dual-backbone object detection model, object detection is performed on the above video stream image to obtain object information corresponding to objects such as excavation equipment and dump trucks in the video stream image; then, according to the above object information, N target areas in the above video stream image are determined; then, according to the objective laws between the excavation operation state of the target area and the operation state of the excavation equipment, the presence or absence of dump trucks, and the positional relationship between the excavation equipment and the dump trucks, the excavation volume corresponding to the current loading of the current target area is determined, and the cumulative excavation volume of the current target area is updated; furthermore, according to the cumulative excavation volumes of all the current target areas in the above N target areas, the total real-time excavation volume corresponding to the construction area to be counted is determined.

[0070] Since the embodiments of the present application determine N target regions in the video stream image based on the target information corresponding to targets such as excavation equipment and dump equipment in the video stream image, and determine the excavation volume corresponding to the current loading of the current target region according to the objective laws between the excavation operation status of the target region and the operation status of the excavation equipment, the presence of dump equipment, and the positional relationship between the excavation equipment and the dump equipment; in this way, the embodiments of the present application can continuously determine the target regions existing in the video stream image and the excavation operation status corresponding to the target regions according to the continuously updated video stream image; on this basis, the embodiments of the present application can continuously determine the excavation volume corresponding to the current loading of the current target region in the video stream image and the cumulative excavation volume of all current target regions in the video stream image. Therefore, the embodiments of the present application can realize real-time monitoring of the earthwork excavation volume based on the analysis of the video stream image at a relatively low cost and at a relatively fast speed. In other words, the embodiments of the present application can save the labor cost, data acquisition cost, and data processing time cost consumed by using unmanned aerial vehicle oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation volume.

[0071] Moreover, the embodiments of the present application use a dual-backbone object detection model to perform object detection on video image frames. Among them, the n feature segmentation modules included in the feature segmentation network in the above dual-backbone object detection model can play a role in aggregating sub-image features at multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features at multiple levels. In this way, the embodiments of the present application can enhance the expression ability and accuracy of the second image features output by the second backbone network, and further improve the accuracy of the object detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a schematic diagram of the application environment of the real-time excavation volume monitoring method based on dual-backbone object detection according to an embodiment of the present application;

[0073] Figure 2 is a schematic diagram of the step flow of the real-time excavation volume monitoring method based on dual-backbone object detection according to an embodiment of the present application;

[0074] Figure 3 is a flowchart of the steps of the method for determining the excavation operation status of the current target region according to an embodiment of the present application;

[0075] Figure 4 is a schematic diagram of the structure of the dual-backbone object detection model according to an embodiment of the present application;

[0076] Figure 5 is a schematic diagram of the structures of the first backbone network 301, the feature segmentation network 302, and the second backbone network 303 according to an embodiment of the present application;

[0077] Figure 6 is a schematic structural diagram of a channel-to-pixel module according to an embodiment of the present application;

[0078] Figure 7 is a schematic structural diagram of a bottleneck module according to an embodiment of the present application;

[0079] Figure 8 is a schematic structural diagram of a second backbone network according to an embodiment of the present application;

[0080] Figure 9 is a schematic structural diagram of a first backbone network according to an embodiment of the present application;

[0081] Figure 10 is a schematic structural diagram of a neck network according to an embodiment of the present application;

[0082] Figure 11 is a schematic structural diagram of a spatial pyramid pooling module according to an embodiment of the present application;

[0083] Figure 12 is a schematic structural diagram of a real-time excavation volume monitoring device based on dual-backbone object detection according to an embodiment of the present application;

[0084] Figure 13 is a schematic structural diagram of a device provided by an embodiment of the present application. Detailed implementation manners

[0085] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0086] The embodiments of the present application can be applied to engineering industries such as hydropower and civil engineering, and are used for real-time monitoring of the corresponding excavation volume for the open-pit excavation operation in the engineering industry.

[0087] Taking the pumped-storage power station project as an example, during the project construction, excavation operations are carried out simultaneously at multiple construction sites. At the same time, due to the continuous change of the on-site construction situation, there is a need to continuously carry out real-time monitoring of the excavation volume based on dual-backbone object detection for multiple construction sites.

[0088] In the traditional technology, for the calculation method of the excavation volume during the engineering construction stage, usually, before and after excavation, the digital elevation model of the excavation area is obtained by using the unmanned aerial vehicle oblique photography method, or the spatial point cloud data of the excavation area is obtained by using the three-dimensional laser scanning technology; then, according to the difference between the digital elevation model or the spatial point cloud data of the excavation area, the volume of the excavation area is calculated; next, according to the volume of the excavation area, the excavation volume is calculated. The above on-site measurement method not only requires certain human and data acquisition costs, but also has the disadvantages of long data processing time and poor real-time performance.

[0089] In view of the technical problems of the traditional technology, such as consuming manpower, high data acquisition cost, long processing time, and poor real-time performance, the embodiment of the present application provides a real-time monitoring method for excavation volume based on dual-backbone object detection. The method specifically includes the following steps:

[0090] Collect video stream images of the construction area to be counted.

[0091] Use the dual-backbone object detection model to perform object detection on the above video stream images to obtain the first target information corresponding to the excavation equipment included in the above video stream images, and the second target information corresponding to the dump truck equipment included in the above video stream images; wherein, the dual-backbone object detection model includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network, and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected with a channel-to-pixel module; n is a positive integer greater than 1.

[0092] Determine N target areas in the above video stream images according to the above first target information and the above second target information; N is a positive integer.

[0093] For the current target area among the N target areas, determine the excavation operation status of the current target area according to the operation status of the excavation equipment in the current target area, whether the current target area contains a dump truck equipment, and the positional relationship between the excavation equipment and the dump truck equipment in the current target area; the above excavation operation status includes: one of the non-excavation status, the excavation-in-progress status, and the loading-in-progress status.

[0094] When the excavation operation status of the current target area is updated from the loading-in-progress status to the non-excavation status or the excavation-in-progress status, it is considered that the current target area has completed the current loading. According to the rated loading capacity corresponding to the dump truck equipment in the current target area, determine the excavation volume corresponding to the current loading of the current target area, and update the cumulative excavation volume of the current target area.

[0095] Determine the total real-time excavation volume corresponding to the construction area to be counted according to the cumulative excavation volumes of all the current target areas among the above N target areas.

[0096] In the embodiments of the present application, first, a video stream image of the construction area to be counted is collected; then, using a dual-backbone object detection model, object detection is performed on the above video stream image to obtain object information corresponding to objects such as excavation equipment and dump trucks in the video stream image; next, according to the above object information, N target areas in the above video stream image are determined; then, according to the objective laws between the excavation operation status of the target area and the operation status of the excavation equipment, the presence or absence of dump trucks, and the positional relationship between the excavation equipment and the dump trucks, the excavation volume corresponding to the current loading of the current target area is determined, and the cumulative excavation volume of the current target area is updated; furthermore, according to the cumulative excavation volumes of all the current target areas in the above N target areas, the total real-time excavation volume corresponding to the construction area to be counted is determined.

[0097] Since, in the embodiments of the present application, N target areas in the video stream image are determined according to the object information corresponding to objects such as excavation equipment and dump trucks in the video stream image, and according to the objective laws between the excavation operation status of the target area and the operation status of the excavation equipment, the presence or absence of dump trucks, and the positional relationship between the excavation equipment and the dump trucks, the excavation volume corresponding to the current loading of the current target area is determined; in this way, the embodiments of the present application can continuously determine the target areas existing in the video stream image and the operation status corresponding to the target areas according to the continuously updated video stream image; on this basis, the embodiments of the present application can continuously determine the excavation volume corresponding to the current loading of the current target area in the video stream image and the cumulative excavation volumes of all the current target areas in the video stream image. Therefore, the embodiments of the present application can realize real-time monitoring of the earthwork excavation volume based on the analysis of the video stream image at a low cost and at a relatively fast speed. In other words, the embodiments of the present application can save the labor cost, data acquisition cost, and data processing time cost consumed by using the unmanned aerial vehicle oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation volume.

[0098] Moreover, in the embodiments of the present application, a dual-backbone object detection model is used to perform object detection on video image frames. Among them, the n feature segmentation modules included in the feature segmentation network in the above dual-backbone object detection model can play the role of aggregating sub-image features at multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse the sub-image features at multiple levels. In this way, the embodiments of the present application can enhance the expression ability and accuracy of the second image features output by the second backbone network, and further improve the accuracy of the object detection results.

[0099] Referring to Figure 1 , a schematic diagram of the application environment of a real-time excavation volume monitoring method based on dual-backbone object detection according to an embodiment of the present invention is shown, where the image acquisition end 101 and the server end 102 can perform data interaction based on a wireless network or a wired network.

[0100] In practical applications, the image acquisition end 101 may be deployed with an image acquisition device having an image acquisition function such as an image sensor. The image acquisition device may acquire a construction video and send the construction video to the server end 102 at a preset time period.

[0101] After receiving the construction video sent by the image acquisition end 101, the server end 102 may extract video stream images from the construction video and use the method of the embodiment of the present application to process the video stream images to determine the excavation volume corresponding to all current target areas in the video stream images of the construction area to be counted.

[0102] It should be noted that as the video stream images are updated, the embodiment of the present application can continuously determine the target areas existing in the video stream images and the excavation operation states corresponding to the target areas; on this basis, the embodiment of the present application can continuously determine the excavation volume corresponding to the current loading of the current target areas in the video stream images and the cumulative excavation volume of all current target areas in the video stream images. Therefore, the embodiment of the present application can realize real-time monitoring of the earth-rock excavation volume at a low cost and at a high speed based on the analysis of the video stream images.

[0103] Method Embodiment 1

[0104] Reference Figure 2 , which shows a schematic flow chart of the steps of a method for real-time monitoring of excavation volume based on dual-backbone object detection according to an embodiment of the present application. The method specifically includes the following steps:

[0105] Step 201, collect video stream images of the construction area to be counted;

[0106] Step 202, use the dual-backbone object detection model to perform object detection on the above video stream images to obtain first target information corresponding to the excavation equipment included in the above video stream images and second target information corresponding to the dump truck equipment included in the above video stream images; wherein, the dual-backbone object detection model specifically includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network, and a detection head network; the feature segmentation network specifically includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected with a channel-to-pixel module; n can be a positive integer greater than 1;

[0107] Step 203, determine N target areas in the above video stream images according to the above first target information and the above second target information; N is a positive integer;

[0108] Step 204: For the current target area among the N target areas, determine the excavation operation status of the current target area according to the operation status of the excavation equipment in the current target area, whether there is a dump truck in the current target area, and the positional relationship between the excavation equipment and the dump truck in the current target area; the excavation operation status includes one of: non-excavation status, excavation-in-progress status, and loading-in-progress status.

[0109] Step 205: When the excavation operation status of the current target area is updated from the loading-in-progress status to the non-excavation status or the excavation-in-progress status, it is considered that the current target area has completed the current loading. Determine the excavation volume corresponding to the current loading of the current target area according to the rated loading capacity of the dump truck in the current target area, and update the cumulative excavation volume of the current target area.

[0110] Step 206: Determine the total real-time excavation volume corresponding to the construction area to be statistically analyzed according to the cumulative excavation volumes of all the current target areas among the N target areas.

[0111] Figure 2 At least one step included in the method shown can be executed by the server. It can be understood that the embodiments of the present application do not limit Figure 2 the specific execution entity of the method shown.

[0112] In step 201, the process of collecting the video stream image specifically includes: after the server receives the construction video sent by the image acquisition end, extract the video stream image from the construction video. Among them, the server can receive the video stream image from the image acquisition end according to a preset time period.

[0113] In the scenario of a hydropower project, multiple video sources can be set, and different video sources can correspond to different video stream images. Among them, the video source can continuously provide continuous video stream images. The embodiments of the present application can process the continuous video stream images provided by any video source to obtain the excavation volume corresponding to the video source. The construction area to be statistically analyzed can be the construction area represented by a certain video source.

[0114] In step 202, a dual-backbone object detection model can be used to perform object detection on the above video stream image to obtain the first target information corresponding to the excavation equipment included in the video stream image and the second target information corresponding to the dump truck included in the video stream image.

[0115] The dual-backbone object detection model is a deep learning model in the field of computer vision, which aims to identify the object information corresponding to the objects in the image. Specifically in the embodiments of this application, the video stream image may include objects such as excavation equipment and dump equipment. Examples of excavation equipment may include excavators, etc. Examples of dump equipment may include dump trucks, etc. In practical applications, the dump truck drives into the excavation site and stops within the working radius of the excavator so that the excavator can easily load the materials into the cargo box of the dump truck.

[0116] The first object information and the second object information belong to the category of object information. The object information may correspond to category information. For example, the category corresponding to the first object information is the excavation equipment category, and the category corresponding to the second object information is the dump equipment category.

[0117] The object information specifically includes: the bounding box corresponding to the object. The bounding box is used to represent the position and range of the object in the video stream image. The bounding box is usually a rectangular box, and its four sides are respectively aligned with the outermost edges of the object, thus enclosing the entire object. The information of the bounding box may include: the upper left corner coordinates, width, and height of the rectangular box; or, the information of the bounding box may include: the center point coordinates, width, and height of the rectangular box.

[0118] The specific process of performing object detection on the above video stream image using object detection technology will be described in detail in Method Embodiment 2.

[0119] In step 203, the target area may be an area in the video stream image that contains an excavation equipment. According to whether the target area contains a dump equipment, the target area can be divided into: the first target area and the second target area. Among them, the first target area contains an excavation equipment and does not contain a dump equipment. The second target area contains an excavation equipment and a dump equipment.

[0120] In an implementation manner of this application, the first object information includes: the first bounding box, and the second object information includes: the second bounding box;

[0121] The process of step 203 for determining the N target areas in the above video stream image according to the above first object information and the above second object information specifically includes:

[0122] Step A1: Determine the number N of target areas according to the number N of the first bounding boxes;

[0123] Step A2: Perform one-to-one matching on the first bounding box and the second bounding box to obtain the corresponding matching result;

[0124] Step A3: If there is a second bounding box that matches the first bounding box, then the smallest closed rectangular area that contains both the first bounding box and the second bounding box is used as the target area;

[0125] Step A4: If there is no second bounding box that matches the first bounding box, then the rectangular area corresponding to the first bounding box is used as the target area.

[0126] In step A1, the number N of target areas can be equal to the number N of the first bounding boxes. In other words, the number of target areas contained in a video stream image can be equal to the number of the first bounding boxes contained in a video stream image.

[0127] In step A2, a one-to-one matching of the first bounding box and the second bounding box can be a distance matching between the first bounding box and the second bounding box.

[0128] In one implementation, the process of step A2 for performing a one-to-one matching of the first bounding box and the second bounding box specifically includes: determining the distance intersection over union (DIoU) index value between a first bounding box and a second bounding box; according to the distance intersection over union index values between a first bounding box and multiple second bounding boxes, selecting the one with the largest distance intersection over union index value from the multiple second bounding boxes as the second bounding box that matches the first bounding box.

[0129] Among them, for the second bounding boxes of M dump trucks in the video stream image, one-to-one matching is performed with the first bounding boxes in the video stream image one by one based on the distance intersection over union index value. The distance intersection over union index value DIoU mn is determined as shown in formula (1).

[0130]

[0131] Among them, A m represents the m-th (m ∈ [1, M], and is a positive integer) second bounding box, and B n represents the n-th (n ∈ [1, N], and is a positive integer) first bounding box. a m , b n respectively represent the center points of the second bounding box A m and the first bounding box B n . ρ mn is the Euclidean distance between the center points a m , b n , and d is the diagonal distance of the smallest closed area that contains both the first bounding box and the second bounding box.

[0132] The distance intersection-over-union metric value reflects the degree of overlap between the first bounding box and the second bounding box, as well as the distance between the center points. Generally speaking, the larger the distance intersection-over-union metric value, the higher the degree of overlap between the two and the closer the distance between the center points.

[0133] For a first bounding box, there are M distance intersection-over-union metric values between it and M second bounding boxes. Then, the largest one of the M distance intersection-over-union metric values can be selected, and the second bounding box corresponding to the largest distance intersection-over-union metric value can be used as the second bounding box matching the first bounding box.

[0134] Step A3 and step A4 can be executed in parallel.

[0135] In practical applications, the N first bounding boxes in the video stream image can be traversed in a first preset order to determine the target regions corresponding to the first bounding boxes one by one. The first preset order can be: from left to right and from bottom to top, or from left to right and from top to bottom.

[0136] In the embodiments of the present application, the N first bounding boxes in the video stream image can also be numbered in a first preset order.

[0137] Numbering can refer to numbering in sequence or designated numbers using any ordered or unordered symbols. In a specific implementation, the numbers of the N first bounding boxes and the numbers of the N target regions can be the same. For example, the numbers of the N first bounding boxes in video stream image A can be respectively: A0001, A0002, …… A000N, and the numbers of the N target regions in video stream image A can also be respectively: A0001, A0002, …… A000N.

[0138] Of course, the numbers of the N first bounding boxes and the numbers of the N target regions can also be different. For example, the numbers of the N first bounding boxes in video stream image A can be respectively: A0001, A0002, …… A000N, and the numbers of the N target regions in video stream image A can be respectively: A+0001, A+0002, …… A+000N.

[0139] During the process of traversing the N first bounding boxes in the video stream image, it can be determined whether there is a second bounding box matching a first bounding box. If so, step A3 can be executed to use the smallest closed rectangle region that contains both the first bounding box and the second bounding box as the target region. If not, step A4 can be executed to use the rectangular region corresponding to the first bounding box as the target region.

[0140] After determining the target area based on step A3 and step A4, information about the target area can be obtained. The information about the target area specifically includes: the number of the target area, the position information of the rectangular frame corresponding to the target area, information such as whether it contains a second bounding box, etc.

[0141] In step 204, for the current target area among the N target areas, the excavation operation status of the current target area can be determined. Wherein, the current target area can represent one target area among the N target areas. It can be understood that the embodiments of the present application can perform the processing of multiple current target areas among the N target areas in parallel or serially. The processing of the current target area can include: the processing of step 204 and step 205. The process of serially performing the processing of multiple current target areas among the N target areas specifically includes: traversing the N target areas in a second preset order from smallest to largest or from largest to smallest according to the number of the target area to obtain a current target area among the N target areas, and serially performing the processing of multiple current target areas among the N target areas according to the second preset order.

[0142] The excavation operation status of the present application specifically includes: one of the unexcavated state, the excavating state, and the loading state; wherein, the unexcavated state means that the current target area has not undergone an excavation operation. The excavating state means that the current target area is undergoing an excavation operation. The loading state means that the current target area is undergoing a loading operation.

[0143] The embodiments of the present application can determine the excavation operation status of the current target area according to the operation status of the excavation equipment in the current target area, whether the current target area contains a dump truck, and the positional relationship between the excavation equipment and the dump truck in the current target area.

[0144] Refer to Figure 3 , which shows a step flowchart of a method for determining the excavation operation status of a current target area according to an embodiment of the present application, and specifically may include the following steps:

[0145] Step 301, determine whether the excavation equipment included in the current target area is in a stationary state to obtain a first judgment result;

[0146] If the first judgment result is yes, then execute step 302: determine that the excavation operation status of the current target area is the unexcavated state;

[0147] If the first judgment result is no, then execute step 303: determine whether the current target area contains a dump truck to obtain a second judgment result;

[0148] If the second judgment result is no, then execute step 304: determine that the excavation operation status of the current target area is the excavating state;

[0149] If the second judgment result is yes, then execute step 305: Determine whether there is an overlap between the dump truck equipment and the excavation equipment in the current target area to obtain a third judgment result;

[0150] If the third judgment result is yes, then execute step 306: Determine that the excavation operation state of the current target area is in the loading state;

[0151] If the third judgment result is no, then execute step 307: Determine that the excavation operation state of the current target area is in the excavation state.

[0152] The embodiments of the present application provide the following determination rules for the excavation operation state:

[0153] The determination rule for the non-excavation state is: The excavation equipment is in a stationary state.

[0154] The determination rules for the excavation state include: The excavator is in a non-stationary state and the current target area does not contain dump truck equipment; or, the current target area contains dump truck equipment and there is no overlap between the dump truck equipment and the excavation equipment.

[0155] The determination rules for the loading state include: The excavation equipment is in a non-stationary state, the current target area contains dump truck equipment, and there is an overlap between the dump truck equipment and the excavation equipment.

[0156] The process of step 301 for determining whether the excavation equipment included in the current target area is in a stationary state specifically includes:

[0157] In the i-th frame to the (i + α)-th frame of the video stream images, if the coordinate differences of the excavation equipment in any two frames of the video stream images are all less than the second threshold, it is determined that the excavation equipment included in the current target area is in a stationary state; where both i and α can be positive integers, and α is greater than i.

[0158] The embodiments of the present application can select two video stream images to be compared in the α video stream images corresponding to the i-th frame to the (i + α)-th frame of the video stream images, and determine whether the coordinate difference between the two video stream images to be compared is less than the second threshold.

[0159] Selecting two video stream images to be compared from the α video stream images, the selection scheme can be P types. If the coordinate differences corresponding to the P selection schemes are all less than the second threshold, it can be determined that the excavation equipment included in the current target area is in a stationary state.

[0160] Considering the influence of signal transmission, camera jitter, etc., when the excavation equipment is in a stationary state, the coordinates of its corresponding first bounding box may have a certain offset. Assume that the upper left pixel coordinates of the first bounding box of the excavation equipment in the i-th frame of the video stream image of the current target area are The lower right pixel coordinates are Set the second threshold corresponding to the coordinate offset as δ and the frame number threshold as α. If formula (2) holds, that is, in the video stream images from the i-th frame to the (i + α)-th frame, the coordinate difference between any two frames is less than the second threshold δ, then it can be determined that the excavation device is in a stationary state.

[0161] Formula (2) calculates and judges the coordinate difference for the upper left pixel coordinates of the i-th frame video stream image and the (i + j)-th frame video stream image. Therefore, represents the abscissa of the upper left pixel of the (i + j)-th frame video stream image, represents the ordinate of the upper left pixel of the (i + j)-th frame video stream image.

[0162]

[0163] It can be understood that formula (2) calculates and judges the coordinate difference for the upper left pixel coordinates of the i-th frame video stream image and the (i + j)-th frame video stream image. As an optional embodiment, actually, the coordinate difference can be calculated and judged for the upper right pixel coordinates, lower left pixel coordinates, or lower right pixel coordinates of the i-th frame video stream image and the (i + j)-th frame video stream image.

[0164] Since in step 203, after determining the target area, information such as the number of the target area, the position information of the rectangular frame corresponding to the target area, and whether it contains the second bounding box is obtained. Therefore, step 303 can judge whether the current target area contains a dump truck according to the information corresponding to whether the current target area contains the second bounding box. Of course, object detection technology can be used to perform object detection on the area image corresponding to the current target area to judge whether the current target area contains a dump truck.

[0165] The process of step 305 judging whether there is an overlap between the dump truck and the excavation device in the current target area specifically includes:

[0166] Step C1: Determine the intersection over union index value between the second bounding box corresponding to the dump truck in the current target area and the first bounding box corresponding to the excavation device in the current target area;

[0167] Step C2: If the intersection over union index value is greater than the first threshold, it is determined that there is an overlap between the dump truck and the excavation device in the current target area.

[0168] Formula (3) shows the calculation process of the intersection over union index value between the second bounding box and the first bounding box.

[0169]

[0170] Wherein, A represents the second bounding box, B represents the first bounding box, A ∩ B represents the intersection area between the second bounding box and the first bounding box, and A ∪ B represents the union area between the second bounding box and the first bounding box.

[0171] In practical applications, when the intersection over union (IoU) index value is greater than 0, it can be considered that there is an overlap between the dump truck equipment and the excavator equipment in the current target area. Therefore, examples of the first threshold can include: 0 or a real number greater than 0.

[0172] In step 205, for the current target area, the change in its excavation operation status can be monitored. If its excavation operation status is updated from the loading status to the non-excavation status or the excavation-in-progress status, it can be considered that the current target area has completed the current loading. Then, according to the rated loading capacity corresponding to the dump truck equipment in the current target area, the excavation volume corresponding to the current loading of the current target area is determined, and the cumulative excavation volume of the current target area is updated. The cumulative excavation volume of the current target area can be the sum of the excavation volumes corresponding to multiple loadings of the current target area.

[0173] Among them, the process of determining the rated loading capacity corresponding to the dump truck equipment in the current target area specifically includes:

[0174] Step D1: Use the dump truck equipment model classification model to determine the target model corresponding to the dump truck equipment in the current target area;

[0175] Step D2: According to the target model, search in the mapping relationship between the search model and the rated loading capacity to determine the rated loading capacity corresponding to the dump truck equipment in the current target area.

[0176] The dump truck equipment model classification model of the embodiments of the present application can have the classification ability of the dump truck equipment model. In other words, it can determine the target model corresponding to the dump truck equipment in the regional image according to the regional image corresponding to the current target area.

[0177] In specific implementation, the embodiments of the present application can label the model categories for the dump truck equipment image dataset, and train the dump truck equipment model classification model according to the dump truck equipment image dataset.

[0178] Here provides a process for obtaining a dump truck equipment image dataset. Specifically, based on the video surveillance system, dump truck equipment images covering different construction scene backgrounds, different lighting conditions, different working postures, etc. can be collected. In addition, combined images of dump truck equipment and excavator equipment during the cooperative excavation operation need to be collected to reflect the actual situation of mutual occlusion between the two types of construction machinery. Therefore, the dump truck equipment image dataset can include: dump truck equipment images and combined images.

[0179] The embodiments of the present application do not limit the specific structure of the dump equipment model classification model. For example, the dump equipment model classification model may include structures of convolutional neural networks such as VGG (Visual Geometry Group).

[0180] The embodiments of the present application may pre - save the mapping relationship between the model and the rated loading capacity. In this way, according to the target model of the dump equipment in the current target area, the mapping relationship can be searched to obtain the rated loading capacity of the dump equipment in the current target area.

[0181] In the embodiments of the present application, when it is monitored that the excavation operation state of the current target area is updated from the loading state to the non - excavated state or the excavating state, it can be considered that the current target area has completed the current loading, and according to the rated loading capacity of the dump equipment in the current target area, the excavation volume corresponding to the current loading in the current target area is determined. The excavation volume corresponding to the current loading may be equal to the rated loading capacity of the dump equipment in the current target area.

[0182] In step 206, the cumulative excavation volumes of all current target areas among the N target areas can be summed up to obtain the total real - time excavation volume corresponding to the construction area to be statistically analyzed.

[0183] In an example of the present application, at the start time of a natural day, the total real - time excavation volume corresponding to the construction area to be statistically analyzed can be set to 0. Subsequently, according to the completion of the current loading in the current target area, the total real - time excavation volume is updated to obtain the continuously accumulated total real - time excavation volume. Therefore, the real - time monitoring process of the excavation volume can be a process of continuously updating the total real - time excavation volume according to the completion of the current loading in the N current target areas.

[0184] In practical applications, if there are multiple construction areas to be statistically analyzed in a hydropower project, the total real - time excavation volume corresponding to each individual construction area to be statistically analyzed can be monitored according to natural days. After the end of a natural day, the total real - time excavation volumes corresponding to the multiple construction areas to be statistically analyzed can be fused. It can be understood that the embodiments of the present application do not limit the specific fusion method.

[0185] In summary, in the real-time excavation volume monitoring method based on dual-backbone object detection according to the embodiments of the present application, the N target regions in the video stream image are determined based on the target information corresponding to targets such as excavation equipment and dump trucks in the video stream image, and the excavation volume corresponding to the current loading of the current target region is determined according to the objective laws among the excavation operation states of the target regions, the operation states of the excavation equipment, the presence or absence of dump trucks, and the positions between the excavation equipment and the dump trucks. In this way, the embodiments of the present application can continuously determine the target regions existing in the video stream image and the operation states corresponding to the target regions based on the continuously updated video stream image. On this basis, the embodiments of the present application can continuously determine the excavation volume corresponding to the current loading of the current target region in the video stream image and the cumulative excavation volume of all current target regions in the video stream image. Therefore, the embodiments of the present application can realize the real-time monitoring of the earth-rock excavation volume based on the analysis of the video stream image at a relatively low cost and at a relatively fast speed. In other words, the embodiments of the present application can save the labor cost, data acquisition cost, and data processing time cost consumed by using the drone oblique photography or 3D laser scanning technology, and can improve the processing efficiency and real-time performance of the earth-rock excavation volume.

[0186] Moreover, in the embodiments of the present application, a dual-backbone object detection model is used to perform object detection on the video image frames. Among them, the n feature segmentation modules included in the feature segmentation network in the above dual-backbone object detection model can play the role of converging sub-image features at multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse the sub-image features at multiple levels. In this way, the embodiments of the present application can enhance the expression ability and accuracy of the second image features output by the second backbone network, and further improve the accuracy of the object detection results.

[0187] Method Embodiment 2

[0188] In the embodiments of the present application, a dual-backbone object detection model can be used to perform object detection on the above video stream image to obtain the first target information corresponding to the excavation equipment included in the above video stream image and the second target information corresponding to the dump truck included in the above video stream image. In a specific implementation, the video stream image can be input into the dual-backbone object detection model to obtain the first target information and the second target information output by the dual-backbone object detection model.

[0189] Refer to Figure 4 , which shows a schematic structural diagram of a dual-backbone object detection model according to an embodiment of the present application, which specifically includes: a first backbone network 401, a feature segmentation network 402, a second backbone network 403, a neck network 404, and a detection head network 405 that are connected in sequence.

[0190] Among them, the first backbone network 401 is used to extract features from the input image to obtain n types of first image features. In the training stage, the input image can be a video stream image sample. In the image recognition stage, the input image can be a real-time video stream image.

[0191] The feature segmentation network 402 is used to perform convolution processing and segmentation processing on the n types of first image features, and output multiple sub-image features obtained by segmentation to the second backbone network 403.

[0192] The second backbone network 403 is used to perform first fusion processing and channel-to-pixel processing on the multiple sub-image features to obtain second image features, and output the second image features to the neck network 404.

[0193] The neck network 404 is used to perform second fusion processing on the above-mentioned second image features to obtain a second fusion processing result.

[0194] The detection head network 405 is used to determine the recognition result according to the above-mentioned second fusion processing result.

[0195] Refer to Figure 5 , which shows a schematic structural diagram of the first backbone network 401, the feature segmentation network 402, and the second backbone network 403 in an embodiment of the present application. Among them, the first backbone network 401 outputs n types of first image features to n feature segmentation modules of the feature segmentation network 402.

[0196] The feature segmentation network 402 specifically includes: n feature segmentation modules, which are respectively denoted as: the first feature segmentation module 421, the second feature segmentation module 422... and the nth feature segmentation module 42n.

[0197] The i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th type of first image feature. The obtained segmentation processing result includes: i types of sub-image features, and outputs the j-th type of sub-image feature to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i types of sub-image features output by the same i-th feature segmentation module are sub-image features with different numbers of channels; one feature fusion module corresponds to one number of channels, and different feature fusion modules correspond to different numbers of channels.

[0198] The second backbone network 403 specifically includes: n feature fusion modules, which are respectively denoted as: the first feature fusion module 431, the second feature fusion module 432... and the nth feature fusion module 43n.

[0199] In a specific implementation, the process of the i-th feature segmentation module in the feature segmentation network performing convolution processing and segmentation processing on the i-th type of first image feature specifically includes:

[0200] Step E1: The i-th feature segmentation module in the feature segmentation network performs convolution processing on the i-th type of first image feature to obtain an image feature with c channels;

[0201] Step E2: According to the channel dimension, the image feature with c channels is segmented into i sub-image features with different channel numbers; the number of channels of the j-th sub-image feature among the i sub-image features is: t * 2 (j-1) . The value of t can be determined by those skilled in the art according to actual application requirements. Examples of the value of t can include: 32, 64, etc.

[0202] The number of channels of the i sub-image features are respectively t * 2 0 , t * 2 1 , t * 2 2 ……, t * 2 (i-1) . Among them, the sum of the corresponding channel numbers of the i sub-image features can be c, that is: t * 2 0 + t * 2 1 + t * 2 2 …… + t * 2 (i-1) = c.

[0203] When n is 5, the i-th feature segmentation module can output the j-th sub-image feature to the connected j-th feature fusion module. Specific examples include:

[0204] For example, the segmentation processing result of the first feature segmentation module 421 includes: 1 type of first sub-image feature with m channels, and 1 type of first sub-image feature with m channels is output to the first feature fusion module 431.

[0205] Another example, the segmentation processing result of the second feature segmentation module 422 includes: 1 type of second sub-image feature with m channels and 1 type of third sub-image feature with 2m channels. 1 type of second sub-image feature with m channels is output to the first feature fusion module 431. 1 type of third sub-image feature with 2m channels is output to the second feature fusion module 432.

[0206] Another example, the segmentation processing result of the third feature segmentation module 423 (not shown in the figure) includes: 1 type of fourth sub-image feature with m channels, 1 type of fifth sub-image feature with 2m channels, and 1 type of sixth sub-image feature with 4m channels. 1 type of fourth sub-image feature with m channels is output to the first feature fusion module 431. 1 type of fifth sub-image feature with 2m channels is output to the second feature fusion module 432. 1 type of sixth sub-image feature with 4m channels is output to the third feature fusion module 433.

[0207] Alternatively, the segmentation processing results of the fourth feature segmentation module 424 (not shown in the figure) include: one kind of seventh sub-image feature with m channels, one kind of eighth sub-image feature with 2m channels, one kind of ninth sub-image feature with 4m channels, and one kind of tenth sub-image feature with 8m channels. One kind of seventh sub-image feature with m channels is output to the first feature fusion module 431. One kind of eighth sub-image feature with 2m channels is output to the second feature fusion module 432. One kind of ninth sub-image feature with 4m channels is output to the third feature fusion module 433. One kind of tenth sub-image feature with 8m channels is output to the fourth feature fusion module 434.

[0208] Alternatively, the segmentation processing results of the fifth feature segmentation module 425 (not shown in the figure) include: one kind of eleventh sub-image feature with m channels, one kind of twelfth sub-image feature with 2m channels, one kind of thirteenth sub-image feature with 4m channels, one kind of fourteenth sub-image feature with 8m channels, and one kind of fifteenth sub-image feature with 16m channels. One kind of eleventh sub-image feature with m channels is output to the first feature fusion module 431. One kind of twelfth sub-image feature with 2m channels is output to the second feature fusion module 432. One kind of thirteenth sub-image feature with 4m channels is output to the third feature fusion module 433. One kind of fourteenth sub-image feature with 8m channels is output to the fourth feature fusion module 434. One kind of fifteenth sub-image feature with 16m channels is output to the fifth feature fusion module 435.

[0209] The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result. The channel-to-pixel module in the second backbone network determines a second image feature according to the first fusion result and outputs the second image feature to the neck network.

[0210] In an alternative implementation of the present application, the second backbone network may further include: a first convolution module connected in front of the feature fusion module;

[0211] Then the process of the i-th feature fusion module in the second backbone network performing a first fusion process on at least one sub-image feature specifically includes:

[0212] Step F1, the i-th feature fusion module receives n - i + 1 sub-image features and the convolution image features output by the first convolution module; the number of channels corresponding to the n - i + 1 sub-image features is the first channel number, and the number of channels corresponding to the convolution image features is the second channel number;

[0213] Step F2, the i-th feature fusion module performs interpolation processing on the n - i + 1 sub-image features, and the number of channels corresponding to the n - i + 1 sub-image features after interpolation processing is the second channel number;

[0214] Step F3: The i-th feature fusion module performs fusion processing on the n - i + 1 sub-image features and the convolutional image features after interpolation processing.

[0215] The first convolutional module, the second convolutional module, or the first convolutional processing module in the embodiments of the present application all belong to the category of convolutional structures. In one example, the above-mentioned convolutional structure specifically includes: at least one convolutional layer, at least one batch normalization layer, and at least one activation function. It can be understood that those skilled in the art can adopt the required convolutional structure according to actual application requirements, and the embodiments of the present application do not limit the specific convolutional structure.

[0216] For example, the number of the first channels corresponding to the n sub-image features received by the first feature fusion module is t * 2 0 . Those skilled in the art can adopt the corresponding interpolation processing technology according to actual application requirements, and the embodiments of the present application do not limit the specific interpolation processing technology.

[0217] The process of the i-th feature fusion module performing fusion processing on the n - i + 1 sub-image features and the convolutional image features after interpolation processing specifically includes: performing addition processing on the values corresponding to the n - i + 1 sub-image features and the convolutional image features after interpolation processing by the i-th feature fusion module, etc. The sub-image features and the convolutional image features can correspond to multi-dimensional matrices, and the above addition processing can be to add the element values of the multi-dimensional matrices.

[0218] In another alternative implementation manner of the present application, the first convolutional module connected in front of the first feature fusion module is used to perform convolutional processing on the video stream image and output the corresponding convolutional image features to the first feature fusion module.

[0219] At least some of the feature fusion modules are connected to a channel-to-pixel module. Refer to Figure 6 , which shows a schematic structural diagram of the channel-to-pixel module according to an embodiment of the present application. The channel-to-pixel module specifically includes: a first convolutional processing module 601, a segmentation processing module 602, M bottleneck modules 603, a connection module 604, and a second convolutional processing module 605. M can be a positive integer greater than 1.

[0220] Among them, the first convolutional processing module 601 is used to perform first convolutional processing on the first fusion result output by the feature fusion module to obtain a first convolutional processing result.

[0221] The splitting processing module 602 is used to split the first convolution processing result into two parts with the same number of channels. These two parts may include: the first part of features and the second part of features. Assuming that the number of channels before splitting is the third number of channels, the number of channels after splitting may be the fourth number of channels, and the third number of channels may be twice the fourth number of channels.

[0222] After the first part of features is processed by M bottleneck modules 603, the obtained bottleneck processing features enter the connection module 604. The first convolution processing result and the second part of features also enter the connection module 604. The connection module 604 is used to perform a connection operation on the first convolution processing result, the second part of features, and the bottleneck processing features to obtain connection features.

[0223] The connection features enter the second convolution processing module 605, and the second convolution processing module 605 restores the number of channels for the connection features. For example, the number of channels is restored from the fourth number of channels to the third number of channels.

[0224] Refer to Figure 7 , which shows a schematic structural diagram of a bottleneck module according to an embodiment of the present application. The bottleneck module specifically includes: a third convolution processing module 701 and a fourth convolution processing module 702.

[0225] Among them, the third convolution processing module 701 is used to reduce the number of channels of the input features to half of the original to obtain a second convolution processing result.

[0226] The fourth convolution processing module 702 is used to double the number of channels of the second convolution processing result to obtain a third convolution processing result. The number of channels of the third convolution processing result is the same as that of the input features. The third convolution processing result is fused with the input features to obtain output features. [[ID=,19]]

[0227] Refer to Figure 8 , which shows a schematic structural diagram of a second backbone network according to an embodiment of the present application. The second backbone network specifically includes: a first convolution module A801, a first feature fusion module 802, a first convolution module B803, a second feature fusion module 804, a first channel-to-pixel module 805, a first convolution module C806, a third feature fusion module 807, a second channel-to-pixel module 808, a first convolution module D809, a fourth feature fusion module 810, a third channel-to-pixel module 811, a first convolution module E812, a fifth feature fusion module 813, and a fourth channel-to-pixel module 814.

[0228] Among them, the first convolution module A801 is used to perform convolution processing on the video stream image and output the corresponding convolution image feature A to the first feature fusion module.

[0229] The first feature fusion module 802 is used to receive 5 sub-image features A from the first feature segmentation module and the convolutional image feature A from the first convolutional module A801, perform interpolation processing on the 5 sub-image features A, and the number of channels corresponding to the 5 sub-image features A after interpolation processing is the second channel number; and, perform fusion processing on the 5 sub-image features A after interpolation processing and the convolutional image feature A, assuming to obtain the first fusion result A.

[0230] The first convolutional module B803 is used to perform convolutional processing on the first fusion result A to obtain the convolutional image feature B.

[0231] The second feature fusion module 804 is used to receive 4 sub-image features B from the second feature segmentation module and the convolutional image feature B from the first convolutional module B803, perform interpolation processing on the 4 sub-image features B, and the number of channels corresponding to the 4 sub-image features B after interpolation processing is the second channel number; and, perform fusion processing on the 4 sub-image features B after interpolation processing and the convolutional image feature B, assuming to obtain the first fusion result B.

[0232] The first channel-to-pixel module 805 is used to determine the intermediate image feature according to the above first fusion result B.

[0233] The first convolutional module C806 is used to perform convolutional processing on the intermediate image feature to obtain the convolutional image feature C.

[0234] The third feature fusion module 807 is used to receive 3 sub-image features C from the third feature segmentation module and the convolutional image feature C from the first convolutional module C806, perform interpolation processing on the 3 sub-image features C, and the number of channels corresponding to the 3 sub-image features C after interpolation processing is the second channel number; and, perform fusion processing on the 3 sub-image features C after interpolation processing and the convolutional image feature C, assuming to obtain the first fusion result C.

[0235] The second channel-to-pixel module 808 is used to determine the second image feature A according to the above first fusion result C and output the second image feature A to the neck network.

[0236] The first convolutional module D809 is used to perform convolutional processing on the second image feature A to obtain the convolutional image feature D.

[0237] The fourth feature fusion module 810 is used to receive 2 sub-image features D from the fourth feature segmentation module and the convolutional image feature D from the first convolutional module D809, perform interpolation processing on the 3 sub-image features D, and the number of channels corresponding to the 2 sub-image features D after interpolation processing is the second channel number; and, perform fusion processing on the 2 sub-image features D after interpolation processing and the convolutional image feature D, assuming to obtain the first fusion result D.

[0238] The third channel to the pixel module 811 is used to determine the second image feature B according to the above first fusion result D and output the second image feature B to the neck network.

[0239] The first convolutional module E812 is used to perform convolutional processing on the second image feature B to obtain a convolutional image feature E.

[0240] The fifth feature fusion module 813 is used to receive 1 sub-image feature E from the fifth feature segmentation module and the convolutional image feature E from the first convolutional module E812, perform interpolation processing on the 1 sub-image feature E, and the number of channels corresponding to the interpolated 1 sub-image feature E is the second channel number; and, perform fusion processing on the interpolated 1 sub-image feature E and the convolutional image feature E, assuming to obtain the first fusion result E.

[0241] The fourth channel to the pixel module 814 is used to determine the second image feature C according to the above first fusion result E and output the second image feature C to the neck network.

[0242] In a specific implementation, the first backbone network may specifically include: a second convolutional module and n - 1 processing units connected in sequence; the processing unit specifically includes: a third convolutional module and a channel-to-pixel module;

[0243] Among them, the second convolutional module is connected to the first feature segmentation module and is used to output the first type of first image feature to the first feature segmentation module; the channel-to-pixel modules included in the n - 1 processing units are respectively connected to the corresponding n - 1 feature segmentation modules and are used to output the first image feature to the corresponding feature segmentation module.

[0244] Refer to Figure 9 , which shows a schematic structural diagram of the first backbone network according to an embodiment of the present application. Among them, the first backbone network specifically includes: a second convolutional module 901, a third convolutional module A902, a fifth channel-to-pixel module 903, a third convolutional module B904, a sixth channel-to-pixel module 905, a third convolutional module C906, a seventh channel-to-pixel module 907, a third convolutional module D908, and an eighth channel-to-pixel module 909 connected in sequence. Among them, for the structure of the fifth channel-to-pixel module 903, please refer to Figure 6 , which will not be elaborated here.

[0245] Among them, the second convolutional module 901 is used to perform convolutional processing on the video stream image to obtain the first image feature A and send the first image feature A to the first feature segmentation module.

[0246] The third convolutional module A902 is used to perform convolutional processing on the first image feature A to obtain the first convolution result.

[0247] The fifth channel to pixel module 903 is used to determine the first image feature B according to the first convolution result and send the first image feature B to the second feature segmentation module.

[0248] The third convolution module B904 is used to perform convolution processing on the first image feature B to obtain a second convolution result.

[0249] The sixth channel to pixel module 905 is used to determine the first image feature C according to the second convolution result and send the first image feature C to the third feature segmentation module.

[0250] The third convolution module C906 is used to perform convolution processing on the first image feature C to obtain a third convolution result.

[0251] The seventh channel to pixel module 907 is used to determine the first image feature D according to the third convolution result and send the first image feature D to the fourth feature segmentation module.

[0252] The third convolution module D908 is used to perform convolution processing on the first image feature D to obtain a fourth convolution result.

[0253] The seventh channel to pixel module 907 is used to determine the first image feature E according to the fourth convolution result and send the first image feature E to the fifth feature segmentation module.

[0254] The neck network is used to perform a second fusion process on the above second image feature to obtain a second fusion process result.

[0255] Refer to Figure 10 , which shows a schematic structural diagram of the neck network according to an embodiment of the present application. The neck network specifically includes: a spatial pyramid pooling module 1001, a first upsampling module 1002, a first connection module 1003, a ninth channel to pixel module 1004, a second upsampling module 1005, a second connection module 1006, a tenth channel to pixel module 1007, a fourth convolution module 1008, a third connection module 1009, an eleventh channel to pixel module 1010, a fifth convolution module 1011, a fourth connection module 1012, and a twelfth channel to pixel module 1013.

[0256] Among them, the spatial pyramid pooling module 1001 is used to receive the second image feature C and perform spatial pyramid pooling processing on the second image feature C. The spatial pyramid pooling processing may include: a convolution operation and a max pooling operation, which can achieve the depth fusion of the second image feature C.

[0257] Refer to Figure 11, which shows a schematic structural diagram of a spatial pyramid pooling module according to an embodiment of the present application. The spatial pyramid pooling module specifically includes: a sixth convolutional module 1101, p max-pooling modules 1102, a fifth connection module 1103, and a seventh convolutional module 1104. p can be a positive integer greater than 1, and the value of p in the figure is 3.

[0258] Among them, the sixth convolutional module 1101 is used to perform convolutional processing on the input second image feature C to obtain a fifth convolutional result.

[0259] The p max-pooling modules 1102 are respectively used to perform max-pooling processing on the input image features to obtain corresponding p max-pooling processing results.

[0260] The fifth convolutional result and the p max-pooling processing results are respectively input into the fifth connection module 1103, and the fifth connection module 1103 performs depth fusion on the fifth convolutional result and the p max-pooling processing results to obtain a corresponding depth fusion result.

[0261] The seventh convolutional module 1104 is used to perform convolutional processing on the depth fusion result to obtain a sixth convolutional result. The sixth convolutional result is provided to the first upsampling module 1002.

[0262] The first upsampling module 1002 is used to perform first upsampling processing on the sixth convolutional result to obtain a first upsampling processing result.

[0263] The first connection module 1003 is used to perform connection processing on the second image feature B and the first upsampling processing result to obtain a first connection result.

[0264] The ninth channel-to-pixel module 1004 is used to determine a first processing result according to the first connection result.

[0265] The second upsampling module 1005 is used to perform second upsampling processing on the first processing result to obtain a second upsampling processing result.

[0266] The second connection module 1006 is used to perform connection processing on the second upsampling processing result and the second image feature A to obtain a second connection result. The above connection processing can be splicing processing of two image features.

[0267] The tenth channel-to-pixel module 1007 is used to determine a second fusion processing result A according to the second connection result. The second fusion processing result A is output to the detection head network.

[0268] The fourth convolutional module 1008 is used to perform convolutional processing on the second fusion processing result A to obtain a sixth convolutional result.

[0269] The third connection module 1009 is used to perform connection processing on the sixth convolution result and the first processing result to obtain a third connection result.

[0270] The eleventh channel-to-pixel module 1010 is used to determine a second fusion processing result B according to the third connection result. The second fusion processing result B is output to the detection head network.

[0271] The fifth convolution module 1011 is used to perform convolution processing on the second fusion result B to obtain a seventh convolution result.

[0272] The fourth connection module 1012 is used to perform connection processing on the sixth convolution result and the seventh convolution result to obtain a fourth connection result.

[0273] The twelfth channel-to-pixel module 1013 is used to determine a second fusion processing result C according to the fourth connection result. The second fusion processing result C is output to the detection head network.

[0274] The detection head network can determine the recognition result according to the above second fusion processing result.

[0275] The detection head network may include: at least one detection module. The above detection module can be used to perform classification and regression calculations respectively according to the second fusion processing result by using a convolution module and a convolutional layer to obtain target information such as the category and position of the target in the video stream image.

[0276] In summary, the n feature segmentation modules of the feature segmentation network in the embodiments of the present application can play the role of converging sub-image features at multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features at multiple levels. In this way, the embodiments of the present application can enhance the expression ability and accuracy of the second image features output by the second backbone network, and further improve the target detection accuracy of the video stream image. On this basis, the accuracy of real-time monitoring of the excavation volume can be improved.

[0277] Experimental analysis was carried out on the dual-backbone target detection model of the embodiments of the present application by using a construction scene target detection benchmark dataset. Among them, 19,404 training set pictures that have been labeled in the construction scene target detection benchmark dataset were used as the training dataset, and 4,000 validation set pictures were used as the test dataset. AP (Average Precision) 50 and AP75 were defined as evaluation indicators. Among them, AP50 represents the average precision when the intersection-over-union threshold is 0.5. AP75 represents the average precision when the intersection-over-union threshold is 0.75.

[0278] The experimental results show that the detection accuracy of the embodiments of the present application on the construction scenario target detection benchmark dataset is better than that of the prior art. Taking the AP50 metric as an example, compared with the prior art, the detection accuracies of the first version, the second version, and the third version of the dual-backbone target detection model are improved by 13.6%, 8.7%, and 4.3% respectively, verifying the effectiveness of the dual-backbone target detection model proposed in the present application. Among them, the first version, the second version, and the third version of the dual-backbone target detection model correspond to different parameter quantities.

[0279] In the embodiments of the present application, the training process of the dual-backbone target detection model may include: forward propagation and backward propagation.

[0280] Among them, forward propagation (Forward Propagation) can calculate the prediction information of the target detection result in sequence according to the parameters of the dual-backbone target detection model in the order from the first backbone network to the detection head network. The prediction information is used to determine the loss information.

[0281] Backward propagation (Backward Propagation) can calculate and update the parameters of the dual-backbone target detection model in sequence according to the loss information in the order from the output layer to the input layer. The dual-backbone target detection model usually adopts the structure of a neural network. The parameters of the dual-backbone target detection model may include: parameters such as the weights of the neural network. Among them, during the backward propagation process, the gradient information of the parameters of the dual-backbone target detection model can be determined, and the parameters of the dual-backbone target detection model can be updated using this gradient information. For example, backward propagation can calculate and store the gradient information of the parameters of the target detection model in sequence according to the chain rule in calculus along the order from the detection head network to the neck network, then to the second backbone network, then to the feature segmentation network, and then to the first backbone network.

[0282] In one implementation, the training process of the dual-backbone target detection model includes:

[0283] Step G1, input the video stream image sample into the dual-backbone target detection model, and the dual-backbone target detection model outputs the prediction result corresponding to the video stream image sample; the prediction result includes: the prediction box information corresponding to the construction object category; the video stream image sample corresponds to the ground truth box information;

[0284] Step G2, determine the loss information corresponding to the ground truth box information and the prediction box information according to the minimum point distance function based on the horizontal rectangle;

[0285] Step G3, update the parameters of the dual-backbone target detection model according to the loss information.

[0286] Among them, the process of obtaining the above video stream image sample specifically includes:

[0287] Step H1: Randomly read four original video stream images from the set of video stream images;

[0288] Step H2: Perform random enhancement operations on the four original video stream images respectively to obtain four enhanced video stream images;

[0289] Step H3: Fuse the four enhanced video stream images into a fused image;

[0290] Step H4: Annotate the true box information for the fused image to obtain a video stream image sample.

[0291] The above random enhancement operations specifically include at least one of the following operations: flipping operation, random scaling operation, random hue transformation operation, and random perspective transformation operation.

[0292] The process of Step H3 fusing the four enhanced video stream images into a fused image specifically includes: placing the four enhanced video stream images on an intermediate image according to a preset offset; cropping the part of the intermediate image that exceeds the size range, and shrinking the range of the true box to prevent out-of-bounds, and finally obtaining a fused image. The size range specifically includes: the coordinate range corresponding to [0,0] to [K,K]. K can be a positive integer.

[0293] Of course, the above process of obtaining a video stream image sample through random enhancement operations based on four images is only an example. In fact, those skilled in the art can obtain a video stream image sample in other ways according to actual application requirements.

[0294] For example, in another example of this application, the process of obtaining a video stream image sample specifically includes: randomly reading an original video stream image from the set of video stream images; annotating the true box information for the original video stream image to obtain a video stream image sample.

[0295] Another example, in yet another example of this application, the process of obtaining a video stream image sample specifically includes: randomly reading an original video stream image from the set of video stream images; performing a random enhancement operation on the original video stream image; annotating the true box information for the enhanced image to obtain a video stream image sample.

[0296] Referring to formula (4), it shows the process of determining the loss information corresponding to the true box information and the predicted box information according to the minimum point distance function based on a horizontal rectangle.

[0297]

[0298] where L MPDIoUIndicates loss information; A' represents the ground truth box; B' represents the predicted box; Represents the pixel coordinates of the upper left corner of the ground truth box; Represents the pixel coordinates of the lower right corner of the ground truth box; Represents the pixel coordinates of the upper left corner of the predicted box, Represents the pixel coordinates of the lower right corner of the predicted box; w and h respectively represent the width and height of the video stream image sample; d1 represents the first distance between the pixel coordinates of the upper left corner of the ground truth box and the predicted box; d2 represents the second distance between the pixel coordinates of the lower right corner of the ground truth box and the predicted box.

[0299] Based on the minimum point distance function of the horizontal rectangle, it helps to ensure that the predicted box is geometrically closer to the ground truth box, especially when the predicted box and the ground truth box have the same aspect ratio but different width and height values. By calculating the distances between the upper left and lower right corners of the predicted box and the ground truth box, the metric is optimized. Since the minimum point distance function is more sensitive to the position error of the predicted box, it may help to improve the detection accuracy of the dual-backbone object detection model.

[0300] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present application are not limited by the described action sequence, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present application.

[0301] Based on the above embodiments, this embodiment also provides a real-time excavation volume monitoring device based on dual-backbone object detection. Refer to Figure 12 , the device may specifically include: a collection module 1201, an object detection 1202, an object area determination module 1203, an excavation operation state determination module 1204, an object area excavation volume determination module 1205, and a to-be-statistical construction area excavation volume determination module 1206.

[0302] Among them, the collection module 1201 is used to collect the video stream image of the to-be-statistical construction area;

[0303] The target detection module 1202 is used to perform target detection on the video stream image by using a dual-backbone target detection model to obtain the first target information corresponding to the mining equipment included in the video stream image and the second target information corresponding to the dump truck equipment included in the video stream image; wherein, the dual-backbone target detection model includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network, and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected with a channel-to-pixel module; n is a positive integer greater than 1;

[0304] The target area determination module 1203 is used to determine N target areas in the video stream image according to the first target information and the second target information; N is a positive integer;

[0305] The excavation operation status determination module 1204 is used to, for the current target area among the N target areas, determine the excavation operation status of the current target area according to the operation status of the mining equipment in the current target area, whether the current target area contains a dump truck equipment, and the positional relationship between the mining equipment and the dump truck equipment in the current target area; the excavation operation status includes: one of an unexcavated status, an excavating status, and a loading status;

[0306] The target area excavation volume determination module 1205 is used to, when the excavation operation status of the current target area is updated from the loading status to the unexcavated status or the excavating status, consider that the current target area has completed the current loading, determine the excavation volume corresponding to the current loading of the current target area according to the rated loading capacity of the dump truck equipment in the current target area, and update the cumulative excavation volume of the current target area;

[0307] The total real-time excavation volume determination module 1206 for the construction area to be statistically analyzed is used to determine the total real-time excavation volume corresponding to the construction area to be statistically analyzed according to the cumulative excavation volumes of all the current target areas among the N target areas.

[0308] Optionally, the first target information specifically includes: a first bounding box, and the second target information specifically includes: a second bounding box;

[0309] The target area determination module includes:

[0310] The target area quantity determination module is used to determine the number N of target areas according to the number N of the first bounding boxes;

[0311] The bounding box matching module is used to perform one-to-one matching on the first bounding box and the second bounding box to obtain the corresponding matching result;

[0312] The first determination module is configured to, if there is a second bounding box that matches the first bounding box, use the smallest closed rectangular area that contains both the first bounding box and the second bounding box as the target area;

[0313] The second determination module is configured to, if there is no second bounding box that matches the first bounding box, use the rectangular area corresponding to the first bounding box as the target area.

[0314] Optionally, the bounding box matching module includes:

[0315] The distance intersection over union metric value determination module is configured to determine the distance intersection over union metric value between a first bounding box and a second bounding box;

[0316] The matching determination module is configured to select, according to the distance intersection over union metric values between a first bounding box and multiple second bounding boxes, the second bounding box with the largest distance intersection over union metric value from the multiple second bounding boxes as the second bounding box that matches the first bounding box.

[0317] Optionally, the excavation operation state determination module includes:

[0318] The first judgment module is configured to judge whether the excavation equipment included in the current target area is in a stationary state to obtain a first judgment result;

[0319] The first determination module is configured to, if the first judgment result is yes, determine that the excavation operation state of the current target area is the non-excavated state;

[0320] The second judgment module is configured to, if the first judgment result is no, judge whether a dump truck is included in the current target area to obtain a second judgment result;

[0321] The second determination module is configured to, if the second judgment result is no, determine that the excavation operation state of the current target area is the excavating state;

[0322] The third judgment module is configured to, if the second judgment result is yes, judge whether there is an overlap between the dump truck and the excavation equipment in the current target area to obtain a third judgment result;

[0323] The third determination module is configured to, if the third judgment result is yes, determine that the excavation operation state of the current target area is the loading state;

[0324] The fourth determination module is configured to, if the third judgment result is no, determine that the excavation operation state of the current target area is the excavating state.

[0325] Optionally, the third judgment module includes:

[0326] An intersection over union (IoU) index value determination module, configured to determine the IoU index value between a second bounding box corresponding to a dump truck device and a first bounding box corresponding to an excavation device in the current target area;

[0327] An overlap determination module, configured to determine that there is an overlap between the dump truck device and the excavation device in the current target area if the IoU index value is greater than a first threshold.

[0328] Optionally, the first determination module is specifically configured to: in the video stream images from the i-th frame to the (i + α)-th frame, if the coordinate differences of the excavation devices in any two video stream images are less than a second threshold, determine that the excavation device included in the current target area is in a stationary state; where i and α are positive integers.

[0329] Optionally, the apparatus further includes:

[0330] A model type determination module, configured to use a dump truck device model classification model to determine the target model corresponding to the dump truck device in the current target area;

[0331] A search module, configured to search in the mapping relationship between the search model and the rated loading capacity according to the target model, and determine the rated loading capacity corresponding to the dump truck device in the current target area.

[0332] In summary, for the real-time excavation volume monitoring apparatus based on dual-backbone object detection in the embodiments of the present application, since the target information corresponding to objects such as excavation devices and dump truck devices in the video stream images is used to determine N target areas in the video stream images, and according to the objective laws between the excavation operation state of the target area and the operation state of the excavation device, the existence of the dump truck device, and the positional relationship between the excavation device and the dump truck device, the excavation volume corresponding to the current loading of the current target area is determined; in this way, the embodiments of the present application can continuously determine the target areas existing in the video stream images and the operation states corresponding to the target areas according to the continuously updated video stream images; on this basis, the embodiments of the present application can continuously determine the excavation volume corresponding to the current loading of the current target area in the video stream images and the cumulative excavation volume of all current target areas in the video stream images. Therefore, the embodiments of the present application can realize the real-time monitoring of the earthwork excavation volume at a low cost and at a high speed based on the analysis of the video stream images. In other words, the embodiments of the present application can save the labor cost, data acquisition cost, and data processing time cost consumed by using the drone oblique photography or 3D laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation volume.

[0333] Moreover, the embodiments of the present application utilize a dual-backbone object detection model to perform object detection on video image frames. Among them, the n feature segmentation modules included in the feature segmentation network in the above dual-backbone object detection model can play a role in aggregating sub-image features at multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features at multiple levels. In this way, the embodiments of the present application can enhance the expression ability and accuracy of the second image features output by the second backbone network, and thus can improve the accuracy of the object detection results.

[0334] The embodiments of the present application also provide a non-volatile readable storage medium, in which one or more modules (programs) are stored. When the one or more modules are applied to a device, the device can be caused to execute instructions (instructions) for each method step in the embodiments of the present application.

[0335] The embodiments of the present application provide one or more machine-readable media, on which instructions are stored. When executed by one or more processors, an electronic device is caused to execute the methods as described in one or more of the above embodiments. In the embodiments of the present application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0336] The embodiments of the present disclosure can be implemented as a device configured as desired using any suitable hardware, firmware, software, or any combination thereof. The device may include: electronic devices such as terminal devices and servers (clusters). Figure 13 Schematically shows an exemplary device 1300 that can be used to implement the various embodiments described in the present application.

[0337] For one embodiment, Figure 13 Shows an exemplary device 1300, which has one or more processors 1302, a control module (chipset) 1304 coupled to at least one of the (one or more) processors 1302, a memory 1306 coupled to the control module 1304, an NVM (non-volatile memory) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0338] The processor 1302 may include one or more single-core or multi-core processors. The processor 1302 may include any combination of general-purpose processors or dedicated processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1300 can act as the terminal device, server (cluster), etc. described in the embodiments of the present application.

[0339] In some embodiments, apparatus 1300 may include one or more computer-readable media (e.g., memory 1306 or non-volatile memory / storage device 1308) having instructions 1314, and one or more processors 1302 coupled to the one or more computer-readable media and configured to execute the instructions 1314 to implement modules to perform the actions described in this disclosure.

[0340] For one embodiment, control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with control module 1304.

[0341] Control module 1304 may include a memory controller module to provide an interface to memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0342] Memory 1306 may be used, for example, to load and store data and / or instructions 1314 for apparatus 1300. For one embodiment, memory 1306 may include any suitable volatile memory, such as, for example, suitable DRAM (Dynamic Random Access Memory). In some embodiments, memory 1306 may include double data rate type four synchronous dynamic random access memory.

[0343] For one embodiment, control module 1304 may include one or more input / output controllers to provide an interface to non-volatile memory / storage device 1308 and the input / output device(s) 1310.

[0344] For example, non-volatile memory / storage device 1308 may be used to store data and / or instructions 1314. Non-volatile memory / storage device 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile disk drives).

[0345] Non-volatile memory / storage device 1308 may include storage resources that are physically part of a device on which apparatus 1300 is installed, or it may be accessible by the device without being part of the device. For example, non-volatile memory / storage device 1308 may be accessed via the input / output device(s) 1310 over a network.

[0346] (One or more) input / output devices 1310 may provide an interface for device 1300 to communicate with any other suitable devices. The input / output devices 1310 may include communication components, audio components, sensor components, etc. The network interface 1312 may provide an interface for device 1300 to communicate through one or more networks. Device 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing a wireless network based on communication standards, such as WiFi (Wireless Fidelity), 2G (2-Generation wireless telephone technology), 3G (3-Generation wireless telephone technology), 4G (4-Generation wireless telephone technology), 5G (5-Generation wireless telephone technology), etc., or a combination thereof for wireless communication.

[0347] For one embodiment, at least one of (one or more) processors 1302 may be logically encapsulated with one or more controllers (e.g., memory controller modules) of the control module 1304. For one embodiment, at least one of (one or more) processors 1302 may be logically encapsulated with one or more controllers of the control module 1304 to form a system-in-package. For one embodiment, at least one of (one or more) processors 1302 may be logically integrated with one or more controllers of the control module 1304 on the same die. For one embodiment, at least one of (one or more) processors 1302 may be logically integrated with one or more controllers of the control module 1304 on the same die to form a system-on-chip.

[0348] In various embodiments, device 1300 may be, but is not limited to: a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touchscreen device, a netbook, etc.) and other terminal devices. In various embodiments, device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, device 1300 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit, and a speaker.

[0349] Among them, a main control chip can be used as a processor or a control module in the detection device. Sensor data, position information, etc. are stored in a memory or a non-volatile memory / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0350] For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments.

[0351] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.

[0352] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and the combination of processes and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for realizing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0353] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device realizes the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0354] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable terminal device provide steps for realizing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0355] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0356] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without more limitations, an element defined by the statement "including one..." does not exclude the presence of additional identical elements in the process, method, article or terminal device including the said element.

[0357] The above has introduced in detail a real-time monitoring method and device for excavation volume based on dual-backbone object detection, an electronic device and a machine-readable medium provided by the present application. Specific examples are used in this text to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scenarios. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A real-time monitoring method for excavation volume based on dual-backbone object detection, characterized in that The method includes: Collecting video stream images of the construction area to be counted; Using a dual-backbone object detection model to perform object detection on the video stream images to obtain first target information corresponding to the excavation equipment included in the video stream images and second target information corresponding to the dump trucks included in the video stream images; wherein, the dual-backbone object detection model includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network, and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected with a channel-to-pixel module; n is a positive integer greater than 1; the first backbone network determines n first image features corresponding to the video stream images; the i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, and the segmentation processing result includes: i sub-image features, and outputs the j-th sub-image feature to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i sub-image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers; Determining N target areas in the video stream images according to the first target information and the second target information; N is a positive integer; For the current target area among the N target areas, determining the excavation operation state of the current target area according to the operation state of the excavation equipment in the current target area, whether there is a dump truck in the current target area, and the positional relationship between the excavation equipment and the dump truck in the current target area; the excavation operation state includes: one of an unexcavated state, an excavating state, and a loading state; When the excavation operation state of the current target area is updated from the loading state to the unexcavated state or the excavating state, it is considered that the current target area has completed the current loading, determining the excavation volume corresponding to the current loading of the current target area according to the rated loading capacity of the dump truck corresponding to the current target area, and updating the cumulative excavation volume of the current target area; Determining the total real-time excavation volume corresponding to the construction area to be counted according to the cumulative excavation volumes of all the current target areas among the N target areas.

2. The method according to claim 1, wherein The first target information includes: a first bounding box, and the second target information includes: a second bounding box; The determining N target areas in the video stream images according to the first target information and the second target information includes: Determining the number N of target areas according to the number N of the first bounding boxes; Performing one-to-one matching on the first bounding boxes and the second bounding boxes to obtain corresponding matching results; If there is a second bounding box that matches a first bounding box, taking the smallest closed rectangle area that contains both the first bounding box and the second bounding box as the target area; If there is no second bounding box that matches the first bounding box, the rectangular area corresponding to the first bounding box is used as the target area.

3. The method according to claim 2, characterized in that, The one-to-one matching of the first bounding box and the second bounding box includes: Determining the distance intersection over union index value between a first bounding box and a second bounding box; According to the distance intersection over union index values between a first bounding box and multiple second bounding boxes, selecting the one with the largest distance intersection over union index value from the multiple second bounding boxes as the second bounding box that matches the first bounding box.

4. The method according to claim 1, wherein For the current target area among the N target areas, determining the excavation operation state of the current target area according to the operation state of the excavation equipment in the current target area, whether the current target area contains a dump truck, and the positional relationship between the excavation equipment and the dump truck in the current target area, includes: Judging whether the excavation equipment included in the current target area is in a stationary state to obtain a first judgment result; If the first judgment result is yes, it is determined that the excavation operation state of the current target area is the unexcavated state; If the first judgment result is no, then judging whether the current target area contains a dump truck to obtain a second judgment result; If the second judgment result is no, it is determined that the excavation operation state of the current target area is the in-excavation state; If the second judgment result is yes, then judging whether there is an overlap between the dump truck and the excavation equipment in the current target area to obtain a third judgment result; If the third judgment result is yes, it is determined that the excavation operation state of the current target area is the loading state; If the third judgment result is no, it is determined that the excavation operation state of the current target area is the in-excavation state.

5. The method according to claim 4, characterized in that The judging whether there is an overlap between the dump truck and the excavation equipment in the current target area includes: Determining the intersection over union index value between the second bounding box corresponding to the dump truck and the first bounding box corresponding to the excavation equipment in the current target area; If the intersection over union index value is greater than the first threshold, it is determined that there is an overlap between the dump truck and the excavation equipment in the current target area.

6. The method according to claim 4, characterized in that, The judging whether the excavation equipment included in the current target area is in a stationary state includes: In the i-th frame to the (i + α)-th frame of video stream images, if the coordinate differences of the excavation equipment in any two frames of video stream images are both less than the second threshold, it is determined that the excavation equipment included in the current target area is in a stationary state; where i and α are positive integers.

7. The method according to claim 1, wherein The method further includes: Using a dump truck model classification model to determine the target model corresponding to the dump truck in the current target area; According to the target model, searching in the mapping relationship between the model and the rated loading capacity to determine the rated loading capacity corresponding to the dump truck in the current target area.

8. The method according to any one of claims 1 to 7, characterized in that The using the dual-backbone object detection model to perform object detection on the video stream images includes: The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result. The channel-to-pixel module in the second backbone network determines a second image feature based on the first fusion result and outputs the second image feature to the neck network; The neck network performs a second fusion process on the second image feature to obtain a second fusion result; The detection head network determines the first target information and the second target information based on the second fusion result.

9. The method according to claim 8, wherein The i-th feature segmentation module in the feature segmentation network performs a convolution process and a segmentation process on the i-th type of first image feature, including: The i-th feature segmentation module in the feature segmentation network performs a convolution process on the i-th type of first image feature to obtain an image feature with c channels; According to the channel dimension, the image features with c channels are segmented into i sub-image features with different numbers of channels; the number of channels of the j-th sub-image feature among the i sub-image features is: t * 2 (j-1) .

10. The method according to claim 8, wherein The second backbone network further includes: a first convolution module connected in front of the feature fusion module; The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature, including: The i-th feature fusion module receives n - i + 1 sub-image features and the convolution image feature output by the first convolution module; the number of channels corresponding to the n - i + 1 sub-image features is the first channel number, and the number of channels corresponding to the convolution image feature is the second channel number; The i-th feature fusion module performs an interpolation process on the n - i + 1 sub-image features, and the number of channels corresponding to the n - i + 1 sub-image features after the interpolation process is the second channel number; The i-th feature fusion module performs a fusion process on the n - i + 1 sub-image features after the interpolation process and the convolution image feature.

11. The method according to claim 10, characterized in that, The first convolution module connected in front of the first feature fusion module is used to perform a convolution process on the video stream image and output the corresponding convolution image feature to the first feature fusion module.

12. The method according to claim 8, wherein The n feature segmentation modules include: a first feature segmentation module, a second feature segmentation module, a third feature segmentation module, a fourth feature segmentation module, and a fifth feature segmentation module; the n feature fusion modules include: a first feature fusion module, a second feature fusion module, a third feature fusion module, a fourth feature fusion module, and a fifth feature fusion module; Among them, the segmentation result of the first feature segmentation module includes: 1 type of m-channel first sub-image feature, and 1 type of m-channel first sub-image feature is output to the first feature fusion module; The segmentation result of the second feature segmentation module includes: 1 type of m-channel second sub-image feature and 1 type of 2m-channel third sub-image feature; 1 type of m-channel second sub-image feature is output to the first feature fusion module; 1 type of 2m-channel third sub-image feature is output to the second feature fusion module; The segmentation processing results of the third feature segmentation module include: one type of fourth sub-image feature with m channels, one type of fifth sub-image feature with 2m channels, and one type of sixth sub-image feature with 4m channels; one type of fourth sub-image feature with m channels is output to the first feature fusion module; one type of fifth sub-image feature with 2m channels is output to the second feature fusion module; one type of sixth sub-image feature with 4m channels is output to the third feature fusion module; The segmentation processing results of the fourth feature segmentation module include: one type of seventh sub-image feature with m channels, one type of eighth sub-image feature with 2m channels, one type of ninth sub-image feature with 4m channels, and one type of tenth sub-image feature with 8m channels; one type of seventh sub-image feature with m channels is output to the first feature fusion module; one type of eighth sub-image feature with 2m channels is output to the second feature fusion module; one type of ninth sub-image feature with 4m channels is output to the third feature fusion module; one type of tenth sub-image feature with 8m channels is output to the fourth feature fusion module; The segmentation processing results of the fifth feature segmentation module include: one type of eleventh sub-image feature with m channels, one type of twelfth sub-image feature with 2m channels, one type of thirteenth sub-image feature with 4m channels, one type of fourteenth sub-image feature with 8m channels, and one type of fifteenth sub-image feature with 16m channels; one type of eleventh sub-image feature with m channels is output to the first feature fusion module; one type of twelfth sub-image feature with 2m channels is output to the second feature fusion module; one type of thirteenth sub-image feature with 4m channels is output to the third feature fusion module; one type of fourteenth sub-image feature with 8m channels is output to the fourth feature fusion module; one type of fifteenth sub-image feature with 16m channels is output to the fifth feature fusion module.

13. A real-time monitoring device for excavation volume based on dual-backbone object detection, characterized in that, The device includes: An acquisition module, configured to acquire video stream images of the construction area to be counted; A target detection module, which is used to perform target detection on the video stream image by using a dual-backbone target detection model to obtain the first target information corresponding to the mining equipment included in the video stream image and the second target information corresponding to the dump truck equipment included in the video stream image; wherein, the dual-backbone target detection model includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network, and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected with a channel-to-pixel module; n is a positive integer greater than 1; the first backbone network determines n first image features corresponding to the video stream image; the i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, and the segmentation processing result includes: i sub-image features, and outputs the j-th sub-image feature to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i sub-image features output by the same i-th feature segmentation module are sub-image features with different numbers of channels; one feature fusion module corresponds to one number of channels, and different feature fusion modules correspond to different numbers of channels; A target area determination module, which is used to determine N target areas in the video stream image according to the first target information and the second target information; N is a positive integer; An excavation operation state determination module, which is used to determine the excavation operation state of the current target area in the N target areas according to the operation state of the mining equipment in the current target area, whether the current target area contains a dump truck equipment, and the positional relationship between the mining equipment and the dump truck equipment in the current target area; the excavation operation state includes: one of an unexcavated state, an excavating state, and a loading state; A target area excavation volume determination module, which is used to, when the excavation operation state of the current target area is updated from the loading state to the unexcavated state or the excavating state, consider that the current target area has completed the current loading, determine the excavation volume corresponding to the current loading of the current target area according to the rated loading capacity of the dump truck equipment in the current target area, and update the cumulative excavation volume of the current target area; A total real-time excavation volume determination module for the construction area to be statistically determined, which is used to determine the total real-time excavation volume corresponding to the construction area to be statistically determined according to the cumulative excavation volumes of all current target areas in the N target areas.

14. An electronic device, characterized in that, Comprising: A processor; And A memory, on which executable code is stored, and when the executable code is executed, the processor is caused to execute the method according to any one of claims 1-12.

15. A machine-readable medium, on which executable code is stored, and when the executable code is executed, the processor is caused to execute the method according to any one of claims 1-12.

Images