Shield slag volume calculation method and equipment based on machine vision

Abstract

The present application discloses a method and device for calculating the volume of shield muck based on machine vision, which relates to the field of image processing. The method comprises: extracting and fusing features of a target depth image and a target RGB image acquired in real time through an image segmentation network based on dual-modal input, and performing image segmentation on the target RGB image according to the obtained fusion features to obtain a muck mask image and the coordinates of a muck boundary box of the target RGB image; calculating the volume of the shield muck by a volume calculation method according to the coordinates of the muck boundary box and the muck mask image; the present application improves the accuracy of the volume calculation of the shield muck on the basis of having good real-time performance, and solves the problem of low accuracy of the shield muck volume calculated by the existing non-contact measurement technology of the shield muck volume under high real-time requirements.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and device for calculating the volume of shield tunneling muck based on machine vision. Background Art

[0002] The Tunnel Boring Machine (TBM) is a large-scale underground construction equipment that integrates multiple functions, including rock tunneling, support, ventilation, and dust removal. It can form a full tunnel section in a single operation by using a roller cutter to break rock. Compared to traditional drill-and-blast methods, the shield method can achieve tunneling speeds 3 to 10 times faster, with high-quality tunnels and minimal environmental disturbance. Currently, shield technology is evolving towards intelligentization. By integrating cutting-edge technologies such as machine vision and deep learning, it is striving to overcome key challenges such as adaptive tunneling in complex strata and unmanned construction. To address the issue of excavation debris generated during excavation, real-time monitoring of the excavation debris volume flow not only reflects excavation efficiency but also establishes a rock-soil-equipment feedback mechanism to enhance project safety and intelligence.

[0003] Currently, a commonly used non-contact measurement technology for shield tunneling muck volume is to collect the volume of shield tunneling muck in a muck truck compartment using a camera. This technology uses a camera to capture an image of the muck truck compartment, segment the captured image to obtain a muck mask image, and finally calculate the volume of the shield tunneling muck based on the muck mask image. However, the inventors of this application have discovered that:

[0004] The existing non-contact measurement method for shield tunneling muck volume has low segmentation accuracy for the muck truck compartment image under high real-time requirements, resulting in low accuracy of the calculated shield tunneling muck volume under high real-time requirements. Summary of the Invention

[0005] The purpose of this application is to provide a method and device for calculating the volume of shield muck based on machine vision, so as to solve the problem of low accuracy of the shield muck volume calculated by the existing non-contact measurement technology under high real-time requirements.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a method for calculating the volume of shield muck based on machine vision, comprising:

[0008] Through the image segmentation network based on dual-modal input, feature extraction and fusion are performed on the target depth image and target RGB image acquired in real time, and the target RGB image is segmented according to the obtained fusion features to obtain the slag mask image and the coordinates of the slag boundary box of the target RGB image;

[0009] According to the coordinates of the muck boundary box and the muck mask image, the volume of the shield muck is calculated by the volume calculation method.

[0010] In the second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that the processor executes the computer program to implement the steps of any one of the above-mentioned methods for calculating shield slag volume based on machine vision.

[0011] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0012] The present application provides a method and device for calculating the volume of shield slag based on machine vision, which obtains the target depth image and target RGB image in real time through a depth camera, and extracts and fuses the features of the target depth image and target RGB image obtained in real time through an image segmentation network based on dual-modal input, and performs image segmentation on the target RGB image according to the obtained fusion features to obtain the slag mask image and the coordinates of the slag boundary box of the target RGB image, and calculates the volume of the shield slag through a volume calculation method based on the coordinates of the slag boundary box and the slag mask image, thereby realizing the real-time calculation of the shield slag volume and ensuring the real-time nature of the calculation results; since the boundary colors of the slag and the slag truck compartment are similar, but the depth information is different, the image segmentation network based on dual-modal input supports dual-modal image (RGB image and depth image) input, and the real-time The acquired target depth image and target RGB image are subjected to feature extraction and fusion, and the dual-modal features (RGB features and depth features) can be complementary fused. The obtained fusion features can more comprehensively cover the features of the slag and the slag truck compartment, making the segmentation network more accurate. Furthermore, the target RGB image is segmented according to the obtained fusion features by the image segmentation network based on the dual-modal input, which can improve the segmentation accuracy of the target RGB image, and thus improve the accuracy of the coordinates of the slag mask image and the slag boundary box of the target RGB image, thereby improving the accuracy of the volume of the shield slag calculated by the volume calculation method based on the coordinates of the slag boundary box and the slag mask image. In summary, the present application realizes the real-time and high-precision calculation of the shield slag volume, and solves the problem that the shield slag volume calculated by the existing non-contact measurement technology of the shield slag volume has low accuracy under high real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 A flowchart of a method for calculating shield tunneling muck volume based on machine vision provided in one embodiment of the present application;

[0015] Figure 2 A schematic diagram of the structure of a backbone network of an image segmentation network based on dual-modal input provided by one embodiment of the present application;

[0016] Figure 3 A schematic diagram of the structure of an image segmentation network based on dual-modal input provided in one embodiment of the present application;

[0017] Figure 4 Provided for an embodiment of this application Figure 2 A schematic structural diagram of the first bimodal feature fusion module;

[0018] Figure 5 A schematic diagram of the shape of a first infinitesimal element provided in one embodiment of the present application;

[0019] Figure 6 A schematic diagram of the shape of a second infinitesimal element provided in one embodiment of the present application;

[0020] Figure 7 A schematic diagram of the functional modules of a shield tunneling muck volume calculation device based on machine vision provided in one embodiment of the present application;

[0021] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0024] In an exemplary embodiment, Figure 1As shown, a method for calculating the volume of shield muck based on machine vision is provided, which is executed by a computer device and includes the following steps 101 to 102.

[0025] In step 101, a dual-modal input-based image segmentation network is used to extract and fuse features of the target depth image and target RGB image acquired in real time, and the target RGB image is segmented based on the obtained fusion features to obtain the slag mask image and the coordinates of the slag boundary box of the target RGB image.

[0026] In the embodiment of the present application, the target depth image and the target RGB image are depth images and RGB images obtained by photographing the muck truck compartment at the slag outlet of the shield machine using a depth camera installed at the slag outlet of the shield machine.

[0027] RGB images can provide appearance features such as texture and color, but they can easily lead to missegmentation of objects with similar colors and textures. Depth images reflect the depth of objects from the depth camera and provide geometric structural features. Because the boundaries between muck and the muck truck compartment have similar colors but different depth information, the image segmentation network based on bimodal input supports bimodal image input (RGB and depth images). This bimodal image segmentation network extracts and fuses features from the target depth image and target RGB image, achieving a complementary fusion of bimodal features (RGB and depth). The resulting fused features more comprehensively cover the features of both the muck and the muck truck compartment, making the segmentation network more accurate. Furthermore, the bimodal image segmentation network uses the fused features to segment the target RGB image, improving segmentation accuracy and reducing the error between the muck mask image and the target RGB image, thereby improving the accuracy of calculated shield muck volumes. The muck bounding box is the minimum rectangular box enclosing the muck area in the target RGB image. The coordinates of the muck bounding box include the coordinates of the four vertices of the muck bounding box.

[0028] Step 102 : Calculate the volume of the muck using a volume calculation method based on the coordinates of the muck boundary box and the muck mask image.

[0029] In the embodiment of the present application, the volume calculation method is not limited and can be set according to actual needs. For example, the volume of shield slag can be calculated using the existing volume calculation method.

[0030] By implementing the above steps 101 to 102, the target depth image and target RGB image are acquired in real time, and feature extraction and fusion are performed on the target depth image and target RGB image acquired in real time through an image segmentation network based on dual-modal input. The target RGB image is segmented according to the obtained fusion features to obtain a slag mask image of the target RGB image and the coordinates of the slag boundary box. The volume of the shield slag is calculated by a volume calculation method based on the coordinates of the slag boundary box and the slag mask image. Therefore, the real-time calculation of the shield slag volume is realized, and the real-time nature of the calculation results is ensured. Since the boundary colors of the slag and the slag truck compartment are similar, but the depth information is different, the image segmentation network based on dual-modal input supports dual-modal image (RGB image and depth image) input. Through the image segmentation network based on dual-modal input, the target depth image and target RGB image acquired in real time are extracted and fused, and the dual-modal features (RGB features and depth features) can be complementary fused. The obtained fusion features can more comprehensively cover the features of the slag and the slag truck compartment, making the segmentation network more accurate. Furthermore, by using the image segmentation network based on dual-modal input to segment the target RGB image according to the obtained fusion features, the segmentation accuracy of the target RGB image can be improved, and the accuracy of the coordinates of the slag mask image and the slag bounding box of the target RGB image can be improved, thereby improving the accuracy of the volume of the shield slag calculated by the volume calculation method based on the coordinates of the slag bounding box and the slag mask image. In summary, the present application realizes real-time and high-precision calculation of the volume of shield slag, solving the problem that the volume of shield slag calculated by the existing non-contact measurement technology of the shield slag volume is low under high real-time requirements. In addition, the image segmentation network based on bimodal input (with reduced parameters) improves the segmentation speed of the target RGB image, thereby improving the calculation speed of the shield slag volume.

[0031] In another exemplary embodiment of the present application, in order to improve the segmentation accuracy of the image segmentation network based on dual-modal input, the image segmentation network based on dual-modal input in step 101 includes an input layer, a backbone network, a neck network and a segmentation layer connected in sequence. The input layer is used to merge channels of each input target RGB image and the corresponding target depth image, and convert the feature map obtained by merging the channels into a matrix , B is the batch dimension, which is the number of target RGB images (consistent with the number of target depth images) that are input into the image segmentation network based on bimodal input at one time. H is the width of the feature map obtained by merging channels, W is the length of the feature map obtained by merging channels. The backbone network is used to Bimodal features are extracted and fused to produce shallow and deep features. These bimodal features include deep image features and RGB image features. The neck network is used to perform a cross-scale fusion operation on the shallow and deep features output by the backbone network, generating fused features at the same scale as the shallow and deep features, respectively. The segmentation layer is used to generate and output at least the coordinates of the soil debris bounding box and the soil debris mask image in the target RGB image based on the fused features output by the neck network.

[0032] In the embodiment of the present application, the target depth image corresponding to each target RGB image refers to the depth image collected at the same time as each target RGB image. Specifically, the target depth image corresponding to each target RGB image refers to the depth image with the same name as each target RGB image.

[0033] In another exemplary embodiment of the present application, in order to further improve the segmentation accuracy of the improved image segmentation algorithm, as shown in FIG. Figure 2 As shown, the backbone network includes a first feature extraction module 201, a second feature extraction module 202, a first bimodal feature fusion module 203, a third feature extraction module 204, a fourth feature extraction module 205, a second bimodal feature fusion module 206, a fifth feature extraction module 207, a sixth feature extraction module 208, a third bimodal feature fusion module 209 and an SPPF spatial pyramid pooling module 2010. The first feature extraction module 201 is used to separate the matrix The second feature extraction module 202 is used to separate the matrix The first bimodal feature fusion module 203 is used to perform feature fusion based on the triple attention mechanism on the bimodal features output by the first feature extraction module 201 and the second feature extraction module 202 to obtain first shallow features (P3). The third feature extraction module 204 is used to perform feature extraction on the output of the first feature extraction module 201. The fourth feature extraction module 205 is used to perform feature extraction on the output of the first bimodal feature fusion module 203. The second bimodal feature fusion module 206 is used to perform feature fusion based on the triple attention mechanism on the bimodal features output by the third and fourth feature extraction modules 204, 205 to obtain second shallow features (P4). The fifth feature extraction module 207 is used to perform feature extraction on the output of the third feature extraction module 204. The sixth feature extraction module 208 is used to perform feature extraction on the output of the second bimodal feature fusion module 206. The third bimodal feature fusion module 209 is used to perform feature fusion based on the triple attention mechanism on the bimodal features output by the fifth feature extraction module 207 and the sixth feature extraction module 208. The SPPF spatial pyramid pooling module 2010 is used to perform pooling operations of different sizes on the output of the third bimodal feature fusion module 209 and to concatenate the feature maps obtained by the pooling operations of different sizes in the channel dimension to obtain the aforementioned deep features (P5).

[0034] In another exemplary embodiment of the present application, in order to further improve the segmentation accuracy of the improved image segmentation algorithm, as shown in FIG. Figure 3 As shown, the first feature extraction module 201 includes the seventh feature extraction module and the eighth feature extraction module. The seventh feature extraction module includes the first separation module D-part, the first convolution module, the second convolution module and the first cross-stage feature fusion module (C2f) connected in sequence. Among them, the first separation module D-part is used to separate the matrix The first convolution module is used to perform a convolution operation on the feature map output by the first separation module D-part. The second convolution module is used to perform a convolution operation on the feature map output by the first convolution module. The first cross-stage feature fusion module (C2f) is used to extract features of different sizes from the feature map output by the second convolution module and perform cross-stage feature fusion on the extracted feature maps of different scales.

[0035] The eighth feature extraction module includes a third convolution module and a second cross-stage feature fusion module. The third convolution module is used to perform a convolution operation on the output of the first cross-stage feature fusion module. The second cross-stage feature fusion module is used to extract features of different sizes on the feature map output by the third convolution module and perform cross-stage feature fusion on the extracted feature maps of different scales.

[0036] In another exemplary embodiment of the present application, in order to further improve the segmentation accuracy of the improved image segmentation algorithm, as shown in FIG. Figure 3 As shown, the second feature extraction module 202 includes the first feature extraction module 201, the ninth feature extraction module, the fourth dual-modal feature fusion module ( Figure 3 The first BTA) and the tenth feature extraction module. The ninth feature extraction module includes the second separation module RGB-part, the fourth convolution module, the fifth convolution module, and the third cross-stage feature fusion module (C2f) connected in sequence. Among them, the second separation module RGB-part is used to separate the matrix The target RGB image part. The fourth convolution module is used to perform a convolution operation on the feature map output by the second separation module RGB-part. The fifth convolution module is used to perform a convolution operation on the feature map output by the third convolution module. The third cross-stage feature fusion module (C2f) is used to extract features of different sizes on the feature map output by the fifth convolution module and perform cross-stage feature fusion on the extracted feature maps of different scales. The fourth bimodal feature fusion module is used to perform feature fusion based on the triple attention mechanism on the output of the seventh feature extraction module of the first feature extraction module 201 and the output of the third cross-stage feature fusion module. The tenth feature extraction module includes a sixth convolution module and a fourth cross-stage feature fusion module. The sixth convolution module is used to perform a convolution operation on the output of the fourth bimodal feature fusion module. The fourth cross-stage feature fusion module is used to extract features of different sizes on the feature map output by the sixth convolution module and perform cross-stage feature fusion on the extracted feature maps of different scales.

[0037] In the embodiment of the present application, the output of the second cross-stage feature fusion module and the output of the fourth bimodal feature fusion module are used as the input of the first bimodal feature fusion module, and the first bimodal feature fusion module outputs P3.

[0038] The feature maps output by the second separation module RGB-part and the first separation module D-part have the same size. The feature maps output by the first cross-stage feature fusion module and the third cross-stage feature fusion module have the same size. The feature maps output by the second cross-stage feature fusion module and the fourth cross-stage feature fusion module have the same size. The fourth bimodal feature fusion module has the same structure as the first bimodal feature fusion module, the second bimodal feature fusion module, and the third bimodal feature fusion module.

[0039] In another exemplary embodiment of the present application, in order to further improve the segmentation accuracy of the improved image segmentation algorithm, as shown in FIG. Figure 3As shown, the third feature extraction module 204 includes a seventh convolution module and a fifth cross-stage feature fusion module. The seventh convolution module is used to perform a convolution operation on the output of the second cross-stage feature fusion module, and the fifth cross-stage feature fusion module is used to extract features of different sizes on the output of the seventh convolution module and perform cross-stage feature fusion on the feature maps of different scales extracted.

[0040] The fourth feature extraction module 205 includes an eighth convolution module and a sixth cross-stage feature fusion module. The eighth convolution module is used to perform a convolution operation on the output of the first bimodal feature fusion module. The sixth cross-stage feature fusion module is used to extract features of different sizes on the output of the eighth convolution module and perform cross-stage feature fusion on the feature maps of different scales extracted.

[0041] In the embodiment of the present application, the feature map sizes output by the fifth cross-stage feature fusion module and the sixth cross-stage feature fusion module are consistent.

[0042] The output of the fifth cross-stage feature fusion module and the output of the sixth cross-stage feature fusion module are used as the input of the second bimodal feature fusion module, and the second bimodal feature fusion module outputs P4.

[0043] In another exemplary embodiment of the present application, in order to further improve the segmentation accuracy of the improved image segmentation algorithm, as shown in FIG. Figure 3 As shown, the fifth feature extraction module 207 includes a ninth convolution module and a seventh stage feature fusion module. The ninth convolution module is used to perform a convolution operation on the output of the fifth cross-stage feature fusion module. The seventh cross-stage feature fusion module is used to extract features of different sizes on the output of the ninth convolution module and perform cross-stage feature fusion on the feature maps of different scales obtained by extraction.

[0044] The sixth feature extraction module 208 includes a tenth convolution module and an eighth cross-stage feature fusion module. The tenth convolution module is used to perform a convolution operation on the output of the second bimodal feature fusion module. The seventh cross-stage feature fusion module is used to extract features of different sizes on the output of the tenth convolution module and perform cross-stage feature fusion on the feature maps of different scales extracted.

[0045] In the embodiment of the present application, the feature map sizes output by the seventh cross-stage feature fusion module and the eighth cross-stage feature fusion module are consistent.

[0046] The output of the seventh cross-stage feature fusion module and the output of the eighth cross-stage feature fusion module are used as the input of the second bimodal feature fusion module.

[0047] In another exemplary embodiment of the present application, in order to further improve the segmentation accuracy of the improved image segmentation algorithm, as shown in FIG. Figure 3As shown, the SPPF spatial pyramid pooling module 2010 includes an eleventh convolution module, multiple serial maximum pooling layers, a first connection layer and a twelfth convolution module in sequence. Among the multiple serial maximum pooling layers, the other maximum pooling layers except the first maximum pooling layer are used to splice the output of the eleventh convolution module and the output of the previous maximum pooling layer and then further perform a maximum pooling operation.

[0048] In the embodiment of the present application, the twelfth convolution module outputs the deep feature P5.

[0049] In another exemplary embodiment of the present application, in order to further improve the segmentation accuracy of the improved image segmentation algorithm, as shown in FIG. Figure 3 As shown, the neck network (Neck) includes a second connection layer, a ninth cross-stage feature fusion module, a thirteenth convolution module, a third connection layer, a tenth cross-stage feature fusion module, a first upsampling module, a fourth connection layer, an eleventh cross-stage feature fusion module, a fourteenth convolution module, a second upsampling module, a fourth connection layer, an eleventh cross-stage feature fusion module, a fourteenth convolution module, a second upsampling module, a fifth connection layer, and a twelfth cross-stage feature fusion module. The second upsampling module is used to upsample the output of the SPPF spatial pyramid pooling module 2010. The third connection layer is used to concatenate the output of the second upsampling module with the output of the second bimodal feature fusion module. The tenth cross-stage feature fusion module is used to extract features of different scales from the output of the third connection layer and perform cross-stage feature fusion on the extracted feature maps of different scales. The first upsampling module is used to upsample the output of the tenth cross-stage feature fusion module. The second connection layer is used to concatenate the output of the first bimodal feature fusion module with the output of the first upsampling module. The ninth cross-stage feature fusion module is used to extract features of different sizes on the output of the second connection layer and perform cross-stage feature fusion on the feature maps of different scales obtained by the extraction. The thirteenth convolution module is used to perform a convolution operation on the output of the ninth cross-stage feature fusion module. The fourth connection layer is used to splice the output of the thirteenth convolution module and the output of the tenth cross-stage feature fusion module. The eleventh cross-stage feature fusion module is used to extract features of different sizes on the output of the fourth connection layer and perform cross-stage feature fusion on the feature maps of different scales obtained by the extraction. The fourteenth convolution module is used to perform a convolution operation on the output of the eleventh cross-stage feature fusion module. The fifth connection layer is used to splice the output of the second upsampling module and the output of the fourteenth convolution module. The twelfth cross-stage feature fusion module is used to extract features of different sizes on the output of the fifth connection layer and perform cross-stage feature fusion on the feature maps of different scales obtained by the extraction.

[0050] In the embodiment of the present application, the output of the ninth cross-stage feature fusion module, the output of the eleventh cross-stage feature fusion module, and the output of the twelfth cross-stage feature fusion module are used as inputs of the segmentation layer.

[0051] In another exemplary embodiment of the present application, in order to further improve the segmentation accuracy of the improved image segmentation algorithm, as shown in FIG. Figure 3 As shown, the segmentation layer includes three Segment modules. The first Segment module performs a convolution operation on the first shallow-layer input features, predicts a soil bounding box, and outputs the pixel coordinates of the four vertices of the predicted soil bounding box. The second Segment module performs a convolution operation on the second shallow-layer input features, predicts and outputs the category of the target object in each target RGB image (the category includes soil, for example, setting the category label of soil to 0). The third Segment module performs a convolution operation on the deep-layer input features, predicts and outputs the mask coefficients for each target RGB image.

[0052] In another exemplary embodiment of the present application, Figure 3 As shown, the structures of the first to twelfth cross-stage feature fusion modules are the same, and all include a fifteenth convolution module, a Split module, a plurality of serial BottleNeck modules, a sixth connection layer, and a sixteenth convolution module connected in sequence. The spilt module is used to convert the output of the fifteenth convolution module (of shape I B×C×H×W ) is divided into I1 along the channel dimension C B×C / 2×H×W and I2 B×C / 2×H×W Two parts, I1 B×C / 2×H×W Input the first BottleNeck module. The output of each BottleNeck module will be used as the input of the next BottleNeck module. The output of each BottleNeck module is I2 output of the Split module. B×C / 2×H×W After splicing at the sixth connection layer, it is input into the sixteenth convolution module.

[0053] In an embodiment of the present application, each BottleNeck module includes a seventeenth convolution module, an eighteenth convolution module and a first adder connected in sequence, and another input of the adder is the input of the seventeenth convolution module of the BottleNeck module.

[0054] In another exemplary embodiment of the present application, the structures of the three Segment modules are the same, such as Figure 3 As shown, each includes three convolution branches, and each convolution branch includes a nineteenth convolution module, a twentieth convolution module and a first convolution layer connected in sequence.

[0055] In this embodiment, each of the three segment sizes generates a bounding box (Bbox) tensor of shape [B, N, 4], a class (Class) tensor of shape [B, N, num] (B represents the batch size, N represents the number of objects in each image, 4 represents the coordinates of the object's bounding box, and num represents the number of object categories to be generated), and mask coefficients of shape [B, N, Q] representing object-specific weights, where Q represents the number of channels in the mask prototype. Deep features are passed through a convolutional layer (Conv2d) to generate a prototype mask. The prototype mask is a universal mask template, and all instance masks are reconstructed based on a linear combination of the prototype mask and the mask coefficients. The bounding box data generated by the segment modules of the three sizes is concatenated and then threshold filtered (Conf) to remove bounding boxes with low confidence. Non-maximum suppression (NMS) is then used to filter overlapping prediction boxes for each object, ultimately retaining the optimal prediction box output (bbox). The class data (Class) generated by the three segment modules is concatenated, and the class with the highest score (argmax) is taken as the class output (Class). The mask coefficients generated by the three segment modules are concatenated, and the mask prototype is matrix multiplied with the concatenated mask coefficients (Masks Coefficients). After passing the sigmoid activation function, it is reshaped into an instance mask with the shape [B, NHW]. The instance mask is scaled to the target RGB image size through the cropping operation (Crop) and then binarized (Threshold) to set the pixel values ​​of the target soil pixels to 255 (white) and the background pixels to 0 (black). The final mask image is output.

[0056] In another exemplary embodiment of the present application, the first bimodal feature fusion module 203, the second bimodal feature fusion module 206, the third bimodal feature fusion module 209 and the fourth bimodal feature fusion module have the same structure and all include a first attention weight extraction module, a second attention weight extraction module, an adder module, a first attention feature map extraction module, a second attention feature map extraction module and a seventh connection layer. Among them, the first attention weight extraction module is used to extract the first attention weight, the second attention weight and the third attention weight, wherein the first attention weight refers to the correlation weight of the channel dimension and width of the input RGB feature map, the second attention weight refers to the correlation weight of the channel dimension and length of the input RGB feature map, and the third attention weight refers to the spatial attention weight of the input RGB feature map. The second attention weight extraction module is used to extract the fourth attention weight, the fifth attention weight and the sixth attention weight, wherein the fourth attention weight refers to the correlation weight of the channel dimension and width of the input depth feature map, the fifth attention weight refers to the correlation weight of the channel dimension and length of the input depth feature map, and the sixth attention weight refers to the spatial attention weight of the input depth feature map. The adder module is used to add the first attention weight and the fourth attention weight to obtain a first joint attention weight, add the second attention weight and the fifth attention weight to obtain a second joint attention weight, and add the third attention weight and the sixth attention weight to obtain a third joint attention weight. The first attention feature map extraction module is used to use the first joint attention weight, the second joint attention weight and the third joint attention weight to perform average weighted fusion on the RGB feature map to obtain an RGB attention feature map. The second attention feature map extraction module is used to use the first joint attention weight, the second joint attention weight and the third joint attention weight to perform average weighted fusion on the depth feature map to obtain a depth attention feature map. The seventh connection layer is used to splice the RGB attention feature map and the depth attention feature map to achieve cross-dimensional and multi-dimensional weight distribution of the extracted feature maps, which can enhance the model's fusion and utilization of multimodal features and improve the performance of segmentation tasks.

[0057] In the embodiment of the present application, the input RGB feature map and depth feature map refer to the RGB feature map and depth feature map input to the first bimodal feature fusion module to the fourth bimodal feature fusion module.

[0058] The core idea of ​​the first bimodal feature fusion module 203, the second bimodal feature fusion module 206, the third bimodal feature fusion module 209 and the fourth bimodal feature fusion module is to dynamically capture the dependencies in each modality by analyzing the feature interactions in the three dimensions of channel-width, channel-height and space of each modality (RGB feature map modality and depth feature map modality), and then share the weights, calculate the joint attention, and then apply the attention to the feature maps of each modality to achieve feature fusion between modalities.

[0059] In another exemplary embodiment of the present application, the first attention weight extraction module includes a first weight extraction branch, a second weight extraction branch, and a third weight extraction branch, and the second attention weight extraction module includes a fourth weight extraction branch, a fifth weight extraction branch, and a sixth weight extraction branch. The first weight extraction branch and the sixth weight extraction branch have the same structure, such as Figure 4 As shown in the figure, each includes a first dimension adjustment module (Permute), a first pooling module (Zpool), and a second convolutional layer (Conv). The first dimension adjustment module is used to rearrange the order of the dimensions of the input RGB feature map or depth feature map according to the following formula:

[0060] ;

[0061] in, I Is the input RGB feature map or the depth feature map, its shape is , 0 means I Batch size B , 1 means I Length H , 2 means I Number of channels C , 3 means I Width W , I cw is the output of the first dimension adjustment module, and its shape is .

[0062] The first pooling module is used to I cw The second convolutional layer is used to perform a convolution operation on the output of the first pooling module.

[0063] The second branch and the fifth branch have the same structure, both including a second dimension adjustment module, a second pooling module and a third convolutional layer (Conv). The second dimension adjustment module is used to rearrange the order of the dimensions of the input RGB feature map or depth feature map according to the following formula:

[0064] ;

[0065] in, I hc is the output of the second dimension adjustment module, and its shape is .

[0066] The second pooling module is used to I hc The third convolutional layer is used to perform a convolution operation on the output of the second pooling module.

[0067] The third and fourth branches have the same structure, both consisting of a third pooling module and a fourth convolutional layer (Conv). The third pooling module performs a pooling operation on the input RGB feature map or depth feature map. The fourth convolutional layer performs a convolution operation on the output of the third pooling module.

[0068] In another exemplary embodiment of the present application, Figure 4 As shown, the structures of the first to third pooling modules are the same, each including a maximum pooling layer, an average pooling layer, and an eighth connection layer. The maximum pooling layer and the average pooling layer are operated in parallel. The maximum pooling layer is used to perform a maximum pooling operation on the input RGB feature map or depth feature map. The average pooling layer is used to perform an average pooling operation on the input RGB feature map or depth feature map. The eighth connection layer is used to splice the outputs of the maximum pooling layer and the average pooling layer.

[0069] In another exemplary embodiment of the present application, Figure 4 As shown, the adder module includes a second adder, a third adder and a fourth adder. The second adder is used to add the first attention weight AR cw and the fourth attention weight AD cw Add up to get the first joint attention weight AS cw The third adder is used to add the second attention weight AR hc and the fifth attention weight AD hc Add up to get the second joint attention weight AS hc The fourth adder is used to add the third attention weight AR hw and the sixth attention weight AD hw Add up to get the third joint attention weight AS hw .

[0070] In the embodiment of this application, , , .

[0071] In another exemplary embodiment of the present application, the first attention feature map extraction module and the second attention feature map extraction module have the same structure, such as Figure 4As shown, they all include a first activation function module, a second activation function module, a third activation function module, a first multiplier, a second multiplier, a third multiplier, a fifth adder, and a sixth adder. Among them, the inputs of the first activation function module, the second activation function module, and the third activation function module correspond to the first joint attention weight AS cw , the second joint attention weight AS hc , the third joint attention weight AS hw The first inputs of the first multiplier, the second multiplier, and the third multiplier correspond to the output of the first activation function module, the output of the second activation function module, and the output of the third activation function module. The second inputs of the first multiplier, the second multiplier, and the third multiplier are the RGB feature map input to the first attention weight extraction module or the depth feature map input to the second attention weight extraction module. The inputs of the fifth adder and the sixth adder are the outputs of the first multiplier, the second multiplier, and the third multiplier. The outputs of the fifth adder and the sixth adder are the inputs of the seventh connection layer.

[0072] In another exemplary embodiment of the present application, the structures of the first to twentieth convolution modules are the same, such as Figure 3 As shown in the figure, each includes a fifth convolutional layer, a batch normalization layer (BatchNorm2d), and an activation function (SiLu). The fifth convolutional layer is used to extract local features of the input feature map, the batch normalization layer is used to prevent gradient explosion and gradient vanishing, and the activation function (SiLu) improves the network's expressive power through nonlinear transformation.

[0073] In another exemplary embodiment of the present application, the above step 102 includes the following steps 301 to 306. Among them:

[0074] Step 301 : In each target depth image, a rectangular sliding window is formed starting from the upper left vertex of the soil boundary box.

[0075] Step 302: perform a micro-element classification operation. The micro-element classification operation includes:

[0076] If the pixel depth value of any of the four vertices of the rectangular sliding window is missing, the pixel area contained in the rectangular sliding window is recorded as a missing element, and the volume of the missing element is recorded as 0.

[0077] If there are no missing pixel depth values ​​at the four vertices of the rectangular sliding window, and:

[0078] If all pixels within the rectangular sliding window are soil pixels, the pixel area contained in the rectangular sliding window is recorded as the first infinitesimal element; wherein, if the pixel value of any pixel within the rectangular sliding window in the corresponding soil mask image is 255, then the pixel within the rectangular sliding window is a soil pixel;

[0079] If the number of pixels belonging to slag in the rectangular sliding window is less than a preset first threshold, the pixel area contained in the rectangular sliding window is recorded as an invalid micro-element, and the volume of the invalid micro-element is recorded as nan;

[0080] If the number of pixels belonging to slag in the rectangular sliding window is not less than the preset first threshold but less than n, the pixel area contained in the rectangular sliding window is recorded as the second infinitesimal element, and n is the total number of pixels in the rectangular sliding window.

[0081] In the embodiment of the present application, the size of the rectangular sliding window, the moving step and the first threshold are not specifically limited and can be set according to actual needs. For example, the size of the rectangular sliding window is set to 3 3. At this time, the vertical movement step size and the horizontal movement step size of the rectangular sliding window are both 3, n is 9, and the first threshold value can be set to 5.

[0082] Step 303: After performing the above-mentioned micro-element classification operation, perform a micro-element volume calculation operation:

[0083] Calculate the volume of the first infinitesimal element according to the three-dimensional space coordinate system of the four vertices of the first infinitesimal element;

[0084] The volume of the second infinitesimal element is calculated according to the four vertices of the second infinitesimal element and the three-dimensional space coordinate system of the soil pixel point.

[0085] Step 304: After performing the microelement volume calculation operation, a loop operation is performed:

[0086] In each target depth image, the rectangular sliding window is moved in sequence from left to right and from top to bottom until it reaches the final position, and the micro-element classification operation and micro-element volume calculation operation are performed after each movement of the rectangular sliding window; the final position is the position where the coordinates of the lower right vertex of the rectangular sliding window exceed the lower right vertex of the muck boundary box.

[0087] Step 305: After executing the loop operation, perform the volume assignment operation:

[0088] The volume of each missing element in each target depth image is assigned a value according to the volumes of the first element and / or the second element of the neighbors of each missing element in each target depth image.

[0089] In this embodiment, since the shape of the slag dropped into the vehicle frame at the slag outlet does not have a particularly prominent element, and the heights of adjacent elements are relatively similar, after the rectangular sliding window has completely calculated the elements within the slag boundary frame, the volume of each missing element is filled in using the volume of the first and / or second elements of each missing element's neighbors. This improves the accuracy of the volume calculation. By summing up the volumes of all non-invalid elements, an approximate value of the captured volume can be obtained.

[0090] Step 306 : After performing the volume assignment operation, the volumes of all missing elements, first elements, and second elements of each target depth image are accumulated to obtain the volume of shield muck in each target depth image.

[0091] In the embodiment of the present application, the volume of the shield muck in each target depth image is obtained by accumulating the volumes of all missing microelements, first microelements, and second microelements in each target depth image using the following formula:

[0092] ;

[0093] in, is the total number of missing elements, After assignment The volume value of the missing element, is the total number of first infinitesimals, For the The volume value of the first infinitesimal element, is the total number of second infinitesimals, For the The volume value of the second infinitesimal element, V E The volume of shield muck in each target depth image.

[0094] In another exemplary embodiment of the present application, the above-mentioned calculation of the volume of the first infinitesimal element based on the three-dimensional space coordinate system of the four vertices of the first infinitesimal element includes the following steps 401 to 402. In which:

[0095] Step 401 : Convert the pixel coordinates of the four vertices of the first infinitesimal element into a three-dimensional space coordinate system to obtain the three-dimensional space coordinates of the four vertices of the first infinitesimal element.

[0096] Step 402: Calculate the volume of the first infinitesimal element according to the spatial three-dimensional coordinates of the four vertices of the first infinitesimal element and the height of the first infinitesimal element using the volume calculation method of a quadrangular prism; wherein the average height of the four vertices of the first infinitesimal element is the absolute value of the difference between the first height value and the second height value, the first height value is the average value of the spatial Z-axis coordinate values ​​of the four vertices of the first infinitesimal element, and the second height value is the height of the depth camera from the bottom of the muck truck compartment.

[0097] In the embodiment of the present application, the dirt microelement in the sliding window can be regarded as having different shapes according to the number of dirt pixels in the sliding window. The shape of the first microelement is as follows: Figure 5 As shown, it is approximately a quadrangular prism with four points of different heights. Projected on the bottom of the car, it can form two triangles with A2B2C2 and B2C2D2 as the base. The vertical height of the bottom of the car from the center of the depth camera is Z 5 can be obtained during installation, then the volume of each first infinitesimal element is the sum of the volumes of the two triangular prisms , the volume of the first infinitesimal element is calculated as follows:

[0098] ;

[0099] ;

[0100] ;

[0101] in, S 1 represents the area of ​​the triangle with the base A2B2C2, S 2 represents the area of ​​the triangle B2C2D2 at the base, and the coordinates of point A1 are , the coordinates of point B1 are , the coordinates of point C1 are , the coordinates of point C1 are , the height of the depth camera from the bottom of the car frame is , the base area of ​​triangle A2B2C2 and the base area of ​​triangle B2C2D2 The calculated value is multiplied by the average height of the four points A1, B1, C1, and C1 to get the volume of the first infinitesimal element.

[0102] In another exemplary embodiment of the present application, the above-mentioned calculation of the volume of the second infinitesimal element based on the four vertices of the second infinitesimal element and the three-dimensional space coordinate system of the soil pixel point includes the following steps 501 to 502. In which:

[0103] Step 501 : convert the pixel coordinates of the four vertices of the second infinitesimal element and the muck pixel point into a three-dimensional space coordinate system to obtain the three-dimensional space coordinates of the four vertices of the second infinitesimal element and the muck pixel point.

[0104] Step 502, calculate the volume of the second infinitesimal element according to the spatial three-dimensional coordinates of the four vertices of the second infinitesimal element and the height of the second infinitesimal element according to the volume calculation method of the quadrangular prism; wherein the height of the second infinitesimal element is the absolute value of the difference between the third height value and the second height value, and the third height value is the average value of the spatial Z-axis coordinate values ​​of all the slag pixel points of the second infinitesimal element.

[0105] In the embodiment of the present application, the shape of the second infinitesimal element is as follows: Figure 6As shown, calculate the volume of the second infinitesimal element As shown in the following formula:

[0106] ;

[0107] Among them, avg(Z) represents the average height of the pixels belonging to the soil pixels in the second microelement (the height of the soil pixels). Z w average value of ).

[0108] In another exemplary embodiment of the present application, the pixel coordinates of the target pixel point are converted to a three-dimensional space coordinate system according to the following steps 601 to 603. The target pixel point includes any vertex of the first infinitesimal element, any vertex of the second infinitesimal element, or any residual pixel point of the second infinitesimal element.

[0109] Step 601: Convert the pixel coordinates of the target pixel point to the camera coordinate system to obtain the coordinates of the target pixel point in the camera coordinate system. The conversion formula is:

[0110] ;

[0111] in,( u , v ) represents the pixel coordinates of the target pixel point, Indicates the coordinates of the optical axis center of the depth camera in the pixel coordinate system, is the focal length of the depth camera on the x-axis (unit: pixel), is the focal length of the depth camera on the y-axis (unit: pixel), , , ) represents the coordinates of the target pixel in the camera coordinate system, and the Z-axis coordinates of the target pixel in the camera coordinate system This is the depth value obtained by the depth camera.

[0112] In the embodiment of the present application, the pixel coordinates of the target pixel point are first converted to the image coordinate system according to the following formula to obtain the coordinates of the target pixel point in the image coordinate system. The conversion formula is:

[0113] ;

[0114] in, and Respectively represent the scaling factors for converting the actual distance in the image coordinate system into the number of pixels in the x-axis direction and the y-axis direction, ( x , y ) represents the coordinates of the target pixel in the image coordinate system.

[0115] Then, the coordinates of the target pixel point in the image coordinate system are converted to the camera coordinate system according to the following formula to obtain the coordinates of the target pixel point in the camera coordinate system. The conversion formula is:

[0116] ;

[0117] in, and Indicates the x-axis and y-axis focal length of the depth camera (unit: mm).

[0118] After the transformation of the above three coordinate systems, the position of the target pixel relative to the center of the depth camera in the camera coordinate system can be obtained ( , , ) calculation formula.

[0119] Pixel coordinates ( u , v ) is the coordinate of the target pixel point in the target depth image in a pixel coordinate system with the upper left corner of the target depth image as the origin (0,0), and the horizontal and vertical coordinates representing the number of rows and columns respectively. x , y ) describes the coordinates of the target pixel relative to the center of the optical axis in a coordinate system with the center of the camera's optical axis as the origin, the horizontal direction perpendicular to the optical axis as the X axis, and the vertical direction perpendicular to the optical axis as the Y axis. The coordinates of the camera coordinate system ( , , ) describes the coordinates of the target pixel relative to the center of the optical axis in a coordinate system with the center of the camera's optical axis as the origin, the optical axis direction as the Z axis, the horizontal direction of the camera perpendicular to the optical axis as the X axis, and the vertical direction of the camera perpendicular to the optical axis as the Y axis. When the depth camera is placed on a horizontal table with the lens facing upward, the pitch angle is 0. When the depth camera lens faces the direction of gravity, the pitch angle is 180°. When the depth camera lens is perpendicular to the direction of gravity and parallel to the ground, the pitch angle is 90°. When the depth camera is placed horizontally with the optical axis perpendicular to the ground, the pitch angle is 180° and the roll angle is 0°. The actual three-dimensional distance of the position represented by the pixel relative to the center of the camera can be obtained.

[0120] Step 602: Calculate the angle of the spatial point corresponding to the target pixel relative to the center of the depth camera according to the following formula:

[0121] ;

[0122] in, Indicates that the target pixel is relative to the center of the depth camera The angle of the axis, Indicates that the target pixel is relative to the center of the depth camera The angle of the axis.

[0123] Step 603: Convert the coordinates of the target pixel in the camera coordinate system to the three-dimensional spatial coordinate system according to the following formula to obtain the three-dimensional spatial coordinates of the target pixel :

[0124] ;

[0125] in, is the pitch angle of the depth camera.

[0126] In the embodiment of the present application, according to the actual needs of the project, the pitch angle of the depth camera may not be perpendicular to the ground or parallel to the ground, and the camera position cannot be guaranteed to remain unchanged. It is necessary to use the IMU module in the depth camera to determine the current camera posture. When the imaging center of the depth camera is determined as the origin, the Z axis is perpendicular to the ground, the pitch angle is 90° and the roll angle is 0°, the horizontal direction of the camera parallel to the ground is the X axis, and the optical axis position of the camera is the Y axis to establish a world coordinate system. At this time, the pitch angle of the camera posture is obtained according to the IMU module of the depth camera, which can be obtained through the three-dimensional spatial coordinates. The calculation formula is the conversion of the target pixel point in the camera coordinate system to the world coordinate system.

[0127] In another exemplary embodiment of the present application, in step 305, the volume of each missing element is assigned based on the volume value of the first element and / or the second element adjacent to each missing element, specifically including:

[0128] The nearest neighbor of each missing element t The average volume value of the number of micro-elements is assigned to the volume value of the missing micro-element, which includes the first micro-element and / or the second micro-element.

[0129] In the embodiment of the present application, the nearest neighbor of each missing microelement t The average volume of each microelement is calculated according to the following formula:

[0130] ;

[0131] in, It is The volume value of the nearest neighbor.

[0132] In another exemplary embodiment of the present application, in order to verify the feasibility of applying the improved image segmentation algorithm of the present application to the calculation of slag volume, experiments are carried out in turn to verify the superiority of the improved image segmentation algorithm and the feasibility of the solution.

[0133] To simulate the loading of muck trucks, this application used boxes proportionally scaled to the size of a single carriage as the muck carrier and sand as the contents to establish an experimental platform. By using volume, shape, camera pitch angle, and camera height as variables, a dataset consisting of RGB images, depth images, and depth information was captured, with the actual volume data serving as the contents. After screening and cleaning, a total of 1,639 data sets were retained. 1,500 of these were used as training and validation datasets for training an image segmentation network based on bimodal input. Labelme software was used to segment and label the muck in the images.

[0134] To verify the superiority of the improved YOLOv8, we conducted a comparative experiment between the improved YOLO algorithm and the original model. This experiment was conducted on a Windows 10 system and the programming platform was PyCharm 2023. The hardware used for training the model was: Nvidia RTX1080TI GPU, Intel 10400F CPU, batch_size parameter was set to 32, and the number of training rounds was 100.

[0135] The speed performance evaluation indicators used are: Parameters / M and FPS (Frames Per Second) for processing a single image. These two indicators can intuitively judge the size and speed of the model.

[0136] The accuracy evaluation indicators for soil image segmentation are: recall (recall), which represents the proportion of all true targets correctly detected by the network; precision (precision), which represents the proportion of correct targets detected by the network; and average precision (AP), which represents the performance of the network under different recall rates. The calculation formula is:

[0137] ;

[0138] ;

[0139] ;

[0140] in, Represents the number of correctly detected targets, Represents the number of true targets that were not detected, Represents the amount of non-target data that was falsely detected.

[0141] The performance of the improved dual-branch OursYOLOv8-seg and the original YOLOv8-seg in the collected slag image dataset is shown in Table 1 below.

[0142] Table 1 Index table before and after improvement

[0143]

[0144] As can be seen from Table 1 above, the improved dual-branch OursYOLOv8-seg in this application has improved speed and accuracy, and the number of parameters and computational complexity have been reduced compared to the original version. The accuracy of soil segmentation can reach 99.5%, and it can accurately separate soil from the background.

[0145] The dataset created during the experiment contains all the information required for the calculation, including two modal images, depth data, camera angle, the height of the bottom of the slag box from the camera center, and the actual volume of the slag. By comparing the volume calculated by this method with the actual volume based on the above data, the accuracy of the slag volume calculation method can be determined. The evaluation metric used to evaluate volume accuracy is the average relative accuracy, which can be expressed as follows:

[0146] ;

[0147] in, is the number of experimental data. A total of 723 sets of data were used to verify the accuracy. The volume calculated from one set of data was The corresponding real volume Divide, calculate the percentage, get the relative error, and then calculate the average relative error of 723 sets of data to get the average relative accuracy. Experimental results on a self-built dataset show that the average relative volume accuracy calculated by this method in the dataset can reach 95.5%, and the calculation time for a single set of data is about 0.4s, which can meet the needs of practical applications.

[0148] Based on the same inventive concept, embodiments of the present application also provide a machine vision-based shield muck volume calculation device for implementing the aforementioned machine vision-based shield muck volume calculation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more machine vision-based shield muck volume calculation device embodiments provided below can be found in the limitations of the machine vision-based shield muck volume calculation method described above and will not be repeated here.

[0149] In an exemplary embodiment, Figure 7 As shown, a shield muck volume calculation device 70 based on machine vision is provided, comprising:

[0150] The image segmentation module 701 is used to extract and fuse features of the target depth image and target RGB image acquired in real time through an image segmentation network based on a dual-modal input, and to segment the target RGB image based on the obtained fusion features to obtain the slag mask image and the coordinates of the slag bounding box of the target RGB image;

[0151] The volume calculation module 702 is used to calculate the volume of shield tunneling muck by a volume calculation method based on the coordinates of the muck boundary box and the muck mask image.

[0152] In the embodiment of the present application, the relevant introduction of the image segmentation network based on dual-modal input is detailed in the description of the above method embodiment, which will not be repeated here.

[0153] In another exemplary embodiment of the present application, the volume calculation module 702 is further configured to:

[0154] In each target depth image, a rectangular sliding window is formed starting from the upper left vertex of the muck bounding box;

[0155] Perform microelement classification operations:

[0156] If the pixel depth value of any of the four vertices of the rectangular sliding window is missing, the pixel area contained in the rectangular sliding window is recorded as a missing element, and the volume of the missing element is recorded as 0;

[0157] If there are no missing pixel depth values ​​at the four vertices of the rectangular sliding window, and:

[0158] If all pixels within the rectangular sliding window are soil pixels, the pixel area contained in the rectangular sliding window is recorded as the first infinitesimal element; wherein, if the pixel value of any pixel within the rectangular sliding window in the corresponding soil mask image is 255, then the pixel within the rectangular sliding window is a soil pixel;

[0159] If the number of pixels belonging to slag in the rectangular sliding window is less than a preset first threshold, the pixel area contained in the rectangular sliding window is recorded as an invalid micro-element, and the volume of the invalid micro-element is recorded as nan;

[0160] If the number of pixels belonging to soil debris in the rectangular sliding window is not less than a preset first threshold but less than n, the pixel area contained in the rectangular sliding window is recorded as a second infinitesimal element, where n is the total number of pixels in the rectangular sliding window;

[0161] After performing the above-mentioned microelement classification operation, perform the microelement volume calculation operation:

[0162] Calculate the volume of the first infinitesimal element according to the three-dimensional space coordinate system of the four vertices of the first infinitesimal element;

[0163] Calculate the volume of the second infinitesimal element according to the three-dimensional space coordinate system of the four vertices of the second infinitesimal element and the soil pixel point;

[0164] After performing the microelement volume calculation operation, perform the loop operation:

[0165] In each target depth image, the rectangular sliding window is moved sequentially from left to right and from top to bottom until it reaches its final position, and the micro-element classification operation and micro-element volume calculation operation are performed after each movement of the rectangular sliding window; the final position is the position where the coordinates of the lower right vertex of the rectangular sliding window exceed the lower right vertex of the muck boundary box;

[0166] The volume of each missing element in each target depth image is assigned a value according to the volumes of the first element and / or the second element of the neighbors of each missing element in each target depth image.

[0167] In the embodiment of the present application, the size of the rectangular sliding window, the moving step and the first threshold are not specifically limited and can be set according to actual needs. For example, the size of the rectangular sliding window is set to 3 3. At this time, the vertical movement step size and the horizontal movement step size of the rectangular sliding window are both 3, n is 9, and the first threshold value can be set to 5.

[0168] In another exemplary embodiment of the present application, the volume calculation module 702 is further configured to:

[0169] Convert the pixel coordinates of the four vertices of the first infinitesimal element into a three-dimensional space coordinate system to obtain the three-dimensional space coordinates of the four vertices of the first infinitesimal element;

[0170] The volume of the first infinitesimal element is calculated according to the spatial three-dimensional coordinates of the four vertices of the first infinitesimal element and the height of the first infinitesimal element according to the volume calculation method of the quadrangular prism; wherein, the average height of the four vertices of the first infinitesimal element is the absolute value of the difference between the first height value and the second height value, the first height value is the average value of the spatial Z-axis coordinate values ​​of the four vertices of the first infinitesimal element, and the second height value is the height of the depth camera from the bottom of the muck truck compartment.

[0171] In another exemplary embodiment of the present application, the volume calculation module 702 is further configured to:

[0172] The pixel coordinates of the four vertices of the second infinitesimal element and the slag pixel point are converted into a three-dimensional space coordinate system to obtain the spatial three-dimensional coordinates of the four vertices of the second infinitesimal element and the slag pixel point;

[0173] The volume of the second infinitesimal element is calculated according to the spatial three-dimensional coordinates of the four vertices of the second infinitesimal element and the height of the second infinitesimal element using the volume calculation method of a quadrangular prism; wherein the height of the second infinitesimal element is the absolute value of the difference between the third height value and the second height value, and the third height value is the average value of the spatial Z-axis coordinate values ​​of all the slag pixel points of the second infinitesimal element.

[0174] In another exemplary embodiment of the present application, the volume calculation module 702 is further configured to:

[0175] Convert the pixel coordinates of the target pixel point to the camera coordinate system to obtain the coordinates of the target pixel point in the camera coordinate system. The conversion formula is:

[0176] ;

[0177] in,( u , v ) represents the pixel coordinates of the target pixel point, Indicates the coordinates of the optical axis center of the depth camera in the pixel coordinate system, is the focal length of the depth camera on the x-axis (unit: pixel), is the focal length of the depth camera on the y-axis (unit: pixel), , , ) represents the coordinates of the target pixel in the camera coordinate system, and the Z-axis coordinates of the target pixel in the camera coordinate system That is the depth value obtained by the depth camera;

[0178] The angle of the spatial point corresponding to the target pixel relative to the center of the depth camera is calculated according to the following formula:

[0179] ;

[0180] in, Indicates that the target pixel is relative to the center of the depth camera The angle of the axis, Indicates that the target pixel is relative to the center of the depth camera The angle of the axis;

[0181] According to the following formula, the coordinates of the target pixel point in the camera coordinate system are converted to the spatial three-dimensional coordinate system to obtain the spatial three-dimensional coordinates of the target pixel point. :

[0182] ;

[0183] in, is the pitch angle of the depth camera.

[0184] In another exemplary embodiment of the present application, the volume calculation module 702 is further configured to:

[0185] The nearest neighbor of each missing element t The average volume value of the number of micro-elements is assigned to the volume value of the missing micro-element, which includes the first micro-element and / or the second micro-element.

[0186] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, memory and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store shield slag volume calculation data based on machine vision. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a shield slag volume calculation method based on machine vision is implemented.

[0187] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0188] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0189] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0190] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0191] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0192] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0193] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0194] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0195] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A shield muck volume calculation method based on machine vision, characterized in that: The shield muck volume calculation method based on machine vision includes: Through the image segmentation network based on dual-modal input, feature extraction and fusion are performed on the target depth image and target RGB image acquired in real time, and the target RGB image is segmented according to the obtained fusion features to obtain the slag mask image and the coordinates of the slag boundary box of the target RGB image; According to the coordinates of the muck boundary box and the muck mask image, the volume of the shield muck is calculated by the volume calculation method; The image segmentation network based on bimodal input includes an input layer, a backbone network, a neck network and a segmentation layer, wherein: The input layer is used to merge channels of each input target RGB image and the corresponding target depth image, and convert the feature map obtained by merging the channels into a matrix ,in, B is the batch dimension, H is the width of the feature map, W is the length of the feature map; The neck network is used to perform a cross-scale fusion operation on the shallow features and deep features output by the backbone network to obtain fused features with the same scale as the shallow features and the deep features respectively; The segmentation layer is used to generate and output at least the coordinates of the muck boundary box and the muck mask image of the target RGB image according to the fusion features output by the neck network; The backbone network includes: The first feature extraction module is used to separate the matrix A target depth image portion, and performing feature extraction on the target depth image portion to obtain a depth image feature; The second feature extraction module is used to separate the matrix , and extracting features from the target RGB image portion to obtain RGB image features; a first bimodal feature fusion module, configured to perform feature fusion based on a triple attention mechanism on the bimodal features output by the first feature extraction module and the second feature extraction module to obtain a first shallow feature; a third feature extraction module, configured to perform feature extraction on the output of the first feature extraction module; a fourth feature extraction module, configured to perform feature extraction on the output of the first bimodal feature fusion module; a second bimodal feature fusion module, configured to perform feature fusion based on a triple attention mechanism on the bimodal features output by the third feature extraction module and the fourth feature extraction module to obtain a second shallow feature; a fifth feature extraction module, configured to perform feature extraction on the output of the third feature extraction module; a sixth feature extraction module, configured to perform feature extraction on the output of the second bimodal feature fusion module; a third bimodal feature fusion module, configured to perform feature fusion based on a triple attention mechanism on the bimodal features output by the fifth feature extraction module and the sixth feature extraction module; The SPPF spatial pyramid pooling module is used to perform pooling operations of different sizes on the output of the third bimodal feature fusion module, and to splice the feature maps obtained by the pooling operations of different sizes in the channel dimension to obtain the deep features.

2. The shield muck volume calculation method based on machine vision according to claim 1 is characterized in that: The first bimodal feature fusion module, the second bimodal feature fusion module and the third bimodal feature fusion module have the same structure and all include: A first attention weight extraction module is used to extract a first attention weight, a second attention weight, and a third attention weight, wherein the first attention weight refers to the correlation weight of the channel dimension and width of the input RGB feature map, the second attention weight refers to the correlation weight of the channel dimension and length of the input RGB feature map, and the third attention weight refers to the spatial attention weight of the input RGB feature map; A second attention weight extraction module is used to extract a fourth attention weight, a fifth attention weight, and a sixth attention weight, wherein the fourth attention weight refers to the correlation weight of the channel dimension and width of the input depth feature map, the fifth attention weight refers to the correlation weight of the channel dimension and length of the input depth feature map, and the sixth attention weight refers to the spatial attention weight of the input depth feature map; an adder module, configured to add the first attention weight and the fourth attention weight to obtain a first joint attention weight, add the second attention weight and the fifth attention weight to obtain a second joint attention weight, and add the third attention weight and the sixth attention weight to obtain a third joint attention weight; A first attention feature map extraction module is used to perform average weighted fusion of the RGB feature map using the first joint attention weight, the second joint attention weight, and the third joint attention weight to obtain an RGB attention feature map; A second attention feature map extraction module is used to perform average weighted fusion of the depth feature map using the first joint attention weight, the second joint attention weight, and the third joint attention weight to obtain a depth attention feature map; The seventh connection layer is used to splice the RGB attention feature map and the depth attention feature map.

3. The shield muck volume calculation method based on machine vision according to claim 2 is characterized in that: The first attention weight extraction module includes a first branch, a second branch, and a third branch, and the second attention weight extraction module includes a fourth branch, a fifth branch, and a sixth branch, wherein: The first branch and the sixth branch have the same structure, and both include: A first dimension adjustment module is configured to rearrange the order of dimensions of the input RGB feature map or the depth feature map according to the following formula: ； in, I is the input RGB feature map or the depth feature map, and its shape is , 0 means I Batch size B , 1 means I Length H , 2 means I Number of channels C , 3 means I Width W , I cw is the output of the first dimension adjustment module, and its shape is ; The first pooling module is used to I cw Perform pooling operations; A second convolutional layer, configured to perform a convolution operation on the output of the first pooling module; The second branch and the fifth branch have the same structure, and both include: The second dimension adjustment module is used to rearrange the order of the dimensions of the input RGB feature map or the depth feature map according to the following formula: ； in, I hc is the output of the second dimension adjustment module, and its shape is ; The second pooling module is used to I hc Perform pooling operations; A third convolutional layer, configured to perform a convolution operation on the output of the second pooling module; The third branch and the fourth branch have the same structure, both comprising: A third pooling module is used to perform a pooling operation on the input RGB feature map or the depth feature map; The fourth convolutional layer is used to perform a convolution operation on the output of the third pooling module.

4. The shield muck volume calculation method based on machine vision according to claim 1 is characterized in that: The volume of the shield tunneling muck is calculated by a volume calculation method based on the coordinates of the muck boundary box and the muck mask image, specifically including: In each of the target depth images, a rectangular sliding window is formed starting from the upper left vertex of the soil boundary box; Perform microelement classification operations: If the pixel depth value of any one of the four vertices of the rectangular sliding window is missing, the pixel area contained in the rectangular sliding window is recorded as a missing element, and the volume of the missing element is recorded as 0; If there are no missing pixel depth values ​​at the four vertices of the rectangular sliding window, and: If all pixels within the rectangular sliding window are soil pixels, the pixel area contained in the rectangular sliding window is recorded as the first infinitesimal element; wherein, if the pixel value of any pixel within the rectangular sliding window in the soil mask image is 255, then the pixel within the rectangular sliding window is a soil pixel; If the number of soil pixel points in the rectangular sliding window is less than a preset first threshold, the pixel area contained in the rectangular sliding window is recorded as an invalid element, and the volume of the invalid element is recorded as nan; If the number of pixels belonging to soil debris in the rectangular sliding window is not less than a preset first threshold but less than n, the pixel area contained in the rectangular sliding window is recorded as a second infinitesimal element, where n is the total number of pixels in the rectangular sliding window; After executing the micro-element classification operation, the micro-element volume calculation operation is performed: Calculating the volume of the first infinitesimal element according to the three-dimensional space coordinate system of the four vertices of the first infinitesimal element; Calculating the volume of the second infinitesimal element according to the three-dimensional space coordinate system of the four vertices of the second infinitesimal element and the soil pixel point; After executing the microelement volume calculation operation, a loop operation is performed: In each of the target depth images, the rectangular sliding window is moved from left to right and from top to bottom until it reaches a final position, and the micro-element classification operation and the micro-element volume calculation operation are performed after each movement of the rectangular sliding window; the final position is the position where the coordinates of the lower right vertex of the rectangular sliding window exceed the lower right vertex of the soil boundary box; After executing the loop operation, perform the volume assignment operation: Assigning a value to the volume of each missing element in each target depth image according to the volume of the first element and / or the second element of the neighboring missing element in each target depth image; After performing the volume assignment operation, the volumes of all missing micro-elements, first micro-elements and second micro-elements of each target depth image are accumulated to obtain the volume of the shield muck.

5. The shield muck volume calculation method based on machine vision according to claim 4 is characterized in that: Calculating the volume of the first infinitesimal element according to the three-dimensional space coordinate system of the four vertices of the first infinitesimal element specifically includes: Convert the pixel coordinates of the four vertices of the first infinitesimal element into a three-dimensional space coordinate system to obtain the three-dimensional space coordinates of the four vertices of the first infinitesimal element; The volume of the first infinitesimal element is calculated according to the spatial three-dimensional coordinates of the four vertices of the first infinitesimal element and the height of the first infinitesimal element using a method for calculating the volume of a quadrangular prism; wherein the average height of the four vertices of the first infinitesimal element is the absolute value of the difference between the first height value and the second height value, the first height value is the average of the spatial Z-axis coordinate values ​​of the four vertices of the first infinitesimal element, and the second height value is the height of the depth camera from the bottom of the muck truck compartment; Calculating the volume of the second infinitesimal element according to the four vertices of the second infinitesimal element and the three-dimensional space coordinate system of the soil pixel point specifically includes: Converting the pixel coordinates of the four vertices of the second infinitesimal element and the muck pixel point into the three-dimensional space coordinate system to obtain the three-dimensional space coordinates of the four vertices of the second infinitesimal element and the muck pixel point; The volume of the second infinitesimal element is calculated according to the spatial three-dimensional coordinates of the four vertices of the second infinitesimal element and the height of the second infinitesimal element using the volume calculation method of a quadrangular prism; wherein the height of the second infinitesimal element is the absolute value of the difference between the third height value and the second height value, and the third height value is the average value of the spatial Z-axis coordinate values ​​of all the slag pixel points of the second infinitesimal element.

6. The shield muck volume calculation method based on machine vision according to claim 5 is characterized in that: The pixel coordinates of a target pixel point are converted to the three-dimensional space coordinate system according to the following steps, where the target pixel point includes any vertex of the first infinitesimal element, any vertex of the second infinitesimal element, or any soil pixel point of the second infinitesimal element: Convert the pixel coordinates of the target pixel point to the camera coordinate system to obtain the coordinates of the target pixel point in the camera coordinate system. The conversion formula is: ； in,( u , v ) represents the pixel coordinates of the target pixel point, Indicates the coordinates of the optical axis center of the depth camera in the pixel coordinate system, is the focal length of the depth camera on the x-axis, is the focal length of the depth camera on the y-axis, ( , , ) represents the coordinate of the target pixel point in the camera coordinate system, and the Z-axis coordinate of the target pixel point in the camera coordinate system That is the depth value obtained by the depth camera; The angle of the spatial point corresponding to the target pixel relative to the center of the depth camera is calculated according to the following formula: ； in, Indicates that the target pixel is relative to the center of the depth camera The angle of the axis, Indicates that the target pixel is relative to the center of the depth camera The angle of the axis; The coordinates of the target pixel point in the camera coordinate system are converted to the spatial three-dimensional coordinate system according to the following formula: : ； in, is the pitch angle of the depth camera.

7. The shield muck volume calculation method based on machine vision according to claim 4 is characterized in that: The assigning a value to the volume of each missing element in each target depth image according to the volume value of the first element and / or the second element adjacent to each missing element in each target depth image specifically includes: The nearest neighbor of each missing element t The average volume value of the micro-elements is assigned to the volume value of the missing micro-element, and the micro-element includes the first micro-element and / or the second micro-element.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the shield slag volume calculation method based on machine vision described in any one of claims 1 to 7.

Images