Image completion method and device for anchor hole

Abstract

The application relates to the technical field of mine engineering equipment, and discloses an image filling method and device for anchor holes; a first drilling rig visual image and a first fuselage visual image in a drilling anchor operation process are acquired; the first drilling rig visual image and the first fuselage visual image are spliced to obtain an input tensor; an initial filling image is generated by using a target generator in a trained image filling model and the input tensor; the consistency of the initial filling image and the first fuselage visual image is checked by using a target discriminator in the trained image filling model to obtain a first checking result; in the case that the first checking result meets a preset consistency condition, the initial filling image is determined as a target filling image of the anchor hole; otherwise, the image under the drilling rig angle and the fuselage angle is re-acquired to execute the image filling process until the target filling image meeting the preset consistency requirement is obtained, so that the difficulty of anchor hole recognition and positioning can be effectively reduced, and the drilling anchor operation efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of mining engineering equipment technology, specifically to an image completion method and apparatus for anchor holes. Background Technology

[0002] Currently, drilling and anchoring robots generally rely on vision sensors to collect images of anchor holes in order to achieve automatic positioning of anchor holes and drilling operations. However, the vision sensors on the drilling rig actuators are easily covered by dust and water mist during the drilling process because they need to be aligned with the anchor holes at close range. This results in partial or large-area obstruction of the collected anchor hole images, making it difficult to identify and locate the anchor holes and seriously reducing the automation efficiency of drilling and anchoring operations. Summary of the Invention

[0003] To address the difficulty in identifying and locating anchor holes, which leads to low efficiency in drilling and anchoring operations, this application provides the following technical solution:

[0004] In a first aspect, embodiments of this application provide an image completion method for anchor holes, including:

[0005] Acquire a first drilling rig visual image and a first machine body visual image during the drilling and anchoring operation; wherein, the first drilling rig visual image represents an image of the anchor hole that is obscured by dust or water mist and is collected from the drilling rig's perspective; the first machine body visual image represents an image containing the anchor hole area collected from the machine body's perspective.

[0006] The first drilling rig visual image and the first machine body visual image are stitched together to obtain the input tensor;

[0007] The initial completed image is generated using the target generator and input tensor in the trained image completion model; whereby the initial completed image represents the image of the complete anchor hole structure obtained by completing the occluded part in the drilling rig visual image.

[0008] The consistency between the initial completed image and the first fuselage visual image is verified using the target discriminator in the trained image completion model, and the first verification result is obtained.

[0009] If the first verification result meets the preset consistency conditions, the initial completed image is determined as the target completed image of the anchor hole; otherwise, images from the drilling rig perspective and the machine body perspective are re-acquired to perform the image completion process until a target completed image that meets the preset consistency requirements is obtained.

[0010] In some embodiments of this application, an initial completed image is generated using the target generator and input tensor in the trained image completion model, including:

[0011] The first feature tensor is obtained by extracting features from the input tensor using the first preset convolutional layer in the target generator; wherein, the first feature tensor includes the positional features of the occluded area, the structure and texture features of the anchor hole;

[0012] The fine-grained features in the first feature tensor are optimized using the preset residual blocks in the target generator to obtain the second feature tensor;

[0013] The noise features in the second feature tensor are removed by using the preset bottleneck layer in the target generator to obtain the third feature tensor;

[0014] The third feature tensor is resized using a pre-set transposed convolutional layer in the target generator to obtain the initial padded image.

[0015] In some embodiments of this application, the consistency between the initial incomplete image and the first fuselage visual image is verified using the target discriminator in the trained image completion model to obtain a first verification result, including:

[0016] The initial completed image and the first fuselage visual image are stitched together to obtain the verification tensor;

[0017] The anchor hole structure features in the verification tensor are extracted using the second preset convolutional layer in the target discriminator; wherein, the anchor hole structure features include the first anchor hole features in the initial completed image and the second anchor hole features in the fuselage visual image;

[0018] The features of the first anchor hole and the features of the second anchor hole are compared to obtain the comparison results;

[0019] The comparison results are reduced in dimensionality using a fully connected layer in the target discriminator to obtain the first vector.

[0020] The activation function in the target discriminator is used to generate a score corresponding to the first vector, and the score is determined as the verification result.

[0021] In some embodiments of this application, the method further includes:

[0022] Obtain the training dataset; wherein, the training dataset includes the second drilling rig visual image and the corresponding second hull visual image; the second drilling rig visual image represents the image with an occlusion mask added to the anchor hole area from the drilling rig's perspective; the second hull visual image represents the hull perspective image without occlusion;

[0023] The initial image completion model is trained based on the training dataset and a preset loss function to obtain the trained image completion model. The preset loss function includes an adversarial loss function and a cycle consistency loss function. The adversarial loss function is used to characterize the degree of difference between the drilling rig visual image after mask region completion by the initial image completion model and the second hull visual image. The cycle consistency loss function is used to characterize the degree of difference between the completed drilling rig visual image and the second drilling rig visual image after the source domain direction is mapped and transformed. The source domain represents the occluded drilling rig visual image domain.

[0024] In some embodiments of this application, an initial image completion model is trained based on a training dataset and a preset loss function to obtain a trained image completion model, including:

[0025] The initial generator in the initial image completion model is used to generate the completed drilling rig visual image corresponding to the second drilling rig visual image;

[0026] The consistency between the completed drilling rig visual image and the second machine body visual image is verified by using the initial discriminator in the initial image completion model, and a second verification result is obtained.

[0027] The adversarial loss function is determined based on the second verification result;

[0028] The cycle consistency loss function is determined based on the mapping and transformation results of the completed drilling rig visual image;

[0029] The total loss function is determined based on the adversarial loss function and the cycle consistency loss function, and the parameters of the initial generator and the initial discriminator are updated based on the total loss function until the updated generator and the updated discriminator meet the convergence condition. The trained image completion model is then determined based on the updated generator and the updated discriminator.

[0030] In some embodiments of this application, the adversarial loss function is determined based on the second verification result, including:

[0031] The first adversarial loss component is determined based on the first recognition score and the second recognition score in the second verification result; wherein, the first recognition score represents the degree to which the initial discriminator recognizes the second hull visual image as a real hull visual image; the second recognition score represents the degree to which the initial discriminator recognizes the completed drilling rig visual image as a real hull visual image.

[0032] The second adversarial loss component is determined based on the third and fourth recognition scores in the second verification results. The third recognition score represents the degree to which the initial discriminator recognizes the second drilling rig visual image as a real masked drilling rig visual image. The fourth recognition score represents the degree to which the initial discriminator recognizes the transformed drilling rig visual image as a real masked drilling rig visual image. The transformed drilling rig visual image is the image obtained by mapping and transforming the source domain direction of the completed drilling rig visual image.

[0033] The adversarial loss function is determined based on the first and second adversarial loss components.

[0034] In some embodiments of this application, the cycle consistency loss function is determined based on the mapping and transformation results of the completed drilling rig visual image, including:

[0035] The cycle consistency loss function is determined based on the first mapping transformation result, the second mapping transformation result, the second drilling rig visual image, and the second hull visual image.

[0036] The first mapping transformation result represents the image obtained after mapping the supplemented drilling rig visual image in the source domain direction; the second mapping transformation result represents the image obtained after mapping the second fuselage visual image in the source domain direction and mapping the transformation result in the target domain direction; the target domain represents the unobstructed fuselage visual image domain.

[0037] In some embodiments of this application, the total loss function is determined based on the adversarial loss function and the cycle consistency loss function, including:

[0038] The total loss function value is determined by multiplying the cycle consistency loss function by the weights and the adversarial loss function.

[0039] In some embodiments of this application, the first drilling rig visual image and the first hull visual image are stitched together to obtain an input tensor, including:

[0040] A perspective transformation is performed on the first fuselage visual image to obtain a transformed fuselage visual image; wherein the transformed fuselage visual image has the same anchor hole observation angle as the first drilling rig visual image.

[0041] The input tensor is obtained by stitching together the first drilling rig visual image and the transformed hull visual image along the channel dimension.

[0042] Secondly, embodiments of this application provide an image completion device for anchor holes, including a processor and a memory storing processor-executable instructions; when the instructions are executed by the processor, the above-mentioned image completion method for anchor holes is implemented.

[0043] Therefore, this application can effectively solve the problem of positioning failure caused by the obstruction of the drilling rig's visual anchor hole during drilling and anchoring operations: First, by synchronously acquiring and stitching dual-field-of-view images, a real anchor hole reference prior is provided for completion; second, the trained image completion model can achieve real-time completion, avoiding the inefficiency of manual occlusion checks; finally, the verification step with preset consistency conditions ensures that the completed image can be directly used for anchor hole positioning of the drilling and anchoring robot, effectively improving the automation accuracy and efficiency of drilling and anchoring operations; thus, it can well adapt to complex scenarios where dust or water mist randomly obstructs the view in the tunnel, ensuring the reliability of the anchor hole image. Attached Figure Description

[0044] To more intuitively illustrate the prior art and this application, several exemplary figures are provided below. It should be understood that the specific shapes and structures shown in the figures should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary figures, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0045] Figure 1 A schematic diagram illustrating the implementation process of the image completion method for anchor holes provided in this application embodiment;

[0046] Figure 2 A schematic diagram of a target generator provided in an embodiment of this application;

[0047] Figure 3 A schematic diagram of a target discriminator provided in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the CycleGAN network structure;

[0049] Figure 5 A schematic diagram of the training dataset provided in the embodiments of this application;

[0050] Figure 6 This is a schematic diagram showing the comparison of experimental results provided in the embodiments of this application;

[0051] Figure 7 A schematic diagram of the composition of the image completion device for the anchor hole proposed in this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Any combination of different embodiments is possible.

[0053] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).

[0054] This application provides an image completion method for anchor holes, such as... Figure 1 As shown, the method for image completion of anchor holes may include the following steps:

[0055] Step 101: Acquire the first drilling rig visual image and the first machine body visual image during the drilling and anchoring operation; wherein, the first drilling rig visual image represents the anchor hole image collected from the drilling rig's perspective and which is obscured by dust or water mist; the first machine body visual image represents the image containing the anchor hole area collected from the machine body's perspective.

[0056] In the embodiments of this application, the anchor hole image completion device can acquire a first drilling rig visual image and a first machine body visual image during the drilling and anchoring operation; wherein, the first drilling rig visual image represents an anchor hole image acquired from the drilling rig's perspective that is obscured by dust or water mist; the first machine body visual image represents an image containing the anchor hole area acquired from the machine body's perspective of the drilling and anchoring robot.

[0057] In the embodiments of this application, the image completion device for anchor holes is an electronic device with data processing and computing capabilities, such as any of the hardware devices with computing functions, such as computers, industrial control servers, intelligent monitoring terminals, and edge computing devices. This application does not make any specific limitation.

[0058] In the embodiments of this application, the first drilling rig visual image can be an anchor hole image collected by the visual sensor mounted on the drilling actuator of the drilling and anchoring robot from the drilling rig's operating perspective. This image is partially or completely obscured by dust and water mist generated during drilling (for example, the upper half of the anchor hole is covered by 20% dust).

[0059] In the embodiments of this application, the first body visual image can be an image containing the anchor hole collected by the visual sensor mounted on the body of the drilling and anchoring robot from a wider body viewpoint. This viewpoint is larger than the drilling rig viewpoint, and the anchor hole area is unobstructed (e.g., the circular outline of the anchor hole and the texture boundary with the rock wall are fully presented).

[0060] Step 102: Stitch together the first drilling rig visual image and the first machine body visual image to obtain the input tensor.

[0061] In the embodiments of this application, after acquiring the first drilling rig visual image and the first machine body visual image during the drilling and anchoring operation, the image completion device for the anchor hole can stitch the first drilling rig visual image and the first machine body visual image together to obtain the input tensor.

[0062] In some embodiments of this application, when the image completion device for the anchor hole stitches together the first drilling rig visual image and the first body visual image to obtain the input tensor, it can perform a perspective transformation on the first body visual image to obtain a transformed body visual image; wherein, the transformed body visual image and the anchor hole observation angle of the first drilling rig visual image are consistent; and the first drilling rig visual image and the transformed body visual image are stitched together by channel dimension to obtain the input tensor.

[0063] In the embodiments of this application, the input tensor can be a 6-channel data tensor obtained by stitching together a 3-channel first drilling rig visual image and a 3-channel transformed fuselage visual image along the channel dimension.

[0064] In the embodiments of this application, the first machine body visual image can be transformed by perspective based on a preset coordinate transformation relationship, which can be the transformation relationship of the drilling rig relative to the machine body coordinate system.

[0065] Step 103: Generate an initial completed image using the target generator and input tensor in the trained image completion model; wherein, the initial completed image represents the image of the complete anchor hole structure obtained by completing the occluded part in the drilling rig visual image.

[0066] In the embodiments of this application, after stitching together the first drilling rig visual image and the first machine body visual image to obtain the input tensor, the anchor hole image completion device can use the target generator in the trained image completion model and the input tensor to generate an initial completion image; wherein, the initial completion image represents the image of the complete anchor hole structure obtained by completing the occluded part in the drilling rig visual image.

[0067] In the embodiments of this application, the target generator refers to the generator module in a trained and parameterized multi-view coupled improved Cycle Consistent Generative Adversarial Network (CycleGAN), which is used to fuse dual-view image information and fill in the occluded region.

[0068] In the embodiments of this application, the target discriminator refers to the discriminator module in the multi-view coupling improved CycleGAN model that has been trained and has fixed parameters, and is used to verify the consistency between the padded image and the real image.

[0069] In some embodiments of this application, when the image completion device for anchor holes generates an initial completed image using the target generator and input tensor in the trained image completion model, it can use a first preset convolutional layer in the target generator to extract features from the input tensor to obtain a first feature tensor. The first feature tensor includes the positional features of the occluded area and the structural and texture features of the anchor holes. A preset residual block in the target generator is used to optimize the fine-grained features in the first feature tensor to obtain a second feature tensor. A preset bottleneck layer in the target generator is used to remove noise features from the second feature tensor to obtain a third feature tensor. A preset transposed convolutional layer in the target generator is used to perform size restoration processing on the third feature tensor to obtain the initial completed image.

[0070] In the embodiments of this application, such as Figure 2 As shown, after the input layer, the first pre-defined convolutional layer may include a 2D reflection pad layer (ReflectionPad2d), a 2D convolutional layer (Conv), an instance normalization layer (InstanceNorm), a first activation function layer (Leaky ReLU), and two subsequent first convolutional modules. Each first convolutional module includes a 2D convolutional layer (Conv), an instance normalization layer (InstanceNorm), and a first activation function layer. The pre-defined residual block may include 6 repeated residual blocks. The preset bottleneck layer (Bottleneck) can include three second convolutional modules, a 2D convolutional layer (Conv), and a batch normalization layer (BN). Each second convolutional module includes a 2D convolutional layer (Conv), a batch normalization layer (BN), and a second activation function layer (ReLU). After the preset bottleneck layer and before the output layer, the preset transposed convolutional layer can include two transposed convolutional modules. Each convolutional module includes a transposed convolutional layer (ConvTranspose), an instance normalization layer (InstanceNorm), and a second activation function layer. In addition, after the preset bottleneck layer and before the output layer, the target generator also includes a 2D reflection padding layer (ReflectionPad2d) and a third activation function layer (Tanh).

[0071] In the embodiments of this application, the preset residual block can be used to enhance the learning of detailed features.

[0072] In embodiments of this application, the second feature tensor may include optimized fine-grained features such as anchor hole edge grayscale and hole diameter.

[0073] In the embodiments of this application, the preset bottleneck layer can be used to screen core features and filter noise.

[0074] In the embodiments of this application, the third feature tensor is the output result of the preset bottleneck layer, which retains only the semantic association features between the anchor hole core structure and the rock wall background.

[0075] In the embodiments of this application, the preset transposed convolutional layer can be used to restore the feature tensor to the image size.

[0076] In the embodiments of this application, optimizations are made to address the problems of blurred and incomplete details in traditional generator-based anchor hole completion: the design of six residual blocks alleviates the degradation problem of deep networks, enabling more accurate learning of fine-grained features such as anchor hole edges and diameters, thus avoiding "blurring" of the completed image; the feature filtering function of the preset bottleneck layer effectively eliminates noise interference from dust and water mist, ensuring the texture consistency between the completed anchor hole and the rock wall background; the size restoration of the preset transposed convolutional layer ensures the compatibility between the completed image and the original image, allowing direct integration with the positioning algorithm of the drilling and anchoring robot. This results in a clear and detailed anchor hole structure after completion.

[0077] Step 104: Use the target discriminator in the trained image completion model to verify the consistency between the initial completed image and the first fuselage visual image, and obtain the first verification result.

[0078] In the embodiments of this application, after the image completion device for anchor holes generates an initial completed image using the target generator and input tensor in the trained image completion model, it can use the target discriminator in the trained image completion model to verify the consistency between the initial completed image and the first fuselage visual image, and obtain a first verification result.

[0079] In the embodiments of this application, the first verification result can be a score in the range of 0 to 1 output by the target discriminator. The closer the score is to 1, the higher the consistency between the initial completed image and the first body visual image.

[0080] In some embodiments of this application, when the anchor hole image completion device verifies the consistency between the initial completed image and the first fuselage visual image using the target discriminator in the trained image completion model and obtains a first verification result, it can stitch the initial completed image and the first fuselage visual image together to obtain a verification tensor; it can extract the anchor hole structural features in the verification tensor using the second preset convolutional layer in the target discriminator; wherein, the anchor hole structural features include the first anchor hole features in the initial completed image and the second anchor hole features in the fuselage visual image; it can compare the first anchor hole features and the second anchor hole features to obtain a comparison result; it can perform dimensionality reduction processing on the comparison result using the fully connected layer in the target discriminator to obtain a first vector; it can generate a score corresponding to the first vector using the activation function in the target discriminator, and determine the score as the verification result.

[0081] In the embodiments of this application, the verification tensor refers to the 6-channel data tensor obtained by stitching the initial completed image and the first fuselage visual image along the channel dimension.

[0082] In the embodiments of this application, such as Figure 3 As shown, after the input layer, the second pre-defined convolutional layer includes a two-dimensional convolutional layer (Conv), a first activation function layer (Leaky ReLU), and three other third convolutional modules. Each third convolutional module includes a two-dimensional convolutional layer (Conv), an instance normalization layer (InstanceNorm), and a Leaky ReLU activation function layer. After the second pre-defined convolutional layer, a fully connected layer (Linear) and a fourth activation function layer (Sigmoid) may be included.

[0083] In the embodiments of this application, the first anchor hole feature can be understood as the completed anchor hole shape, position and other features, and the second anchor hole feature can be understood as the actual anchor hole shape, position and other features in the fuselage visual image.

[0084] In embodiments of this application, a fully connected layer can be used to convert a feature map into a one-dimensional vector.

[0085] In the embodiments of this application, the activation function can be the Sigmoid function, which is used to output a consistency score in the range of 0 to 1.

[0086] In the embodiments of this application, the dual-input verification tensor design allows the discriminator to directly compare the anchor hole features of the completed image with those of the real image, avoiding the one-sidedness of single-image verification; the feature extraction of the second preset convolutional layer focuses on the core structure (shape and position) of the anchor hole, rather than irrelevant background noise; the combination of the fully connected layer and the Sigmoid function achieves consistent quantitative scoring, making the quality of the completion result measurable.

[0087] Step 105: If the first verification result meets the preset consistency conditions, the initial completed image is determined as the target completed image of the anchor hole; otherwise, the image completion process is performed again by re-acquiring images from the drilling rig perspective and the machine body perspective until the target completed image that meets the preset consistency requirements is obtained.

[0088] In the embodiments of this application, the anchor hole image completion device verifies the consistency between the initial completed image and the first fuselage visual image using the target discriminator in the trained image completion model. After obtaining the first verification result, if the first verification result meets the preset consistency conditions, the initial completed image can be determined as the target completed image of the anchor hole; otherwise, the image completion process is performed again by re-acquiring images from the drilling rig perspective and the fuselage perspective until a target completed image that meets the preset consistency requirements is obtained.

[0089] In the embodiments of this application, the preset consistency condition can be that the first verification result is greater than or equal to the preset consistency score threshold. The specific value of the preset consistency score threshold is not limited in this application. For example, the preset consistency score threshold is 0.8. When the first verification result is ≥0.8, that is, the matching degree between the supplemented image and the real anchor hole reaches more than 80%, it can be determined that the preset consistency condition is met.

[0090] In the embodiments of this application, the target completion image refers to a completion image that meets the preset consistency conditions; the target completion image is free from dust or water mist obstruction, the anchor hole structure is complete, and the semantics are consistent with the real anchor hole.

[0091] In some embodiments of this application, before generating the initial completed image using the target generator and input tensor in the trained image completion model, the anchor hole image completion method may further include the following steps:

[0092] Step 201: Obtain the training dataset; wherein, the training dataset includes the second drilling rig visual image and the corresponding second hull visual image; the second drilling rig visual image represents the image with an occlusion mask added to the anchor hole area under the drilling rig view; the second hull visual image represents the hull view image without occlusion.

[0093] In the embodiments of this application, the anchor hole image completion device can acquire a training dataset before generating an initial completed image using the target generator and input tensor in the trained image completion model; wherein, the training dataset includes a second drilling rig visual image and a corresponding second fuselage visual image; the second drilling rig visual image represents an image with an occlusion mask added to the anchor hole region under the drilling rig's viewpoint; the second fuselage visual image represents an unoccluded fuselage viewpoint image.

[0094] Step 202: Train the initial image completion model based on the training dataset and the preset loss function to obtain the trained image completion model. The preset loss function includes an adversarial loss function and a cycle consistency loss function. The adversarial loss function is used to characterize the degree of difference between the drilling rig visual image after the masked region is completed by the initial image completion model and the second hull visual image. The cycle consistency loss function is used to characterize the degree of difference between the completed drilling rig visual image and the second drilling rig visual image after the source domain direction is mapped and transformed. The source domain represents the occluded drilling rig visual image domain.

[0095] In the embodiments of this application, after acquiring the training dataset, the image completion device for anchor holes can train the initial image completion model based on the training dataset and a preset loss function to obtain the trained image completion model.

[0096] Understandably, adding a mask to the anchor hole area in the drilling rig's visual image can simulate dust / water mist obstruction.

[0097] In the embodiments of this application, the masked second drilling rig visual image accurately simulates the random occlusion scene of dust / water mist in the tunnel, improving the generalization ability of the model; the adversarial loss function ensures the visual realism of the completed image, and the cycle consistency loss function ensures the semantic coherence between the completed image and the drilling rig scene. The combination of the two avoids the problem of the model "freely creating" anchor holes.

[0098] In some embodiments of this application, when the image completion device for anchor holes trains an initial image completion model based on a training dataset and a preset loss function to obtain a trained image completion model, it can use the initial generator in the initial image completion model to generate a completed drilling rig visual image corresponding to the second drilling rig visual image; use the initial discriminator in the initial image completion model to verify the consistency between the completed drilling rig visual image and the second rig visual image to obtain a second verification result; determine an adversarial loss function based on the second verification result; determine a cycle consistency loss function based on the mapping and transformation result of the completed drilling rig visual image; determine a total loss function based on the adversarial loss function and the cycle consistency loss function; and update the parameters of the initial generator and the initial discriminator based on the total loss function until the updated generator and the updated discriminator meet the convergence condition; and determine the trained image completion model based on the updated generator and the updated discriminator.

[0099] In embodiments of this application, the initial generator characterizes an untrained multi-view coupling improved CycleGAN generator module.

[0100] In the embodiments of this application, the initial discriminator represents an untrained multi-view coupling improved CycleGAN discriminator module.

[0101] In the embodiments of this application, the second verification result characterizes the consistency score between the completed drilling rig visual image output by the initial discriminator and the second hull visual image.

[0102] In the embodiments of this application, the convergence condition is not limited. For example, the convergence condition may be that after the model is trained iteratively 500 times, the loss function value stabilizes below 0.01.

[0103] In some embodiments of this application, when determining the adversarial loss function based on the second verification result, the image completion device for the anchor hole can determine the first adversarial loss component based on the first recognition score and the second recognition score in the second verification result; wherein, the first recognition score represents the degree to which the initial discriminator recognizes the second hull visual image as a real hull visual image; the second recognition score represents the degree to which the initial discriminator recognizes the completed drilling rig visual image as a real hull visual image; the second adversarial loss component is determined based on the third recognition score and the fourth recognition score in the second verification result; wherein, the third recognition score represents the degree to which the initial discriminator recognizes the second drilling rig visual image as a real masked drilling rig visual image; the fourth recognition score represents the degree to which the initial discriminator recognizes the converted drilling rig visual image as a real masked drilling rig visual image, the converted drilling rig visual image being the image obtained after mapping and transforming the completed drilling rig visual image in the source domain direction; the adversarial loss function is determined based on the first adversarial loss component and the second adversarial loss component.

[0104] In the embodiments of this application, the first adversarial loss component represents the visual matching loss between the completed drilling rig visual image and the real hull visual image.

[0105] In embodiments of this application, the second adversarial loss component represents the style matching loss between the transformed drilling rig visual image and the real masked drilling rig visual image.

[0106] In some embodiments of this application, the method for determining the first adversarial loss component based on the first identification score and the second identification score in the second verification result can be expressed as the following formula:

[0107] (1);

[0108] in, For the first confrontation loss component, As the first identification score, For the second identification score; This means taking a sample y from the target domain Y space (the second fuselage visual image), where the target domain is the clear fuselage visual image. ; This indicates that sample x is taken from the source domain X space (the second drilling rig visual image), where the source domain is the drilling rig visual image. , is the image of the occluded anchor hole; define mapping G as from the source domain To the target domain The mapping, This refers to the set of occluded drilling rig visual images mapped to clear rig hull visual images; it can be understood as a completed drilling rig visual image; mapping F is the target domain. To the source domain The mapping, This represents a discriminator neural network corresponding to the target domain, used to distinguish... and .

[0109] In some embodiments of this application, the method for determining the second adversarial loss component based on the third and fourth identification scores in the second verification result can be expressed as the following formula:

[0110] (2);

[0111] in, For the second confrontation loss component, For the third identification score, The fourth identification score; This represents a discriminator neural network corresponding to the source domain, used to distinguish... and , This represents the set of images that approximate the source domain after mapping F.

[0112] In some embodiments of this application, when determining the cycle consistency loss function based on the mapping transformation result of the completed drilling rig visual image, the image completion device for the anchor hole can determine the cycle consistency loss function based on the first mapping transformation result, the second mapping transformation result, the second drilling rig visual image, and the second fuselage visual image; wherein, the first mapping transformation result represents the image obtained after mapping transformation of the completed drilling rig visual image in the source domain direction; the second mapping transformation result represents the image obtained by mapping transformation of the second fuselage visual image in the source domain direction and mapping transformation of the transformation result in the target domain direction; the target domain represents the unobstructed fuselage visual image domain.

[0113] In some embodiments of this application, the method for determining the cycle consistency loss function based on the mapping and transformation results of the completed drilling rig visual image can be expressed as the following formula:

[0114] (3);

[0115] in, Let the cycle consistency loss function be... This is the result of the first mapping transformation. This is the result of the second mapping transformation. This is the second drilling rig visual image corresponding to the source domain. This is the second fuselage visual image corresponding to the target domain.

[0116] In some embodiments of this application, when determining the total loss function based on the adversarial loss function and the cycle consistency loss function, the image completion device for anchor holes can determine the value of the total loss function based on the product of the cycle consistency loss function and the weights, as well as the adversarial loss function.

[0117] In some embodiments of this application, the method for determining the total loss function value based on the product of the cycle consistency loss function and the weights, and the adversarial loss function, can be expressed as the following formula:

[0118] + (4);

[0119] in, As weight.

[0120] This application provides an image completion method for anchor holes, which breaks through the limitations of traditional single-view anchor hole images. It innovatively adopts a simultaneous acquisition scheme of dual fields of view, namely "drilling rig vision (occluded anchor hole) + machine vision (unobstructed, wide-angle view)," to provide a priori information on the actual anchor hole structure for completion. This effectively solves problems such as shape distortion and positional offset of anchor hole images when the occlusion rate of the drilling rig view is high, significantly improving the consistency between the completed anchor hole image and the actual anchor hole structure, and effectively supporting the precise positioning requirements of drilling and anchoring robots. The structures of the generator and discriminator were specifically improved. The generator, through a multi-stage process of "feature extraction from the first preset convolutional layer, fine-grained optimization of the preset residual block, noise filtering from the preset bottleneck layer, and size restoration from the preset transposed convolutional layer," can accurately capture the structure and texture features of the anchor hole and avoid completion distortion. The discriminator adopts a dual-input stitching verification mode. After extracting the anchor hole structure features through the second preset convolutional layer, dimensionality reduction through the fully connected layer, and quantization through the activation function, it outputs a consistency score to achieve accurate verification of the completion results. This can completely solve the problem of poor anchor hole image quality leading to difficulty in anchor hole positioning, thereby greatly improving the efficiency of drilling and anchoring operations.

[0121] Based on the above embodiments, in another embodiment of this application, considering that after the drilling point and anchor hole positions are obtained, the drilling rig can complete the drilling and anchoring operations according to three-dimensional coordinates, but in actual work, drilling is accompanied by dust and water mist. The mixture of the two will obstruct the drilling rig's vision sensor, making it difficult for subsequent anchor hole identification and positioning; this embodiment studies the supplementary visual information of the machine body and drilling rig through multi-field coupling, and relies on the image obtained by the machine body vision to repair and fill in the obstructed area of ​​the drilling rig vision.

[0122] CycleGAN is a deep learning framework for unsupervised image-to-image translation. It achieves image translation by learning the mapping relationship between two image domains. Its structure is as follows: Figure 4As shown, the CycleGAN network mainly consists of two generators and two discriminators. The discriminator includes a convolutional layer (Conv) and an instance normalization layer (InstanceNorm). The generator comprises an encoder, a transformer, a decoder, a residual block, and a transposed convolutional layer (ConvTranspose), responsible for converting the source domain image to the target domain image and vice versa, forming a loop. The encoding structure uses a convolutional neural network to extract features from the input image, compressing the image into 256 64×64 feature vectors. By combining dissimilar features of the image, the feature vectors in the source domain are converted to feature vectors in the target domain. Four residual blocks are used to improve the feature extraction capability of the generator and prevent degradation phenomena in deep networks. Finally, transposed convolution is used to recover low-level features from the feature vectors, obtaining the generated image.

[0123] The discriminator takes an image as input and attempts to predict whether it is the original image or the generator's output. Its structure consists of five convolutional layers: the first layer converts the input's 3 channels to 64 channels, halving the image size. Subsequent layers gradually extract more complex features, increasing the output channel count from 128 to 512. Each convolutional layer also halves the image size. Except for the last layer, instance normalization is applied after each convolutional layer to improve model stability and training speed. The Leaky ReLU activation function is used to help mitigate the "dead neuron" phenomenon. The final layer maps the feature map to an output representing the probability that the input image is real or fake.

[0124] In the embodiments of this application, the single image input of the original CycleGAN is changed to a dual-channel input, that is, the occluded drilling rig visual image and the corresponding fuselage visual image are input simultaneously to form a 6-channel input tensor; the number of residual blocks is increased to 6 to enhance feature extraction capability and alleviate the gradient vanishing problem; a bottleneck layer is introduced after the residual blocks, and the feature dimension is further optimized through three convolutional layers to improve the recovery capability of complex textures; at the same time, the input of the discriminator is also expanded to 6 channels, and the fuselage visual image and the generated image are concatenated in the channel dimension; the terminal convolutional layer is replaced with a fully connected layer, and a sigmoid activation function is used to achieve global discrimination capability and improve the accuracy of evaluating the realism of the generated image; an adversarial loss and a cycle consistency loss are combined; the adversarial loss ensures that the generated image is visually close to the target domain image; the cycle consistency loss ensures the consistency of the content structure of the image during the transformation process.

[0125] In the embodiments of this application, the input layer is modified to have dual inputs. The image captured by the camera's vision is used as the target to generate a completed image. The blurred image is used as the processing object. The number of input channels is increased from 3 (single image) to 6. The two images are concatenated along the channel dimension to form an input tensor. Secondly, the number of residual blocks is increased to 6. More residual connections improve feature extraction capabilities and help alleviate the gradient vanishing problem. Finally, a bottleneck layer, Bottleneck, is defined and added after the residual blocks. The bottleneck layer consists of three main convolutional layers to further process the features generated by the residual layer, optimize feature dimensions, and capture more complex feature information.

[0126] In the embodiments of this application, the structure of the discriminator is improved. Its input end is also modified to 6 channels. The machine vision image and the image generated by the generator are concatenated in the channel dimension. The terminal convolution is modified to a fully connected layer and the sigmoid activation function is applied. The flattened feature map is mapped to a single output. The output value usually represents the degree of authenticity of the input image. The closer the value is to 1, the more authentic it is, i.e., a real image. The closer it is to 0, the more fake it is.

[0127] In the embodiments of this application, the structure of the improved generator is as described above. Figure 2 As shown, firstly, the input layer is modified to have dual inputs, using the image captured by the camera's vision as the target and generating a completed image. The blurred image is then processed, and the number of input channels is increased from 3 to 6. The two images are then concatenated along the channel dimension to form an input tensor. Secondly, the number of residual blocks is increased to 6, improving feature extraction capabilities through more residual connections and helping to alleviate the gradient vanishing problem. Finally, a bottleneck layer, Bottleneck, is defined and added after the residual blocks. This bottleneck layer consists of three main convolutional layers, further processing the features generated by the residual layer, optimizing the feature dimensions, and capturing more complex feature information.

[0128] For example, the occluded visual image of the drilling rig and the transformed visual image of the hull are stitched together in the channel dimension to form a 6-channel tensor. This directly provides the generator with both the occluded local details and clear global semantic information, guiding it to perform more evidence-based completion. The generator extracts deep features from the 6-channel input through an encoder, 6 residual blocks, and a bottleneck layer. The decoder then generates a complete and clear visual image of the drilling rig based on the fused features.

[0129] In the original discriminator, one input is the original image, and the other is the image generated by the generator. The two discriminators can only distinguish whether an image is the original image or the generator's output, but cannot determine its generation quality. Therefore, this application improves the discriminator structure by modifying its input to also 6 channels, concatenating the camera's visual image and the generator-generated image along the channel dimension, and changing the terminal convolution to a fully connected layer. A sigmoid activation function is applied, mapping the flattened feature map to a single output. The output value typically represents the realism of the input image; the closer the value is to 1, the more realistic it is (i.e., a real image), and the closer it is to 0, the more likely it is a forgery.

[0130] In the embodiments of this application, for the improved network, the loss function adopts adversarial loss and cycle consistency loss. The adversarial loss is used to ensure that the image generated by the generator looks like the image in the target domain. This application divides the image completion work of the CycleGAN network into processes, defining mapping G as the mapping from the source domain X to the target domain Y, and mapping F as the mapping from the target domain Y to the source domain X. The adversarial loss can be determined based on the aforementioned formulas (1) and (2).

[0131] In the embodiments of this application, the cycle consistency loss ensures that the consistency between the input and generated images is maintained during the conversion process. That is, if image x is first converted to image y, and then image y is converted back to the original image x, the conversion result should be as consistent as possible with the original image x. The cycle consistency loss can be determined based on the aforementioned formula (3). The overall loss function can be determined based on the aforementioned formula (4).

[0132] For example, during network training, the loss function continuously calculates adversarial loss and cycle consistency loss through forward and backward propagation, thereby updating the parameters of the generator and discriminator, enabling them to learn how to utilize multi-view information for high-quality completion. Specific steps may include: using a preset coordinate transformation relationship to perform perspective transformation on the rig's visual image, generating a reference image (transformed rig visual image) aligned with the drilling rig's visual image perspective; concatenating the occluded image and the aligned reference image along the channel dimension to form a 6-channel tensor input generator; processing the 6-channel tensor by the improved generator, the network learns to extract effective features from it to fill in the missing parts, ultimately outputting a completed and clear drilling rig visual image; concatenating the completed image with the reference image again, and inputting it into the improved discriminator. The discriminator's task is to determine whether it is realistic enough, outputting a probability value close to 1 (true) or 0 (false). Furthermore, during model training, the loss function can be calculated based on the discriminator's output, and the network weights of the generator and discriminator can be updated through the backpropagation algorithm, driving their continuous updates. As can be seen, by using deep convolutional networks to extract deep features from the stitched data, the network automatically learns how to repair missing areas of damaged images from the texture and structure of reference images.

[0133] Furthermore, to verify the effectiveness of the improved CycleGAN network proposed in this application for image restoration of the vision sensor of a coal mine drilling robot, an ablation experiment was designed. Before the experiment, 5732 anchor hole images were collected from different angles in a simulated tunnel environment using both the robot's own vision and the drilling rig's vision. To verify the image completion effect, masks were added to 2866 drilling rig vision images in the dataset to simulate the occlusion of the drilling rig's vision sensor by dust and water mist generated during drilling operations. The mask positions were all located at the anchor holes (if the anchor holes were not occluded, their position information could be directly extracted without image completion). All images in the dataset were 1280×620 resolution. To achieve quantitative comparison, three levels of mask coverage were set: 20%, 50%, and 80%, respectively. Figure 5 As shown, 8598 masked visual images of the drilling rig can be generated.

[0134] The evaluation metrics selected for the image completion experiment were Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Fréchet Inception Distance (FID). PSNR is a commonly used metric for evaluating the quality of reconstructed images. It assesses image quality by comparing the difference between the reconstructed image and the original image; a higher value indicates better image quality. PSNR can be expressed as follows:

[0135] (5);

[0136] Where R represents the maximum possible value of a pixel in the image; MSE represents the mean squared error, which can be calculated using the following formula:

[0137] (6);

[0138] Where I represents the original image, K represents the reconstructed image, and M and N are the height and width of the image, respectively.

[0139] SSIM is an index that measures the similarity between two images in terms of brightness, contrast, and structure. It is more in line with human visual perception. The closer the value is to 1, the better the image structure is preserved. It can be calculated using the following formula:

[0140] (7);

[0141] in, and It is the average of the x and y values ​​of the image. and It is the variance of the x and y values ​​of the image. It is covariance. and It is a constant to prevent the denominator from being zero.

[0142] FID (Features Identifier) ​​is a metric used to evaluate the quality of images generated by a generative model. It measures the similarity between generated and real images by comparing their feature distributions; FID can be expressed as the following formula:

[0143] (8);

[0144] in, and These are the feature mean values ​​of the real image and the generated image, respectively. and These are the covariance matrices of the features of the real and generated images, respectively. Represents the trace of a matrix.

[0145] To verify the image completion effect of the improved CycleGAN network in this paper, an ablation experiment was designed to compare the four models shown in Table 1 under three-level mask coverage.

[0146]

[0147] Table 2 below shows the results of the quantitative comparison:

[0148]

[0149] As shown in Table 2, all metrics of models A through D are optimized under the same coverage, with model D (the image completion model of this application) achieving the best performance across all metrics. This indicates that a clear image of the fuselage provides crucial prior structural information, the global discriminator (fully connected layer) can better evaluate the realism of the generated image, and the bottleneck layer enhances the recovery capability of complex textures through feature compression. However, as the mask coverage increases, the performance of each model decreases under all three metrics. The degree of image blurring directly affects the completion effect, increasing the difficulty of image restoration. Model D effectively mitigates this problem compared to the original model.

[0150] Furthermore, such as Figure 6 The figure shows a qualitative comparison of four models on a self-built dataset. The boxes in the figure represent the mask coverage area, and the ellipses highlight the obvious differences between the generated and original images. As can be seen from the figure, Model D performs better than the other three models in restoring the anchor hole image. Model A, being an unsupervised restoration, generates a somewhat arbitrary restored image, resulting in inconsistencies in color and semantics. In contrast, Models B and C, due to the input of the aircraft's visual image, restore some semantic information to a certain extent, but the mask boundaries are blurred, and there are some differences in lines and colors. Model D solves the semantic and color discontinuities and further optimizes the boundary information. Although the anchor hole angle is slightly off, it is more realistic compared to the original image.

[0151] For example, when constructing a self-built dataset, in a simulated coal mine roadway environment, a drilling and anchoring robot can be used to collect original images of anchor holes from both the drilling rig's perspective and the machine's perspective. Then, for the anchor hole area in the original drilling rig perspective image, occlusion masks with different coverage rates are manually added, setting three occlusion ratios of 20%, 50%, and 80% respectively, to simulate the random occlusion of visual images by dust and water mist in actual drilling and anchoring operations, forming occluded drilling rig perspective samples. Then, each occluded drilling rig perspective sample can be paired one-to-one with an unoccluded machine's perspective sample collected under the same working conditions and the same anchor hole to form "occluded-clear" paired samples. All paired samples are then subjected to uniform size normalization and grayscale standardization processing to remove invalid samples such as blurry or misaligned samples, ensuring that the dataset format is uniform and the annotation is standardized, thus constituting a self-built dataset.

[0152] In summary, compared with existing technologies, the improved CycleGAN network proposed in this application outperforms the original model in terms of PSNR, SSIM, and FID in anchor hole image completion and repair. Moreover, it can still produce semantically and color-consistent results even when the image occlusion area is large, and it has a certain generalization ability for different degrees of image damage.

[0153] Based on the above embodiments, another embodiment of this application provides an image completion device for anchor holes, such as... Figure 7 As shown, the image completion device 1 for anchor holes proposed in this application embodiment may include a processor 11 and a memory 12 storing instructions executable by the processor 11; further, the image completion device 1 for anchor holes may also include a communication interface 13 and a bus 14 for connecting the processor 11, the memory 12 and the communication interface 13.

[0154] In the embodiments of this application, the processor 11 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and this application embodiment does not specifically limit it. The memory 12 can be connected to the processor 11, wherein the memory 12 is used to store executable program code, which includes computer operation instructions. The memory 12 may include high-speed RAM memory, and may also include non-volatile memory, such as at least two disk drives.

[0155] In embodiments of this application, bus 14 is used to connect communication interface 13, processor 11 and memory 12 to enable communication between these devices.

[0156] In embodiments of this application, memory 12 is used to store instructions and data.

[0157] Furthermore, the processor 11 can be used to acquire a first drilling rig visual image and a first body visual image during the drilling and anchoring operation; wherein, the first drilling rig visual image represents an image of the anchor hole that is obscured by dust or water mist, acquired from the drilling rig's perspective; the first body visual image represents an image containing the anchor hole area, acquired from the body perspective of the drilling and anchoring robot; the first drilling rig visual image and the first body visual image are stitched together to obtain an input tensor; an initial completed image is generated using the target generator in the trained image completion model and the input tensor; wherein, the initial completed image represents an image of the complete anchor hole structure obtained by completing the obscured parts in the drilling rig visual image; the consistency between the initial completed image and the first body visual image is verified using the target discriminator in the trained image completion model to obtain a first verification result; if the first verification result meets the preset consistency conditions, the initial completed image is determined as the target completed image of the anchor hole; otherwise, the images from the drilling rig's perspective and the body's perspective are reacquired to perform the image completion process until a target completed image that meets the preset consistency requirements is obtained.

[0158] In practical applications, the aforementioned memory 12 can be volatile memory, such as random-access memory (RAM), or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 11.

[0159] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0160] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment.

[0161] Specifically, the program instructions corresponding to the image completion method for anchor holes in this embodiment can be stored on storage media such as optical discs and hard disks. When the program instructions corresponding to the image completion method for anchor holes in the storage media are read or executed by an image completion device for anchor holes, the following steps are included:

[0162] Acquire a first drilling rig visual image and a first machine body visual image during the drilling and anchoring operation; wherein, the first drilling rig visual image represents an image of the anchor hole that is obscured by dust or water mist and is collected from the drilling rig's perspective; the first machine body visual image represents an image containing the anchor hole area collected from the machine body's perspective.

[0163] The first drilling rig visual image and the first machine body visual image are stitched together to obtain the input tensor;

[0164] The initial completed image is generated using the target generator and input tensor in the trained image completion model; whereby the initial completed image represents the image of the complete anchor hole structure obtained by completing the occluded part in the drilling rig visual image.

[0165] The consistency between the initial completed image and the first fuselage visual image is verified using the target discriminator in the trained image completion model, and the first verification result is obtained.

[0166] If the first verification result meets the preset consistency conditions, the initial completed image is determined as the target completed image of the anchor hole; otherwise, images from the drilling rig perspective and the machine body perspective are re-acquired to perform the image completion process until a target completed image that meets the preset consistency requirements is obtained.

[0167] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0168] This application is described with reference to schematic and / or block diagrams illustrating the implementation of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each step and / or block in the schematic and / or block diagrams, as well as combinations thereof, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more steps of the schematic and / or one or more blocks of the block diagrams.

[0169] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0171] The above embodiments are merely preferred embodiments provided to fully illustrate this application, and the scope of protection of this application is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this application are all within the scope of protection of this application.

Claims

1. A method for image completion of anchor holes, characterized in that, The methods include: Acquire a first drilling rig visual image and a first machine body visual image during the drilling and anchoring operation; wherein, the first drilling rig visual image represents an image of the anchor hole that is obscured by dust or water mist and is collected from the drilling rig's perspective; the first machine body visual image represents an image containing the anchor hole area collected from the machine body's perspective. The first drilling rig visual image and the first machine body visual image are stitched together to obtain the input tensor; The first feature tensor is obtained by using the first preset convolutional layer of the target generator in the trained image completion model to extract features from the input tensor; wherein, the first feature tensor includes the positional features of the occluded area, the structure and texture features of the anchor hole; The fine-grained features in the first feature tensor are optimized using the preset residual block of the target generator to obtain the second feature tensor; The noise features in the second feature tensor are removed using the preset bottleneck layer of the target generator to obtain the third feature tensor; The third feature tensor is subjected to size restoration processing using the preset transposed convolutional layer in the target generator to obtain an initial padded image; wherein, the initial padded image represents the image of the complete anchor hole structure obtained by padded the occluded part in the drilling rig visual image; The initial completed image and the first fuselage visual image are stitched together to obtain a verification tensor; The anchor hole structure features in the verification tensor are extracted using the second preset convolutional layer of the target discriminator in the trained image completion model; wherein, the anchor hole structure features include the first anchor hole features in the initial completion image and the second anchor hole features in the fuselage visual image; The features of the first anchor hole and the features of the second anchor hole are compared to obtain the comparison results; The comparison result is reduced in dimensionality using the fully connected layer in the target discriminator to obtain a first vector. The activation function in the target discriminator is used to generate a score corresponding to the first vector, and the score is determined as the first verification result; If the first verification result meets the preset consistency conditions, the initial completed image is determined as the target completed image of the anchor hole; otherwise, images from the drilling rig perspective and the machine body perspective are re-acquired to perform the image completion process until a target completed image that meets the preset consistency requirements is obtained.

2. The image completion method for anchor holes according to claim 1, characterized in that, The method further includes: Obtain a training dataset; wherein the training dataset includes a second drilling rig visual image and a corresponding second hull visual image; the second drilling rig visual image represents an image with an occlusion mask added to the anchor hole area from the drilling rig's perspective; the second hull visual image represents an unoccluded hull perspective image; The initial image completion model is trained based on the training dataset and the preset loss function to obtain the trained image completion model. The preset loss function includes an adversarial loss function and a cycle consistency loss function. The adversarial loss function characterizes the difference between the drill rig visual image completed by the initial image completion model and the second rig visual image. The cycle consistency loss function characterizes the difference between the completed drill rig visual image and the second drill rig visual image after mapping and transforming the source domain direction. The source domain characterizes the occluded drill rig visual image domain.

3. The image completion method for anchor holes according to claim 2, characterized in that, The step of training the initial image completion model based on the training dataset and a preset loss function to obtain the trained image completion model includes: The initial generator in the initial image completion model is used to generate the completed drilling rig visual image corresponding to the second drilling rig visual image; The consistency between the completed drilling rig visual image and the second machine body visual image is verified using the initial discriminator in the initial image completion model to obtain a second verification result; The adversarial loss function is determined based on the second verification result; The cycle consistency loss function is determined based on the mapping and transformation results of the completed drilling rig visual image; The total loss function is determined based on the adversarial loss function and the cycle consistency loss function, and the parameters of the initial generator and the initial discriminator are updated based on the total loss function until the updated generator and the updated discriminator meet the convergence condition. The trained image completion model is then determined based on the updated generator and the updated discriminator.

4. The image completion method for anchor holes according to claim 3, characterized in that, The step of determining the adversarial loss function based on the second verification result includes: The first adversarial loss component is determined based on the first recognition score and the second recognition score in the second verification result; wherein, the first recognition score represents the degree to which the initial discriminator recognizes the second hull visual image as a real hull visual image; the second recognition score represents the degree to which the initial discriminator recognizes the completed drilling rig visual image as a real hull visual image. The second adversarial loss component is determined based on the third and fourth identification scores in the second verification result; wherein, the third identification score represents the degree to which the initial discriminator identifies the second drilling rig visual image as a real masked drilling rig visual image; the fourth identification score represents the degree to which the initial discriminator identifies the transformed drilling rig visual image as a real masked drilling rig visual image, and the transformed drilling rig visual image is the image obtained by mapping and transforming the source domain direction of the completed drilling rig visual image; The adversarial loss function is determined based on the first adversarial loss component and the second adversarial loss component.

5. The image completion method for anchor holes according to claim 4, characterized in that, The determination of the cycle consistency loss function based on the mapping and transformation result of the completed drilling rig visual image includes: The cycle consistency loss function is determined based on the first mapping transformation result, the second mapping transformation result, the second drilling rig visual image, and the second hull visual image. Wherein, the first mapping transformation result represents the image obtained after mapping transformation of the supplemented drilling rig visual image in the source domain direction; the second mapping transformation result represents the image obtained after mapping transformation of the second fuselage visual image in the source domain direction and mapping transformation of the transformation result in the target domain direction; the target domain represents the unobstructed fuselage visual image domain.

6. The image completion method for anchor holes according to claim 5, characterized in that, The determination of the total loss function based on the adversarial loss function and the cycle consistency loss function includes: The total loss function value is determined based on the product of the cycle consistency loss function and the weights, and the adversarial loss function.

7. The image completion method for anchor holes according to claim 1, characterized in that, The step of stitching together the first drilling rig visual image and the first machine body visual image to obtain the input tensor includes: A perspective transformation is performed on the first fuselage visual image to obtain a transformed fuselage visual image; wherein the transformed fuselage visual image has the same anchor hole observation angle as the first drilling rig visual image. The input tensor is obtained by stitching the first drilling rig visual image and the transformed hull visual image together along the channel dimension.

8. An image completion device for anchor holes, characterized in that, The image completion device for anchor holes includes a processor and a memory storing executable instructions of the processor; when the executable instructions are executed by the processor, the image completion method for anchor holes as described in any one of claims 1 to 7 is implemented.

Images