A method, apparatus and device for completing a depth map of a rotating target

Abstract

The application discloses a kind of rotating target depth map hole completion method, device and equipment, utilize the rotating target detection model based on connecting pipe to identify connecting pipe in color chart, obtain the positioning result of rotating rectangular frame containing inclination angle, then the horizontal rectangular frame corresponding to rotating rectangular frame positioning result is mapped into depth chart, based on rotating rectangular frame positioning result, the regional depth value and depth missing point coordinates of horizontal rectangular frame are extracted, the multiscale similarity of missing point and neighborhood is calculated, the completion order of depth missing point is determined in combination with target inclination angle, to complete the depth missing point in horizontal rectangular frame according to the calculation result of multiscale similarity and Euclidean distance completion order.It solves the technical problem that the rotation angle constraint of connecting pipe is often ignored when combining depth information in prior art, leading to the hole in depth chart cannot be accurately completed, it is difficult to help unmanned aerial vehicle to more accurately identify and locate connecting pipe, influence the three-dimensional reconstruction effect of connecting pipe.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a rotating target depth map hole completion method, device and equipment. BACKGROUND

[0002] In the maintenance process of overhead transmission lines, the splicing pipe, as a key component, plays a role in connecting the transmission line. The traditional splicing pipe inspection and maintenance mostly rely on manual operation, which is not only inefficient, but also has certain safety risks. As an efficient inspection tool, the unmanned aerial vehicle is gradually applied in the detection and maintenance of the transmission line. However, in the process of automatic inspection of the unmanned aerial vehicle, the positioning and identification of the splicing pipe still face many challenges. The existing research on the identification and positioning of the splicing pipe does not consider how to effectively solve the problem of depth map hole completion, especially for the splicing pipe with an arbitrary angle in a long-distance aerial photography. The existing technology often ignores the rotation angle constraint of the splicing pipe when combining depth information, which leads to the fact that the holes in the depth map cannot be accurately completed, and it is difficult to help the unmanned aerial vehicle to more accurately identify and position the splicing pipe, thereby affecting the three-dimensional reconstruction effect of the splicing pipe. SUMMARY

[0003] The present application provides a rotating target depth map hole completion method, device and equipment, which is used to solve the technical problem that the existing technology often ignores the rotation angle constraint of the splicing pipe when combining depth information, which leads to the fact that the holes in the depth map cannot be accurately completed, and it is difficult to help the unmanned aerial vehicle to more accurately identify and position the splicing pipe, thereby affecting the three-dimensional reconstruction effect of the splicing pipe.

[0004] Therefore, the present application provides a rotating target depth map hole completion method, which comprises the following steps:

[0005] Based on the rotating target data set of the overhead transmission line splicing pipe, a rotating target detection model for identifying the overhead transmission line splicing pipe is trained;

[0006] A color map and a depth map including a rotating target splicing pipe are obtained by a depth camera;

[0007] The splicing pipe in the color map is identified based on the rotating target detection model, and a rotating rectangular frame positioning result of the splicing pipe is obtained, wherein the rotating rectangular frame positioning result comprises an inclination angle of the rotating rectangular frame;

[0008] A horizontal rectangular frame corresponding to the rotating rectangular frame positioning result is mapped to the depth map, and the area depth value and the depth missing point coordinates of the horizontal rectangular frame in the depth map are extracted;

[0009] The multi-scale similarity of the missing points in the depth map and the neighborhood in the color map is calculated;

[0010] According to the rotation rectangular frame positioning result and the Euclidean distance from the depth missing point in the depth map to a straight line passing through the center point of the rotation rectangular frame and perpendicular to the long side of the rotation rectangular frame, a completion order is set;

[0011] Based on the calculation result of the multi-scale similarity, the depth missing points in the horizontal rectangular frame are completed according to the completion order and a depth value completion formula.

[0012] Optionally, based on the calculation result of the multi-scale similarity, the depth missing points in the horizontal rectangular frame are completed according to the completion order, and then further comprising:

[0013] According to the inclination angle of the rotation rectangular frame, the distance value of the depth camera and the connecting pipe is extracted.

[0014] Optionally, the rotation target detection model is a rotation target detection model based on a yolov5_OBB network structure.

[0015] Optionally, the horizontal rectangular frame corresponding to the rotation rectangular frame positioning result is mapped to the depth map, and the regional depth value and the depth missing point coordinates of the horizontal rectangular frame are extracted, comprising:

[0016] According to the rotation rectangular frame positioning result, the horizontal rectangular frame corresponding to the rotation rectangular frame positioning result is calculated based on the positioning result of the rotation rectangular frame.

[0017] The horizontal rectangular frame is mapped to a horizontal rectangular frame in the depth map, and the regional depth value of the horizontal rectangular frame in the depth map is extracted.

[0018] Based on the regional depth value, the depth missing point coordinates of the horizontal rectangular frame in the depth map are extracted by using a threshold method.

[0019] Optionally, the multi-scale similarity between the missing point in the depth map and the neighborhood in the color map is calculated, comprising:

[0020] Based on the structural similarity principle, the multi-scale similarity between the missing point in the depth map and the neighborhood in the color map is calculated, and the multi-scale similarity calculation model is:

[0021]

[0022] wherein, is a multi-scale similarity function, is a pixel value corresponding to a 3*3 region with the pixel point p as the center, is a pixel value corresponding to a 3*3 region with q as the center, q is a neighboring pixel point of the pixel point p, is a brightness comparison function, a contrast comparison function, a structure comparison function, a weight factor of a weight factor of a weight factor of

[0023] Optionally, the depth value completion formula is:

[0024]

[0025] wherein, a depth value to be completed at a same coordinate position in a corresponding depth map, a multi-scale similarity of two adjacent pixels and in a color image, a depth value at a same coordinate position in a corresponding depth map.

[0026] Optionally, according to the rotation rectangular frame positioning result and the Euclidean distance from the depth missing point in the depth map to a straight line passing through the center point of the rotation rectangular frame and being perpendicular to the long side of the rotation rectangular frame, a completion order is set, comprising:

[0027] determining, according to the rotation rectangular frame positioning result, the depth missing point in the depth map to the straight line passing through the center point of the rotation rectangular frame and being perpendicular to the long side of the rotation rectangular frame;

[0028] calculating the Euclidean distance from the depth missing point in the depth map to the straight line;

[0029] ordering the Euclidean distance from the depth missing point in the rotation rectangular frame in the depth map to the straight line from small to large to determine the completion order.

[0030] Optionally, according to the tilt angle of the rotation rectangular frame, a distance value of the depth camera and the connecting pipe is extracted, comprising:

[0031] according to the tilt angle of the rotation rectangular frame, extracting, in the rotation target depth map, a mean value of depth values within a neighborhood preset range along a center line passing through the center point of the rotation rectangular frame, the mean value being the distance value of the depth camera and the connecting pipe;

[0032] the center line is:

[0033]

[0034] wherein, a tilt angle of a rotation rectangular frame, is a point on the center line, is a coordinate of a center point of the rotated rectangular frame.

[0035] The second aspect of the present application provides a device for completing a hole in a depth map of a rotating target, comprising:

[0036] A model construction module is configured to train a rotating target detection model for identifying an overhead transmission line splicing pipe based on a rotating target data set of the overhead transmission line splicing pipe.

[0037] An image acquisition module is configured to acquire a color image and a depth image including a rotating target splicing pipe through a depth camera.

[0038] A target identification module is configured to identify the splicing pipe in the color image based on the rotating target detection model, and obtain a rotated rectangular frame positioning result of the splicing pipe, wherein the rotated rectangular frame positioning result includes an inclination angle of the rotated rectangular frame.

[0039] A depth processing module is configured to map a horizontal rectangular frame corresponding to the rotated rectangular frame positioning result to the depth image, and extract a region depth value and a depth missing point coordinate of the horizontal rectangular frame in the depth image.

[0040] A similarity calculation module is configured to calculate a multi-scale similarity between the missing point in the depth image and a neighborhood in the color image.

[0041] A completion order determination module is configured to set a completion order according to the rotated rectangular frame positioning result and a Euclidean distance from a depth missing point in the depth image to a straight line passing through a center point of the rotated rectangular frame and being perpendicular to a long side of the rotated rectangular frame.

[0042] A completion module is configured to complete the depth missing point in the horizontal rectangular frame based on a calculation result of the multi-scale similarity, the completion order, and a depth value completion formula.

[0043] Optionally, the device further comprises:

[0044] A distance calculation module is configured to extract a distance value between the depth camera and the splicing pipe according to the inclination angle of the rotated rectangular frame.

[0045] Optionally, the rotating target detection model is a rotating target detection model based on a yolov5_OBB network structure.

[0046] Optionally, the depth processing module is specifically configured to:

[0047] According to the rotated rectangular frame positioning result, a horizontal rectangular frame corresponding to the rotated rectangular frame positioning result is calculated based on the positioning result of the rotated rectangular frame.

[0048] mapping the horizontal rectangular frame to a horizontal rectangular frame in the depth map, and extracting a region depth value of the horizontal rectangular frame in the depth map;

[0049] based on the region depth value, extracting a depth missing point coordinate of the horizontal rectangular frame in the depth map by using a threshold method.

[0050] Optionally, the similarity calculation module is specifically configured to:

[0051] based on a structural similarity principle, calculating a multi-scale similarity of the missing point and a neighborhood in the depth map in the color map, the multi-scale similarity calculation model being:

[0052]

[0053] wherein, is a multi-scale similarity function, is a corresponding pixel value in a 3*3 region centered on a pixel point p in the depth map, is a corresponding pixel value in a 3*3 region centered on q, q being a neighboring pixel point of the pixel point p, is a luminance comparison function, is a contrast comparison function, is a structure comparison function, is a weight factor of is a weight factor of is a weight factor of

[0054] Optionally, the depth value completion formula is:

[0055]

[0056] wherein, is a depth value to be completed at a same coordinate position in the corresponding depth map, is a multi-scale similarity of two neighboring pixels and in the color map, is a depth value at the same coordinate position in the corresponding depth map.

[0057] Optionally, the completion sequence determination module is specifically configured to:

[0058] determining, according to the rotated rectangular frame positioning result, a straight line passing through a center point of the rotated rectangular frame and being perpendicular to a long side of the rotated rectangular frame;

[0059] calculating Euclidean distances from the depth missing points in the depth map to the straight line;

[0060] ordering the Euclidean distances from the depth missing points in the rotating rectangular frame in the depth map to the straight line in ascending order to determine a completion order.

[0061] Optionally, the distance calculation module is specifically configured to:

[0062] extracting, according to the tilt angle of the rotating rectangular frame, a mean value of depth values of the rotating target depth map in a neighborhood preset range along a center line from the center point of the rotating rectangular frame, the mean value being a distance value of the depth camera and the connecting pipe;

[0063] the center line is:

[0064]

[0065] wherein, the tilt angle of the rotating rectangular frame, the point on the center line, the center point coordinate of the rotating rectangular frame.

[0066] The third aspect of the present application provides a rotating target depth map hole completion device, the device comprising a processor and a memory:

[0067] The memory is used to store program code and transmit the program code to the processor;

[0068] The processor is used to execute the rotating target depth map hole completion method according to the instructions in the program code.

[0069] As can be seen from the above technical solutions, the rotating target depth map hole completion method provided by the present application has the following advantages:

[0070] The rotating target depth map hole completion method provided by the present application uses a connecting pipe-based rotating target detection model to identify the connecting pipe in the color image, obtains a rotating rectangular frame positioning result containing a tilt angle, and then maps a horizontal rectangular frame corresponding to the rotating rectangular frame positioning result to the depth map. Based on the rotating rectangular frame positioning result, the region depth value and the depth missing point coordinate of the horizontal rectangular frame are extracted, the multi-scale similarity of the missing point and the neighborhood is calculated, the completion order of the depth missing point is determined in combination with the target tilt angle, and the depth missing point in the horizontal rectangular frame is completed according to the calculation result of the multi-scale similarity and the Euclidean distance completion order, thereby solving the technical problem that the prior art often ignores the rotation angle constraint of the connecting pipe when combining depth information, resulting in that the hole in the depth map cannot be accurately completed, making it difficult to help the unmanned aerial vehicle to more accurately identify and locate the connecting pipe, and affecting the three-dimensional reconstruction effect of the connecting pipe. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a flowchart illustrating a method for completing holes in a rotating target depth map provided in an embodiment of the present invention;

[0073] Figure 2 This is a schematic diagram of SSIM similarity calculation for adjacent pixels in a 3×3 region provided in an embodiment of the present invention;

[0074] Figure 3 This is a schematic diagram illustrating the calculation of depth missing point completion distance provided in an embodiment of the present invention;

[0075] Figure 4 This is an example diagram of the depth map based on angle constraints and SSIM completion provided in the embodiments of the present invention;

[0076] Figure 5 This is a schematic diagram of a rotating target depth map hole filling device provided in an embodiment of the present invention. Detailed Implementation

[0077] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] For easier understanding, please refer to Figure 1 This invention provides an embodiment of a method for completing holes in a rotating target depth map, comprising:

[0079] Step 101: Based on the rotating target dataset of overhead power transmission line splice pipes, train a rotating target detection model for recognizing overhead power transmission line splice pipes.

[0080] It should be noted that the rotating target refers to a target with a rotating direction that needs to be recognized and positioned in the detection process. Real aerial footage of the adapter pipe is used, and a large three-dimensional software Blender is applied to design different application scenarios of the adapter pipe to obtain high-fidelity virtual images of the adapter pipe under different conditions. Data enhancement technology is used to generate an adapter pipe image set suitable for diversified application scenarios on the basis of the above two image sets. The adapter pipe is labeled by using a rotating target special labeling software roLabelImg on the basis of the adapter pipe image set to make a rotating target data set of the overhead power transmission line adapter pipe. A rotating target detection model for recognizing the overhead power transmission line adapter pipe is trained by using the rotating target data set. In an embodiment, the rotating target detection model is a rotating target detection model based on a yolov5_OBB network structure.

[0081] Step 102, obtaining a color image and a depth image including a rotating target adapter pipe by using a depth camera.

[0082] It should be noted that the depth camera is used to obtain the color image including the rotating target adapter pipe and the depth image that has been aligned with the color image.

[0083] Step 103, identifying the adapter pipe in the color image based on the rotating target detection model to obtain a rotating rectangular frame positioning result of the adapter pipe, the rotating rectangular frame positioning result including an inclination angle of the rotating rectangular frame.

[0084] It should be noted that the rotating target detection model is used to identify the adapter pipe in the color image including the rotating target adapter pipe obtained by the depth camera, and the rotating target detection model outputs a rotating rectangular frame positioning result of the adapter pipe in the rotating target color image with a confidence higher than 0.9. The rotating rectangular frame positioning result is denoted as , wherein, is a center point coordinate of the rotating rectangular frame, w is a width of the rotating rectangular frame, and h is a height of the rotating rectangular frame. is an inclination angle of the rotating rectangular frame.

[0085] Step 104, mapping a horizontal rectangular frame corresponding to the rotating rectangular frame positioning result to the depth image to extract a region depth value and a depth missing point coordinate of the horizontal rectangular frame in the depth image.

[0086] It should be noted that first, the rotating rectangular frame positioning result in the rotating target color image is to calculate a corresponding horizontal rectangular frame , wherein, , , , and are four corner point coordinates of the horizontal rectangular frame, and the four corner point coordinates of the horizontal rectangular frame are determined according to the following formula:

[0087]

[0088]

[0089]

[0090]

[0091] Then the horizontal rectangular frame is mapped to the horizontal rectangular frame in the depth map, and the region depth value of is extracted, , the pixel coordinates in the region corresponding to the depth value.

[0092] Based on the extracted region depth value, a threshold method is used to extract the depth missing point coordinates of the horizontal rectangular frame in the depth map. Set the depth threshold , when , the position is recorded as a depth missing point, and the depth missing point set is recorded as , and N is the total number of depth value missing points.

[0093] Step 105, calculate the multi-scale similarity of the missing point in the depth map and the neighborhood in the color map.

[0094] It should be noted that, as Figure 2 indicated, the multi-scale similarity of p and the neighborhood q in the rotated target color map is calculated by applying the structural similarity (SSIM) principle. First, the region in the rotated target color map is converted to a grayscale image by the cv2.COLOR_BGR2GRAY() function, and the similarity between the pixel point p and the neighborhood pixel point q in the grayscale image is measured by brightness, contrast and structure. The multi-scale similarity calculation model is:

[0095]

[0096] wherein, is the multi-scale similarity function, is the corresponding pixel value in the 3x3 region centered on the pixel point p in the depth map, is the corresponding pixel value in the 3x3 region centered on q, q is the neighboring pixel point of p, is the brightness comparison function, is the contrast comparison function, is the structure comparison function, is the weight factor of , is a weight factor, is a weight factor, is a weight factor. is a weight factor.

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] wherein, is the mean value in a 3x3 region centered at pixel p, is the mean value in a 3x3 region centered at pixel q, is the variance in a 3x3 region centered at pixel p, is the variance in a 3x3 region centered at pixel q, is the covariance of the 3x3 region in which pixels p, q are located, , , is a constant for preventing the initial value from being 0.

[0104] Step 106, according to the Euclidean distance from the rotated rectangular frame positioning result and the depth missing point in the depth map to the straight line passing through the center point of the over-rotated rectangular frame and on the long side of the rotated rectangular frame, the completion order is set.

[0105] It should be noted that, as shown in Figure 3 , the inclination angle of the rotated rectangular frame is , the center point coordinates are , and the straight line passing through the center point coordinates and perpendicular to the long side of the rotated rectangular frame is:

[0106]

[0107] The vertical distance from the depth missing point to the straight line is:

[0108]

[0109] As can be seen from the formula of the vertical distance , the size of is affected by the inclination angle of the rotated rectangular frame​ Constraints. The horizontal bounding box region in the depth map. deep missing points according to The values ​​are sorted in ascending order.

[0110] Step 107: Based on the calculation results of multi-scale similarity, complete the missing depth points in the horizontal rectangle according to the completion order and depth value completion formula.

[0111] It should be noted that the formula for depth value completion is:

[0112]

[0113] in, for The depth value that needs to be supplemented at the same coordinate position on the corresponding depth map. The position in the color image is two adjacent pixels. and Multiscale similarity, for The depth value at the same coordinate position on the corresponding depth map.

[0114] The effect of depth map completion based on angle constraints and SSIM is as follows: Figure 4 As shown.

[0115] The rotating target depth map hole completion method provided by this invention utilizes a rotating target detection model based on splice tubes to identify splice tubes in a color image, obtaining a rotating rectangular box location result including tilt angle. Then, the horizontal rectangular box corresponding to the rotating rectangular box location result is mapped to the depth map. Based on the rotating rectangular box location result, the region depth value and depth missing point coordinates of the horizontal rectangular box are extracted. The multi-scale similarity between the missing point and its neighborhood is calculated. The completion order of the depth missing points is determined by combining the target tilt angle. Thus, the depth missing points in the horizontal rectangular box are completed according to the multi-scale similarity calculation results and the Euclidean distance completion order. This solves the technical problem that existing technologies often ignore the rotation angle constraint of the splice tube when combining depth information, resulting in inaccurate hole completion in the depth map, making it difficult for UAVs to more accurately identify and locate splice tubes, and affecting the 3D reconstruction effect of the splice tube.

[0116] In one embodiment, step 107 is followed by:

[0117] Step 108: Extract the distance between the depth camera and the splice tube based on the tilt angle of the rotating rectangle.

[0118] It should be noted that the center point of the rotated rectangle was extracted from the depth map. Along the centerline within a preset range in the neighborhood (the preset range is denoted as...) , , w is the width of the rotated rectangle frame), the mean value of the depth value in the rotated rectangle frame is the distance value between the depth camera and the joint pipe.

[0119] The equation of the center line is expressed as:

[0120]

[0121] , wherein, is the inclination angle of the rotated rectangle frame, is a point on the center line, is the coordinate of the center point of the rotated rectangle frame.

[0122] The distance between the depth camera and the joint pipe can be expressed as:

[0123]

[0124] , wherein distance is the distance between the depth camera and the joint pipe.

[0125] The distance value between the depth camera and the joint pipe can be used for subsequent distance measurement of the joint pipe by the depth camera carried by the unmanned aerial vehicle.

[0126] For ease of understanding, please refer to Figure 5 An embodiment of a rotating target depth map hole completion device is provided in the application, which comprises:

[0127] A model construction module is configured to train a rotating target detection model for identifying overhead power transmission line joint pipes based on a rotating target data set of the overhead power transmission line joint pipes.

[0128] An image acquisition module is configured to acquire a color image and a depth image including a rotating target joint pipe through a depth camera.

[0129] A target recognition module is configured to identify the joint pipe in the color image based on the rotating target detection model, and obtain a rotating rectangle frame positioning result of the joint pipe, wherein the rotating rectangle frame positioning result includes an inclination angle of the rotating rectangle frame.

[0130] A depth processing module is configured to map a horizontal rectangle frame corresponding to the rotating rectangle frame positioning result to the depth image, and extract regional depth values and depth missing point coordinates of the horizontal rectangle frame in the depth image.

[0131] A similarity calculation module is configured to calculate multi-scale similarity of the missing points in the depth image and the neighborhood in the color image.

[0132] A completion order determination module is configured to set a completion order according to the rotating rectangle frame positioning result and the Euclidean distance from the depth missing points in the depth image to a straight line passing through the center point of the rotated rectangle frame and perpendicular to the long side of the rotated rectangle frame.

[0133] The completion module is configured to complete the depth missing points in the horizontal rectangular frame according to a completion sequence and a depth value completion formula based on the calculation result of the multi-scale similarity.

[0134] In one embodiment, the method further comprises:

[0135] The distance calculation module is configured to extract a distance value of the depth camera and the connecting pipe according to the tilt angle of the rotated rectangular frame.

[0136] In one embodiment, the rotated target detection model is a rotated target detection model based on a yolov5_OBB network structure.

[0137] In one embodiment, the depth processing module is specifically configured to:

[0138] According to the rotated rectangular frame positioning result, the horizontal rectangular frame corresponding to the rotated rectangular frame positioning result is calculated based on the positioning result of the rotated rectangular frame.

[0139] The horizontal rectangular frame in the depth map is mapped as a horizontal rectangular frame in the depth map, and the region depth value of the horizontal rectangular frame in the depth map is extracted.

[0140] Based on the region depth value, the depth missing point coordinates of the horizontal rectangular frame in the depth map are extracted by using a threshold method.

[0141] In one embodiment, the similarity calculation module is specifically configured to:

[0142] Based on the structural similarity principle, the multi-scale similarity of the missing points in the depth map and the neighborhood in the color map is calculated, and the multi-scale similarity calculation model is:

[0143]

[0144] wherein, is a multi-scale similarity function, is a pixel value corresponding to the pixel point p in a 3*3 region, is a pixel value corresponding to the pixel point q in a 3*3 region, q is a neighboring pixel point of the pixel point p, is a luminance comparison function, is a contrast comparison function, is a structure comparison function, is a weight factor of is a weight factor of is a weight factor of is a weight factor of is a weight factor of

[0145] In one embodiment, the depth value completion formula is:

[0146]

[0147] wherein, is the depth value of the same coordinate position in the corresponding depth map, is the multi-scale similarity of the two adjacent pixels in the color image and is the depth value of the same coordinate position in the corresponding depth map.

[0148] In one embodiment, the completion order determination module is specifically configured to:

[0149] determine a straight line in the depth map from the depth missing point to the center point of the rotated rectangular frame and perpendicular to the long side of the rotated rectangular frame according to the rotated rectangular frame positioning result;

[0150] calculate the Euclidean distance from the depth missing point in the depth map to the straight line;

[0151] sort the Euclidean distances from the depth missing points in the rotated rectangular frame in the depth map to the straight line in ascending order to determine the completion order.

[0152] In one embodiment, the distance calculation module is specifically configured to:

[0153] extract the mean value of the depth values of the center point of the rotated rectangular frame along the center line within the preset range of the neighborhood in the rotated target depth map according to the inclination angle of the rotated rectangular frame, and the mean value is the distance value of the depth camera and the connecting pipe;

[0154] the center line is:

[0155]

[0156] wherein, is the inclination angle of the rotated rectangular frame, is the point on the center line, is the coordinate of the center point of the rotated rectangular frame.

[0157] An embodiment of a rotating target depth map hole completion device is also provided in the application, and the device comprises a processor and a memory:

[0158] The memory is used to store program codes and transmit the program codes to the processor;

[0159] The processor is used to execute the rotating target depth map hole completion method in each rotating target depth map hole completion method embodiment in the application according to the instructions in the program codes.

[0160] ​The rotating target depth map hole completion device and the equipment provided in the application are used for executing the rotating target depth map hole completion method provided in the application, the principle and the technical effects obtained are the same as those of the rotating target depth map hole completion method provided in the application, and details are not repeated here.

[0161] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for completing a depth map hole of a rotating target, characterized in that, The method comprises the following steps: training a rotating target detection model for identifying overhead power line splicing pipes based on a rotating target data set of overhead power line splicing pipes; acquiring a color image and a depth image of the rotating target splicing pipe through a depth camera; identifying the splicing pipe in the color image based on the rotating target detection model to obtain a rotating rectangular frame positioning result of the splicing pipe, wherein the rotating rectangular frame positioning result comprises an inclination angle of the rotating rectangular frame; mapping a horizontal rectangular frame corresponding to the rotating rectangular frame positioning result to the depth image to extract regional depth values and depth missing point coordinates of the horizontal rectangular frame in the depth image; calculating multi-scale similarities between the missing points in the depth image and the neighborhood in the color image; setting a completion order according to the rotating rectangular frame positioning result and the Euclidean distance from the depth missing points in the depth image to a straight line passing through the center point of the rotating rectangular frame and being perpendicular to the long side of the rotating rectangular frame; completing the depth missing points in the horizontal rectangular frame according to the completion order and a depth value completion formula based on the calculation result of the multi-scale similarities; setting a completion order according to the rotating rectangular frame positioning result and the Euclidean distance from the depth missing points in the depth image to a straight line passing through the center point of the rotating rectangular frame and being perpendicular to the long side of the rotating rectangular frame, comprising: determining the straight line passing through the center point of the rotating rectangular frame and being perpendicular to the long side of the rotating rectangular frame according to the rotating rectangular frame positioning result; calculating the Euclidean distance from the depth missing points in the depth image to the straight line; sorting the Euclidean distances from the depth missing points in the rotating rectangular frame in the depth image to the straight line in the order from small to large to determine the completion order; completing the depth missing points in the horizontal rectangular frame according to the completion order based on the calculation result of the multi-scale similarities, and further comprising: extracting the mean value of the depth values of the depth camera and the splicing pipe within a preset range of the neighborhood along the center line passing through the center point of the rotating rectangular frame in the rotating target depth image according to the inclination angle of the rotating rectangular frame, wherein the mean value is the distance value of the depth camera and the splicing pipe; the center line is: ； wherein, is the angle of inclination of the rotated rectangular frame, is the point on the center line, is the rotated rectangular frame center point coordinate.

2. The method of claim 1, wherein, the rotating target detection model is a rotating target detection model based on a yolov5_OBB network structure.

3. The method of claim 1, wherein, mapping the horizontal rectangular frame corresponding to the rotating rectangular frame positioning result to the depth image to extract the regional depth values and the depth missing point coordinates of the horizontal rectangular frame, comprising: calculating the horizontal rectangular frame corresponding to the rotating rectangular frame positioning result based on the positioning result of the rotating rectangular frame according to the rotating rectangular frame positioning result; mapping the horizontal rectangular frame as a horizontal rectangular frame in the depth image to extract the regional depth values of the horizontal rectangular frame in the depth image; extracting the depth missing point coordinates of the horizontal rectangular frame in the depth image by using a threshold method based on the regional depth values.

4. The method of claim 1, wherein, calculating the multi-scale similarities between the missing points in the depth image and the neighborhood in the color image, comprising: calculating the multi-scale similarities between the missing points in the depth image and the neighborhood in the color image based on the structural similarity principle, wherein the multi-scale similarity calculation model is: ； in, For multi-scale similarity functions, Let p be the pixel value within a 3×3 region centered on pixel p in the depth map. Let p be the pixel value within a 3×3 region centered at q, where q represents the neighboring pixels of pixel p. This is a brightness comparison function. This is a contrast comparison function. For structural comparison functions, for Weighting factors for Weighting factors for Weighting factors.

5. The method of claim 4, wherein, the depth value completion formula is: ； wherein, is the depth value of the same coordinate position in the corresponding depth map, is the multi-scale similarity of the two adjacent pixels in the color image and is the depth value of the same coordinate position in the corresponding depth map.​ 6. A device for filling holes in a rotating target depth map, characterized in that, The method comprises the following steps: a model construction module is used to train a rotating target detection model for identifying overhead transmission line splicing pipes based on a rotating target data set of the overhead transmission line splicing pipes; an image acquisition module is used to acquire a color image and a depth image of the rotating target splicing pipe through a depth camera; a target identification module is used to identify the splicing pipe in the color image based on the rotating target detection model, and obtain a rotating rectangular frame positioning result of the splicing pipe, wherein the rotating rectangular frame positioning result comprises an inclination angle of the rotating rectangular frame; a depth processing module is used to map a horizontal rectangular frame corresponding to the rotating rectangular frame positioning result to the depth image, and extract the area depth value and depth missing point coordinates of the horizontal rectangular frame in the depth image; a similarity calculation module is used to calculate the multi-scale similarity of the missing points in the depth image and the neighborhood in the color image; a completion order determination module is used to set a completion order according to the rotating rectangular frame positioning result and the Euclidean distance from the depth missing points in the depth image to a straight line passing through the center point of the rotating rectangular frame and perpendicular to the long side of the rotating rectangular frame; a completion module is used to complete the depth missing points in the horizontal rectangular frame based on the calculation result of the multi-scale similarity, according to the completion order and the depth value completion formula. According to the rotating rectangular frame positioning result and the Euclidean distance from the depth missing points in the depth image to a straight line passing through the center point of the rotating rectangular frame and perpendicular to the long side of the rotating rectangular frame, the completion order is set, which comprises: determining the Euclidean distance from the depth missing points in the depth image to the straight line passing through the center point of the rotating rectangular frame and perpendicular to the long side of the rotating rectangular frame according to the rotating rectangular frame positioning result; calculating the Euclidean distance from the depth missing points in the depth image to the straight line; sorting the Euclidean distance from the depth missing points in the rotating rectangular frame in the depth image to the straight line in ascending order to determine the completion order; based on the calculation result of the multi-scale similarity, completing the depth missing points in the horizontal rectangular frame according to the completion order, and then comprising: extracting the average value of the depth values within a preset range of the neighborhood along the center line passing through the center point of the rotating rectangular frame in the rotating target depth image according to the inclination angle of the rotating rectangular frame, wherein the average value is the distance value between the depth camera and the splicing pipe; the center line is: ； wherein, is the angle of inclination of the rotated rectangular frame, is the point on the center line, is the rotated rectangular frame center point coordinate.

7. An apparatus for completing a depth map hole of a rotating target, the apparatus comprising: the device comprises a processor and a memory: the memory is used to store program code and transmit the program code to the processor; the processor is used to execute the instructions in the program code to perform the rotating target depth image hole completion method according to any one of claims 1-5.

Images