Method, device and equipment for complementing holes in depth map of rotating target

Abstract

The invention discloses a rotating target depth map hole completion method, device and equipment, and the method comprises the steps: recognizing a splicing sleeve in a color map through a rotating target detection model based on the splicing sleeve, obtaining a rotating rectangular frame positioning result containing an inclination angle, and mapping a horizontal rectangular frame corresponding to the rotating rectangular frame positioning result into a depth map, extracting a region depth value and a depth missing point coordinate of a horizontal rectangular frame based on a rotating rectangular frame positioning result, calculating multi-scale similarity between the missing point and a neighborhood, and determining a completion sequence of the depth missing point by combining a target inclination angle; and the depth missing points in the horizontal rectangular frame are completed according to the calculation result of the multi-scale similarity and the Euclidean distance completion sequence. The technical problems that in the prior art, when depth information is combined, rotation angle constraint of the splicing sleeve is often ignored, so that holes in a depth map cannot be accurately completed, it is difficult to help an unmanned aerial vehicle to identify and position the splicing sleeve more accurately, and the three-dimensional reconstruction effect of the splicing sleeve is affected are solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and equipment for filling holes in a rotating target depth map. Background Art

[0002] In the maintenance process of overhead transmission lines, the joint pipe, as a key component, plays the role of connecting the transmission lines. Traditional joint pipe inspection and maintenance mostly rely on manual operation, which is not only inefficient but also has certain safety risks. As an efficient inspection tool, drones are gradually being used in the inspection and maintenance of transmission lines. However, in the process of automated drone inspection, the positioning and identification of joint pipes still face many challenges. Existing research on the identification and positioning of joint pipes has not considered how to effectively solve the problem of filling holes in depth maps, especially when facing joint pipes with arbitrary angles during mid- and long-distance aerial photography. Existing technologies often ignore the rotation angle constraints of joint pipes when combining depth information, resulting in the inability to accurately fill holes in the depth map, making it difficult to help drones more accurately identify and locate joint pipes, affecting the three-dimensional reconstruction effect of joint pipes. Summary of the invention

[0003] The present invention provides a rotating target depth map hole filling method, device and equipment, which are used to solve the technical problem that the existing technology often ignores the rotation angle constraint of the connecting pipe when combining depth information, resulting in the inability to accurately fill the holes in the depth map, making it difficult to help drones more accurately identify and locate the connecting pipe, and affecting the three-dimensional reconstruction effect of the connecting pipe.

[0004] In view of this, the present invention provides a method for filling holes in a rotation target depth map, comprising:

[0005] Based on the rotating target dataset of overhead transmission line joint pipes, a rotating target detection model for identifying overhead transmission line joint pipes is trained;

[0006] Acquire a color image and a depth image including a rotating target connecting tube by a depth camera;

[0007] Based on the rotating target detection model, the connecting pipe in the color image is identified to obtain a rotating rectangular frame positioning result of the connecting pipe, wherein the rotating rectangular frame positioning result includes a tilt angle of the rotating rectangular frame;

[0008] Mapping the horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame to the depth map, and extracting the regional depth value and the coordinates of the depth missing point of the horizontal rectangular frame in the depth map;

[0009] Calculating multi-scale similarity of missing points and neighborhoods in the depth map in the color map;

[0010] Setting a completion order according to the positioning result of the rotated rectangular frame and the Euclidean distance from the depth missing point in the depth map to a straight line passing through the center point of the rotated rectangular frame and perpendicular to the long side of the rotated rectangular frame;

[0011] Based on the calculation result of the multi-scale similarity, the depth missing points in the horizontal rectangular box are completed according to the completion order and the depth value completion formula.

[0012] Optionally, based on the calculation result of the multi-scale similarity, the depth missing points in the horizontal rectangular box are completed in the completion order, and then the method further includes:

[0013] According to the tilt angle of the rotating rectangular frame, the distance value between the depth camera and the connecting pipe is extracted.

[0014] Optionally, the rotating target detection model is a rotating target detection model based on the yolov5_OBB network structure.

[0015] Optionally, mapping a horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame to the depth map, and extracting a regional depth value and a depth missing point coordinate of the horizontal rectangular frame, comprises:

[0016] According to the positioning result of the rotated rectangular frame, a horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame is calculated based on the positioning result of the rotated rectangular frame;

[0017] Mapping the horizontal rectangular frame to a horizontal rectangular frame in the depth map, and extracting a regional depth value of the horizontal rectangular frame in the depth map;

[0018] Based on the regional depth value, a threshold method is used to extract the coordinates of the depth missing points of the horizontal rectangular box in the depth map.

[0019] Optionally, calculating, in the color map, a multi-scale similarity between a missing point and a neighborhood in the depth map includes:

[0020] Based on the principle of structural similarity, the multi-scale similarity between the missing points and the neighborhood in the depth map is calculated in the color map. The multi-scale similarity calculation model is:

[0021]

[0022] in, is the multi-scale similarity function, is the pixel value corresponding to the 3×3 area centered at the pixel point p in the depth map, is the corresponding pixel value in the 3×3 area with q as the center, q is the adjacent pixel point of pixel point p, is the brightness comparison function, is the contrast comparison function, is the structure comparison function, for The weight factor of for The weight factor of for The weight factor of .

[0023] Optionally, the depth value completion formula is:

[0024]

[0025] in, for The depth value to be completed at the same coordinate position in the corresponding depth map, The positions of two adjacent pixels in the color map and The multi-scale similarity of for The depth value at the same coordinate position in the corresponding depth map.

[0026] Optionally, the completion order is set according to the positioning result of the rotated rectangular frame and the Euclidean distance from the depth missing point in the depth map to a straight line passing through the center point of the rotated rectangular frame and perpendicular to the long side of the rotated rectangular frame, including:

[0027] Determine, according to the positioning result of the rotated rectangular frame, a straight line from the depth missing point in the depth map to the center point of the rotated rectangular frame and perpendicular to the long side of the rotated rectangular frame;

[0028] Calculating the Euclidean distance from the depth missing point in the depth map to the straight line;

[0029] The Euclidean distances from the depth missing points in the rotated rectangular frame in the depth map to the straight line are sorted in order from small to large to determine the completion order.

[0030] Optionally, extracting the distance value between the depth camera and the connection pipe according to the tilt angle of the rotating rectangular frame includes:

[0031] According to the tilt angle of the rotating rectangular frame, the average of the depth values ​​along the center line passing through the center point of the rotating rectangular frame within a preset range of the neighborhood is extracted from the rotating target depth map, and the average is the distance value between the depth camera and the connecting pipe;

[0032] The center line is:

[0033]

[0034] in, is the tilt angle of the rotated rectangle, is a point on the center line, The coordinates of the center point of the rotated rectangle.

[0035] A second aspect of the present invention provides a rotating target depth map hole filling device, comprising:

[0036] A model building module, used for training a rotating target detection model for identifying an overhead transmission line joint pipe based on a rotating target dataset of an overhead transmission line joint pipe;

[0037] An image acquisition module, used to acquire a color image and a depth image including a rotating target connecting tube through a depth camera;

[0038] A target recognition module, used for recognizing the connecting pipe in the color image based on the rotating target detection model, and obtaining a rotating rectangular frame positioning result of the connecting pipe, wherein the rotating rectangular frame positioning result includes a tilt angle of the rotating rectangular frame;

[0039] A depth processing module, used for mapping the horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame into the depth map, and extracting the regional depth value and the coordinates of the depth missing point of the horizontal rectangular frame in the depth map;

[0040] A similarity calculation module, used for calculating the multi-scale similarity between the missing points and the neighborhood in the depth map in the color map;

[0041] A completion order determination module, used to set a completion order according to the positioning result of the rotated rectangular frame and the Euclidean distance from the depth missing point in the depth map to a straight line passing through the center point of the rotated rectangular frame and perpendicular to the long side of the rotated rectangular frame;

[0042] A completion module is used to complete the depth missing points in the horizontal rectangular box according to the completion order and the depth value completion formula based on the calculation result of the multi-scale similarity.

[0043] Optionally, it also includes:

[0044] The distance calculation module is used to extract the distance value between the depth camera and the connection tube according to the tilt angle of the rotating rectangular frame.

[0045] Optionally, the rotating target detection model is a rotating target detection model based on the yolov5_OBB network structure.

[0046] Optionally, the depth processing module is specifically used for:

[0047] According to the positioning result of the rotated rectangular frame, a horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame is calculated based on the positioning result of the rotated rectangular frame;

[0048] Mapping the horizontal rectangular frame to a horizontal rectangular frame in the depth map, and extracting a regional depth value of the horizontal rectangular frame in the depth map;

[0049] Based on the regional depth value, a threshold method is used to extract the coordinates of the depth missing points of the horizontal rectangular box in the depth map.

[0050] Optionally, the similarity calculation module is specifically used to:

[0051] Based on the principle of structural similarity, the multi-scale similarity between the missing points and the neighborhood in the depth map is calculated in the color map. The multi-scale similarity calculation model is:

[0052]

[0053] in, is the multi-scale similarity function, is the pixel value corresponding to the 3×3 area centered at pixel point p in the depth map, is the corresponding pixel value in the 3×3 area centered on q, q is the adjacent pixel point of pixel point p, is the brightness comparison function, is the contrast comparison function, is the structure comparison function, for The weight factor of for The weight factor of for The weight factor of .

[0054] Optionally, the depth value completion formula is:

[0055]

[0056] in, for The depth value to be completed at the same coordinate position in the corresponding depth map, The positions of two adjacent pixels in the color map and The multi-scale similarity of for The depth value at the same coordinate position in the corresponding depth map.

[0057] Optionally, the completion order determination module is specifically used to:

[0058] Determine, according to the positioning result of the rotated rectangular frame, a straight line from the depth missing point in the depth map to the center point of the rotated rectangular frame and perpendicular to the long side of the rotated rectangular frame;

[0059] Calculating the Euclidean distance from the depth missing point in the depth map to the straight line;

[0060] The Euclidean distances from the depth missing points in the rotated rectangular frame in the depth map to the straight line are sorted in order from small to large to determine the completion order.

[0061] Optionally, the distance calculation module is specifically used to:

[0062] According to the tilt angle of the rotating rectangular frame, the average of the depth values ​​along the center line passing through the center point of the rotating rectangular frame within a preset range of the neighborhood is extracted from the rotating target depth map, and the average is the distance value between the depth camera and the connecting pipe;

[0063] The center line is:

[0064]

[0065] in, is the tilt angle of the rotated rectangle, is a point on the center line, The coordinates of the center point of the rotated rectangle.

[0066] A third aspect of the present invention provides a rotation target depth map hole completion device, the device comprising a processor and a memory:

[0067] The memory is used to store program code and transmit the program code to the processor;

[0068] The processor is used to execute the rotation target depth map hole filling method described in any one of the first aspects of the present invention according to the instructions in the program code.

[0069] It can be seen from the above technical solutions that the rotation target depth map hole filling method provided by the present invention has the following advantages:

[0070] The rotating target depth map hole completion method provided by the present invention utilizes a rotating target detection model based on a connecting pipe to identify a connecting pipe in a color map, obtains a rotating rectangular frame positioning result including a tilt angle, and then maps a horizontal rectangular frame corresponding to the rotating rectangular frame positioning result to a depth map, extracts a regional depth value and a coordinate of a depth missing point of the horizontal rectangular frame based on the rotating rectangular frame positioning result, calculates a multi-scale similarity between the missing point and the neighborhood, and determines a completion order of the depth missing point in combination with the target tilt angle, thereby completing the depth missing points in the horizontal rectangular frame according to the calculation result of the multi-scale similarity and the Euclidean distance completion order, thereby solving the technical problem that the prior art often ignores the rotation angle constraint of the connecting pipe when combining depth information, resulting in the inability to accurately complete the holes in the depth map, making it difficult to help the drone more accurately identify and locate the connecting pipe, and affecting the three-dimensional reconstruction effect of the connecting pipe. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 A schematic diagram of a flow chart of a rotation target depth map hole filling method provided in an embodiment of the present invention;

[0073] Figure 2 A schematic diagram of calculating the SSIM similarity of adjacent pixels in a 3×3 region provided in an embodiment of the present invention;

[0074] Figure 3 A schematic diagram of depth missing point completion distance calculation provided in an embodiment of the present invention;

[0075] Figure 4 This is an example diagram of a depth map completed based on angle constraints and SSIM provided in an embodiment of the present invention;

[0076] Figure 5 It is a structural schematic diagram of a rotating target depth map hole filling device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] For easier understanding, see Figure 1 The present invention provides an embodiment of a method for filling a hole in a rotating target depth map, comprising:

[0079] Step 101: Based on a rotating target dataset of overhead transmission line joint pipes, a rotating target detection model for identifying overhead transmission line joint pipes is trained.

[0080] It should be noted that a rotating target refers to a target with a rotating direction that needs to be identified and located during the detection process. Real aerial photography images of joint pipes are used, and large-scale three-dimensional software Blender is used to design different application scenarios of joint pipes, obtain high-fidelity virtual images of joint pipes under different conditions, and use data enhancement technology to generate joint pipe image sets suitable for diversified application scenarios based on the above two image sets. Based on the joint pipe image set, the joint pipes are annotated using the special annotation software roLabelImg for rotating targets to create a rotating target data set for overhead transmission line joint pipes. The rotating target data set is used to train a rotating target detection model for identifying overhead transmission line joint pipes. In one embodiment, the rotating target detection model is a rotating target detection model based on the yolov5_OBB network structure.

[0081] Step 102: Acquire a color image and a depth image of the rotating target connecting pipe through a depth camera.

[0082] It should be noted that the depth camera is used to obtain a color image of the rotating target connecting pipe and a depth image aligned with the color image.

[0083] Step 103: Identify the connecting pipe in the color image based on the rotating target detection model, and obtain the rotating rectangular frame positioning result of the connecting pipe, wherein the rotating rectangular frame positioning result includes the tilt angle of the rotating rectangular frame.

[0084] It should be noted that the rotating target detection model is used to identify the connecting pipe in the color image including the rotating target connecting pipe obtained by the depth camera, and the rotating target detection model outputs the rotating rectangular frame positioning result of the connecting pipe with a confidence level higher than 0.9 in the rotating target color image. The rotating rectangular frame positioning result is recorded as ,in, is the coordinate of the center point of the rotating rectangle, w is the width of the rotating rectangle, and h is the height of the rotating rectangle. The tilt angle of the rotated rectangle.

[0085] Step 104 : Map the horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame into the depth map, and extract the regional depth value and the coordinates of the depth missing point of the horizontal rectangular frame in the depth map.

[0086] It should be noted that the result is first located according to the rotation rectangle in the rotation target color image. Calculate the corresponding horizontal rectangular box ,in, , , ,and They are the coordinates of the four corner points of the horizontal rectangular box, and the formula for determining the coordinates of the four corner points of the horizontal rectangular box is:

[0087]

[0088] Then add the horizontal rectangular frame Mapped to a horizontal rectangular box in the depth map ,extract The area depth value , , for Pixel coordinates within the region The corresponding depth value.

[0089] Based on the extracted regional depth value, the threshold method is used to extract the coordinates of the depth missing points of the horizontal rectangular box in the depth map. Set the depth threshold ,when When Denoted as the depth missing point, the depth missing point set is , N is the total number of missing depth value points.

[0090] Step 105: Calculate the multi-scale similarity between the missing points and the neighborhood in the depth map in the color map.

[0091] It should be noted that if Figure 2 As shown in the figure, the structural similarity (SSIM) principle is applied to calculate the multi-scale similarity between the rotation target color map p and the neighborhood q. First, The region is converted into a grayscale image by the cv2.COLOR_BGR2GRAY() function. The similarity between the grayscale image pixel p and the neighboring pixel q is measured by brightness, contrast and structure. The multi-scale similarity calculation model is:

[0092]

[0093] in, is the multi-scale similarity function, is the pixel value corresponding to the 3×3 area centered at pixel point p in the depth map, is the corresponding pixel value in the 3×3 area centered on q, q is the adjacent pixel point of pixel point p, is the brightness comparison function, is the contrast comparison function, is the structure comparison function, for The weight factor of for The weight factor of for The weight factor of .

[0094]

[0095] in, is the mean value in a 3×3 region centered at pixel p, is the mean value in a 3×3 region centered on pixel q, is the variance in a 3×3 region centered on pixel p, is the variance in a 3×3 region centered on pixel q, is the covariance of the 3×3 region where pixels p and q are located, , , A constant used to prevent the initial value from being 0.

[0096] Step 106 : Set a completion order according to the positioning result of the rotated rectangular frame and the Euclidean distance from the depth missing point in the depth map to a straight line passing through the center point of the rotated rectangular frame and on the long side of the rotated rectangular frame.

[0097] It should be noted that vertical Figure 3 As shown, the tilt angle of the rotating rectangular frame is , the center point coordinates are , then the straight line passing through the center point coordinates and perpendicular to the long side of the rotated rectangle is:

[0098]

[0099] Depth Missing Points To the straight line Vertical distance for:

[0100]

[0101] From the vertical distance It can be seen from the formula that The size of the rotated rectangle is affected by the tilt angle of The horizontal box area in the depth map Depth missing point according to The values ​​are sorted in order from small to large.

[0102] Step 107: Based on the calculation result of the multi-scale similarity, the depth missing points in the horizontal rectangular frame are completed according to the completion order and the depth value completion formula.

[0103] It should be noted that the depth value completion formula is:

[0104]

[0105] in, for The depth value to be completed at the same coordinate position in the corresponding depth map, The positions of two adjacent pixels in the color map and The multi-scale similarity of for The depth value at the same coordinate position in the corresponding depth map.

[0106] The effect of completing the depth map based on angle constraints and SSIM is as follows Figure 4 shown.

[0107] The rotating target depth map hole completion method provided by the present invention utilizes a rotating target detection model based on a connecting pipe to identify a connecting pipe in a color map, obtains a rotating rectangular frame positioning result including a tilt angle, and then maps a horizontal rectangular frame corresponding to the rotating rectangular frame positioning result to a depth map, extracts a regional depth value and a coordinate of a depth missing point of the horizontal rectangular frame based on the rotating rectangular frame positioning result, calculates a multi-scale similarity between the missing point and the neighborhood, and determines a completion order of the depth missing points in combination with the target tilt angle, thereby completing the depth missing points in the horizontal rectangular frame according to the calculation result of the multi-scale similarity and the Euclidean distance completion order, thereby solving the technical problem that the prior art often ignores the rotation angle constraint of the connecting pipe when combining depth information, resulting in the inability to accurately complete the holes in the depth map, making it difficult to help the drone identify and locate the connecting pipe more accurately, and affecting the three-dimensional reconstruction effect of the connecting pipe.

[0108] In one embodiment, step 107 further includes:

[0109] Step 108: extract the distance value between the depth camera and the connection pipe according to the tilt angle of the rotating rectangular frame.

[0110] It should be noted that the center point of the rotated rectangular frame is extracted from the depth map Along the center line in the neighborhood preset range (the preset range is recorded as , , w is the width of the rotated rectangular box), and the mean is the distance between the depth camera and the connecting tube.

[0111] The equation of the center line is expressed as:

[0112]

[0113] in, is the tilt angle of the rotated rectangle, is a point on the center line, The coordinates of the center point of the rotated rectangle.

[0114] The distance between the depth camera and the connecting pipe can be expressed as:

[0115]

[0116] Wherein, distance is the distance between the depth camera and the connecting pipe.

[0117] The calculated distance between the depth camera and the connecting pipe can be used for subsequent distance measurement of the connecting pipe by a depth camera carried by a UAV.

[0118] For easier understanding, see Figure 5 The present invention provides an embodiment of a rotating target depth map hole filling device, comprising:

[0119] A model building module, used for training a rotating target detection model for identifying overhead transmission line joint pipes based on a rotating target dataset of overhead transmission line joint pipes;

[0120] An image acquisition module, used for acquiring a color image and a depth image including a rotating target connecting tube through a depth camera;

[0121] A target recognition module is used to recognize the connecting pipe in the color image based on the rotating target detection model, and obtain the rotating rectangular frame positioning result of the connecting pipe, wherein the rotating rectangular frame positioning result includes the tilt angle of the rotating rectangular frame;

[0122] A depth processing module is used to map the horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame into a depth map, and extract the regional depth value and the coordinates of the depth missing point of the horizontal rectangular frame in the depth map;

[0123] A similarity calculation module is used to calculate the multi-scale similarity between missing points and their neighborhoods in the depth map in the color map;

[0124] A completion order determination module, used to set the completion order according to the positioning result of the rotated rectangular frame and the Euclidean distance from the depth missing point in the depth map to the straight line passing through the center point of the rotated rectangular frame and perpendicular to the long side of the rotated rectangular frame;

[0125] The completion module is used to complete the missing depth points in the horizontal rectangular box according to the completion order and depth value completion formula based on the calculation results of multi-scale similarity.

[0126] In one embodiment, it further includes:

[0127] The distance calculation module is used to extract the distance value between the depth camera and the connecting tube according to the tilt angle of the rotating rectangular frame.

[0128] In one embodiment, the rotating target detection model is a rotating target detection model based on the yolov5_OBB network structure.

[0129] In one embodiment, the depth processing module is specifically used for:

[0130] According to the positioning result of the rotated rectangular frame, a horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame is calculated based on the positioning result of the rotated rectangular frame;

[0131] Mapping the horizontal rectangular box to a horizontal rectangular box in the depth map, and extracting the regional depth value of the horizontal rectangular box in the depth map;

[0132] Based on the regional depth value, the threshold method is used to extract the coordinates of the depth missing points of the horizontal rectangular box in the depth map.

[0133] In one embodiment, the similarity calculation module is specifically used for:

[0134] Based on the principle of structural similarity, the multi-scale similarity between the missing points and the neighborhood in the depth map is calculated in the color map. The multi-scale similarity calculation model is:

[0135]

[0136] in, is the multi-scale similarity function, is the pixel value corresponding to the 3×3 area centered at pixel point p in the depth map, is the corresponding pixel value in the 3×3 area centered on q, q is the adjacent pixel of pixel p, is the brightness comparison function, is the contrast comparison function, is the structure comparison function, for The weight factor of for The weight factor of for The weight factor of .

[0137] In one embodiment, the depth value completion formula is:

[0138]

[0139] in, for The depth value to be completed at the same coordinate position in the corresponding depth map, The positions of two adjacent pixels in the color map and The multi-scale similarity of for The depth value at the same coordinate position in the corresponding depth map.

[0140] In one embodiment, the completion order determination module is specifically used to:

[0141] According to the positioning result of the rotating rectangular frame, determine a straight line from the depth missing point in the depth map to the center point of the rotating rectangular frame and perpendicular to the long side of the rotating rectangular frame;

[0142] Calculate the Euclidean distance from the depth missing point to the straight line in the depth map;

[0143] The Euclidean distances from the depth missing points in the rotated rectangular box in the depth map to the straight line are sorted in order from small to large to determine the completion order.

[0144] In one embodiment, the distance calculation module is specifically used for:

[0145] According to the tilt angle of the rotating rectangular frame, the average of the depth values ​​along the center line of the rotating rectangular frame in the preset range of the neighborhood is extracted from the rotating target depth map, and the average is the distance value between the depth camera and the connecting pipe;

[0146] The center line is:

[0147]

[0148] in, is the tilt angle of the rotated rectangle, is a point on the center line, The coordinates of the center point of the rotated rectangle.

[0149] The present invention also provides an embodiment of a rotating target depth map hole filling device, the device comprising a processor and a memory:

[0150] The memory is used to store the program code and transmit the program code to the processor;

[0151] The processor is used to execute the rotated target depth map hole filling method in each rotated target depth map hole filling method embodiment of the present invention according to the instructions in the program code.

[0152] The rotating target depth map hole filling device and apparatus provided in the present invention are used to execute the rotating target depth map hole filling method provided in the present invention. The principle and technical effects achieved are the same as those of the rotating target depth map hole filling method provided in the present invention, and will not be repeated here.

[0153] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for filling holes in a rotating target depth map, characterized in that: include: Based on the rotating target dataset of overhead transmission line joint pipes, a rotating target detection model for identifying overhead transmission line joint pipes is trained; Acquire a color image and a depth image including a rotating target connecting tube by a depth camera; Based on the rotating target detection model, the connecting pipe in the color image is identified to obtain a rotating rectangular frame positioning result of the connecting pipe, wherein the rotating rectangular frame positioning result includes a tilt angle of the rotating rectangular frame; Mapping the horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame to the depth map, and extracting the regional depth value and the coordinates of the depth missing point of the horizontal rectangular frame in the depth map; Calculating multi-scale similarity of missing points and neighborhoods in the depth map in the color map; Setting a completion order according to the positioning result of the rotated rectangular frame and the Euclidean distance from the depth missing point in the depth map to a straight line passing through the center point of the rotated rectangular frame and perpendicular to the long side of the rotated rectangular frame; Based on the calculation result of the multi-scale similarity, the depth missing points in the horizontal rectangular box are completed according to the completion order and the depth value completion formula.

2. The rotation target depth map hole filling method according to claim 1, characterized in that: Based on the calculation result of the multi-scale similarity, the depth missing points in the horizontal rectangular frame are completed according to the completion order, and then the method further includes: According to the tilt angle of the rotating rectangular frame, the distance value between the depth camera and the connecting pipe is extracted.

3. The rotation target depth map hole filling method according to claim 1, characterized in that: The rotating target detection model is a rotating target detection model based on the yolov5_OBB network structure.

4. The rotation target depth map hole filling method according to claim 1, characterized in that: Mapping the horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame to the depth map, and extracting the regional depth value and the coordinates of the depth missing point of the horizontal rectangular frame, including: According to the positioning result of the rotated rectangular frame, a horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame is calculated based on the positioning result of the rotated rectangular frame; Mapping the horizontal rectangular frame to a horizontal rectangular frame in the depth map, and extracting a regional depth value of the horizontal rectangular frame in the depth map; Based on the regional depth value, a threshold method is used to extract the coordinates of the depth missing points of the horizontal rectangular box in the depth map.

5. The rotation target depth map hole filling method according to claim 1, characterized in that: Calculating the multi-scale similarity of the missing point in the depth map and the neighborhood in the color map includes: Based on the principle of structural similarity, the multi-scale similarity between the missing points and the neighborhood in the depth map is calculated in the color map. The multi-scale similarity calculation model is: in, is the multi-scale similarity function, is the pixel value corresponding to the 3×3 area centered at pixel point p in the depth map, is the corresponding pixel value in the 3×3 area centered on q, q is the adjacent pixel point of pixel point p, is the brightness comparison function, is the contrast comparison function, is the structure comparison function, for The weight factor of for The weight factor of for The weight factor of .

6. The rotation target depth map hole filling method according to claim 5, characterized in that: The depth value completion formula is: in, for The depth value to be completed at the same coordinate position in the corresponding depth map, The positions of two adjacent pixels in the color map and The multi-scale similarity of for The depth value at the same coordinate position in the corresponding depth map.

7. The rotation target depth map hole filling method according to claim 1, characterized in that: According to the positioning result of the rotated rectangular frame and the Euclidean distance from the depth missing point in the depth map to a straight line passing through the center point of the rotated rectangular frame and perpendicular to the long side of the rotated rectangular frame, a completion order is set, including: Determine, according to the positioning result of the rotated rectangular frame, a straight line from the depth missing point in the depth map to the center point of the rotated rectangular frame and perpendicular to the long side of the rotated rectangular frame; Calculating the Euclidean distance from the depth missing point in the depth map to the straight line; The Euclidean distances from the depth missing points in the rotated rectangular frame in the depth map to the straight line are sorted in order from small to large to determine the completion order.

8. The rotation target depth map hole filling method according to claim 2, characterized in that: Extracting the distance value between the depth camera and the connection pipe according to the tilt angle of the rotating rectangular frame includes: According to the tilt angle of the rotating rectangular frame, the average of the depth values ​​along the center line passing through the center point of the rotating rectangular frame within a preset range of the neighborhood is extracted from the rotating target depth map, and the average is the distance value between the depth camera and the connecting pipe; The center line is: in, is the tilt angle of the rotated rectangle, is a point on the center line, The coordinates of the center point of the rotated rectangle.

9. A rotating target depth map hole filling device, characterized in that: include: A model building module, used for training a rotating target detection model for identifying overhead transmission line joint pipes based on a rotating target dataset of overhead transmission line joint pipes; An image acquisition module, used for acquiring a color image and a depth image including a rotating target connecting tube through a depth camera; A target recognition module, used for recognizing the connecting pipe in the color image based on the rotating target detection model, and obtaining a rotating rectangular frame positioning result of the connecting pipe, wherein the rotating rectangular frame positioning result includes a tilt angle of the rotating rectangular frame; A depth processing module, used for mapping the horizontal rectangular frame corresponding to the positioning result of the rotated rectangular frame into the depth map, and extracting the regional depth value and the coordinates of the depth missing point of the horizontal rectangular frame in the depth map; A similarity calculation module, used for calculating the multi-scale similarity between the missing points and the neighborhood in the depth map in the color map; A completion order determination module, used to set a completion order according to the positioning result of the rotated rectangular frame and the Euclidean distance from the depth missing point in the depth map to a straight line passing through the center point of the rotated rectangular frame and perpendicular to the long side of the rotated rectangular frame; A completion module is used to complete the depth missing points in the horizontal rectangular box according to the completion order and the depth value completion formula based on the calculation result of the multi-scale similarity.

10. A rotating target depth map hole filling device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the rotation target depth map hole filling method according to any one of claims 1-8 according to the instructions in the program code.

Images