Image processing method and electronic device

Abstract

The application discloses an image processing method and an electronic device, and relates to the technical field of image processing, wherein the method comprises the following steps: extracting a first geometric edge line set of an initial depth image according to the initial depth image corresponding to a target monocular image; screening geometric edge lines with gradient scores greater than a target score threshold from a depth gradient image generated according to the initial depth image and an initial edge line set of the target monocular image to determine a second geometric edge line set; and fusing the first geometric edge line set and the second geometric edge line set to determine image geometric edge lines of the target monocular image and construct a final geometric edge line set. The method solves the problems of serious texture interference, insufficient utilization of depth information, difficulty in balancing the number and purity of edge lines, and insufficient real-time performance in the prior art, and achieves the technical effects of filtering texture noise while ensuring the number and robustness of edge lines and meeting the needs of multi-sensor fusion tasks.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and electronic device. Background Technology

[0002] Image geometric edge lines serve as a crucial link connecting two-dimensional textures and three-dimensional structures. The quality of their extraction directly determines the reliability of feature matching, thereby affecting the accuracy of extrinsic parameter calibration. They also play an important role in tasks such as three-dimensional scene reconstruction and robot SLAM (Simultaneous Localization and Mapping).

[0003] In related technologies, image edge extraction methods are mainly divided into two categories. One category is image edge extraction based on grayscale gradients, such as directly applying the LSD (Line Segment Detector) algorithm to the original RGB image (Red-Green-Blue image) or grayscale image to extract edge lines. The other category is image edge extraction methods based on depth information, such as first acquiring or generating a depth image, then marking edge points on the depth image, and finally obtaining geometric edge lines by fitting straight lines. Some studies can also use deep learning semantic segmentation models to perform pixel-level semantic annotation on images, extracting object contour pixels and fitting them as edge lines.

[0004] However, in related technologies, image edge extraction methods based on grayscale gradients are easily affected by texture and lighting interference, misclassifying texture edges as geometric edges, and the extracted edges have low geometric edge purity, which may be difficult to meet task requirements; image edge extraction methods based on depth information have insufficient edge quantity and rely on depth estimation accuracy, which can easily lead to missed detections in some scenarios and have weak generalization ability; edge extraction methods based on semantic segmentation rely on large-scale labeled data, which has high labeling costs and long cycles, and lacks real-time performance, which urgently needs to be addressed. Summary of the Invention

[0005] This application provides an image processing method and electronic device to at least solve the problems in related technologies, such as severe texture interference, insufficient utilization of depth information, difficulty in balancing the number of edge lines and the purity of geometric edges, and inability to effectively guarantee real-time performance, making it difficult to meet the needs of real-time scenarios.

[0006] This application provides an image processing method, comprising the following steps: determining an initial depth image corresponding to a target monocular image, and extracting a first set of geometric edge lines of the target monocular image based on the initial depth image; generating a depth gradient map of the target monocular image based on the initial depth image, and using the depth gradient map and the initial set of edge lines of the target monocular image to filter geometric edge lines with gradient scores greater than a target score threshold to determine a second set of geometric edge lines; fusing the first set of geometric edge lines and the second set of geometric edge lines to determine the image geometric edge lines of the target monocular image, and constructing a final set of geometric edge lines of the target monocular image.

[0007] This application also provides an image processing apparatus, comprising: an extraction module, configured to determine an initial depth image corresponding to a target monocular image, and extract a first set of geometric edge lines of the target monocular image based on the initial depth image; a filtering module, configured to generate a depth gradient map of the target monocular image based on the initial depth image, and filter geometric edge lines with gradient scores greater than a target score threshold using the depth gradient map and the initial set of edge lines of the target monocular image to determine a second set of geometric edge lines; and a fusion module, configured to fuse the first set of geometric edge lines and the second set of geometric edge lines to determine the image geometric edge lines of the target monocular image, and construct a final set of geometric edge lines of the target monocular image.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described image processing methods.

[0009] This application also provides a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described image processing methods.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described image processing methods.

[0011] This application allows for the construction of a final geometric edge set for a target monocular image by combining a first set of geometric edge lines extracted from the processed initial depth image corresponding to the target monocular image and a second set of geometric edge lines filtered based on a depth gradient map generated from the processed initial depth image. This addresses the technical problems of severe texture interference, insufficient utilization of depth information, difficulty in balancing the number of edge lines with geometric edge purity, and inability to effectively guarantee real-time performance in related image edge extraction methods, thus failing to meet the demands of real-time scenarios. It achieves a technical approach based on "depth estimation empowering geometric feature recognition + gradient filtering to remove texture interference + dual-source edge line fusion and completion." Depth information of the scene in the target monocular image is obtained through depth estimation, texture noise in the image is filtered by gradient filtering, and dual-source geometric edge lines are fused to ensure the quantity and robustness of geometric edge lines. This effectively solves the problems of texture interference and insufficient edge line quantity, achieving scene adaptability and high real-time performance. Furthermore, by employing a lightweight monocular depth estimation model and an efficient LSD edge detection algorithm, millisecond-level processing is achieved while maintaining extraction accuracy, meeting the real-time requirements of different scenarios. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of an image processing method provided according to an embodiment of this application; Figure 2 This is a schematic diagram comparing the original monocular image and the initial depth image according to an embodiment of this application; Figure 3 This is a depth gradient map according to an embodiment of this application; Figure 4 This is a schematic diagram of the depth map edge line according to an embodiment of this application; Figure 5 This is a schematic diagram of the initial edge lines of the original RGB image according to an embodiment of this application; Figure 6 This is a schematic diagram of the filtered edge line features according to one embodiment of this application; Figure 7 This is a flowchart of an image geometric edge line extraction method that integrates depth estimation and gradient filtering according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an image processing apparatus provided according to an embodiment of this application.

[0014] Figure label: Among them, 10 is an image processing device; 100 is an extraction module, 200 is a filtering module and 300 is a fusion module. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0016] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0017] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Before explaining the image processing methods in the embodiments of this application, the technical principles of image edge lines and other technologies involved in the embodiments of this application will be explained first.

[0019] In multi-sensor perception systems, cameras, with their advantages of high resolution and rich texture information, complement the high-precision 3D position information of LiDAR (Light Detection and Ranging). The fusion of the two is the core foundation for achieving environmental perception in autonomous driving and precise navigation for robots. However, the effective fusion of LiDAR and cameras first requires the calibration of their extrinsic parameters. The accuracy of this process is highly dependent on the accuracy of matching LiDAR point cloud features with image features.

[0020] Image geometric edge lines serve as a crucial link connecting two-dimensional textures and three-dimensional structures. The quality of their extraction directly determines the reliability of feature matching, thereby affecting the accuracy of extrinsic parameter calibration. They also play an important role in tasks such as 3D scene reconstruction and robot SLAM.

[0021] The core objective of image edge extraction is to filter out regions with significant changes in grayscale or structure from a two-dimensional pixel matrix. Line edges (i.e., edge lines) are key features reflecting the geometric structure of an object. Therefore, edge extraction techniques in related technologies mainly revolve around "grayscale change detection" or "geometric structure analysis," with the core idea being to fit continuous line features by identifying grayscale differences or spatial structure changes between pixels. However, in practical applications, the extraction of image geometric edge lines faces multiple real-world challenges.

[0022] First, natural scenes contain a large amount of texture information. Traditional edge extraction methods based on gray-level gradients are prone to misjudging texture edges as geometric edges, resulting in a large number of "false features". Illumination conditions such as strong light, shadow, and backlight can cause abnormal local gray-level distribution in the image, resulting in broken edge lines or false edges, which is difficult to meet the requirements of tasks such as LiDAR camera calibration.

[0023] Furthermore, traditional images are two-dimensional planar projections, which lose the depth information of three-dimensional scenes and cannot distinguish between "texture changes on the same plane" and "geometric transitions between different planes".

[0024] Although some methods have introduced depth image-assisted extraction, they are limited by the generalization ability of the depth estimation model and can only extract a small number of edge lines, which is difficult to meet the feature quantity requirements of subsequent calibration and optimization. If the extraction is solely based on the original image, it is difficult to guarantee the geometric purity of the edge lines, and it cannot effectively meet the requirements of tasks such as LiDAR camera calibration.

[0025] To address the shortcomings of image geometric edge line extraction in related technologies, this application designs an image geometric edge line extraction method that integrates depth estimation and gradient filtering. This method can fuse depth information and gradient filtering, ensuring the number and robustness of edge lines while filtering texture noise, thus meeting the needs of multi-sensor fusion tasks. Compared with other technologies, it can effectively improve the geometric purity, number, scene adaptability, and real-time performance of edge lines.

[0026] The embodiments of this application provide an image processing method, and the method is described in detail in conjunction with the execution flow of image processing.

[0027] Specifically, Figure 1 This is a flowchart of an image processing method provided according to an embodiment of this application.

[0028] like Figure 1 As shown, the image processing method includes the following steps: In step S101, an initial depth image corresponding to the target monocular image is determined, and a first set of geometric edge lines of the initial depth image is extracted based on the initial depth image.

[0029] It is understandable that the target monocular image here refers to a specific monocular image object for which geometric edge line extraction is required. Here, a monocular image can be understood as an RGB image, that is, a single RGB image captured by a single camera.

[0030] The initial depth image can be understood here as the raw depth map predicted directly from the target monocular image using a depth model, without optimization, correction, fusion, or other processing steps. In the initial depth image, each pixel stores not color, but rather the distance of that pixel to the camera, clearly indicating how far the object in the image is from the camera.

[0031] For example, Figure 2 This is a schematic diagram comparing the original monocular image and the initial depth image according to an embodiment of this application. Figure 2 As shown, this application may, but is not limited to, employ the lightweight monocular depth estimation model Depth Anything V2 to perform monocular image estimation on the original monocular image, i.e., the target monocular image in the embodiments of this application. Perform depth prediction to generate an initial depth image. .

[0032] This lightweight monocular depth estimation model, based on the Transformer architecture, can generate dense and high-precision depth images from a single RGB image, with a resolution comparable to the original image, i.e., the target monocular image. Consistency Depth values ​​in the initial depth image Reflecting each pixel The relative distance from the scene point to the camera.

[0033] Based on this initial depth image, embodiments of this application can extract a first set of geometric edge lines from the initial depth image. It should be noted that the first set of geometric edge lines here refers to the set of image geometric edge lines extracted from the initial depth image corresponding to the target monocular image. The term "first" here is merely a designation to distinguish it from other edge line sets and has no other meaning.

[0034] Optionally, in one embodiment of this application, extracting a first set of geometric edge lines from a target monocular image based on an initial depth image includes: filtering the initial depth image to obtain a processed initial depth image; and extracting a first set of geometric edge lines based on the processed initial depth image.

[0035] In some embodiments, considering that the generated initial depth image contains a lot of noise, in order to improve the accuracy of geometric edge line extraction, this application may, but is not limited to, perform a certain Gaussian filtering on the initial depth image, thereby extracting the first set of geometric edge lines based on the Gaussian-filtered initial depth image.

[0036] For example, this application may, but is not limited to, employ the following Gaussian filter on the initial depth image. Gaussian filtering is performed to remove depth estimation noise, resulting in the processed initial depth image. : In this embodiment of the application, the kernel size of the Gaussian filter can be, but is not limited to, set to... The standard deviation can be, but is not limited to, set to 1. The filtering formula can be expressed, but is not limited to, as:

[0037] in, , Both represent the positional offset of the convolution kernel. , Representing pixels of coordinates and coordinate, Gaussian function:

[0038] in, It represents pi (π).

[0039] Denoising depth image It can more accurately reflect the true depth distribution of the scene.

[0040] The embodiments of this application can filter noise in the initial depth map, reduce interference from invalid pixels, and help to make the depth gradient distribution smoother, thereby accurately locating the real physical edge and avoiding the extraction of false edge lines. At the same time, it can improve the continuity and integrity of the edge lines, reduce the error of subsequent line fitting and similarity matching, and lay a high-quality data foundation for subsequent deduplication, optimization and other steps.

[0041] Optionally, in one embodiment of this application, extracting a first set of geometric edge lines based on the processed initial depth image includes: calculating the horizontal and vertical gradients of each pixel in the processed initial depth image; detecting pixel regions in the processed initial depth image whose gradient angles meet the target requirements based on the horizontal and vertical gradients; and extracting the first set of geometric edge lines based on the pixel regions.

[0042] In some embodiments, after performing Gaussian filtering on the initial depth image, this application can extract the geometric edge lines in the processed initial depth image to obtain a first set of geometric edge lines.

[0043] For example, this application can, but is not limited to, the initial depth image after denoising using Gaussian filtering. The LSD algorithm is applied to extract the processed initial depth image. The geometric edge lines in the data are used to obtain the initial set of geometric edge lines. (i.e., the first set of geometric edge lines): Figure 3 This is a depth gradient map according to one embodiment of this application. Figure 3 As shown, embodiments of this application can, but are not limited to, use the Sobel operator to calculate the initial depth image after Gaussian filtering. Horizontal and vertical gradients for each pixel:

[0044]

[0045] in, Represents each pixel The horizontal gradient, Represents each pixel The vertical gradient, This represents the convolution operation.

[0046] Additionally, gradient magnitude The calculation method is as follows: It can reflect the intensity of changes in depth; gradient angle The calculation method is as follows: It can reflect the direction of change in depth.

[0047] Furthermore, considering that there will still be some noise in the processed initial depth image, and that the image gradient is prone to clutter, jitter, and false edges, directly fitting a straight line will result in serious deviation. Therefore, this embodiment of the application can detect pixel regions in the processed initial depth image whose gradient angles meet the target requirements by calculating the horizontal and vertical gradients, and then extract the geometric edge lines used to form the first geometric edge line set within these pixel regions.

[0048] Here, the target requirement can be understood as the standards / requirements that the pixel region where the geometric edge line is extracted should meet. For example, the gradient angle of each pixel within the region where the geometric edge line is extracted should have a certain consistency, that is, the orientation of pixels within the pixel region should be consistent.

[0049] This application embodiment can filter out pixel regions that meet certain requirements by calculating the horizontal and vertical gradients of each pixel in the processed initial depth image, thereby determining real edge pixels with consistent orientation based on pixel-level gradients and eliminating random and undirected noise. It ensures that the gradient angle of the region from which geometric edge lines are extracted should meet certain consistency, so that edge lines can be extracted based on pixel regions with uniform angles. This ensures that the extracted geometric edge lines are regular in orientation and fit the actual physical edge, greatly reducing scattered false edges and improving the accuracy and consistency of the edge line set.

[0050] Optionally, in one embodiment of this application, detecting pixel regions in the processed initial depth image whose gradient angles meet the target requirements based on horizontal and vertical gradients includes: calculating the gradient magnitude and gradient angle of each pixel based on horizontal and vertical gradients; and determining pixel regions that meet the target requirements based on pixel regions whose gradient magnitude is greater than the target gradient threshold and pixel regions whose gradient angle is less than the target angle threshold.

[0051] In actual implementation, when selecting pixel regions that meet certain requirements based on the horizontal and vertical gradients of each pixel in the processed initial depth image, this application may, but is not limited to, calculate the gradient magnitude and gradient angle of each pixel in the processed initial depth image based on the horizontal and vertical gradients of each pixel. Pixel regions with gradient magnitudes greater than the target gradient threshold and gradient angles less than the target angle threshold are pixel regions that meet certain requirements.

[0052] Here, the target gradient threshold refers to the minimum intensity boundary value used to determine whether a pixel belongs to a valid edge. Only when the gradient magnitude is greater than a certain gradient threshold can it be considered that the depth change at this pixel is significant enough to be a real edge or contour, rather than noise, smooth areas, or weak textures in the depth map; pixels with values ​​below this threshold are directly excluded, thus ensuring that only points with sufficiently reliable edge intensity remain in the pixel region.

[0053] The target angle threshold here refers to the upper limit of the gradient angle difference between any two pixels in a predetermined area when determining whether multiple pixels belong to the same straight line within an acceptable directional range. When the gradient angle difference between any two pixels in the area is less than the threshold, it indicates that the edge directions of the two pixels are basically consistent and belong to the same straight line structure. If the gradient angle difference between any two pixels in the area exceeds (is greater than or equal to) the threshold, it means that the directions of the two pixels are not consistent and cannot be classified into the same straight line support area. This ensures that the pixel directions within the pixel area are consistent and the collinearity is reliable.

[0054] In simple terms, the gradient threshold can constrain whether the edge line is sufficiently like an edge, and the angle threshold can constrain whether multiple pixels on the edge line are sufficiently like the same straight line. By combining the gradient magnitude being greater than a certain gradient threshold and the gradient angle being less than a certain angle threshold, the embodiments of this application can accurately circle the pixel region that truly belongs to the edge of a straight line.

[0055] It should be noted that the specific target gradient threshold and target angle threshold can be set or adjusted by those skilled in the art according to actual needs and circumstances. The embodiments in this application are only illustrative and do not impose specific limitations.

[0056] For example, this application may, but is not limited to, use the LSD algorithm to cluster pixels into "straight line support regions" (i.e., pixel regions that meet the target requirements in the embodiments of this application) by analyzing the consistency of gradient angles. The difference in gradient angles of pixels in the same region is less than a certain threshold, and the gradient magnitude is greater than the depth gradient threshold.

[0057] In other words, the embodiments of this application can use the LSD algorithm to first detect the straight line support region. That is, based on the consistency of pixel gradient angle and gradient magnitude threshold, pixels with similar directions and belonging to the same real edge are grouped into connected regions. At the same time, noise and invalid pixels with chaotic gradient direction and insufficient magnitude are filtered out, thereby ensuring that the pixels used later are real collinear and reliable and stable points.

[0058] It should be noted that in the embodiments of this application, the pixel region that meets certain requirements, namely the line support region, is to determine that the pixels in the region used to extract the edge line meet certain directional consistency. Each edge line corresponds to a line support region. Therefore, in the entire processed initial depth image, there can be at least one pixel region that meets certain requirements, that is, one or more.

[0059] The embodiments of this application can combine the gradient magnitude and gradient angle of each pixel to obtain the straight line support region, thereby providing accurate input for subsequent least squares straight line fitting, so that the fitted straight line segment is accurate in position and stable in angle. Then, short pseudo-lines are removed by length filtering, and finally a clean, accurate set of edge lines that can reflect the real structure can be extracted from the depth image.

[0060] Optionally, in one embodiment of this application, extracting a first set of geometric edge lines based on a pixel region includes: performing least squares line fitting on multiple pixels within the pixel region to obtain at least one line segment; filtering out short line segments with a length less than a first length threshold from the at least one line segment to obtain the first set of geometric edge lines.

[0061] In other embodiments, after obtaining a pixel region that meets certain requirements, this application can perform least squares line fitting on all pixels within the pixel region (line support region) to obtain at least one line segment (one or more line segments), including but not limited to information such as the starting point, ending point coordinates, and angle of the line segment.

[0062] Furthermore, to ensure that all obtained line segments are valid geometric edge lines, embodiments of this application can filter out line segments whose length is less than a first length threshold. The first set of geometric edge lines can be obtained by considering the short straight line segments and the remaining straight line segments after filtering. ,like Figure 4 As shown, Figure 4 This is a schematic diagram of the edge line of a depth map according to an embodiment of this application.

[0063] Here, the first length threshold can be understood as the minimum length limit that must be met when filtering line segments. Only when a line segment is greater than or equal to this length threshold can it be used as a geometric edge line in the final set of first geometric edge lines.

[0064] It should be noted that the specific first length threshold can be set or adjusted by those skilled in the art according to actual needs and circumstances. The embodiments in this application are only illustrative and do not impose specific limitations.

[0065] This application embodiment can aggregate pixels with consistent orientation into regular straight line segments, accurately restore the linear features of real physical edges, and remove scattered short straight line segments formed by noise or local pseudo edges, reducing redundancy and making the final first geometric edge line set more concise and in line with the actual scene structure, effectively improving the effectiveness and integrity of edge lines.

[0066] Step S102: Generate a depth gradient map of the target monocular image based on the initial depth image, and use the depth gradient map and the initial edge line set of the target monocular image to filter geometric edge lines with gradient scores greater than the target score threshold to determine the second geometric edge line set.

[0067] Those skilled in the art will understand that the number of geometric edge lines that can be extracted from depth images is relatively small, generally making it difficult to meet the feature quantity requirements of subsequent calibration and optimization.

[0068] Based on this, in some embodiments, this application can generate a depth gradient map of the target monocular image based on the initial depth image, that is, after performing Gaussian filtering on the initial depth image, the depth gradient map of the target monocular image is generated using the processed initial depth image.

[0069] Then, embodiments of this application can use the depth gradient map and the initial set of edge lines of the target monocular image to filter geometric edge lines with gradient scores greater than the target score threshold to determine the second set of geometric edge lines.

[0070] Here, the initial set of edge lines in the target monocular image can be understood as the set of edge lines composed of geometric edge lines extracted from the original target monocular image without any processing, including both geometric and texture edge lines in the target monocular image. For example... Figure 5 As shown, Figure 5 This is a schematic diagram of the initial edge lines of the original RGB image in one embodiment of this application, all of which are directly extracted by LSD.

[0071] Furthermore, the target score threshold can be understood here as the minimum gradient score limit that the initial edge lines in the pre-defined initial edge line set should satisfy. Only initial edge lines in the initial edge line set that are greater than or equal to this value can potentially become geometric edge lines in the final second geometric edge line set.

[0072] The gradient score here can be understood as a quantized score calculated for each edge line in the initial edge line set, based on its pixel coordinate set, tilt angle, and depth gradient map. In this embodiment, the gradient score can, but is not limited to, using the depth gradient component perpendicular to the edge line's own direction as the core indicator to measure the reliability of the edge line, thereby filtering out valid geometric edge lines.

[0073] This application embodiment can combine edge lines extracted directly from the depth image with edge lines filtered by gradients from the original image, thereby effectively making up for the problem of insufficient edge lines in the depth estimation model in large-area planes and distant object scenes, so as to ensure the feature quantity requirements of subsequent calibration optimization and other tasks.

[0074] Optionally, in one embodiment of this application, generating a depth gradient map of a target monocular image based on an initial depth image includes: calculating a gradient magnitude map and a gradient angle map of the target monocular image based on the processed initial depth image; and determining the depth gradient map based on the gradient magnitude map and the gradient angle map.

[0075] In some embodiments, when generating a depth gradient map of a target monocular image based on an initial depth image, the gradient magnitude and gradient angle map of the target monocular image can be calculated using the processed initial depth image, and the depth gradient map of the target monocular image can be determined by combining the gradient magnitude map and the gradient angle map.

[0076] For example, this application may, but is not limited to, still employ a lightweight monocular depth estimation model (such as DepthAnything V2) to the denoised initial depth image. Calculate the dense depth gradient map .

[0077] Among them, the dense depth gradient map is composed of the gradient magnitude map. and gradient angle diagram These two corresponding dense images together completely describe the dense depth gradient map. Information on the depth changes of each pixel.

[0078] That is, dense depth gradient map Each pixel location has a gradient value, and the gradient value at each pixel location is determined by the gradient magnitude map. and gradient angle diagram The magnitude and angle of the gradient at the corresponding position are determined. Among them, the gradient magnitude (size) refers to the length / intensity of the gradient value, which is a positive number; the gradient angle (direction) refers to the direction of the gradient, which is an angle.

[0079] The embodiments of this application can decompose the intensity and direction of depth changes into two structured dense maps, which can not only accurately filter out edge pixels with sufficient intensity, but also cluster pixels in the same direction according to the angle, making the representation of depth gradient clearer and quantifiable, and providing accurate and hierarchical gradient data support for subsequent edge extraction.

[0080] Optionally, in one embodiment of this application, a second set of geometric edge lines is determined by filtering geometric edge lines with gradient scores greater than a target score threshold using an initial edge line set of a depth gradient map and a target monocular image. This includes: obtaining a set of pixel coordinates for each edge line in the initial edge line set and calculating the tilt angle of each edge line; calculating the gradient score of each edge line based on the tilt angle and calculating the average length of the edge lines in the initial edge line set; and determining a target score threshold based on the average length of the edge lines, so as to obtain the second set of geometric edge lines based on the edge lines with gradient scores greater than the target score threshold.

[0081] As one possible approach, when using the depth gradient map and the initial edge line set of the target monocular image to filter edge lines with a length greater than a certain length threshold to determine the second geometric edge line set, this application may, but is not limited to, first obtain the pixel coordinate set of each edge line in the initial edge line set in order to calculate the tilt angle of each edge line.

[0082] Here, the set of pixel coordinates for each edge line can be understood as the set of coordinates of all pixels along that edge line. It should be noted that the coordinates of each pixel along each edge line are relative to the target monocular image.

[0083] In other words, the embodiments of this application may, but are not limited to, first construct a certain coordinate system based on the target monocular image. For example, a certain coordinate system may be constructed with the lower left corner of the target monocular image as the origin, the horizontal direction as the X coordinate, and the vertical direction as the Y coordinate. Alternatively, a certain coordinate system may be constructed with the center point of the target monocular image (the intersection of the horizontal and vertical midlines) as the origin, the horizontal midline as the X coordinate, and the vertical midline as the Y coordinate.

[0084] Therefore, each pixel on each edge line is in a coordinate system and has its own coordinates. Based on these coordinates, the tilt angle of each edge line (the angle between the edge line and the X or Y coordinate) can be calculated.

[0085] Furthermore, since the depth gradient direction corresponding to a geometric edge line is usually perpendicular to the edge line itself (such as the boundary line between a wall and the ground, where the depth gradient direction is perpendicular to the boundary line), embodiments of this application may, but are not limited to, select the vertical component of the depth gradient as a screening criterion for each edge line in the initial edge line set. Define a gradient score .

[0086] Furthermore, embodiments of this application can determine the target score threshold for filtering edge lines in the initial edge line set in other embodiments based on the average length of all edge lines in the initial edge line set.

[0087] Ultimately, all edge lines in the initial edge line set whose gradient scores are greater than the target score threshold can be retained, and the set of all retained edge lines is the second geometric edge line set.

[0088] This application embodiment can calculate the gradient score by combining the tilt angle of the edge line, thereby quantifying the effectiveness of the gradient feature of each edge line. Furthermore, this application embodiment can determine the screening threshold by the average length of the edge line, thereby avoiding the limitation of fixed thresholds, adapting to actual data, and ensuring that the final selected second edge line set is more in line with the scenario and has higher effectiveness.

[0089] Optionally, in one embodiment of this application, calculating the gradient score of each edge line based on the tilt angle includes: calculating the proportion of the depth gradient component of each edge line in the direction perpendicular to the edge line based on the depth gradient angle and tilt angle of each pixel in each edge line; and calculating the gradient score of each edge line based on the proportion of the components.

[0090] As can be understood from the descriptions of other embodiments, this application can calculate the gradient score of each edge line based on the tilt angle from the initial set of edge lines.

[0091] Specifically, in this embodiment, the proportion of the depth gradient component in the direction perpendicular to the edge line can be calculated based on the depth gradient angle and tilt angle of each pixel in each edge line, and then the gradient score of each edge line in the initial edge line set can be calculated based on the proportion of the component.

[0092] As will be understood by those skilled in the art, a depth gradient map is generated from a processed initial depth image. It contains gradient magnitude plot and gradient angle diagram .

[0093] The gradient magnitude map is a complete and dense map formed by storing the gradient magnitude of each pixel in the processed initial depth image according to the original pixel position. Each pixel value is the gradient magnitude (depth gradient magnitude) at that position.

[0094] The horizontal and vertical gradients of each pixel are obtained through The gradient magnitude map can be calculated. Each pixel Similarly, for the gradient magnitude, simply use the processed initial depth image in this formula. Horizontal gradient of each pixel and vertical gradient Replace with gradient magnitude map Each pixel The horizontal and vertical gradients are sufficient. Among them, the gradient magnitude diagram... Each pixel The horizontal and vertical gradients are calculated in the same way as the processed initial depth image. Horizontal gradient of each pixel and vertical gradient The calculation method is the same.

[0095] Furthermore, the gradient angle map is another complete and dense map formed by storing the gradient angles calculated for each pixel in the processed initial depth image, corresponding one-to-one with the original pixel positions. Each pixel value represents the gradient angle (depth gradient angle) at that location. This is achieved by combining the horizontal and vertical gradients of each pixel... The gradient angle of each pixel can be calculated. Similarly, the processed initial depth image in this formula can be used. Horizontal gradient of each pixel and vertical gradient Replace with gradient magnitude map Each pixel The horizontal and vertical gradients are sufficient.

[0096] Then, the embodiments of this application can be implemented through Get each pixel The depth gradient angle is used to combine each pixel. The tilt angle of each edge line is calculated, and the proportion of the depth gradient (depth gradient information (gradient magnitude, gradient angle) of all pixels on the edge line) in the direction perpendicular to the edge line is calculated.

[0097] Ultimately, based on this component proportion, the gradient score of each edge line in the initial edge line set can be calculated.

[0098] This application embodiment can combine the depth gradient angle of each pixel in each edge line and the tilt angle of the edge line to calculate the proportion of the depth gradient component corresponding to each edge line in the direction perpendicular to the edge line. The authenticity of the edge line is quantified by "whether the direction of depth change is perpendicular to the edge". The gradient score of each edge line is calculated based on the proportion of the component, so that the selected edge lines are closer to the real physical boundary than false edges.

[0099] Optionally, in one embodiment of this application, calculating the gradient score of each edge line based on the component ratio includes: obtaining the depth gradient angle and depth gradient magnitude of each pixel in each edge line; and calculating the gradient score of each edge line based on the component ratio and the depth gradient angle and depth gradient magnitude of each pixel.

[0100] In some embodiments, when calculating the gradient score of each edge line in the initial edge line set based on the proportion of the component of the depth gradient corresponding to each edge line in the direction perpendicular to the edge line, this application needs to combine the depth gradient angle and depth gradient magnitude of each pixel in each edge line.

[0101] For example, this application can process each initial edge line in the initial edge line set. The Bresenham algorithm (Bresenham's Line Algorithm, an integer arithmetic algorithm for drawing lines) is used to obtain the coordinates of all the pixels contained within it. ( For the edge line The number of pixels, (Represents the pixel index). Then, calculate each edge line. Angle (i.e., the angle of inclination of the straight line), then the edge line gradient score The calculation formula can be expressed, but is not limited to, as follows:

[0102] in, For pixels Depth gradient angle, Used to calculate the proportion of the depth gradient component in the direction perpendicular to the edge line; For pixels Depth gradient magnitude; weighting coefficients ,in For the edge line Length, It is the smallest side length among all the side lengths of the original image.

[0103] The embodiments of this application can effectively utilize the characteristic that the depth gradient direction corresponding to the geometric edge line is usually perpendicular to the edge line direction, define a gradient score based on the vertical component of the depth gradient, and achieve accurate screening of the initial edge lines of the original image, retaining edge lines with actual geometric significance.

[0104] Optionally, in one embodiment of this application, determining the target score threshold based on the average length of the edge lines includes: obtaining the target side length of the target monocular image; and determining the target score threshold based on the target side length and the average length of the edge lines.

[0105] Based on the descriptions of other embodiments, it is understood that the embodiments of this application can determine the target score threshold based on the average length of all edge lines in the initial edge line set, and then obtain the second geometric edge line set based on the edge lines whose gradient scores are greater than the target score threshold.

[0106] In some embodiments, to further ensure the robustness of the edge lines, this application may introduce constraints on the edge line length and image size—because longer edge lines are more likely to be true geometric edges.

[0107] Specifically, in determining the target score threshold based on the average length of all edge lines in the initial edge line set, this application embodiment may, but is not limited to, obtain the target side length of the target monocular image, and then combine the target side length and the average length of all edge lines in the initial edge line set to determine the target score threshold.

[0108] Here, the target side length can be understood as the largest side length among all sides of the original target monocular image.

[0109] Target score threshold The calculation method can be, but is not limited to, expressed as follows:

[0110] in, For the edge line Length, For the initial set of edge lines The average length of all edge lines in the middle. This is the maximum side length among all sides of the original RGB image.

[0111] If the edge line gradient score If so, it is determined to be the edge line. For valid geometric edge lines, all retained geometric edge lines together form the set of geometric edge lines. That is, the set of second geometric edge lines. For example... Figure 6 As shown, Figure 6 This is a schematic diagram of the filtered edge line features according to one embodiment of this application.

[0112] The embodiments of this application can dynamically adjust the filtering threshold (target score threshold) based on the image size, edge line length, etc. of the original RGB image without manual intervention. It can effectively adapt to RGB images with different texture complexities, different lighting conditions, and different scene structures, and has extremely strong generalization ability.

[0113] Step S103: Fuse the first set of geometric edge lines and the second set of geometric edge lines to determine the image geometric edge lines of the target monocular image and construct the final set of geometric edge lines of the target monocular image.

[0114] In some embodiments, after obtaining the processed initial depth image The first set of geometric edge lines obtained, and the depth image Calculated dense depth gradient map and the original image After filtering the initial set of edge lines to obtain the second set of geometric edge lines, this application can fuse the two to determine the final geometric edge lines of the target monocular image. The set of all geometric edge lines is the final geometric edge line set of the target monocular image.

[0115] This application's embodiments utilize a technical approach of "depth estimation-enabled geometric feature recognition, gradient filtering to remove texture interference, and dual-source edge line fusion for completion." It obtains depth information of the scene in the target monocular image through depth estimation, filters texture noise in the image using gradient filtering, and fuses dual-source geometric edge lines to ensure the quantity and robustness of geometric edges. This effectively solves the problems of texture interference and insufficient edge line quantity, achieving scene adaptability and high real-time performance. Furthermore, its lightweight monocular depth estimation model and efficient LSD edge detection algorithm can achieve millisecond-level processing on embedded devices while maintaining extraction accuracy, meeting the real-time requirements of various scenarios such as autonomous driving and robot navigation.

[0116] Optionally, in one embodiment of this application, fusing the first set of geometric edge lines and the second set of geometric edge lines includes: calculating the similarity between each edge line in the first set of geometric edge lines and each edge line in the second set of geometric edge lines; and fusing the first set of geometric edge lines and the second set of geometric edge lines based on the similarity.

[0117] In some embodiments, since the first set of geometric edge lines and the second set of geometric edge lines are respectively the first set of geometric edge lines obtained from the processed initial depth image and the second set of geometric edge lines obtained by filtering the dense depth gradient map calculated from the depth image and the initial set of edge lines of the original image, there may be duplicate geometric edge lines in the first set of geometric edge lines and the second set of geometric edge lines.

[0118] Based on this, embodiments of this application can calculate the similarity between each edge line in the first geometric edge line set and each edge line in the second geometric edge line set, so as to remove edge lines with excessive similarity in the two geometric edge line sets according to the similarity, and avoid duplicate edge lines in the final geometric edge line set.

[0119] The embodiments of this application can calculate the similarity between each edge line in the first set of geometric edge lines and each edge line in the second set of geometric edge lines, so as to accurately determine whether there are duplicate edge lines in the two sets based on the similarity, so as to ensure that the edge lines in the final set of geometric edge lines are all valid and non-repeating.

[0120] Optionally, in one embodiment of this application, fusing a first set of geometric edge lines and a second set of geometric edge lines to determine the geometric edge lines of the target monocular image and constructing a final set of geometric edge lines for the target monocular image includes: removing the shorter edge line from two edge lines in the first set of geometric edge lines or the second set of geometric edge lines whose similarity satisfies the target similarity threshold, provided that the similarity satisfies the target similarity threshold; merging the first set of geometric edge lines and the second set of geometric edge lines after removing the shorter edge line, or merging the first set of geometric edge lines and the second set of geometric edge lines after removing the shorter edge line, to obtain the final set of geometric edge lines.

[0121] In actual implementation, when this application fuses each edge line in the first geometric edge line set and each edge line in the second geometric edge line set based on their similarity, the first geometric edge line set and the second geometric edge line set can be mainly achieved through the comparison results of similarity and target similarity threshold.

[0122] Specifically, if any two edge lines in the first set of geometric edge lines and the second set of geometric edge lines have a similarity that satisfies the target similarity threshold, then in this embodiment of the application, the shorter edge line among the two edge lines that satisfy the target similarity threshold can be removed from the first set of geometric edge lines or the second set of geometric edge lines, and the longer edge line can be retained.

[0123] Here, the target similarity threshold can be understood as a pre-set critical value used to determine whether two edge lines are duplicate edges. It can be used to measure whether two edge lines are similar enough to determine whether they are duplicate extraction results of the same physical edge. When the calculated similarity value of any two edge lines from the first and second geometric edge line sets is greater than or equal to the threshold, these two lines can be considered as duplicate detections of the same real physical edge; conversely, if the similarity is lower than the threshold, they can be determined as two different edge lines, and both should be retained.

[0124] By performing similar operations on each edge line in the first set of geometric edge lines and each edge line in the second set of geometric edge lines, the repeated shorter geometric edge lines in the two sets of geometric edge lines can be effectively removed. By merging the first set of geometric edge lines and the second set of geometric edge lines after removing these edge lines, the final set of geometric edge lines can be obtained.

[0125] The embodiments of this application can accurately remove redundant edge lines that are extracted repeatedly through similarity calculation, and retain longer and more complete effective edges, thereby effectively avoiding the duplication and redundancy of two sets of edge lines, making the final geometric edge line set more concise and complete, conforming to the real physical edge features, and improving the accuracy and practicality of edge extraction results.

[0126] Optionally, in one embodiment of this application, calculating the similarity between each edge line in the first set of geometric edge lines and each edge line in the second set of geometric edge lines includes: obtaining the angle and midpoint of each edge line in the first set of geometric edge lines, and the angle and midpoint of each edge line in the second set of geometric edge lines; and calculating the similarity based on the angle and midpoint of each edge line in the first set of geometric edge lines and the angle and midpoint of each edge line in the second set of geometric edge lines.

[0127] Based on the relevant descriptions in other embodiments, it can be understood that this application can calculate the similarity between each edge line in the first set of geometric edge lines and each edge line in the second set of geometric edge lines, thereby filtering out extremely similar geometric edge lines in the first set of geometric edge lines and the second set of geometric edge lines.

[0128] In actual implementation, when calculating the similarity between each edge line in the first set of geometric edge lines and each edge line in the second set of geometric edge lines, this application may, but is not limited to, calculate the similarity based on the angle and midpoint of each edge line in the first set of geometric edge lines and the angle and midpoint of each edge line in the second set of geometric edge lines.

[0129] For example, this application can compute two edge lines from any first set of geometric edge lines and second set of geometric edge lines. and (from respectively) and The similarity between the two edge lines is used to determine if the similarity exceeds a threshold. If the similarity exceeds a threshold, the edge line is considered a duplicate and the longer one is retained.

[0130] In this embodiment of the application, the edge line similarity calculation may, but is not limited to, using a dual index of "direction + distance": let the edge line The angle is Midpoint is ) edge line The angle is Midpoint is If the similarity is... for:

[0131] in, It can reflect the angular similarity of two edge lines (range of values) Take a positive value here. ), Reflects the similarity of midpoint distances (range of values) ), Here is the kernel function, where This represents the pixel value of the distance attenuation coefficient.

[0132] It should be noted that 0.6 and 0.4 are the weighting coefficients for angle similarity and midpoint distance similarity, respectively. In practical applications, these coefficients can be set or adjusted by those skilled in the art according to the actual situation. The 0.6 and 0.4 in this embodiment are only illustrative examples and are not intended to impose any specific limitations.

[0133] After removing shorter edge lines from the repeated edge lines, the remaining edge lines are merged to obtain the image geometric edge lines of the target monocular image, resulting in the final set of image geometric edge lines. ( (Number of repeating edge lines).

[0134] This application embodiment can use a dual index of "direction + distance" to calculate the similarity of edge lines in the first and second geometric edge line sets, thereby effectively achieving efficient deduplication and fusion of dual-source edge lines and ensuring the integrity and uniqueness of the edge line sets. Furthermore, this application embodiment can dynamically adjust the weight coefficients of the direction and distance indices through a weighted approach, thereby effectively adapting to different computational needs in different scenarios and significantly improving its flexibility and generalization.

[0135] Optionally, in one embodiment of this application, the first set of geometric edge lines and the second set of geometric edge lines are fused to determine the image geometric edge lines of the target monocular image and to construct the final set of geometric edge lines of the target monocular image, including: obtaining the start coordinates, end coordinates, angle, and length of each edge line in the first set of geometric edge lines and the second set of geometric edge lines; and determining the final set of geometric edge lines based on the start coordinates, end coordinates, angle, and length of each edge line.

[0136] In some embodiments, this application may use a set of edge lines extracted based on an initial depth image. The set of edge lines filtered by gradient from the original image The images are then fused to obtain the final set of geometric edge lines.

[0137] In this embodiment, each edge line in the final geometric edge line set includes, but is not limited to, information such as start coordinates, end coordinates, angle, and length, so that it can be directly used for subsequent LiDAR-camera feature matching, extrinsic parameter calibration and other tasks.

[0138] The embodiments of this application can obtain information such as the starting point coordinates, ending point coordinates, angle, and length of each edge line, so as to effectively support various tasks that require accurate extraction of image geometric edge lines, such as LiDAR camera extrinsic parameter calibration, 3D scene reconstruction, robot navigation SLAM, intelligent monitoring target tracking, and industrial product defect detection.

[0139] The image processing method in this application will be explained in detail below with a specific embodiment.

[0140] Figure 7 This is a flowchart illustrating an embodiment of an image geometric edge extraction method that integrates depth estimation and gradient filtering, according to this application. Figure 7 As shown, taking the LiDAR camera extrinsic parameter calibration scenario for autonomous driving as an example, its workflow can be represented, but is not limited to, as follows: (1) Data preparation: Collect RGB image datasets for autonomous driving scenarios, including but not limited to images of different scenarios such as urban roads, highways, and suburban roads, covering different lighting conditions such as strong light, shadow, and backlight. The image resolution can be, but is not limited to, uniformly set to 1920×1080. At the same time, point cloud data of the corresponding scenarios can be collected through LiDAR devices for subsequent edge line matching verification; (2) According to the overall framework of the system, depth image generation and optimization are performed on each original RGB image. The initial depth image is generated using the Depth Anything V2 model, and the Gaussian filter convolution kernel size is set to 5×5 with a standard deviation of 1.5. Edge lines are extracted using the LSD algorithm, and the set of edge lines directly extracted from the depth image is obtained by using straight line support region detection and short straight line segment filtering threshold. ; (3) Perform depth gradient map calculation and original image edge line screening. After calculating the depth gradient map, apply the LSD algorithm to the original image to extract the initial edge lines, set the weight coefficients in the edge line gradient score calculation, and use the gradient score of each edge line and a certain score threshold to screen out the effective geometric edge lines, retain the edge lines with geometric significance, and obtain the set of screened edge lines. ; (4) Perform dual-source edge line fusion, set the weight coefficient, distance attenuation coefficient (pixel), and similarity threshold in the similarity calculation, and obtain the final geometric edge line set after removing duplicate edge lines; (5) Match the extracted edge lines with the geometric edge points extracted from the LiDAR point cloud to verify whether the geometric purity and number of the edge lines meet the requirements of external parameter calibration.

[0141] Experimental results show that the method in this embodiment extracts edge lines with a false feature ratio of less than 10%, which effectively improves the geometric purity of the edge lines. The number of edge lines is increased by more than 80% compared with the pure depth extraction method, which can effectively support the LiDAR camera extrinsic calibration task.

[0142] Additionally, the image processing method in this application embodiment can be applied not only to autonomous driving LiDAR camera extrinsic calibration scenarios, but also to tasks requiring precise extraction of image geometric edge lines, such as 3D scene reconstruction, robot navigation SLAM, intelligent monitoring target tracking, and industrial product defect detection.

[0143] Furthermore, the depth estimation model used in this application embodiment can be replaced with other lightweight monocular depth estimation models, and the edge detection algorithm can also be replaced with other efficient line detection algorithms, as long as the core idea of ​​integrating depth estimation and gradient screening is maintained.

[0144] Based on the dual-source edge line fusion strategy in the embodiments of this application, combining it with other multi-source edge line fusion methods, such as edge lines obtained by semantic segmentation, can further improve the robustness and integrity of the edge lines.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0146] Embodiments of this application also provide an image processing apparatus 10, including: an extraction module 100, a filtering module 200, and a fusion module 300.

[0147] The extraction module 100 is used to determine the initial depth image corresponding to the target monocular image, and extract the first set of geometric edge lines of the initial depth image based on the initial depth image. The filtering module 200 is used to generate a depth gradient map of the target monocular image based on the initial depth image, and to filter geometric edge lines with gradient scores greater than the target score threshold using the depth gradient map and the initial edge line set of the target monocular image to determine a second set of geometric edge lines. The fusion module 300 is used to fuse the first set of geometric edge lines and the second set of geometric edge lines to determine the image geometric edge lines of the target monocular image and construct the final set of geometric edge lines of the target monocular image.

[0148] Optionally, in one embodiment of this application, the extraction module 100 includes: a processing unit, configured to perform filtering processing on the initial depth image to obtain a processed initial depth image; and an extraction unit, configured to extract a first set of geometric edge lines based on the processed initial depth image.

[0149] Optionally, in one embodiment of this application, the extraction unit includes: a first calculation subunit, used to calculate the horizontal and vertical gradients of each pixel in the processed initial depth image; a detection subunit, used to detect pixel regions in the processed initial depth image whose gradient angles meet the target requirements based on the horizontal and vertical gradients; and an extraction subunit, used to extract a first set of geometric edge lines based on the pixel regions.

[0150] Optionally, in one embodiment of this application, the detection subunit includes: a calculation subunit, used to calculate the gradient magnitude and gradient angle of each pixel based on the horizontal gradient and the vertical gradient; and a determination subunit, used to determine the pixel region that meets the target requirements based on the pixel region whose gradient magnitude is greater than the target gradient threshold and the pixel region whose gradient angle is less than the target angle threshold.

[0151] Optionally, in one embodiment of this application, the extraction sub-unit includes: a fitting sub-unit, used to perform least squares line fitting on multiple pixels within a pixel region to obtain at least one line segment; and a filtering sub-unit, used to filter short line segments with a length less than a first length threshold from the at least one line segment to obtain a first set of geometric edge lines.

[0152] Optionally, in one embodiment of this application, the screening module 200 includes: a first calculation unit, configured to calculate the gradient magnitude map and gradient angle map of the target monocular image based on the processed initial depth image; and a first determination unit, configured to determine the depth gradient map based on the gradient magnitude map and gradient angle map.

[0153] Optionally, in one embodiment of this application, the filtering module 200 includes: a first acquisition unit, used to acquire the set of pixel coordinates of each edge line in the initial edge line set and calculate the tilt angle of each edge line; a second calculation unit, used to calculate the gradient score of each edge line based on the tilt angle and calculate the average length of the edge lines in the initial edge line set; and a second determination unit, used to determine a target score threshold based on the average length of the edge lines, so as to obtain a second geometric edge line set based on the edge lines whose gradient scores are greater than the target score threshold.

[0154] Optionally, in one embodiment of this application, the second determining unit includes: a first acquiring subunit, used to acquire the target side length of the target monocular image; and a determining subunit, used to determine a target score threshold based on the target side length and the average length of the edge line.

[0155] Optionally, in one embodiment of this application, the second calculation unit includes: a second calculation subunit, configured to calculate the proportion of the depth gradient component of each edge line in the direction perpendicular to the edge line based on the depth gradient angle and tilt angle of each pixel in each edge line; and a third calculation subunit, configured to calculate the gradient score of each edge line according to the proportion of the components.

[0156] Optionally, in one embodiment of this application, the third calculation subunit includes: an acquisition subunit, configured to acquire the depth gradient angle and depth gradient magnitude of each pixel in each edge line; and a calculation subunit, configured to calculate the gradient score of each edge line based on the component ratio and the depth gradient angle and depth gradient magnitude of each pixel.

[0157] Optionally, in one embodiment of this application, the fusion module 300 includes: a third calculation unit, configured to calculate the similarity between each edge line in the first geometric edge line set and each edge line in the second geometric edge line set; and a fusion unit, configured to fuse the first geometric edge line set and the second geometric edge line set according to the similarity.

[0158] Optionally, in one embodiment of this application, the fusion module 300 includes: a removal unit, configured to remove the shorter edge line from two edge lines in a first geometric edge line set or a second geometric edge line set whose similarity satisfies the target similarity threshold, provided that the similarity satisfies the target similarity threshold; and a merging unit, configured to merge the first geometric edge line set and the second geometric edge line set after removing the shorter edge line, or to merge the first geometric edge line set and the second geometric edge line set after removing the shorter edge line, to obtain a final geometric edge line set.

[0159] Optionally, in one embodiment of this application, the third calculation unit includes: a second acquisition subunit, used to acquire the angle and midpoint of each edge line in the first geometric edge line set, and the angle and midpoint of each edge line in the second geometric edge line set; and a fourth calculation subunit, used to calculate the similarity based on the angle and midpoint of each edge line in the first geometric edge line set and the angle and midpoint of each edge line in the second geometric edge line set.

[0160] Optionally, in one embodiment of this application, the fusion module 300 includes: a second acquisition unit, used to acquire the start coordinates, end coordinates, angle, and length of each edge line in the first geometric edge line set and the second geometric edge line set; and a third determination unit, used to determine the final geometric edge line set based on the start coordinates, end coordinates, angle, and length of each edge line.

[0161] For a description of the features in the embodiment corresponding to the image processing apparatus, please refer to the relevant description in the embodiment corresponding to the image processing method, which will not be repeated here.

[0162] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described image processing method embodiments.

[0163] Embodiments of this application also provide a non-volatile computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described image processing method embodiments at runtime.

[0164] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0165] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described image processing method embodiments.

[0166] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described image processing method embodiments.

[0167] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0168] The foregoing has provided a detailed description of the image processing method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. An image processing method, characterized in that, Includes the following steps: Determine an initial depth image corresponding to the target monocular image, and extract a first set of geometric edge lines from the initial depth image based on the initial depth image; A depth gradient map of the target monocular image is generated based on the initial depth image, and geometric edge lines with gradient scores greater than a target score threshold are filtered using the depth gradient map and the initial edge line set of the target monocular image to determine a second set of geometric edge lines. The first set of geometric edge lines and the second set of geometric edge lines are fused to determine the image geometric edge lines of the target monocular image, and the final set of geometric edge lines of the target monocular image is constructed. The step of using the depth gradient map and the initial edge line set of the target monocular image to filter geometric edge lines with gradient scores greater than a target score threshold to determine a second geometric edge line set includes: obtaining the pixel coordinate set of each edge line in the initial edge line set and calculating the tilt angle of each edge line; calculating the gradient score of each edge line based on the tilt angle and calculating the average length of the edge lines in the initial edge line set; determining the target score threshold based on the average length of the edge lines, so as to obtain the second geometric edge line set based on the edge lines with gradient scores greater than the target score threshold. The step of calculating the gradient score of each edge line based on the tilt angle includes: calculating the proportion of the depth gradient component of each edge line in the direction perpendicular to the edge line based on the depth gradient angle of each pixel in each edge line and the tilt angle; and calculating the gradient score of each edge line based on the proportion of the components. The expression for the gradient score is as follows: in, For the edge line gradient score, For pixels Depth gradient angle, Used to calculate the proportion of the depth gradient component in the direction perpendicular to the edge line; For pixels Depth gradient magnitude; weighting coefficients ,in For the edge line Length, It is the minimum side length among all the side lengths of the original image. For the edge line The number of pixels, Indicates the pixel index; The step of fusing the first set of geometric edge lines and the second set of geometric edge lines includes: calculating the similarity between each edge line in the first set of geometric edge lines and each edge line in the second set of geometric edge lines; and fusing the first set of geometric edge lines and the second set of geometric edge lines based on the similarity. The step of fusing the first set of geometric edge lines and the second set of geometric edge lines to determine the geometric edge lines of the target monocular image and constructing the final set of geometric edge lines of the target monocular image includes: removing the shorter edge line from two edge lines in the first set of geometric edge lines or the second set of geometric edge lines whose similarity meets the target similarity threshold, provided that the similarity meets the target similarity threshold; merging the first set of geometric edge lines after removing the shorter edge line and the second set of geometric edge lines, or merging the first set of geometric edge lines and the second set of geometric edge lines after removing the shorter edge line, to obtain the final set of geometric edge lines. The step of calculating the similarity between each edge line in the first set of geometric edge lines and each edge line in the second set of geometric edge lines includes: obtaining the angle and midpoint of each edge line in the first set of geometric edge lines, and the angle and midpoint of each edge line in the second set of geometric edge lines; and calculating the similarity based on the angle and midpoint of each edge line in the first set of geometric edge lines and the angle and midpoint of each edge line in the second set of geometric edge lines.

2. The image processing method according to claim 1, characterized in that, The step of extracting the first set of geometric edge lines from the initial depth image includes: The initial depth image is filtered to obtain the processed initial depth image; Based on the processed initial depth image, the first set of geometric edge lines is extracted.

3. The image processing method according to claim 2, characterized in that, The step of extracting the first set of geometric edge lines based on the processed initial depth image includes: Calculate the horizontal and vertical gradients of each pixel in the processed initial depth image; Based on the horizontal gradient and the vertical gradient, detect pixel regions in the processed initial depth image whose gradient angles meet the target requirements; The first set of geometric edge lines is extracted based on the pixel region.

4. The image processing method according to claim 3, characterized in that, The step of detecting pixel regions in the processed initial depth image whose gradient angles satisfy the target requirements based on the horizontal and vertical gradients includes: Based on the horizontal gradient and the vertical gradient, calculate the gradient magnitude and gradient angle of each pixel; Based on the pixel regions where the gradient magnitude is greater than the target gradient threshold and the pixel regions where the gradient angle is less than the target angle threshold, the pixel regions that meet the target requirements are determined.

5. The image processing method according to claim 3, characterized in that, The step of extracting the first set of geometric edge lines based on the pixel region includes: Perform least-squares line fitting on multiple pixels within the pixel region to obtain at least one line segment; Filter at least one of the line segments whose length is less than a first length threshold to obtain the first set of geometric edge lines.

6. The image processing method according to claim 1, characterized in that, The step of generating a depth gradient map of the target monocular image based on the initial depth image includes: Based on the processed initial depth image, the gradient magnitude map and gradient angle map of the target monocular image are calculated; The depth gradient map is determined based on the gradient magnitude map and the gradient angle map.

7. The image processing method according to claim 1, characterized in that, Determining the target score threshold based on the average length of the edge lines includes: Obtain the target side length of the target monocular image; The target score threshold is determined based on the target side length and the average length of the edge line.

8. The image processing method according to claim 1, characterized in that, The step of calculating the gradient score of each edge line based on the component proportion includes: Obtain the depth gradient angle and depth gradient magnitude of each pixel in each edge line; Based on the component proportions and the depth gradient angle and depth gradient magnitude of each pixel, the gradient score of each edge line is calculated.

9. The image processing method according to claim 1, characterized in that, The process of fusing the first set of geometric edge lines and the second set of geometric edge lines to determine the image geometric edge lines of the target monocular image and constructing the final set of geometric edge lines of the target monocular image includes: Obtain the start coordinates, end coordinates, angle, and length of each edge line in the first set of geometric edge lines and the second set of geometric edge lines; The final set of geometric edge lines is determined based on the starting point coordinates, ending point coordinates, angle, and length of each edge line.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the image processing method as described in any one of claims 1 to 9.

Images