Visual image processing method, device, equipment and program product

Abstract

The invention relates to the field of image processing, in particular to a visual image processing method and device, equipment and a program product. The method comprises the following steps: acquiring a binocular image, and generating a first disparity map through a stereo matching network; obtaining a second disparity map through extreme value filtering; generating an edge mask through edge detection; performing edge filtering on the second disparity map according to the edge mask to obtain a third disparity map; and horizontal displacement is obtained according to a parallax value in the third parallax image, parameters are calibrated through a camera, the horizontal displacement is converted into depth information, and a three-dimensional point cloud is generated according to the positions of the pixel points and the depth information. According to the method, a real boundary is extracted through edge filtering, parallax of a boundary area is forcibly corrected by using an edge mask, a transition zone of network prediction is suppressed, burrs or outliers of point clouds after conversion can be effectively reduced, scattered point cloud noise is eliminated, the object contour boundary is clearer, and the perception precision of a visual system is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to visual image processing methods, devices, equipment and program products. Background Art

[0002] At present, the point cloud data estimated by the deep learning-based stereo matching method has noise at the boundaries. This noise is caused by the characteristics of the underlying smooth function approximator of deep learning. It cannot be effectively suppressed by simply modifying the network structure, thereby reducing the perception accuracy of the visual system. Summary of the Invention

[0003] In view of this, the embodiments of the present application provide a visual image processing method, device, equipment and program product, which can accurately reconstruct image boundaries, make object contour boundaries clearer, and help improve the perception accuracy of the visual system.

[0004] A first aspect of an embodiment of the present application provides a visual image processing method, the method comprising:

[0005] Obtain binocular images and generate a first disparity map using a pre-trained deep learning-based stereo matching network;

[0006] performing extreme value filtering on the first disparity map to obtain a second disparity map;

[0007] performing edge detection on the second disparity map to generate an edge mask;

[0008] performing edge filtering on the second disparity map according to the edge mask to obtain a third disparity map;

[0009] The horizontal displacement of each pixel in the third disparity map is calculated based on the disparity value of the pixel, and the horizontal displacement is converted into depth information of the pixel in three-dimensional space through camera calibration parameters. A three-dimensional point cloud is generated based on the position of the pixel in the third disparity map and the depth information.

[0010] In combination with the first aspect, in a first possible implementation manner of the first aspect, performing extreme value filtering on the first disparity map to obtain the second disparity map includes:

[0011] Counting and sorting all disparity values ​​in the first disparity map;

[0012] Determine the range of parallax values ​​that need to be filtered out based on the preset maximum value percentage and minimum value percentage;

[0013] Filter out corresponding pixel points within the disparity value range to generate the second disparity map.

[0014] In combination with the first aspect, in a second possible implementation manner of the first aspect, performing edge detection on the second disparity map to generate an edge mask includes:

[0015] Smoothing the second disparity map using a Gaussian filter to obtain a smoothed image;

[0016] Obtaining the gradient intensity and gradient direction of the smoothed image;

[0017] According to the gradient intensity and gradient direction, a non-maximum suppression algorithm is used to retain the local maximum value to generate a preliminary edge, and an edge mask is generated according to the preliminary edge.

[0018] In combination with the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, generating an edge mask according to the preliminary edge includes:

[0019] Performing a double thresholding process on the preliminary edge to determine a first edge pixel and a second edge pixel, wherein the gradient intensity of the first edge pixel is greater than the gradient intensity of the second edge pixel;

[0020] Traversing the second edge pixels to determine whether there is a first edge pixel in its neighborhood;

[0021] If so, the second edge pixel is retained; otherwise, the second edge pixel is removed to generate the edge mask.

[0022] In combination with the second possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, obtaining the gradient strength and gradient direction of the smoothed image includes:

[0023] Calculating the gradients of the smoothed image in the horizontal and vertical directions respectively;

[0024] Calculating the gradient strength of each pixel based on the gradients in the horizontal and vertical directions;

[0025] According to the gradients in the horizontal and vertical directions, a four-quadrant inverse tangent function is used to calculate the gradient direction of each pixel.

[0026] In combination with the first aspect, in a fifth possible implementation manner of the first aspect, performing edge filtering on the second disparity map according to the edge mask to obtain a third disparity map includes:

[0027] identifying an edge region in the second disparity map according to the edge mask;

[0028] Filtering is performed on the pixels in the edge area to remove edge noise and generate the third disparity map.

[0029] In combination with the first aspect, in a sixth possible implementation of the first aspect, converting the horizontal displacement into depth information of the pixel point in three-dimensional space by using camera calibration parameters includes:

[0030] The horizontal displacement of the pixel is converted into depth information of the pixel in three-dimensional space through camera calibration parameters such as camera focal length, baseline distance and principal point coordinates.

[0031] A second aspect of the embodiments of the present application provides a visual image processing device, the device comprising:

[0032] A first disparity map generating unit is configured to acquire a binocular image and generate a first disparity map using a pre-trained deep learning-based stereo matching network;

[0033] an extreme value filtering unit, configured to perform extreme value filtering on the first disparity map to obtain a second disparity map;

[0034] an edge detection unit, configured to perform edge detection on the second disparity map and generate an edge mask;

[0035] an edge filtering unit, configured to perform edge filtering on the second disparity map according to the edge mask to obtain a third disparity map;

[0036] The point cloud generation unit is used to calculate the horizontal displacement of each pixel point in the third disparity map based on the disparity value of the pixel point, convert the horizontal displacement into the depth information of the pixel point in three-dimensional space through camera calibration parameters, and generate a three-dimensional point cloud based on the position of the pixel point in the third disparity map and the depth information.

[0037] A third aspect of an embodiment of the present application provides a visual image processing device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the visual image processing device implements a method as described in any one of the first aspects.

[0038] A fourth aspect of the embodiments of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect or its various implementations.

[0039] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0040] A sixth aspect of the present application provides a chip for implementing the methods described in each implementation of the first aspect. Specifically, the chip includes a processor configured to retrieve and execute a computer program from a memory, causing a device equipped with the chip to execute the methods described in the first aspect or its implementations.

[0041] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain a first disparity map by processing binocular images through a pre-trained deep learning-based stereo matching network, and optimize the first disparity map by using extreme value filtering to obtain a second disparity map, thereby effectively removing obvious outliers in the first disparity map, and then extracting edge features of the second disparity map, generating an edge mask, performing edge filtering based on the edge mask to obtain a third disparity map, determining the horizontal displacement of the pixel points based on the third disparity map, converting the horizontal displacement into depth information in three-dimensional space according to the camera calibration parameters to determine a three-dimensional point cloud, extracting the real boundary through edge filtering, and using the edge mask to forcibly correct the disparity of the boundary area, suppressing the transition zone predicted by the network, and compensating for the boundary blur problem of pure deep learning. It can effectively reduce the "burrs" or outliers of the converted point cloud, filter out scattered point cloud noise, make the object contour boundary clearer, and help improve the perception accuracy of the visual system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 This is a schematic diagram of an implementation scenario of a visual image processing method provided in an embodiment of the present application;

[0044] Figure 2 This is a schematic diagram of an implementation flow of a visual image processing method provided in an embodiment of the present application;

[0045] Figure 3 1 is a schematic diagram of an implementation flow of a method for performing extreme value filtering on a first disparity map provided in an embodiment of the present application;

[0046] Figure 4 This is a schematic diagram of an implementation flow of a method for edge detection and edge mask generation provided in an embodiment of the present application;

[0047] Figure 5 is a schematic diagram of a visual image processing device provided in an embodiment of the present application;

[0048] Figure 6It is a schematic diagram of a visual image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0050] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0051] Stereo matching is a hot topic in the field of computer vision. It aims to estimate the disparity of matching pixels in binocular images and restore dense correspondences between image pairs. Disparity is the displacement of pixels in an image pair. Based on this, the depth information of each pixel can be inferred through triangulation to construct an object model in three-dimensional space. Because disparity is closely related to depth, stereo matching not only lays the foundation for depth map generation, but also provides key data for many high-level computer vision tasks, and its application scenarios are extremely wide. In the field of three-dimensional reconstruction, it helps the system generate accurate terrain models; in robot navigation, robots rely on depth information to perceive the environment and avoid obstacles; in the field of augmented reality, it provides real-world depth information for virtual overlay and interaction of objects.

[0052] Traditional stereo matching algorithms often rely on manually designed matching cost functions and engineered regularization strategies, addressing the disparity estimation problem by defining pixel-to-pixel matching metrics (such as color, texture, or structural similarity) and optimization strategies (such as global optimization or local smoothness constraints). Although these methods perform well in specific scenarios, they rely on complex feature design and manual parameter tuning, and are less effective when handling large-scale or complex scenes. With the rapid development of computer vision, especially the rise of deep learning technology, methods based on convolutional neural networks (CNNs) have gradually become mainstream. CNNs can automatically learn features and perform end-to-end optimization, offering significant advantages when processing large-scale stereo datasets. Through training, CNNs can learn rich image features from large amounts of annotated data and effectively perform depth estimation. CNN-based stereo matching methods have significantly improved matching accuracy and processing speed in practical applications.

[0053] However, CNNs are essentially smooth function approximators. When predicting pixel-level depth, they blur depth values ​​in boundary areas, resulting in transitions at the disparity map boundaries. When converting depth maps to point clouds, these transitions create "scattered pixels" at object boundaries. These "scattered pixels" stem from inaccurate depth estimation, causing point cloud data to appear anomalies at object edges. This can lead to accumulated errors in subsequent processing steps, compromising overall visual system performance.

[0054] To solve the above problems, the present invention proposes a method for visual image processing. Figure 1 This is a schematic diagram of an implementation scenario for this method, which includes a binocular camera and a visual image processing device. The binocular camera can be used to capture binocular images. Furthermore, to ensure that the binocular images captured by the binocular camera can be effectively converted into point cloud information, it is necessary to obtain the camera's focal length, baseline distance, and camera principal point coordinates.

[0055] The visual image processing device can be a dedicated hardware module or a software module integrated in the computing system, which is used to perform noise reduction on the generated point cloud data.

[0056] A binocular camera can capture binocular images of the target scene in real time. These images are input into a visual image processing device, which generates a first disparity map through a pre-trained deep learning-based stereo matching network. Subsequently, a series of optimization processing steps, including extreme value filtering, edge detection, and edge filtering, gradually optimize the quality of the disparity map. Finally, point cloud data is generated based on the optimized disparity map and camera parameters. During this process, the visual image processing device performs noise reduction on the generated point cloud data to improve its quality, particularly accuracy in boundary areas, thereby providing more reliable data support for subsequent 3D reconstruction, scene analysis, or other applications.

[0057] Figure 2 A schematic diagram of an implementation flow of a visual image processing method provided in an embodiment of the present application is detailed as follows:

[0058] In S201 , a binocular image is acquired, and a first disparity map is generated through a pre-trained deep learning-based stereo matching network.

[0059] When acquiring binocular images, the left and right cameras in a binocular camera must synchronously capture images of the same scene, ensuring that timestamps are aligned to avoid motion blur. For example, using parallel binocular cameras with a baseline distance of 12 cm to capture an indoor scene at a resolution of 1280 × 720, the left and right images must be calibrated to eliminate lens distortion and align epipolar lines. The binocular images provide the input for stereo matching, and calibrated binocular images can reduce parallax calculation errors.

[0060] Deep learning-based stereo matching networks can use models such as PSMNet (Pyramid Stereo Matching Network) or GCNet (Geometric Constraint Network). They use convolutional neural networks to extract features from the left and right images, calculate matching costs, and ultimately regress disparity values. For example, PSMNet aggregates multi-scale features through a pyramid pooling module. After pre-training on a dataset, it generates a first disparity map for the input binocular image within a predetermined disparity range. Low-texture areas (such as white walls) may appear noisy due to missing features.

[0061] In S202 , extreme value filtering is performed on the first disparity map to obtain a second disparity map.

[0062] Extreme value filtering targets outliers in the first disparity map and removes them by statistically analyzing the distribution of disparity values ​​within a local window. This operation can eliminate flying points caused by occlusion or mismatching (such as an abnormal 200-pixel disparity value that appears in the background).

[0063] In the embodiment of the present application, when performing extreme value filtering on the first disparity map to obtain the second disparity map, the following can be done: Figure 3 As shown, including:

[0064] In S301 , all disparity values ​​in the first disparity map are counted and sorted.

[0065] The disparity map is generated by a binocular neural network, where each pixel corresponds to a disparity value, representing the depth information of that point in three-dimensional space. To perform extreme value filtering, all disparity values ​​must first be counted and sorted. The purpose of sorting is to quickly locate extreme points to be filtered out based on percentage ranges. For example, if the disparity map resolution is 640x480, there are 307,200 disparity values, which are arranged from smallest to largest to form an ordered list.

[0066] In S302 , the range of disparity values ​​to be filtered out is determined according to the preset maximum value percentage and minimum value percentage.

[0067] Extreme filtering is used to remove outliers in the first disparity map that may be caused by noise or misestimation. These outliers are usually distributed in the minimum or maximum range of the disparity values. By setting the minimum percentage Pmin and the maximum percentage Pmax, the disparity range that needs to be filtered can be determined, and the range less than the minimum percentage is determined as the minimum range, and the range greater than the maximum percentage is determined as the maximum range. For example, if Pmin is set to 2% and Pmax is set to 99%, the disparity values ​​in the top 2% and bottom 1% after sorting will be considered outliers. Specifically, for 307,200 disparity values, the first 6,144 and last 3,072 disparity values ​​will be marked as ranges that need to be filtered.

[0068] In S303 , corresponding pixel points within the disparity value range are filtered out to generate the second disparity map.

[0069] After determining the disparity ranges that need to be filtered, pixels within these ranges can be removed from the disparity map. This can be done by setting the disparity values ​​of these points to invalid values ​​or simply removing them from the disparity map. For example, for the first 2% and last 1% of disparity values, the corresponding pixels will be marked as invalid, thus generating a second disparity map that has been filtered using extreme values. This step can effectively reduce noise in the disparity map and improve the accuracy of subsequent processing.

[0070] Extreme value filtering removes extreme values ​​from the disparity map, which are often caused by weak texture, low lighting, or errors in binocular matching. By filtering out these outliers, the quality of the disparity map can be significantly improved, making subsequent edge detection and point cloud generation more accurate. For example, in low-light conditions, a binocular neural network may misestimate the disparity values ​​of certain pixels, causing them to be significantly greater or less than the true value. Extreme value filtering can effectively identify and remove these misestimates, thereby improving the robustness and accuracy of the overall system. Practical Applications of Extreme Value Filtering. Extreme value filtering has a wide range of applications in binocular vision systems, especially in complex scenarios. For example, in autonomous driving, vehicles need to obtain real-time depth information of their surroundings. However, areas with low lighting or texture can increase noise in the disparity map. Extreme value filtering can remove this noise, making the generated depth information more reliable and thus improving the safety of autonomous driving systems. Furthermore, extreme value filtering can significantly improve system performance in areas such as robot navigation and 3D reconstruction.

[0071] In S203 , edge detection is performed on the second disparity map to generate an edge mask.

[0072] Edge detection can identify object boundaries by analyzing changes in pixel values ​​in a disparity map. In a disparity map, object boundaries typically appear as sudden changes in pixel values, which can be detected using gradient calculations or specialized filters (such as Sobel and Canny). For example, when using the Canny edge detection algorithm, the second disparity map is first Gaussian filtered to smooth out noise. The gradient magnitude and direction are then calculated, and then non-maximum suppression and double thresholding are performed to generate an edge mask.

[0073] The edge mask is a binary image in which edge pixels are marked as 1 and non-edge pixels are marked as 0. This step can be used to accurately extract object boundary information in the disparity map, providing a basis for subsequent edge filtering.

[0074] Among them, the process of edge detection to generate edge mask can be as follows Figure 4 As shown, including:

[0075] In S401 , a Gaussian filter is used to smooth the second disparity map to obtain a smoothed image.

[0076] Gaussian filter can blur the image by weighted averaging, thereby reducing the impact of noise. The definition of Gaussian filter can be expressed as:

[0077]

[0078] Where (x, y) is the pixel coordinate, σ is the standard deviation of the Gaussian function, and G(x, y) represents the Gaussian filter. Applying the Gaussian filter to an image and performing smoothing on the image can be expressed as:

[0079] I s (x,y)=I(x,y)*G(x,y)

[0080] Where I(x,y) is the input image, I s (x,y) is the output image, and * represents the convolution operation, which is to calculate the convolution of each pixel value with the Gaussian filter.

[0081] In disparity maps, noise often manifests as irregular fluctuations in pixel values. Gaussian filters can effectively suppress these fluctuations, making the image smoother. For example, using a 3x3 Gaussian kernel with a weight distribution where the center pixel has the highest weight and the surrounding pixels have gradually decreasing weights ensures that the image is smoothed while retaining key edge information.

[0082] In S402, the gradient intensity and gradient direction of the smoothed image are obtained.

[0083] The gradient magnitude reflects the severity of the change in pixel value in an image, while the gradient direction indicates the direction of this change. To obtain the gradient magnitude and gradient direction of a smoothed image, the horizontal and vertical gradients of the smoothed image can be calculated separately. Based on the horizontal and vertical gradients, the gradient magnitude of each pixel can be calculated. The gradient direction of the pixel can also be calculated using the four-quadrant inverse method.

[0084] In a disparity map, edges typically appear as sudden changes in pixel values, so these edges can be effectively detected by calculating the gradient strength and direction. For example, the Sobel operator can be used to calculate the gradient. The Sobel operator calculates the horizontal and vertical gradients separately through a convolution operation, then calculates the gradient strength using the Pythagorean theorem and the gradient direction using the inverse tangent function.

[0085] The gradient calculation formula can be expressed as:

[0086]

[0087] Among them G x (x,y) and G y (x,y) are the gradients in the x and y directions respectively, G x and G y Convolution kernels in two directions.

[0088] After obtaining the gradients in two directions, the gradient strength and gradient direction of the disparity map can be calculated:

[0089]

[0090] Where M(x,y) is the gradient strength, θ(x,y) is the gradient direction, and atan2 is the four-quadrant inverse tangent function.

[0091] In S403 , a non-maximum suppression algorithm is used to retain the local maximum value according to the gradient intensity and gradient direction, thereby generating a preliminary edge.

[0092] The non-maximum suppression algorithm compares the gradient strength of each pixel with the gradient strength of its neighboring pixels, retaining the local maximum and thus obtaining finer edges. For example, for each pixel, if its gradient strength is greater than the gradient strength of its neighboring pixels, the pixel is retained as an edge point; otherwise, it is suppressed. This effectively removes redundant information in edge detection and makes the edges clearer.

[0093] Generating an edge mask according to the preliminary edge may be as shown in S404-S406:

[0094] In S404 , a double thresholding process is performed on the preliminary edge to determine a first edge pixel and a second edge pixel, wherein the gradient intensity of the first edge pixel is greater than the gradient intensity of the second edge pixel.

[0095] Dual thresholding can be used to classify edge pixels by setting two thresholds (e.g., first threshold t1 = 0.1, second threshold t2 = 0.3). Pixels with gradient strengths above the second threshold t2 (e.g., 0.35) are marked as first edge pixels (strong edge pixels), pixels between the first and second thresholds t1 and t2 (e.g., 0.2) are marked as second edge pixels (weak edge pixels), and pixels below the first threshold t1 (e.g., 0.05) are directly rejected. This classification method effectively distinguishes significant edges from noise and reduces the probability of gradient regions being mistakenly identified as edges.

[0096] In S405 , the second edge pixels are traversed to determine whether there is a first edge pixel in the neighborhood thereof.

[0097] When traversing the second edge pixel, you need to define a neighborhood range (such as a 3×3 or 5×5 pixel area). For example, if the coordinates of a weak edge pixel are (5,5), check whether there are strong edge pixels with gradient values ​​exceeding t2 in the surrounding 8 neighborhoods.

[0098] In S406 , if the second edge pixel exists, the second edge pixel is retained; otherwise, the second edge pixel is removed to generate the edge mask.

[0099] If the gradient strength of the pixel (5,6) in the neighborhood is 0.4 (strong edge pixel), the weak edge pixel (5,5) is retained; if the gradients of all pixels in the neighborhood are less than t2 (e.g., the highest is 0.25), the weak edge pixel is discarded. The logic behind neighborhood judgment can be based on the principle of edge continuity. The edges of real objects are typically locally coherent, and a weak edge connected to a strong edge may be part of the true edge. For example, when detecting the edge of a glass, some areas may have a gradient strength of only 0.25 (weak edge) due to reflection, but the gradient of the adjacent area is 0.4 (strong edge). Connecting these areas can complete the broken edge contour.

[0100] Completing the edge mask effectively reduces noise interference and preserves true edges. For example, in a binocular disparity map, scattered noise between leaves can produce isolated weak edges (gradient 0.15-0.2). These points can be removed through neighborhood checking. However, weak edges on the boundaries of real objects are retained because they are adjacent to strong edges. The final mask can more accurately reflect the scene structure.

[0101] In practice, threshold selection must be tailored to the image characteristics. For high-contrast images (such as industrial parts inspection, where the contrast is greater than the first contrast threshold), the second threshold t2 can be set to the first value, such as 0.5, to filter out subtle textures. For low-light images (such as nighttime road monitoring, where the contrast is less than the second contrast threshold, which is less than the first contrast threshold), the first threshold t1 may be lowered to the second value, such as 0.05, to capture subtle edges. The neighborhood size can also be dynamically adjusted based on the scenario. For example, a larger neighborhood (7×7) results in wider edges, while a smaller neighborhood results in finer edges.

[0102] An edge mask is a binary image in which edge pixels are marked as 1 and non-edge pixels are marked as 0. Generating an edge mask facilitates specific operations on edge regions during subsequent processing. For example, in a disparity map, the edge mask can be used to guide edge noise filtering, limiting filtering to edge regions while preserving disparity information in non-edge regions. This improves the perception accuracy of the binocular system and reduces scattered point cloud noise.

[0103] In S204 , edge filtering is performed on the second disparity map according to the edge mask to obtain a third disparity map.

[0104] The purpose of edge filtering is to remove noise near edges while preserving object boundaries. Specifically, edge regions in the second disparity image can be identified based on the edge mask, and pixels in the edge regions can be filtered to remove edge noise and generate a third disparity image.

[0105] Each pixel in the second disparity map can be traversed to check its corresponding value in the edge mask. If the pixel is located at the edge (i.e., the median value of the edge mask is 1), its original disparity value is retained; if the pixel is located in the non-edge area (i.e., the median value of the edge mask is 0), its disparity value is smoothed, such as using a mean filter or a median filter. For example, for a non-edge pixel, the average of the disparity values ​​of all pixels in its 3x3 neighborhood can be taken as the new disparity value. This filtering method can effectively reduce the noise in the non-edge area while ensuring the clarity and accuracy of the object boundary. Through the above steps, the generated third disparity map significantly reduces the edge noise while retaining the object boundary. This edge denoising method is of great significance in practical applications. For example, in three-dimensional reconstruction and autonomous driving scenarios, a clear disparity map can improve the accuracy of depth estimation, thereby improving the overall performance of the system. In addition, the introduction of edge filtering can also reduce the computational complexity in subsequent processing steps and improve the efficiency of the algorithm.

[0106] In S205, a horizontal displacement of each pixel in the third disparity map is obtained based on the disparity value of the pixel, the horizontal displacement is converted into depth information of the pixel in three-dimensional space through camera calibration parameters, and a three-dimensional point cloud is generated based on the position of the pixel in the third disparity map and the depth information.

[0107] The key to converting a disparity map to a point cloud is to use internal parameters such as the camera focal length f, the baseline distance B, and the principal point coordinates (u0, v0) to calculate the three-dimensional coordinates through triangulation. For example, when the disparity value of a pixel (u, v) is d, the point cloud coordinates can be expressed as:

[0108]

[0109] Where is the disparity value of the pixel, f is the focal length of the camera, B is the binocular baseline distance, is the coordinate of the camera principal point, and is the point cloud coordinate.

[0110] In the process of generating point cloud data according to the third disparity map and camera parameters, the camera focal length, baseline distance and principal point coordinates can be first obtained as the basis for generating point cloud data.

[0111] The camera focal length determines the image scale, the baseline distance is the horizontal distance between the two lenses in a binocular camera, and the principal point coordinates are the center of the image plane. These parameters together define the camera's geometric characteristics and are key to the subsequent calculation of 3D coordinates. For example, assuming a camera focal length of 1000 pixels, a baseline distance of 200 mm, and a principal point coordinate of (640, 360) at the image center, these parameters are used to convert the pixel information in the disparity map into real-world 3D coordinates.

[0112] For the disparity value of each pixel in the third disparity map, the horizontal displacement of the pixel in the third disparity map image is calculated. The horizontal displacement of the pixel is converted into the depth information of the pixel in the three-dimensional space through the camera calibration parameters of the camera focal length, baseline distance and principal point coordinates. Then, combined with the position of the pixel in the third disparity map image, the three-dimensional coordinates of the pixel in the three-dimensional space are calculated. For example, assuming that the disparity value of a pixel is 50 pixels, the depth value corresponding to the pixel can be calculated according to the formula. Combined with the camera focal length and baseline distance, the X, Y, and Z coordinates of the pixel in the three-dimensional space can be further inferred. This process maps the two-dimensional image information to the three-dimensional space, providing a basis for the generation of point cloud data.

[0113] 3D point cloud data consists of a large number of 3D coordinate points, each representing a spatial location in the scene. By converting each pixel in the disparity map into 3D coordinates, a complete 3D point cloud model can be constructed. For example, if the disparity map resolution is 1280x720, the resulting point cloud data will contain 1280x720 3D coordinate points. This point cloud data can be used for applications such as 3D reconstruction and object detection, providing high-precision spatial information for subsequent computer vision tasks.

[0114] Through the above steps, the process of converting the disparity map into point cloud data achieves the mapping from a two-dimensional image to a three-dimensional space. This process not only improves the perception accuracy of the binocular system but also provides a reliable data foundation for subsequent three-dimensional analysis and processing. For example, in autonomous driving scenarios, the generated point cloud data can be used to identify obstacles on the road and support vehicle path planning. In industrial inspection, point cloud data can be used to measure the size and shape of objects, improving inspection accuracy and efficiency.

[0115] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0116] Figure 5 A schematic diagram of a visual image processing device provided in an embodiment of the present application, the device comprising:

[0117] A first disparity map generating unit 501 is configured to acquire a binocular image and generate a first disparity map using a pre-trained deep learning-based stereo matching network;

[0118] an extreme value filtering unit 502, configured to perform extreme value filtering on the first disparity map to obtain a second disparity map;

[0119] an edge detection unit 503, configured to perform edge detection on the second disparity map to generate an edge mask;

[0120] an edge filtering unit 504, configured to perform edge filtering on the second disparity map according to the edge mask to obtain a third disparity map;

[0121] The point cloud generation unit 505 is used to calculate the horizontal displacement of each pixel point in the third disparity map based on the disparity value of the pixel point, convert the horizontal displacement into the depth information of the pixel point in three-dimensional space through camera calibration parameters, and generate a three-dimensional point cloud based on the position of the pixel point in the third disparity map and the depth information.

[0122] Figure 5 The visual image processing device shown, and Figure 2 The visual image processing method shown corresponds to.

[0123] Figure 6 Schematic diagram of a visual image processing device provided in an embodiment of the present application. Figure 6 As shown, the visual image processing device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60, such as a visual image processing program. When the processor 60 executes the computer program 62, the steps of the visual image processing method embodiments described above are implemented. Alternatively, when the processor 60 executes the computer program 62, the functions of the modules / units described in the apparatus embodiments described above are implemented.

[0124] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 62 in the visual image processing device 6.

[0125] The visual image processing device 6 can be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The visual image processing device can include, but is not limited to, a processor 60 and a memory 61. It will be understood by those skilled in the art that Figure 6It is only an example of a visual image processing device 6 and does not constitute a limitation of the visual image processing device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the visual image processing device may also include input and output devices, network access devices, buses, etc.

[0126] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0127] The memory 61 can be an internal storage unit of the visual image processing device 6, such as a hard disk or memory of the visual image processing device 6. The memory 61 can also be an external storage device of the visual image processing device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the visual image processing device 6. Furthermore, the memory 61 can also include both an internal storage unit and an external storage device of the visual image processing device 6. The memory 61 is used to store the computer program and other programs and data required by the visual image processing device. The memory 61 can also be used to temporarily store data that has been output or is to be output.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0129] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0130] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0133] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0134] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0135] In addition, an embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the methods in the above-mentioned implementation manners.

[0136] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A visual image processing method, characterized in that: The method comprises: Obtain binocular images and generate a first disparity map using a pre-trained deep learning-based stereo matching network; performing extreme value filtering on the first disparity map to obtain a second disparity map; performing edge detection on the second disparity map to generate an edge mask; performing edge filtering on the second disparity map according to the edge mask to obtain a third disparity map; The horizontal displacement of each pixel in the third disparity map is calculated based on the disparity value of the pixel, and the horizontal displacement is converted into depth information of the pixel in three-dimensional space through camera calibration parameters. A three-dimensional point cloud is generated based on the position of the pixel in the third disparity map and the depth information.

2. The method according to claim 1, characterized in that The performing extreme value filtering on the first disparity map to obtain a second disparity map includes: Counting and sorting all disparity values ​​in the first disparity map; Determine the range of parallax values ​​that need to be filtered out based on the preset maximum value percentage and minimum value percentage; Filter out corresponding pixel points within the disparity value range to generate the second disparity map.

3. The method according to claim 1, characterized in that The performing edge detection on the second disparity map to generate an edge mask includes: Smoothing the second disparity map using a Gaussian filter to obtain a smoothed image; Obtaining the gradient intensity and gradient direction of the smoothed image; According to the gradient intensity and gradient direction, a non-maximum suppression algorithm is used to retain the local maximum value to generate a preliminary edge, and an edge mask is generated according to the preliminary edge.

4. The method according to claim 3, characterized in that Generating an edge mask according to the preliminary edge comprises: Performing a double thresholding process on the preliminary edge to determine a first edge pixel and a second edge pixel, wherein the gradient intensity of the first edge pixel is greater than the gradient intensity of the second edge pixel; Traversing the second edge pixels to determine whether there is a first edge pixel in its neighborhood; If so, the second edge pixel is retained; otherwise, the second edge pixel is removed to generate the edge mask.

5. The method according to claim 3, characterized in that The obtaining of the gradient intensity and gradient direction of the smoothed image comprises: Calculating the gradients of the smoothed image in the horizontal and vertical directions respectively; Calculating the gradient strength of each pixel based on the gradients in the horizontal and vertical directions; According to the gradients in the horizontal and vertical directions, a four-quadrant inverse tangent function is used to calculate the gradient direction of each pixel.

6. The method according to claim 1, characterized in that The performing edge filtering on the second disparity map according to the edge mask to obtain a third disparity map includes: identifying an edge region in the second disparity map according to the edge mask; Filtering is performed on the pixels in the edge area to remove edge noise and generate the third disparity map.

7. The method according to claim 1, characterized in that The horizontal displacement is converted into depth information of the pixel point in three-dimensional space through camera calibration parameters, including: The horizontal displacement of the pixel is converted into depth information of the pixel in three-dimensional space through camera calibration parameters such as camera focal length, baseline distance and principal point coordinates.

8. A visual image processing device, characterized in that: The device comprises: A first disparity map generating unit is configured to acquire a binocular image and generate a first disparity map using a pre-trained deep learning-based stereo matching network; an extreme value filtering unit, configured to perform extreme value filtering on the first disparity map to obtain a second disparity map; an edge detection unit, configured to perform edge detection on the second disparity map and generate an edge mask; an edge filtering unit, configured to perform edge filtering on the second disparity map according to the edge mask to obtain a third disparity map; The point cloud generation unit is used to calculate the horizontal displacement of each pixel point in the third disparity map based on the disparity value of the pixel point, convert the horizontal displacement into the depth information of the pixel point in three-dimensional space through camera calibration parameters, and generate a three-dimensional point cloud based on the position of the pixel point in the third disparity map and the depth information.

9. A visual image processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the visual image processing device implements the method according to any one of claims 1 to 7.

10. A computer program product comprising computer program instructions, characterized in that When the computer program is executed, the method according to any one of claims 1 to 7 is performed.

Images