Image processing method and device, equipment and storage medium

Abstract

The invention is suitable for the technical field of images, and provides an image processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining images collected according to a preset interval; extracting a sub-aperture image based on the image; calculating to obtain a cost body according to the sub-aperture image; performing focusing feature extraction on the sub-aperture image to obtain a focusing feature map; calculating to obtain a decision diagram based on the focusing feature map; and according to the decision diagram, fusing the cost bodies, and obtaining three-dimensional point cloud data of an object corresponding to the image based on a fusion result. According to the method, the measurement efficiency can be improved, an optimization strategy of cost body fusion guided by a decision diagram is adopted, the measurement accuracy can be kept while the sampling number is low, the depth continuity can be kept, the advantages of adjacent cost bodies in respective depth-of-field ranges are effectively utilized, complementary information of multi-focal-plane light field data is effectively utilized, and the measurement accuracy is improved. And a measurement blind area and depth information distortion are avoided.

Description

Technical Field

[0001] This invention belongs to the field of image technology, and particularly relates to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous development of automation technology in manufacturing, especially in precision manufacturing fields such as electronic components and semiconductor packaging, the demand for high-speed, high-precision Automatic Optical Inspection (AOI) systems is increasing, particularly in the three-dimensional dimensional inspection of complex structural parts. Existing three-dimensional measurement technologies can be mainly divided into the following three categories: multi-view stereo vision (MVS), depth cameras based on active light sources (such as structured light and time-of-flight, ToF technology), and passive depth estimation methods based on monocular imaging.

[0003] The first type of method—multi-view stereo vision (MVS)—acquires images from multiple perspectives of the same scene and reconstructs 3D shapes using feature matching and disparity calculation between images. In recent years, with the development of deep learning, end-to-end stereo matching methods based on deep neural networks, such as MVSNet, have emerged, which can improve reconstruction accuracy to some extent. However, MVS systems typically require simultaneous acquisition by multiple cameras, resulting in complex structures, difficult calibration, and high system integration costs. The second type of method—active depth cameras (including structured light and Time-of-Flight)—obtains depth information by projecting specific coded light sources onto the scene and analyzing the phase or time difference of the reflected signals. This type of method has the advantages of fast measurement speed and real-time performance, but in actual industrial inspection, it is prone to light scattering, overexposure, or signal loss when facing highly reflective metal surfaces or transparent materials, leading to measurement blind spots and distortion of depth information, limiting its application in complex surface measurement. The third type of method—passive depth estimation based on monocular imaging—relies on the geometric and optical features within the image (such as shadows, textures, and focus blur) to infer depth information. Among them, the Shape from Focus (SFF) method, which reconstructs 3D structures by analyzing the sharpness changes of images at different focal planes, has been widely used in small-scale scenarios such as microscopic measurement. However, this method typically requires acquiring hundreds of images at different focal planes, resulting in large data volumes, slow acquisition speeds, and low reconstruction efficiency, making it difficult to meet the real-time inspection needs of industrial production lines. In summary, existing 3D measurement technologies all have certain limitations when dealing with complex industrial samples such as highly reflective, transparent materials, and high aspect ratio microstructures. For example, multi-viewpoint methods are complex, costly, and have poor real-time performance; active light source methods suffer from measurement blind spots and ambient light interference; and passive monocular methods have low data acquisition efficiency and limited depth of field.

[0004] Therefore, there is an urgent need for a new three-dimensional measurement method to meet the application needs of the industrial AOI field. Summary of the Invention

[0005] The present invention provides an image processing method, apparatus, device, and storage medium, which aims to solve at least one technical problem in the background art.

[0006] The present invention is implemented as follows: an image processing method, the method comprising: Acquire images at preset intervals; Based on the image, a sub-aperture image is extracted; Based on the sub-aperture image, the three-dimensional cost volume is calculated; Focusing feature extraction is performed on the sub-aperture image to obtain a focusing feature map; Based on the focused feature map, a decision map is calculated; Based on the decision map, the three-dimensional cost volume is fused, and the three-dimensional point cloud data of the object corresponding to the image is obtained based on the fusion result.

[0007] Furthermore, the step of extracting the sub-aperture image based on the image includes: Based on the arrangement pattern of the microlens array, the macro-pixel region corresponding to each microlens in the image is determined; Within each macro-pixel region, determine the coordinates of target pixels with the same relative position; Traverse the image and extract the pixel value located at the target pixel coordinates in each macro pixel region; The extracted pixel values ​​are recombined according to the spatial arrangement order of the microlens array to obtain a sub-aperture image corresponding to the target pixel coordinates.

[0008] Further, the step of calculating the three-dimensional cost volume based on the sub-aperture image includes: Determine the disparity search range, and select multiple candidate disparity values ​​within the disparity search range; For each candidate disparity value, the sub-aperture image is translated according to its positional relationship with the central main view in the sub-aperture image. Calculate the matching cost between the translated sub-aperture image and the central main view; The matching costs corresponding to all the candidate disparity values ​​are stacked according to the disparity direction to construct a three-dimensional cost volume.

[0009] Further, the step of extracting focusing features from the sub-aperture image to obtain a focusing feature map includes: A mean filter is used to smooth the central main view in the sub-aperture image to obtain a smoothed image; Calculate the absolute value of the grayscale difference between the central main view and the smoothed image to obtain a coarse focus feature map; Using the central main view as a guide image, the coarse focus feature map is subjected to guided filtering to obtain a focus feature map.

[0010] Furthermore, the step of calculating the decision map based on the focused feature map includes: Multiple focused feature maps corresponding to different acquisition positions are obtained, and the multiple focused feature maps are stacked in the depth direction; Iterate through each pixel coordinate and compare the pixel values ​​of the multiple focused feature maps at the same pixel coordinate; Select the layer number corresponding to the focused feature map with the largest pixel value as the initial decision value of the pixel coordinate, and generate an initial decision map; Based on the initial decision map, the central main view in the sub-aperture image is initially fused to obtain an initial fused image; Using the initial fused image as a guide image, the initial decision image is subjected to guided filtering to obtain the final decision image.

[0011] Further, the step of fusing the three-dimensional cost volume according to the decision map and obtaining the three-dimensional point cloud data of the object corresponding to the image based on the fusion result includes: The displacement of each acquisition position is obtained from the image, and the total disparity range of all three-dimensional cost volumes under a unified reference and the disparity overlap area between adjacent cost volumes are determined based on the displacement. Within the disparity overlap region, the pixel values ​​of the decision map are used as weights to perform linear interpolation weighted fusion on the cost values ​​of adjacent cost bodies; By combining the cost value of non-overlapping areas with the cost value of the merged overlapping areas, a panoramic deep fusion cost body is constructed. Based on the panoramic depth fusion cost volume, the three-dimensional point cloud data of the object corresponding to the image is obtained.

[0012] Further, the step of obtaining the 3D point cloud data of the object corresponding to the image based on the panoramic depth fusion cost volume includes: The depth label corresponding to the minimum cost is extracted from the fusion result to obtain the disparity map; Scale mapping is performed on the disparity map to obtain the true depth value; The true depth value is used to convert the depth map into three-dimensional point cloud data using a coordinate projection formula.

[0013] This invention also provides an image processing apparatus, the apparatus comprising: The acquisition module is used to acquire images collected at preset intervals; The first extraction module is used to extract a sub-aperture image based on the image; The first calculation module is used to calculate the three-dimensional cost volume based on the sub-aperture image; The second extraction module is used to extract the focus features from the sub-aperture image to obtain a focus feature map; The second calculation module is used to calculate a decision map based on the refocused image; The fusion module is used to fuse the three-dimensional cost volume according to the decision map, and obtain the three-dimensional point cloud data of the object corresponding to the image based on the fusion result.

[0014] This invention also provides an image processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the image processing method described above.

[0015] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method described above.

[0016] The beneficial effects achieved by this invention are as follows: acquiring images collected at preset intervals; extracting sub-aperture images based on the images; calculating a three-dimensional cost volume based on the sub-aperture images; extracting focusing features from the sub-aperture images to obtain a focusing feature map; calculating a decision map based on the focusing feature map; fusing the three-dimensional cost volumes based on the decision map; and obtaining three-dimensional point cloud data of the object corresponding to the image based on the fusion result. The cost volume fusion strategy guided by the decision map can maintain measurement accuracy with a low sampling rate, while effectively utilizing the advantages of adjacent cost volumes within their respective depth ranges while maintaining depth continuity. It also effectively utilizes the complementary information of multi-focal surface light field data, avoiding measurement blind spots and depth information distortion. Attached Figure Description

[0017] Figure 1 This is a flowchart of the image processing method in Embodiment 1 of the present invention; Figure 2 yes Figure 1 A schematic diagram of a specific implementation method for step S20; Figure 3 yes Figure 1 A schematic diagram of a specific implementation method for step S30; Figure 4 yes Figure 1 A schematic diagram of a specific implementation of step S40; Figure 5 yes Figure 1 A schematic diagram of a specific implementation method for step S50; Figure 6 This is a schematic diagram of the decision diagram in Embodiment 1 of the present invention; Figure 7 This is a schematic diagram of the structure of an image processing device according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 9 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] The present invention will now be described in detail through specific embodiments.

[0020] Please see Figure 1 , Figure 1 A flowchart illustrating an image processing method provided in an embodiment of the present invention includes the following steps: Step S10: Acquire images collected at preset intervals.

[0021] In practice, a light field camera is used to acquire images of the object that needs to be measured in three dimensions. The object that needs to be measured in three dimensions can be semiconductor, microelectronic or other devices.

[0022] During image acquisition, a light field camera can be mounted on a displacement stage. The stage moves around the object to be measured in three dimensions, allowing for image acquisition from different positions. Specifically, the displacement stage moves around the object according to preset equal intervals. Precise positioning is achieved, and the stage can be stopped at each position to facilitate image acquisition by the light field camera. Therefore, the light field camera performs one image acquisition operation when the stage is stationary. By performing multiple image acquisition operations, multiple frames of original light field images with different focal positions are obtained. A preset equal-interval step size is used. The distance can be 1 meter, 2 meters, etc., and the specific setting can be determined based on the size of the object being photographed; no limitation is made here. Since the light field image simultaneously records the spatial position and angle information of the light rays, only about 5 frames of light field images are needed to cover the depth information of hundreds of layers in the original confocal system. Therefore, the number of image frames acquired can be 5-10, thereby reducing the number of images acquired and increasing the image acquisition speed.

[0023] A light field camera, by configuring a microlens array behind the main lens, can simultaneously record the spatial position and angle information of light rays in a single exposure, thus acquiring the complete light distribution of a scene without increasing the complexity of acquisition. This characteristic allows the raw light field image to combine the depth perception capability of multi-view stereo vision with the structural simplicity of a monocular imaging system, providing a new approach for high-precision 3D measurement. Specifically, the image acquired by the light field camera is a microlens array image, which is four-dimensional light field data containing light direction information, i.e., containing four-dimensional coordinates L(u, v, s, t). Here, (s, t) represents the global coordinates of a pixel on the image sensor inside the light field camera, and (u, v) represents the position coordinates on the main lens (or aperture) of the light field camera.

[0024] Image acquisition can be achieved using a single light field camera, eliminating the need for simultaneous acquisition by multiple cameras and reducing the difficulty of image acquisition.

[0025] Step S20: Based on the image, extract the sub-aperture image.

[0026] In the specific implementation process, after the image is acquired, a microlens array calibration can be performed on the acquired image. Specifically, a white reference image under uniform illumination conditions is acquired using a light field camera. The white reference image can eliminate interference from scene content. The arrangement rules of the microlens array of the light field camera are obtained from the white reference image, and the coordinates of the center of each microlens in the pixel coordinate system are calculated according to the arrangement rules to determine the position of the macropixel. Based on the position of the macropixel, the corresponding N×N pixel region under each microlens can be found, and the sub-aperture image can be extracted.

[0027] In some embodiments of the invention, such as Figure 2 As shown, a scheme for extracting sub-aperture images is provided. In S20, that is, the process of extracting sub-aperture images based on the image, the specific steps include the following steps S21-S24: S21, Based on the arrangement pattern of the microlens array, determine the macro-pixel region corresponding to each microlens in the image; S22, within each macro-pixel region, determine the coordinates of target pixels with the same relative position; S23, traverse the image and extract the pixel value located at the target pixel coordinates in each macro pixel region; S24, according to the spatial arrangement order of the microlens array, the extracted pixel values ​​are recombined to obtain a sub-aperture image corresponding to the target pixel coordinates.

[0028] Specifically, based on the arrangement of the microlens array, the macro-pixel region corresponding to each microlens in the image is determined; within each macro-pixel region, the coordinates of target pixels with the same relative position are determined, for example, the coordinates (i, j) of target pixels with relative positions are determined, where i, j ∈ [0, N-1]. This serves as the target viewpoint. For the (m, n)th macro-pixel region in the acquired image, a traversal is performed (m, n ∈ [0, M-1]): the pixel with coordinates (i, j) is extracted from the N x N local pixels of this macro-pixel region. The extracted pixel value is placed at the (m, n)th pixel position of the sub-aperture image I_{i,j} to be generated. After traversing all M x M macro-pixel regions, the extracted pixel values ​​are recombined to obtain a complete image taken from a fixed viewpoint (i, j) (i.e., (u, v)), which is the sub-aperture image. The sub-aperture image can be further defined as four-dimensional light field data L(u, v, s, t).

[0029] Step S30: Calculate the three-dimensional cost volume based on the sub-aperture image.

[0030] After obtaining the sub-aperture images, a 3D cost volume is calculated based on them. Different focusing effects can be presented by translating and superimposing the sub-aperture images according to certain rules. Thinking from a reverse reasoning perspective, objects located on the focal plane exhibit maximum sharpness and contrast, while areas deviating from the focal plane experience defocusing and blurring due to parallax inconsistencies. Specifically, the parallax estimation process can be achieved by constructing a cost volume. Using four-dimensional light field data acquired by a light field camera in the same scene, within a preset parallax search space, the matching cost between multiple views and the central main view is calculated for each candidate parallax value. The cost maps under all candidate parallaxes are then stacked into a 3D structure along the depth direction, thus constructing a complete 3D cost volume.

[0031] In some embodiments of the invention, such as Figure 3 As shown, a scheme for obtaining a three-dimensional cost volume is provided. In S30, that is, the process of calculating the three-dimensional cost volume based on the sub-aperture image, the specific steps include the following S31-S34: S31, determine the disparity search range, and select multiple candidate disparity values ​​within the disparity search range; S32, for each candidate disparity value, perform a translation operation on the sub-aperture image according to the positional relationship of the sub-aperture image relative to the central main view in the sub-aperture image; S33, calculate the matching cost between the translated sub-aperture image and the central main view; S34, the matching costs corresponding to all the candidate disparity values ​​are stacked according to the disparity direction to construct a three-dimensional cost volume.

[0032] Specifically, the image from the center viewpoint of the light field, i.e., the central front view, is usually chosen as the reference image. The image from the center viewpoint of the light field is a sub-aperture image composed of the central local coordinates extracted from all macropixel regions. For example, after completing the microlens array calibration, the central local coordinates are calculated based on the size of each macropixel. Specifically, assuming the macropixels are 10×10 (N=10), the central local coordinates can be (…). , If the pixel is extracted from coordinate (5,5) and stitched together, the resulting sub-aperture image is the image from the center viewpoint of the light field.

[0033] After selecting the image from the center viewpoint of the light field, for each other sub-aperture image, the multi-view geometric properties of the light field are utilized to map it to the coordinate system of the reference image. Specifically, a three-dimensional cost volume is constructed through disparity estimation. First, the dimensional structure of the three-dimensional cost volume is determined, i.e., the disparity search range is determined. The cost volume is a three-dimensional array with dimensions H (image height), W (image width), and D (number of disparity levels). H and W correspond to the spatial resolution of the reference image, and D represents the discretized disparity search range; for example, a disparity range from 0 to 63 pixels corresponds to D=64 levels. Next, pixel-by-pixel disparity hypothesis calculation is performed. First, within the disparity search range, a series of possible disparity values ​​are sampled with a certain step size for subsequent matching cost calculation. For each pixel position (x, y) in the reference image, and each possible disparity hypothesis d, epipolar projection is first performed. A pixel point p_ref=(x, y) is taken in the reference image, assuming the disparity of this point is d, meaning that this point will have a horizontal offset in other viewpoint images. For the k-th viewpoint image, the formula can be used... The position of pixel p_ref is predicted. For each candidate disparity value, the sub-aperture image is translated based on its position relative to the central main view within the sub-aperture image, and the predicted position of pixel p_ref is the translated position. Here, Δu_k represents the baseline scaling factor of the viewpoint relative to the reference viewpoint. In a uniformly arranged light field, this factor is typically a simple viewpoint index difference. Then, the matching cost is calculated. Based on the predicted position of pixel p_ref, a pixel block centered at p_k is extracted from the current viewpoint image, and a pixel block of the same size centered at p_ref is extracted from the reference image. The similarity or difference between the two pixel blocks is then calculated, i.e., the matching cost between the translated sub-aperture image and the central main view is calculated, yielding the cost value. Common similarity calculation methods include sum of absolute differences and sum of squared differences. The matching costs of all N sub-aperture images and the reference image under the current disparity d are calculated, and these costs are aggregated into a comprehensive cost (e.g., taking the mean or variance), which serves as the final cost body for the current pixel under the current disparity assumption.

[0034] Finally, there is the traversal and filling process. For each pixel position in the reference image and for each possible value within the disparity search range, the calculated final cost is filled into the corresponding position in the 3D array, thus constructing the complete 3D cost volume data structure, that is, obtaining the 3D cost volume.

[0035] Step S40: Extract focus features from the sub-aperture image to obtain a focus feature map.

[0036] Specifically, firstly, the central front view of the sub-aperture image extracted in step S20 is obtained. A mean filter is used to smooth the central front view, removing detail textures to obtain a smoothed image. Next, the absolute value of the grayscale difference between the central front view and the smoothed image is calculated. This difference reflects the high-frequency edge information of the image, thus obtaining a coarse focus feature map. Finally, to optimize edges and reduce noise interference, the central front view is used as a guide image to perform guided filtering on the coarse focus feature map, thereby obtaining a focus feature map with clear edges and rich high-frequency information.

[0037] In some embodiments of the invention, such as Figure 4 As shown, a scheme for obtaining a focus feature map is provided. In S40, that is, the process of extracting focus features from the sub-aperture image to obtain a focus feature map, the specific steps include the following S41-S42: S41, the central main view in the sub-aperture image is smoothed using a mean filter, and the absolute value of the difference between the central main view and the smoothed image is calculated to obtain a coarse focus feature map; S42, using the central main view as a guide image, perform guided filtering enhancement on the coarse focus feature map to obtain a focus feature map.

[0038] Specifically, a guided filter algorithm is employed, which can preserve image edge gradient information while smoothing noise. First, the center front view in the sub-aperture image is... Defined as a guide image, the coarse focus feature map calculated in step S41 is used. Defined as the input image. The guided filter assumes that the guided image... Local window In the middle, output image (i.e., the focal feature map to be determined) and the guiding image A linear relationship exists:

[0039] in, and These are the linear coefficients within a local window. This is achieved by minimizing the output image. With input image The coefficients are calculated by solving for the differences between (i.e., the coarsely focused feature maps). and Finally, the coarse focus feature map is filtered using the calculated linear coefficients, resulting in an image where high-frequency edge information is enhanced and noise is suppressed; this is the focus feature map. This process effectively transfers the structural texture information from the central main view to the focused feature map, correcting discontinuous areas in the coarse feature map.

[0040] Step S50: Calculate the decision map based on the focused feature map.

[0041] Specifically, firstly, the central master views and their corresponding focused feature maps from different acquisition locations are acquired. All focused feature maps are then stacked along the depth direction to construct a 3D feature volume. Using a "winner-take-all" (WTA) strategy, focused feature values ​​at different levels are compared pixel-by-pixel, and the level index corresponding to the maximum response value is selected to generate an initial decision map. To eliminate edge artifacts caused by level jumps, the initial decision map is used to initially fuse the central master views, generating an initial fused image. Finally, using the initial fused image as a guide image, guided filtering optimization is performed on the initial decision map to obtain a final decision map with smooth and accurate edges.

[0042] In some embodiments of the invention, such as Figure 5 As shown, a scheme for obtaining a decision graph is provided, in which feature 55 is focused in S50.

[0043] S51, acquire multiple focused feature maps corresponding to different acquisition positions, and stack the multiple focused feature maps in the depth direction; S52, traverse each pixel coordinate and compare the pixel values ​​of the multiple focused feature maps at the same pixel coordinate; S53, Select the layer number corresponding to the focused feature map with the largest pixel value as the initial decision value of the pixel coordinate, and generate an initial decision map; S54, Based on the initial decision map, the central main view in the sub-aperture image is initially fused to obtain an initial fused image; S55, using the initial fused image as a guide image, perform guided filtering on the initial decision image to obtain a decision image.

[0044] The specific algorithm is implemented as follows: Focused feature maps from all locations are stacked along the depth direction to form a 3D feature volume. A winner-takes-all strategy is used to compare the sharpness response value (i.e., contrast) of each pixel in the depth dimension. For each pixel coordinate, the index corresponding to the layer with the highest sharpness value is selected as the optimal focusing layer for that pixel, thus generating the initial decision map. To avoid edge artifacts caused by layer-level abrupt changes in the initial decision map, pixels are first extracted from each central main view based on the initial decision map to synthesize an initial fused image. Subsequently, this initial fused image is used as a guide image to perform guided filtering on the initial decision map. Utilizing the edge-preserving properties of guided filtering, the decision boundaries are smoothed, ultimately resulting in a smooth and accurate final decision map.

[0045] Step S60: Based on the decision map, the three-dimensional cost volume is fused, and the three-dimensional point cloud data of the object corresponding to the image is obtained based on the fusion result.

[0046] Specifically, firstly, based on the displacement step size during light field camera acquisition... This determines the overlapping region in disparity space between two adjacent cost volumes. For example, suppose there are two cost volumes. and ), parallax range Due to the disparity offset caused by displacement, there is an overlapping region in the fused disparity space. This overlapping region corresponds to when... Intervals. Within these overlapping regions, a decision graph is used for optimal selection. For ease of implementation and visualization, this invention constructs a schematic decision graph, such as... Figure 6 As shown. Taking the fusion of two cost bodies as an example, the range of the decision graph is defined as follows. Based on this decision map, we define the overlapping area as follows: The value within is selectively merged. Specifically, the extent of this overlapping region simultaneously corresponds to... The disparity interval is as well as The disparity interval is Then, the cost map of the overlapping region is extracted, and the weights of the fused overlapping region are determined based on the decision map. This identifies the reliable region within the overlapping region. After obtaining the reliable region, to achieve smooth and continuous cost fusion, a one-dimensional linear interpolation algorithm is used for fusion, the expression of which is: , ,

[0047]

[0048]

[0049] in: For decision graphs, These are serial numbers for different levels. For mask label values, This is the label value after rounding down.

[0050] The above fusion algorithm effectively utilizes the advantages of adjacent cost volumes within their respective depth ranges while maintaining depth continuity, and effectively leverages the complementary information of multi-focal plane light field data, avoiding measurement blind spots and depth information distortion. When extended to general scenarios of multi-cost volume fusion, multiple cost volumes may overlap at the same depth level. In this case, a decision graph can effectively select the two optimal cost volumes for fusion calculation.

[0051] In practical applications, the handling of boundary cases can be further considered. For example, when the decision graph parameters... In some cases, the fusion formula requires a third cost map, which may not exist in the actual system. To address this boundary case, we adopt a more robust "Winner-Take-All" (WTA) strategy:

[0052] This strategy effectively avoids the uncertainty of interpolation calculation under boundary conditions while ensuring the robustness of the algorithm.

[0053] The step of obtaining the 3D point cloud data of the object corresponding to the image based on the fusion result specifically includes: Specifically, the fused cost volume is the panoramic depth cost volume. The fused panoramic depth cost volume is then processed using a "winner-take-all" (WTA) strategy to extract the depth label corresponding to the minimum cost, resulting in a complete disparity map. Subsequently, scale mapping is performed according to system calibration parameters (pixel spacing, focal length, baseline length) to recover the true depth values. Finally, the depth map is converted into 3D point cloud data using coordinate projection formulas, thus obtaining the 3D point cloud data of the corresponding objects contained in the image. Specifically, for each pixel (u, v) in the depth map and its depth value Z (or... ), its 3D coordinates in the camera coordinate system (Xc, Yc, )for: =( ) =( ) = .

[0054] The cost volume fusion strategy guided by decision graph can maintain measurement accuracy while reducing the number of samples. At the same time, by weighted fusion of overlapping regions, it can effectively utilize the advantages of adjacent cost volumes within their respective depth ranges while maintaining depth continuity, fully integrate the complementary information of multi-focal surface light field data, and avoid measurement blind spots and depth information distortion.

[0055] In summary, the image processing method in this embodiment acquires images at preset intervals using a single light field camera, eliminating the need for simultaneous acquisition by multiple cameras and reducing the difficulty of image acquisition. Based on the images, sub-aperture images are extracted; a three-dimensional cost volume is calculated based on the sub-aperture images; focusing features are extracted from the sub-aperture images to obtain a focusing feature map; a decision map is calculated based on the focusing feature map; and the three-dimensional cost volumes are fused based on the decision map, obtaining the three-dimensional point cloud data of the object corresponding to the image based on the fusion result. The cost volume fusion strategy guided by the decision map maintains measurement accuracy with a low sampling rate, effectively utilizes the advantages of adjacent cost volumes within their respective depth ranges while maintaining depth continuity, and effectively leverages the complementary information of multi-focal surface light field data, avoiding measurement blind spots and depth information distortion.

[0056] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0057] In one embodiment, an image processing apparatus is provided, which corresponds one-to-one with the image processing method in Embodiment 1 above. For example... Figure 7 As shown, the image processing device includes an acquisition module 101, a first extraction module 102, a first calculation module 103, a second extraction module 104, a second calculation module 105, and a fusion module 106. Detailed descriptions of each functional module are as follows: The acquisition module 101 is used to acquire images collected at preset intervals; The first extraction module 102 is used to extract a sub-aperture image based on the image; The first calculation module 103 is used to calculate the three-dimensional cost volume based on the sub-aperture image; The second extraction module 104 is used to extract the focusing features of the sub-aperture image to obtain a focusing feature map; The second calculation module 105 is used to calculate a decision map based on the focused feature map; The fusion module 106 is used to fuse the three-dimensional cost volume according to the decision map, and obtain the three-dimensional point cloud data of the object corresponding to the image based on the fusion result.

[0058] The image processing apparatus provided by this invention acquires images at preset intervals using a single light field camera, eliminating the need for simultaneous acquisition by multiple cameras and reducing the difficulty of image acquisition. Based on the images, sub-aperture images are extracted; a three-dimensional cost volume is calculated based on the sub-aperture images; focusing features are extracted from the sub-aperture images to obtain a focusing feature map; a decision map is calculated based on the focusing feature map; and the three-dimensional cost volumes are fused based on the decision map to obtain the three-dimensional point cloud data of the object corresponding to the image. The cost volume fusion strategy guided by the decision map maintains measurement accuracy with a low sampling rate, effectively utilizes the advantages of adjacent cost volumes within their respective depth ranges while maintaining depth continuity, and effectively leverages the complementary information of multi-focal surface light field data, avoiding measurement blind spots and depth information distortion.

[0059] For specific limitations regarding the image processing apparatus, please refer to the limitations on the image processing method above, which will not be repeated here. Each module in the aforementioned image processing apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0060] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of an image processing method on the server side.

[0061] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of an image processing method on the client side.

[0062] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire images at preset intervals; Based on the image, a sub-aperture image is extracted; Based on the sub-aperture image, the three-dimensional cost volume is calculated; Focusing feature extraction is performed on the sub-aperture image to obtain a focusing feature map; Based on the focused feature map, a decision map is calculated; Based on the decision map, the three-dimensional cost volume is fused, and the three-dimensional point cloud data of the object corresponding to the image is obtained based on the fusion result.

[0063] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire images at preset intervals; Based on the image, a sub-aperture image is extracted; Based on the sub-aperture image, the three-dimensional cost volume is calculated; Focusing feature extraction is performed on the sub-aperture image to obtain a focusing feature map; Based on the focused feature map, a decision map is calculated; Based on the decision map, the three-dimensional cost volume is fused, and the three-dimensional point cloud data of the object corresponding to the image is obtained based on the fusion result.

[0064] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0065] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0066] The embodiments described above are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention. In the description of this specification, the reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image processing method, characterized in that, The method includes: Acquire images at preset intervals; Based on the image, a sub-aperture image is extracted; Based on the sub-aperture image, the three-dimensional cost volume is calculated; Focusing feature extraction is performed on the sub-aperture image to obtain a focusing feature map; Based on the focused feature map, a decision map is calculated; Based on the decision map, the three-dimensional cost volume is fused, and the three-dimensional point cloud data of the object corresponding to the image is obtained based on the fusion result.

2. The image processing method as described in claim 1, characterized in that, The step of extracting the sub-aperture image based on the image includes: Based on the arrangement pattern of the microlens array, the macro-pixel region corresponding to each microlens in the image is determined; Within each macro-pixel region, determine the coordinates of target pixels with the same relative position; Traverse the image and extract the pixel value located at the target pixel coordinates in each macro pixel region; The extracted pixel values ​​are recombined according to the spatial arrangement order of the microlens array to obtain a sub-aperture image corresponding to the target pixel coordinates.

3. The image processing method as described in claim 1, characterized in that, The step of calculating the three-dimensional cost volume based on the sub-aperture image includes: Determine the disparity search range, and select multiple candidate disparity values ​​within the disparity search range; For each candidate disparity value, the sub-aperture image is translated according to its positional relationship with the central main view in the sub-aperture image. Calculate the matching cost between the translated sub-aperture image and the central main view; The matching costs corresponding to all the candidate disparity values ​​are stacked according to the disparity direction to construct a three-dimensional cost volume.

4. The image processing method as described in claim 1, characterized in that, The step of extracting focusing features from the sub-aperture image to obtain a focusing feature map includes: A mean filter is used to smooth the central main view in the sub-aperture image to obtain a smoothed image; Calculate the absolute value of the grayscale difference between the central main view and the smoothed image to obtain a coarse focus feature map; Using the central main view as a guide image, the coarse focus feature map is subjected to guided filtering to obtain a focus feature map.

5. The image processing method as described in claim 1, characterized in that, The step of calculating the decision map based on the focused feature map includes: Multiple focused feature maps corresponding to different acquisition positions are obtained, and the multiple focused feature maps are stacked in the depth direction; Iterate through each pixel coordinate and compare the pixel values ​​of the multiple focused feature maps at the same pixel coordinate; Select the layer number corresponding to the focused feature map with the largest pixel value as the initial decision value of the pixel coordinate, and generate an initial decision map; Based on the initial decision map, the central main view in the sub-aperture image is initially fused to obtain an initial fused image; Using the initial fused image as a guide image, the initial decision image is subjected to guided filtering to obtain a decision image.

6. The image processing method as described in claim 1, characterized in that, The step of fusing the three-dimensional cost volume according to the decision map and obtaining the three-dimensional point cloud data of the object corresponding to the image based on the fusion result includes: The displacement of each acquisition position is obtained from the image, and the total disparity range of all three-dimensional cost volumes under a unified reference and the disparity overlap area between adjacent cost volumes are determined based on the displacement. Within the disparity overlap region, the pixel values ​​of the decision map are used as weights to perform linear interpolation weighted fusion on the cost values ​​of adjacent cost bodies; By combining the cost value of non-overlapping areas with the cost value of the merged overlapping areas, a panoramic deep fusion cost body is constructed. Based on the panoramic depth fusion cost volume, the three-dimensional point cloud data of the object corresponding to the image is obtained.

7. The image processing method as described in claim 6, characterized in that, The step of obtaining the 3D point cloud data of the object corresponding to the image based on the panoramic depth fusion cost volume includes: Using a winner-takes-all strategy, the panoramic deep fusion cost volume is traversed to extract the disparity label corresponding to the optimal matching cost and obtain the initial disparity map; Based on a predetermined mapping relationship between disparity and physical depth, the initial disparity map is scaled to convert the disparity values ​​into true depth values, thereby obtaining a depth map. Using the coordinate projection formula, the pixel coordinates in the depth map are mapped to the actual depth values ​​in three-dimensional space to generate the three-dimensional point cloud data.

8. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire images collected at preset intervals; The extraction module is used to extract a sub-aperture image based on the image; The first calculation module is used to calculate the three-dimensional cost volume based on the sub-aperture image; The second calculation module is used to calculate a decision map based on the focused feature map; The fusion module is used to fuse the three-dimensional cost volume according to the decision map, and obtain the three-dimensional point cloud data of the object corresponding to the image based on the fusion result.

9. An image processing apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image processing method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image processing method as described in any one of claims 1-7.

Images