A multi-view three-dimensional reconstruction method and system based on depth recovery

Abstract

The application relates to the technical field of three-dimensional reconstruction, and provides a multi-view three-dimensional reconstruction method and system based on depth-of-field recovery, which comprises the following steps: acquiring multi-view images of a scene to be reconstructed; for each image, a PSO-BP neural network is used to extract a defocused blur area, and depth-of-field recovery is performed on the defocused blur area of the image; for each image after depth-of-field recovery, feature points are detected; by matching the feature points between different images, a reconstruction result is obtained; wherein the PSO-BP neural network updates the weight and threshold of the BP neural network by cross-executing a PSO algorithm and a simplex optimization algorithm, and in the iteration process, the inertia weight of the PSO algorithm is updated based on the evaluation number. Not only the local search capability of the PSO-BP neural network is improved, but also the depth-of-field recovery efficiency is greatly improved, and the details of the three-dimensional reconstruction model are improved.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, and in particular relates to a multi-view 3D reconstruction method and system based on depth of field restoration. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of computer vision, multi-view 3D reconstruction technology is being used more and more widely. For example, detailed 3D city models, as the foundational data for urban planning, construction, management, and informatization, are increasingly being used and are an important component of urban spatial data frameworks. At the same time, the public's demand for the completeness and detail of 3D models is also increasing.

[0004] When part of an image scene is out of focus, it will cause blurring. Blur is a common type of image degradation. It often causes the image to lose some original image details and information. This not only reduces the information expressed by the image, but also affects other computer vision tasks that take images as input. For example, blurring will cause the loss of details in 3D reconstruction. Therefore, studying the deblurring task of images has high research value and practical application value in the field of 3D reconstruction. Summary of the Invention

[0005] To address the technical problems mentioned above, this invention provides a multi-view 3D reconstruction method and system based on depth-of-field restoration. It not only improves the local search capability of the PSO-BP neural network by making the inertial weight nonlinear and using a local solution algorithm in the PSO algorithm stage, but also measures the out-of-focus blurred areas of the image based on the PSO-BP neural network's blur measurement method, performing depth-of-field restoration in these areas. This significantly improves the efficiency of depth-of-field restoration and enhances the detail of the 3D reconstructed model.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The first aspect of the present invention provides a multi-view 3D reconstruction method based on depth-of-field restoration, comprising:

[0008] Acquire multi-view images of the scene to be reconstructed;

[0009] For each image, a PSO-BP neural network is used to extract the out-of-focus blurred region and perform depth-of-field restoration on the out-of-focus blurred region of the image;

[0010] For each depth-of-field restored image, feature points are detected;

[0011] The reconstruction result is obtained by matching feature points between different images;

[0012] The PSO-BP neural network updates the weights and thresholds of the BP neural network by cross-executing the PSO algorithm and the simplex optimization algorithm, and updates the inertia weights of the PSO algorithm based on the number of evaluations during the iteration process.

[0013] Furthermore, for each depth-of-field restored image, foreground region segmentation, grayscale processing, and sharpening processing are performed sequentially to detect feature points.

[0014] Furthermore, the specific steps for extracting the out-of-focus blurred region are as follows:

[0015] Divide the image into blocks;

[0016] Extract the feature parameters of each block;

[0017] The extracted feature parameters are input into the PSO-BP neural network to obtain the out-of-focus blurred region.

[0018] Furthermore, the specific steps of the cross-execution of the PSO algorithm and the simplex optimization algorithm are as follows:

[0019] Step (1): Initialize the number of iterations p = 1;

[0020] Step (2): Combining the global optimum, the PSO algorithm performs several iterations on all particles and outputs the global optimum solution;

[0021] Step (3): The simplex optimization algorithm iterates using the global optimal solution output by the PSO algorithm as the initial value. When the minimum error of the simplex optimization algorithm is satisfied or the maximum number of iterations of the simplex optimization algorithm is reached, the global optimal solution obtained by the simplex optimization algorithm is output.

[0022] Step (4): Let p = p + 1, and determine whether the number of iterations p has reached the maximum number of iterations. If so, proceed to step (5); otherwise, take the global optimal solution obtained by the simplex optimization algorithm as the global optimal solution and return to step (2).

[0023] Step (5): Output the global optimal solution obtained by the simplification algorithm.

[0024] Furthermore, the inertia weight for the t-th iteration is:

[0025]

[0026] r(t+1)=4r(t)(1-r(t)),r(0)=rand

[0027] Among them, w min and w maxAll are set values, FEs(t) is the number of evaluations in the t-th iteration, and MaxFEs is the maximum number of evaluations.

[0028] A second aspect of the present invention provides a multi-view 3D reconstruction system based on depth-of-field restoration, comprising:

[0029] The image acquisition module is configured to acquire multi-view images of the scene to be reconstructed.

[0030] The depth-of-field restoration module is configured to: for each image, use a PSO-BP neural network to extract the out-of-focus blurred areas and restore the depth of field of the out-of-focus blurred areas of the image;

[0031] The feature point detection module is configured to detect feature points for each depth-restored image.

[0032] The reconstruction module is configured to obtain reconstruction results by matching feature points between different images;

[0033] The PSO-BP neural network updates the weights and thresholds of the BP neural network by cross-executing the PSO algorithm and the simplex optimization algorithm, and updates the inertia weights of the PSO algorithm based on the number of evaluations during the iteration process.

[0034] Furthermore, the feature point detection module is specifically configured to: for each depth-restored image, perform foreground region segmentation, grayscale processing, and sharpening processing in sequence, and then detect feature points.

[0035] Furthermore, the inertia weight for the t-th iteration is:

[0036]

[0037] r(t+1)=4r(t)(1-r(t)),r(0)=rand

[0038] Among them, w min and w max All are set values, FEs(t) is the number of evaluations in the t-th iteration, and MaxFEs is the maximum number of evaluations.

[0039] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a multi-view 3D reconstruction method based on depth-of-field restoration as described above.

[0040] A fourth aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a multi-view 3D reconstruction method based on depth-of-field restoration as described above.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] This invention provides a multi-view 3D reconstruction method based on depth-of-field restoration, which measures the out-of-focus and blurred areas of an image and performs depth-of-field restoration in these areas, greatly improving restoration efficiency and saving resources.

[0043] This invention provides a multi-view 3D reconstruction method based on depth of field restoration, which uses an image deblurring algorithm to remove blur from out-of-focus images, restores depth of field to make the images clearer, and applies the processed images to 3D reconstruction, thereby improving the details of the reconstructed 3D model.

[0044] The present invention provides a multi-view 3D reconstruction method based on depth of field restoration, which improves the nonlinear inertial weighting method and increases the probability of escaping local optima.

[0045] This invention provides a multi-view 3D reconstruction method based on depth of field restoration, which uses a local solution algorithm in the PSO algorithm stage to improve local search capability. Attached Figure Description

[0046] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0047] Figure 1 This is a flowchart of a multi-view 3D reconstruction method based on depth-of-field restoration according to Embodiment 1 of the present invention;

[0048] Figure 2 This is a flowchart of the improved PSO algorithm according to Embodiment 1 of the present invention. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0051] Example 1

[0052] This embodiment provides a multi-view 3D reconstruction method based on depth-of-field restoration.

[0053] When an object is located on the focusing plane of the lens, light emitted from a point on the object can be projected onto the image plane as a single image point. However, for points not on the focusing plane, a blurring circle composed of many image points will be formed, resulting in out-of-focus blur and consequently, loss of detail during 3D reconstruction.

[0054] This embodiment provides a multi-view 3D reconstruction method based on depth of field restoration. It uses an image deblurring algorithm to remove blur from out-of-focus images, restores depth of field to make the images clearer, and applies the processed images to 3D reconstruction to improve the details of the reconstructed 3D model.

[0055] This embodiment provides a multi-view 3D reconstruction method based on depth-of-field restoration. It receives input raw image information, acquires a full-focus image and a depth map of the scene, and performs 3D reconstruction using the acquired full-focus image. Figure 1 As shown, the specific steps include the following:

[0056] Step 1: Obtain initial data.

[0057] The initial data includes: multi-view original images of the scene to be reconstructed, image exterior orientation elements, camera intrinsic parameters, and coordinates of the object to be reconstructed.

[0058] Step 2: Depth of field restoration.

[0059] Most commonly used structured illumination 3D imaging technologies today use CCD / CMOS cameras as the collection device for structured illumination signals. Capturing clearly focused fringe signals is crucial for accurately reconstructing the target's 3D topography. When multiple objects in the foreground and background need to be reconstructed, the distance between them may exceed the depth of field of the CCD / CMOS camera, causing the structured illumination fringe signals to become out of focus, thus affecting the reconstruction of the target's 3D topography. Therefore, restoring out-of-focus and blurred areas in the image is of great significance.

[0060] Step 201: Measurement of local defocusing blur in the image.

[0061] This step employs a blur measurement method based on an improved PSO-BP neural network to measure the out-of-focus blurry areas of the image. Performing depth-of-field restoration in these areas can greatly improve restoration efficiency and save resources.

[0062] Particle Swarm Optimization (PSO) and Backpropagation (BP) Neural Networks (PSOs) are complementary in their characteristics. PSO utilizes gradient descent to adjust weights towards the negative gradient of its error function, achieving rapid error reduction and strong local fine-tuning capabilities, but it is prone to getting trapped in local optima. Backpropagation, on the other hand, is characterized by its swarm search strategy and information exchange among individuals within the swarm. Its search is independent of gradient information, resulting in strong algorithm robustness. However, traditional PSO suffers from low local search capability and is susceptible to getting trapped in local optima. Therefore, some improvements have been made to the traditional PSO algorithm.

[0063] During the iteration process, the inertia weight of the PSO algorithm is updated based on the number of evaluations. That is, the inertia weight is improved in the following way:

[0064]

[0065]

[0066] Among them, w min =0.4, w max =0.9; r(t) is the random number generated by the iteration of equation (5), FEs(t) is the number of evaluations at time t, and MaxFEs is the maximum number of evaluations.

[0067] The non-linear nature of this inertia weighting can increase the probability of escaping local optima.

[0068] To improve local search capabilities, the PSO algorithm employs a local solution algorithm—the Nelder-Mead algorithm. This local solution uses an iterative optimization strategy based on the concept of simplification to construct solutions, and is an algorithm for finding local minima of multivariate functions. For an N-dimensional optimization problem, an N+1-dimensional simplex is initially constructed, the function values ​​at the vertices are calculated, and then the vertex function values ​​are analyzed and compared. New vertices and simplexes are constructed until the convergence condition is met. Therefore, it has the advantages of not requiring differentiability of the function and being able to converge to a local minimum relatively quickly.

[0069] The PSO algorithm has strong global search capabilities, while the simplex optimization algorithm has strong local search capabilities. Combining the two and leveraging their complementary strengths can significantly improve optimization accuracy. Figure 2 As shown.

[0070] The improved PSO algorithm is used to optimize the initial weights and thresholds of the network. The fitness function of the PSO algorithm and the simplex optimization algorithm is the error between the predicted value and the true value of the neural network. A single particle in the particle swarm is regarded as its position vector in N-dimensional space, where N refers to the number of weights.

[0071] like Figure 2 As shown, the specific steps of the improved PSO algorithm are as follows:

[0072] Step (1): Initialize the parameters and initialize the number of iterations p=1. Set the number of particles, initial particle position and velocity, maximum number of iterations, number of iterations M of the PSO algorithm, minimum error of the simplex optimization algorithm, and maximum number of iterations of the simplex optimization algorithm.

[0073] Step (2): Based on the global optimum gbest, the PSO algorithm performs M iterations on all particles and outputs the global optimum solution gbest1 at this time; the global optimum gbest when the number of iterations p=1 is selected from the initial particles;

[0074] Step (3): The simplex optimization algorithm uses the global optimal solution gbest1 output by the PSO algorithm after M iterations as the initial value for iteration. When the minimum error of the simplex optimization algorithm is satisfied or the maximum number of iterations of the simplex optimization algorithm is reached, the global optimal solution gbest2 obtained by the simplex optimization algorithm at this time is output.

[0075] Step (4): Use the global optimal solution gbest2 obtained by the simplex optimization algorithm to guide the iteration of the PSO algorithm. Specifically, let p = p + 1, and determine whether the number of iterations p has reached the maximum number of iterations. If so, execute step (5); otherwise, take the global optimal solution gbest2 as the global optimal gbest and return to step (2).

[0076] Step (5): Output the global optimal solution obtained by the final simplex optimization algorithm as the weights and thresholds of the BP neural network.

[0077] The specific steps are as follows:

[0078] (1) Divide the image into blocks;

[0079] (2) Extract the feature parameters of each block;

[0080] (3) Input the extracted parameters into the improved PSO-BP neural network for training;

[0081] (4) Measure the blurred areas of the image.

[0082] Step 202: Perform depth-of-field restoration on the out-of-focus and blurred areas of the image. Specific steps include:

[0083] (1) Set up coded aperture convolution kernels of different sizes.

[0084] Encoded apertures alter the imaging of point light sources, improving the discriminative power between different convolutions. By deconvolving with kernels of different sizes, coded apertures can more easily distinguish between correct and incorrect scales in their deconvolution results. This is because, for coded apertures, both excessively large and small kernels will exhibit ringing effects after deconvolution, while correctly sized kernels will not.

[0085] (2) Select a local window for each pixel in the out-of-focus blurry area of ​​the image.

[0086] (3) Deconvolve this local window using convolution kernels of different sizes.

[0087] (4) Calculate the convolution error corresponding to different convolution kernels using the following formula:

[0088]

[0089] Among them, E k c represents the convolution error corresponding to the k-th convolution kernel. k Indicates the encoding aperture convolution kernel, x i y represents the local image obtained after deconvolution. i This represents the image corresponding to the local window of this pixel.

[0090] (5) Select the convolution kernel with the smallest error.

[0091] (6) Use the convolution kernel at this time to perform depth recovery on the pixel to obtain its corresponding clear image.

[0092] Step 3: Segment the foreground region of the image.

[0093] Image region segmentation can reduce subsequent computation and reduce the generation of redundant point clouds.

[0094] Step 4: Grayscale processing.

[0095] In the grayscale conversion process based on the segmentation results, grayscale processing is performed based on brightness. The corresponding grayscale processing expression is:

[0096]

[0097] Where Gray(i,j) represents the gray value at coordinate (i,j) in the image, R(i,j) is the brightness value of the R component at coordinate (i,j) in the image, G(i,j) is the brightness value of the G component at coordinate (i,j) in the image, and B(i,j) is the brightness value of the B component at coordinate (i,j) in the image.

[0098] Grayscale processing not only preserves the chromaticity and brightness characteristics of the original image, but also reduces invalid information such as color.

[0099] Step 5: Sharpening.

[0100] Sharpening can enhance boundaries, highlight object details, or enhance blurred image details.

[0101] Step 6: Image feature point detection.

[0102] The specific steps for feature point extraction are as follows:

[0103] Step 601: Construct a scale space dependent on a Gaussian filter;

[0104] Step 602: Detect key points based on scale space;

[0105] Step 603: Detect key points through curve fitting and obtain the corresponding extreme points;

[0106] Step 604: Remove edge points;

[0107] Step 605: Determine the main direction of the feature points;

[0108] Step 606: Construct a descriptor based on the main direction of the feature points.

[0109] Step 7: Feature point matching.

[0110] Step 701: Use Euclidean distance to determine whether there is similarity between two descriptors and obtain preliminary matching results.

[0111] Step 702: Use the Ransac random sampling consensus method as the final purification method to eliminate mismatches in the initial matching results and obtain the best matching results.

[0112] Step 8: 3D reconstruction.

[0113] Specifically, feature points extracted from images can yield a large amount of point cloud data. Then, a TIN triangulation is constructed from the matched 3D point cloud (the best matching pair result), and the texture image is automatically mapped onto the corresponding TIN triangular facets, ultimately forming a refined real-world 3D model.

[0114] Example 2

[0115] This embodiment provides a multi-view 3D reconstruction system based on depth-of-field restoration, which specifically includes:

[0116] The image acquisition module is configured to acquire multi-view images of the scene to be reconstructed.

[0117] The depth-of-field restoration module is configured to: for each image, use a PSO-BP neural network to extract the out-of-focus blurred areas and restore the depth of field of the out-of-focus blurred areas of the image;

[0118] The feature point detection module is configured to detect feature points for each depth-restored image.

[0119] The reconstruction module is configured to obtain reconstruction results by matching feature points between different images;

[0120] The PSO-BP neural network updates the weights and thresholds of the BP neural network by cross-executing the PSO algorithm and the simplex optimization algorithm, and updates the inertia weights of the PSO algorithm based on the number of evaluations during the iteration process.

[0121] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0122] Example 3

[0123] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a multi-view 3D reconstruction method based on depth-of-field restoration as described in Embodiment 1 above.

[0124] Example 4

[0125] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the multi-view 3D reconstruction method based on depth-of-field restoration as described in Embodiment 1 above.

[0126] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0127] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0128] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0129] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0130] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0131] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-view 3D reconstruction method based on depth-of-field restoration, characterized in that, include: Acquire multi-view images of the scene to be reconstructed; For each image, a PSO-BP neural network is used to extract the out-of-focus blurred region and perform depth-of-field restoration on the out-of-focus blurred region of the image; For each depth-of-field restored image, feature points are detected; The reconstruction result is obtained by matching feature points between different images; In this process, the PSO-BP neural network updates the weights and thresholds of the BP neural network by cross-executing the PSO algorithm and the simplex optimization algorithm. During the iteration process, the inertia weights of the PSO algorithm are updated based on the number of evaluations. That is, the inertia weights are improved in the following way: in, , ; For the random numbers generated during the iteration, for Number of evaluations at any given moment This represents the maximum number of evaluations. This inertial weight is applied in a non-linear manner to increase the probability of escaping local optima.

2. The multi-view 3D reconstruction method based on depth-of-field restoration as described in claim 1, characterized in that, For each depth-of-field restored image, foreground region segmentation, grayscale processing, and sharpening processing are performed sequentially, and feature points are detected.

3. The multi-view 3D reconstruction method based on depth-of-field restoration as described in claim 1, characterized in that, The specific steps for extracting the out-of-focus blurred region are as follows: Divide the image into blocks; Extract the feature parameters of each block; The extracted feature parameters are input into the PSO-BP neural network to obtain the out-of-focus blurred region.

4. The multi-view 3D reconstruction method based on depth-of-field restoration as described in claim 1, characterized in that, The specific steps of the cross-execution PSO algorithm and the simplex optimization algorithm are as follows: Step (1): Initialize the number of iterations p = 1; Step (2): Combining the global optimum, the PSO algorithm performs several iterations on all particles and outputs the global optimum solution; Step (3): The simplex optimization algorithm iterates using the global optimal solution output by the PSO algorithm as the initial value. When the minimum error of the simplex optimization algorithm is satisfied or the maximum number of iterations of the simplex optimization algorithm is reached, the global optimal solution obtained by the simplex optimization algorithm is output. Step (4): Let p = p + 1, and determine whether the number of iterations p has reached the maximum number of iterations. If so, execute step (5); otherwise, take the global optimal solution obtained by the simplex optimization algorithm as the global optimal solution and return to step (2). Step (5): Output the global optimal solution obtained by the simplification optimization algorithm.

5. A multi-view 3D reconstruction system based on depth-of-field restoration, characterized in that, include: The image acquisition module is configured to acquire multi-view images of the scene to be reconstructed. The depth-of-field restoration module is configured to: for each image, use a PSO-BP neural network to extract the out-of-focus blurred areas and restore the depth of field of the out-of-focus blurred areas of the image; The feature point detection module is configured to detect feature points for each depth-restored image. The reconstruction module is configured to obtain reconstruction results by matching feature points between different images; In this process, the PSO-BP neural network updates the weights and thresholds of the BP neural network by cross-executing the PSO algorithm and the simplex optimization algorithm. During the iteration process, the inertia weights of the PSO algorithm are updated based on the number of evaluations. That is, the inertia weights are improved in the following way: in, , ; For the random numbers generated during the iteration, for Number of evaluations at any given moment This represents the maximum number of evaluations. This inertial weight is applied in a non-linear manner to increase the probability of escaping local optima.

6. The multi-view 3D reconstruction system based on depth-of-field restoration as described in claim 5, characterized in that, The feature point detection module is specifically configured to: for each depth-restored image, perform foreground region segmentation, grayscale processing, and sharpening processing in sequence, and then detect feature points.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multi-view 3D reconstruction method based on depth-of-field restoration as described in any one of claims 1-4.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multi-view 3D reconstruction method based on depth-of-field restoration as described in any one of claims 1-4.

Images