A multi-focus fusion DOF three-dimensional reconstruction method based on a Siamese network

Abstract

The application discloses a multi-focus fusion DOF three-dimensional reconstruction method based on a Siamese network. The method collects multi-focus plane images of an object to be measured, generates a standardized multi-focus plane image sequence in combination with a calibration parameter set containing a focal plane depth, a threshold parameter group and a weight parameter, inputs adjacent images into a weight sharing Siamese network, extracts a focus feature map in a unified feature space through CNN, FPN and DCN, calculates a focus confidence map according to the focus feature map, and constructs a focus decision map in combination with a focus response difference; pixel-level fusion is completed relying on the decision map and the confidence map, a pixel-level focus index matrix is updated through recursive comparison, index abnormal jumps are corrected by using depth continuity constraints and regional correction weights, an optimized focus index matrix is obtained, depth data and preliminary reconstruction results are generated based on the optimized focus index matrix, abnormal areas are corrected in a reverse tracing manner, and finally, the DOF three-dimensional reconstruction results are output. The method can reduce misjudgment, suppress depth jumps, and improve continuity and accuracy.

Description

Technical Field

[0001] This invention relates to the fields of industrial vision, computer vision, machine learning and optical 3D reconstruction technology, and in particular to a multi-focus fusion DOF 3D reconstruction method based on Siamese networks. Background Technology

[0002] DOF (Direction of Focus) 3D reconstruction typically involves acquiring images of the object under test at different focal planes, determining the sharpness of each pixel or local region at different focal planes, and then calculating the corresponding depth based on the optimal focus position to generate a 3D reconstruction result. This type of method has high application value in scenarios such as precision inspection, microstructure measurement, and complex surface reconstruction, but its reconstruction accuracy largely depends on the accuracy of the focus area determination.

[0003] Existing methods typically rely on sharpness features such as image gradient, edge intensity, and local variance for focus determination. For regions with clear textures and well-defined boundaries, these methods meet basic requirements. However, for regions with weak textures, high reflectivity, strong noise, or curved edge transitions, a single sharpness feature is easily affected by brightness variations, noise disturbances, and edge diffusion, leading to unstable determination of the focus source for local pixels. Such misjudgments may not be obvious in 2D fused images, but when further converted to depth data, they can cause reconstruction anomalies such as depth jumps, isolated points, local discontinuities, or surface discontinuities.

[0004] With the development of deep learning technology, some solutions have begun to use neural networks to extract multifocal image features to improve the ability to identify focal regions. However, existing processing methods mostly focus on improving the sharpness of the fused image itself, and do not adequately constrain the correlation between the focusing result and subsequent depth reconstruction. If depth data is directly generated based solely on the focusing result output by the network, it is still difficult to avoid the impact of local misjudgments on the 3D reconstruction result. Therefore, it is necessary to propose a DOF 3D reconstruction scheme that can form a continuous processing relationship between multifocal image feature extraction, focusing source determination, depth index correction, and reconstruction result feedback, in order to improve the reconstruction stability under complex imaging conditions. Summary of the Invention

[0005] This invention provides a multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks, comprising: S10. Acquire multi-focal plane images of the object under test using a DOF camera, and generate a standardized multi-focal plane image sequence by combining a calibration parameter set containing focal plane depth, threshold parameter set, and weight parameters. S20. Input two adjacent multi-focal plane images into a Siamese network with a weight-sharing structure, and generate focal feature maps in the same feature space through CNN, FPN and DCN. S30. Generate a focusing decision map based on the focusing feature map, and calculate a focusing confidence map to characterize the reliability of the depth source; S40. Perform pixel-level fusion based on the focus decision map and focus confidence map, and update the pixel-level focus index matrix during the recursive comparison process; S50. Construct depth continuity constraints and region correction weights based on the calibration parameter set, and correct abnormal index jumps in the pixel-level focus index matrix to obtain the optimized focus index matrix. S60. Generate pixel-level depth data and preliminary 3D reconstruction results based on the optimized focusing index matrix, perform reverse pixel-level source correction on abnormal reconstruction areas, and output the final DOF 3D reconstruction results.

[0006] The multi-focal plane fusion DOF 3D reconstruction method based on Siamese networks, as described above, involves acquiring multi-focal plane images of the object under test using a DOF camera and combining them with a calibration parameter set including focal plane depth, threshold parameters, and weight parameters to generate a standardized multi-focal plane image sequence, including: S101. Read the acquisition trigger time, focal plane drive position and camera calibration parameters of each image according to the focal plane acquisition order, and write the focal plane depth, threshold parameter group and weight parameters into the calibration parameter set. S102. Based on the calibration parameter set, perform spatial registration, size unification, brightness normalization and noise suppression processing on multiple images, and filter image data according to the registration threshold in the calibration parameter set to obtain a standardized multi-focal plane image sequence.

[0007] The multi-focal plane fusion DOF 3D reconstruction method based on Siamese networks, as described above, involves inputting two adjacent multi-focal plane images into a Siamese network with a weight-sharing structure. Focus feature maps in the same feature space are generated through CNN, FPN, and DCN, including: S201. Input two adjacent multi-focal plane images into the first feature extraction branch and the second feature extraction branch respectively. The first feature extraction branch and the second feature extraction branch use the same convolution kernel parameters, normalization parameters and feature mapping parameters. S202. In the first feature extraction branch and the second feature extraction branch, the basic focusing features are extracted by CNN, the multi-scale focusing features are fused by FPN, and the sampling offset is learned by DCN based on the local structural response to obtain the focusing feature maps corresponding to the two images.

[0008] The above-described multi-focus fusion DOF 3D reconstruction method based on Siamese networks includes generating a focus decision map from the focus feature map and calculating a focus confidence map to characterize the reliability of the depth source, comprising: S301. Normalize the feature response intensity, local texture energy, noise variance and neighborhood feature consistency of the same pixel location in the focus feature maps of the two images, and calculate the focus confidence of the pixel location according to the weight parameters in the calibration parameter set. S302. Determine the focus source at the pixel location based on the focus response difference between the two images at the same pixel location and the response judgment conditions in the calibration parameter set, and generate a focus decision map and a focus confidence map.

[0009] The above-described multi-focus fusion DOF 3D reconstruction method based on Siamese networks includes pixel-level fusion based on focus decision maps and focus confidence maps, and updating the pixel-level focus index matrix during recursive alignment, comprising: S401. In the first round of recursive comparison, the focus source image number corresponding to each pixel position in the focus decision map is written into the pixel-level focus index matrix, and the focus confidence corresponding to the same pixel position is written into the historical confidence matrix. S402. In subsequent recursive comparisons, the focus index number of the target pixel is updated according to the current focus confidence, historical confidence, neighborhood consistency condition and confidence update condition in the calibration parameter set, and the original focus index number of the target pixel is retained when the update condition is not met.

[0010] The above-described multi-focus fusion DOF 3D reconstruction method based on Siamese networks includes constructing depth continuity constraints and region correction weights according to a calibration parameter set, and correcting abnormal index jumps in the pixel-level focus index matrix to obtain an optimized focus index matrix, including: S501. Convert the focal plane number of the target pixel and the focal plane number of the neighboring pixels in the pixel-level focus index matrix into calibration depth values ​​respectively, and determine the abnormal index jump pixels according to the calibration depth difference, boundary response value and jump judgment conditions in the calibration parameter set. S502. Generate region correction weights based on the texture energy, brightness saturation ratio, and neighborhood index concentration of the region where the abnormal index jump pixel is located, and correct the focus index number of the abnormal index jump pixel according to the region correction weights to obtain the optimized focus index matrix.

[0011] The above-described multi-focus fusion DOF 3D reconstruction method based on Siamese networks includes generating pixel-level depth data and preliminary 3D reconstruction results based on an optimized focus index matrix, performing reverse pixel-level source correction on reconstructed anomaly regions, and outputting the final DOF 3D reconstruction result, including: S601. Map the optimal focus source image number of each pixel position in the optimized focus index matrix to the corresponding focal plane calibration depth, and combine the pixel coordinates with the corresponding focal plane calibration depth to generate pixel-level depth data; S602. Determine the reconstruction anomaly region in the preliminary 3D reconstruction result based on the reconstruction anomaly conditions in the calibration parameter set, reverse map the reconstruction anomaly region to the corresponding pixel position, correct the focus index number based on the corresponding pixel position, update the pixel-level depth data, and output the final DOF 3D reconstruction result.

[0012] This invention also provides a multi-focus fusion DOF three-dimensional reconstruction system based on Siamese networks, comprising: The image sequence construction module is used to acquire multi-focal plane images of the object under test through a DOF camera, and combine them with a calibration parameter set containing focal plane depth, threshold parameter set and weight parameters to generate a standardized multi-focal plane image sequence. The focus feature extraction module is used to input two adjacent multi-focal plane images into a Siamese network with a weight-sharing structure, and generate focus feature maps in the same feature space through CNN, FPN and DCN; The focus decision generation module is used to generate a focus decision map based on the focus feature map and calculate a focus confidence map to characterize the reliability of the depth source. The index recursive update module is used to perform pixel-level fusion based on the focus decision map and the focus confidence map, and update the pixel-level focus index matrix during the recursive comparison process; The index constraint optimization module is used to construct depth continuity constraints and region correction weights based on the calibration parameter set, and to correct abnormal index jumps in the pixel-level focus index matrix to obtain an optimized focus index matrix. The reconstruction feedback correction module is used to generate pixel-level depth data and preliminary 3D reconstruction results based on the optimized focusing index matrix, and to perform reverse pixel-level source correction on abnormal reconstruction areas, and output the final DOF 3D reconstruction results.

[0013] The beneficial effects achieved by this invention are as follows: This invention unifies and standardizes multi-focal-plane images, ensuring consistent spatial and parameter correspondences between different focal-plane images. Based on this, it utilizes a weight-sharing Siamese network to extract features from adjacent focal-plane images according to the same rules, making the focus response comparison more stable. Simultaneously, by combining factors such as texture, noise, and neighborhood consistency to calculate focus confidence, it can reduce focus misjudgments in areas with weak texture, high reflectivity, and noise. Furthermore, this invention progressively updates the pixel-level focus index through a recursive fusion method and corrects abnormal index jumps before depth mapping, reducing depth abrupt changes caused by local focus errors. After forming the preliminary 3D reconstruction results, it can also reverse-correct the focus index of corresponding pixels based on the reconstructed abnormal regions, thereby improving the continuity, accuracy, and reliability of the final DOF 3D reconstruction results. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0015] Figure 1 This is a flowchart of a multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks provided in Embodiment 1 of this application; Figure 2 This is a schematic diagram of the Siamese network focusing feature extraction structure provided in Embodiment 1 of this application; Figure 3 This is a schematic diagram of the multi-focal plane image recursive fusion and pixel-level focus index update process provided in Embodiment 1 of this application; Figure 4 This is a schematic diagram of a multi-focus fusion DOF three-dimensional reconstruction system based on a Siamese network, provided in Embodiment 2 of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1 like Figure 1As shown, Embodiment 1 of this application provides a multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks, including the following steps: S10. Acquire multi-focal plane images of the object under test using a DOF camera, and generate a standardized multi-focal plane image sequence by combining a calibration parameter set containing focal plane depth, threshold parameter set, and weight parameters. In this embodiment, the DOF camera performs multiple imaging operations on the object under test according to a set focal plane acquisition sequence, obtaining multiple images corresponding to different focal plane positions. For each image, the acquisition trigger time and focal plane drive position are recorded during acquisition, and the corresponding image number, focal plane number, and focal plane depth are determined by combining the calibration parameter set. Specifically, this includes the following sub-steps: S101. Read the acquisition trigger time, focal plane drive position and camera calibration parameters of each image according to the focal plane acquisition order, and write the focal plane depth, threshold parameter group and weight parameters into the calibration parameter set. Specifically, each image is assigned an image number according to the order in which it enters the acquisition sequence, and the focal plane is assigned an image number according to the order in which the focal plane drive positions are arranged in the calibration stroke. The correspondence between the focal plane drive positions and the focal plane depths is pre-recorded in the camera calibration parameters, and the focal plane depth corresponding to each focal plane number is determined based on this correspondence. When multiple images are acquired at the same focal plane position, the multiple images have different image numbers but correspond to the same focal plane number.

[0018] The calibration parameter set includes focal plane depth, threshold parameter set, and weight parameters. The threshold parameter set includes registration threshold, response threshold, update threshold, jump threshold, boundary threshold, correction threshold, and reconstruction anomaly threshold; the weight parameters include focus confidence calculation weight and region correction calculation weight. A correspondence is established between image IDs and acquisition trigger times, and between focal plane IDs and focal plane depths. The threshold parameter set and weight parameters are written into the calibration parameter set as processing parameters.

[0019] S102. Based on the calibration parameter set, perform spatial registration, size unification, brightness normalization and noise suppression processing on multiple images, and filter image data according to the registration threshold in the calibration parameter set to obtain a standardized multi-focal plane image sequence.

[0020] Specifically, a reference image is selected from the multi-focal-plane images, and spatial registration is performed on the remaining images using the reference image as a reference, so that the corresponding test areas in different focal-plane images form a unified pixel coordinate relationship. After spatial registration is completed, the registration residual of each image relative to the reference image is calculated, and valid images are selected according to the registration threshold in the calibration parameter set.

[0021] For valid images, size unification, brightness normalization, and noise suppression are performed. The processed image data is then correlated with the corresponding image ID, focal plane ID, pixel coordinates, and focal plane depth to obtain a standardized multi-focal plane image sequence. For images that do not meet the registration threshold, their image ID and focal plane ID are recorded, and they are marked as abnormal images.

[0022] S20. Input two adjacent multi-focal plane images into a Siamese network with a weight-sharing structure, and generate focal feature maps corresponding to the two images in the same feature space through CNN, FPN and DCN. In this embodiment, two adjacent multifocal plane images are selected from the standardized multifocal plane image sequence as the input images for the current round, and then input into a Siamese network with a weight-sharing structure. This Siamese network processes the two input images through two structurally consistent feature extraction branches, and outputs focus feature maps corresponding to the two images, providing feature data for the subsequent generation of focus confidence maps and focus decision maps.

[0023] like Figure 2 As shown, input image A and input image B enter the first and second feature extraction branches of the Siamese network, respectively. The two feature extraction branches employ a shared weight structure and each branch incorporates a CNN, an FPN, and a DCN. The CNN extracts the basic focus features of the input images, the FPN fuses focus responses at different scales, and the DCN adaptively adjusts the sampling positions of focus boundaries and irregular structural regions. The first feature extraction branch outputs Feature Map A, and the second feature extraction branch outputs Feature Map B, which serve as the focus feature maps for the two input images.

[0024] S201. Input two adjacent multi-focal plane images into the first feature extraction branch and the second feature extraction branch respectively. The first feature extraction branch and the second feature extraction branch use the same convolution kernel parameters, normalization parameters and feature mapping parameters. Specifically, in a standardized multifocal plane image sequence, two adjacent multifocal plane images are selected according to the focal plane number or the order of the effective image sequence, and are used as input image A and input image B, respectively. Input image A is input into the first feature extraction branch, and input image B is input into the second feature extraction branch.

[0025] The first and second feature extraction branches have the same network hierarchical structure. Both branches share convolution kernel parameters, normalization parameters, and feature mapping parameters, ensuring that both input images are processed using the same feature extraction rules. The convolution kernel parameters determine how the convolutional layer extracts local texture, edge contours, and grayscale variations; the normalization parameters constrain the numerical distribution of intermediate feature responses; and the feature mapping parameters control the mapping relationship between features at different levels.

[0026] After dual-branch processing with shared parameters, the first feature extraction branch generates branch features corresponding to input image A, and the second feature extraction branch generates branch features corresponding to input image B. The two branch features have a consistent spatial resolution correspondence and feature channel organization, enabling subsequent focus response comparisons at corresponding pixel locations and their neighborhoods.

[0027] S202. In the first feature extraction branch and the second feature extraction branch, the basic focusing features are extracted by CNN, the multi-scale focusing features are fused by FPN, and the sampling offset is learned by DCN based on the local structural response to obtain the focusing feature maps corresponding to the two images.

[0028] Specifically, in each feature extraction branch, the CNN performs multi-layer convolution processing on the input image to extract basic focusing features at different levels. These basic focusing features include local texture features, edge contour features, gray-level abrupt change features, and detail structure responses. Focused regions typically exhibit strong local texture and edge responses, while out-of-focus regions usually show edge diffusion and weakened local responses. Therefore, basic focusing features can reflect the differences in focusing states across different regions of the image.

[0029] FPN receives feature maps from different layers of the CNN and performs multi-scale fusion of shallow detail features and deep structural features. Shallow detail features are used to preserve subtle textures and local edge variations, while deep structural features are used to represent the focus response distribution over a larger neighborhood. After fusion by FPN, multi-scale focus features that simultaneously contain local details and region structural information are obtained.

[0030] In the process of feature extraction or feature enhancement, DCN learns the sampling offset based on the local structural response of the input features and adjusts the convolution sampling position based on the sampling offset. For focused boundaries, irregular contours, and curved transition regions, the variable sampling position can reduce the restriction of fixed convolution sampling on the boundary shape, allowing branch features to better express local edge changes and irregular structural responses.

[0031] After processing by CNN, FPN, and DCN, the first feature extraction branch outputs Feature Map A corresponding to input image A, and the second feature extraction branch outputs Feature Map B corresponding to input image B. Feature Map A and Feature Map B have the same spatial size and feature channel structure, and serve as the focused feature maps corresponding to the two images.

[0032] S30. Calculate a focus confidence map to characterize the reliability of the depth source based on the focus feature map, and generate a focus decision map by combining the focus response differences; In this embodiment, based on Feature Map A and Feature Map B obtained in step S20, the focus responses of the two input images at the same pixel positions are compared, and the focus confidence is calculated by combining local texture, noise fluctuation, and neighborhood consistency. The focus confidence is used to represent the reliability of selecting a certain focal plane image as the focus source at the current pixel position, and the focus decision map is used to record the focus source selection result corresponding to that pixel position. Specifically, it includes the following sub-steps: S301. Normalize the feature response intensity, local texture energy, noise variance and neighborhood feature consistency of the same pixel location in the focus feature maps of the two images, and calculate the focus confidence of the pixel location according to the weight parameters in the calibration parameter set. Specifically, let the focus feature map corresponding to the input image A be... The focus feature map corresponding to input image B is The pixel position is denoted as pixel position The neighborhood window is denoted as For any input image ,in Extract the image at the pixel position respectively. The feature response intensity, local texture energy, noise variance, and neighborhood consistency at a given location are used to calculate the focus evidence value for that pixel location.

[0033] To reduce misjudgments of a single sharpness metric in areas with weak texture, high reflectivity, or noise, this embodiment employs a multi-factor weighted method to calculate image sharpness. At pixel position Focused evidence value Its expression is as follows: ,in, Representing an image At pixel position The focus of evidence value; Representing an image The focused feature map at the pixel location The eigenvector at that location; This represents the L2 norm of the eigenvector; Represented by pixel position A neighborhood window centered on the center; Indicates the number of pixels within the neighborhood window; This represents the position of any pixel within the neighborhood window; Representing an image The focused feature map at the neighborhood pixel location The eigenvector at that location; Representing an image At pixel position Local texture energy at the location; Representing an image In the neighborhood window Average texture energy within; Representing an image At pixel position The noise variance estimate at point ; This represents the feature difference between neighboring pixels and the target pixel; This represents the spatial distance between neighboring pixels and the target pixel; and All of these are weight parameters in the calibration parameter set; This is a positive, stable term. In the above formula, the first term represents the prominence of the target pixel's feature response relative to the average feature response of its neighborhood; the second term represents the degree of change in the target pixel's local texture energy relative to the neighborhood's texture level; the third term represents the suppressive effect of noise variance on the focus evidence value; and the fourth term represents the degree of feature consistency between the target pixel and its neighboring pixels. Through the above calculations, the pixel positions of images A and B can be obtained respectively. Focused evidence value and .

[0034] In obtaining and Then, the image with the larger focus evidence value was identified as the candidate focus source image, denoted as... ,in Pixel position Select candidate focus source images Focus confidence level The calculation is as follows: ,in, Indicates pixel position Select image Focus confidence level when used as a source of focus; Indicates candidate focused source image At pixel position The focus of evidence value; and These represent the pixel positions of image A and image B, respectively. The focus of evidence value; Represents an exponential function; This represents the difference in focus evidence between two images at this pixel location; It is a positive and stable term.

[0035] The aforementioned focus confidence score consists of two parts. The first part determines the relative advantage of candidate focus sources based on the distribution of focus evidence values ​​between the two images; the second part suppresses the confidence score of low-discrimination locations based on the degree of difference in focus evidence values ​​between the two images. After calculating the confidence score for each pixel location in the above manner, a focus confidence score map with the same spatial size as the input image is obtained.

[0036] S302. Determine the focus source at the pixel location based on the focus response difference between the two images at the same pixel location and the response judgment conditions in the calibration parameter set, and associate the focus source with the corresponding focus confidence to generate a focus decision map.

[0037] Specifically, after obtaining the focus evidence values ​​corresponding to images A and B, for the same pixel location... The focus response difference at the pixel location is used for judgment. If the focus evidence value of image A is higher than that of image B, and the difference in focus evidence between the two satisfies the response judgment condition in the calibration parameter set, then image A is determined as the focus source at that pixel location; if the focus evidence value of image B is higher than that of image A, and the difference in focus evidence between the two satisfies the response judgment condition in the calibration parameter set, then image B is determined as the focus source at that pixel location.

[0038] When two images are at pixel positions If the difference in focus evidence at a given location does not meet the response determination criteria, the pixel location is marked as a low-discrimination location. For low-discrimination locations, a supplementary determination is made by combining the focus confidence of the pixel location with the distribution of already determined focus sources within its neighborhood to determine the focus source of the pixel location.

[0039] After the above processing is completed, a focus decision map is generated. The focus decision map is stored in the form of a pixel matrix, with each pixel location recording the corresponding focus source image number. The focus confidence map has the same spatial size as the focus decision map and corresponds one-to-one with the pixel locations. The focus decision map is used for pixel-level fusion in step S40, and the focus confidence map is used for the recursive update of the pixel-level focus index matrix in step S40.

[0040] S40. Perform pixel-level fusion based on the focus decision map and focus confidence map, recursively compare the fusion result with the next focal plane image, and update the pixel-level focus index matrix. In this embodiment, based on the focus decision map and focus confidence map generated in step S30, pixel-level fusion is performed on the two current input images, and the focus source image number corresponding to each pixel position is written into the pixel-level focus index matrix. Subsequently, the fusion result of the current round is input into the Siamese network for recursive comparison with the next focal plane image until all effective image processing in the multi-focal plane image sequence is completed.

[0041] like Figure 3 As shown, after the DOF camera acquires multiple focal plane images, it first uses the first and second focal plane images as a set of input images for focus determination, obtaining the focus decision map, focus confidence map, and fusion result for the current round. Then, this fusion result is compared with the next focal plane image to form a new input image set, and the pixel-level focus index matrix is ​​updated based on the new focus decision result. Through this iterative process, the pixel-level focus index matrix records the optimal focus source image number for each pixel position among the multiple focal plane images.

[0042] Specifically, it includes the following sub-steps: S401. In the first round of recursive comparison, the focus source image number corresponding to each pixel position in the focus decision map is written into the pixel-level focus index matrix, and the focus confidence corresponding to the same pixel position is written into the historical confidence matrix. Specifically, in the first round of recursive comparison, the first and second valid images from the standardized multi-focal plane image sequence are input into the Siamese network, and corresponding focus decision maps and focus confidence maps are generated according to step S30. Each pixel position in the focus decision map records the image number selected as the focus source in the current two images, and the same pixel position in the focus confidence map records the confidence value of that focus source.

[0043] For pixel position If the focus decision map shows that the focus source at this location is the first valid image, then the image ID of the first valid image is written into the corresponding position in the pixel-level focus index matrix; if the focus decision map shows that the focus source at this location is the second valid image, then the image ID of the second valid image is written into the corresponding position in the pixel-level focus index matrix. Simultaneously, the focus confidence score corresponding to this pixel location is written into the historical confidence matrix, ensuring a one-to-one spatial correspondence between the pixel-level focus index matrix and the historical confidence matrix.

[0044] After the index is written, pixel-level fusion is performed on the first and second valid images based on the focus decision map. To facilitate pixel-level fusion, the focus decision map is converted into a binary fusion mask. , where, when pixel position When the focus source is the first valid image, The value is 1; when the pixel position When the focus source is the second valid image, The value is 0. Pixel-level blending is performed as follows: ,in, This indicates the pixel position of the fusion result in the current round. Pixel value at; This represents the binary fusion mask obtained by transforming the focused decision graph; Indicates the pixel position of the first valid image. Pixel value at; Indicates the pixel position of the second valid image. The pixel value at that location. When When the value is close to 1, the fusion result is taken from the first valid image; when... When the value is close to 0, the fusion result is taken from the second valid image. This yields the first round of fusion results, which are then used as one of the input images for subsequent recursive comparisons.

[0045] S402. In subsequent recursive comparisons, the previous fusion result and the next focal plane image are input into the Siamese network, and the focus index number of the target pixel is updated according to the current focus confidence, historical confidence, neighborhood consistency condition and confidence update condition in the calibration parameter set. If the update condition is not met, the original focus index number of the target pixel is retained.

[0046] Specifically, after completing the first round of recursive comparison, the fusion result of the previous round is used as one input image for the current round, and the next valid image in the standardized multi-focal plane image sequence is used as another input image for the current round. Both are then input into the Siamese network. After the Siamese network outputs the focus feature map corresponding to the current round, the focus decision map and focus confidence map for the current round are regenerated according to step S30.

[0047] For the target pixel location The system reads the focus source image number corresponding to the current round of focus decision map and the current focus confidence level corresponding to the current round of focus confidence map. Simultaneously, it reads the historical confidence level recorded for that pixel location from the historical confidence matrix and the original focus index number recorded for that pixel location from the pixel-level focus index matrix.

[0048] When the current focus confidence satisfies the confidence update conditions in the calibration parameter set, and the focus source distribution within the target pixel's neighborhood satisfies the neighborhood consistency condition, the pixel-level focus index matrix is ​​updated according to the focus source type given by the current round's focus decision map. If the current round's focus source is the next effective focal plane image, the focus index number at that pixel position in the pixel-level focus index matrix is ​​updated to the image number corresponding to the next effective focal plane image, and the confidence at that pixel position in the historical confidence matrix is ​​updated to the current focus confidence. If the current round's focus source is the fusion result from the previous round, the original focus index number already recorded at that pixel position in the pixel-level focus index matrix is ​​used, and the confidence at that pixel position in the historical confidence matrix is ​​updated according to the current focus confidence. When the current focus confidence does not satisfy the confidence update conditions in the calibration parameter set, or the focus source distribution within the target pixel's neighborhood does not satisfy the neighborhood consistency condition, the original focus index number and original historical confidence at that pixel position are retained.

[0049] After updating the index for the current round, pixel-level fusion is performed based on the focus decision map for that round to obtain a new fusion result. This new fusion result, along with the next valid focal plane image, forms the input image group, and the recursive comparison, index update, and pixel-level fusion process is repeated until all valid images in the standardized multi-focal plane image sequence have been processed. The final pixel-level focus index matrix records the corresponding best focus source image number for each pixel position.

[0050] S50. Construct depth continuity constraints and region correction weights based on the calibration parameter set, and correct abnormal index jumps in the pixel-level focus index matrix to obtain the optimized focus index matrix. In this embodiment, based on the pixel-level focus index matrix obtained in step S40, and combined with the focal plane depth in the calibration parameter set, a depth consistency judgment is made on the index changes between adjacent pixels. For index jump positions that do not conform to the local depth change pattern, a region correction weight is further generated by combining the texture energy, brightness saturation ratio, and neighborhood index concentration of the region where the position is located, and the corresponding focus index number is corrected according to the region correction weight to obtain an optimized focus index matrix.

[0051] S501. Convert the focal plane number of the target pixel and the focal plane number of the neighboring pixels in the pixel-level focus index matrix into calibration depth values ​​respectively, and determine the abnormal index jump pixels according to the calibration depth difference, boundary response value and jump judgment conditions in the calibration parameter set. Specifically, let the pixel-level focus index matrix be... The target pixel position is The neighborhood window of the target pixel is Pixel-level focus index matrix Each pixel location records a focus source image number, which corresponds to a focal plane number. Based on the focal plane depth in the calibration parameter set, the target pixel location is... The corresponding focal plane number is converted to a calibration depth value, and the neighborhood window is... The focal plane number corresponding to each neighboring pixel is converted into the corresponding calibration depth value.

[0052] After depth transformation, the calibration depth difference between the target pixel and its neighboring pixels is calculated. If the calibration depth of the target pixel differs significantly from the calibration depths of most pixels in the neighborhood, it indicates that the target pixel may have an isolated index jump. To distinguish between true structural boundaries and anomalous index jumps, this embodiment also calculates the boundary response value at the target pixel. The boundary response value can be determined by the edge response in the focused feature map, the image gradient response, or the DCN sampling offset response. If the calibration depth difference meets the jump threshold in the calibration parameter set, and the boundary response value does not meet the true boundary conditions, then the target pixel is determined to be an anomalous index jump pixel. Anomalous index jump pixels can be determined according to the following conditions: in, Indicates the target pixel position Its neighboring windows The difference in calibration depth between them; Represents the position of the target pixel in the pixel-level focus index matrix. The calibrated depth value corresponding to the focused source image number; Indicates the location of neighboring pixels Sum the calibration depth values ​​corresponding to the focus source image number; Indicates the number of pixels within the neighborhood window; This represents the threshold value for the transition in the calibration parameter set; Indicates the target pixel position Boundary response value at; Represents the boundary threshold in the calibration parameter set; Indicates an abnormal index jump flag, when This indicates that the target pixel is an anomalous index jump pixel. This indicates that the target pixel is not considered an abnormal index jump pixel.

[0053] In the above judgment logic, the calibration depth difference is used to determine whether there is a sudden change in the depth distribution of the target pixel compared to its neighborhood, and the boundary response value is used to distinguish whether the sudden change corresponds to the true structural boundary. If the calibration depth difference is large but the boundary response value is weak, then the position is more likely to be an abnormal index jump caused by a focusing misjudgment; if the calibration depth difference is large and the boundary response value is strong, then the position can be retained as a depth change at the true structural boundary.

[0054] S502. Generate region correction weights based on the texture energy, brightness saturation ratio, and neighborhood index concentration of the region where the abnormal index jump pixel is located, and correct the focus index number of the abnormal index jump pixel according to the region correction weights to obtain the optimized focus index matrix.

[0055] Specifically, for the target pixel identified as having an abnormal index jump in step S501, the texture energy, brightness saturation ratio, and neighborhood index concentration of the local region where the pixel is located are further extracted. Texture energy is used to characterize the texture richness of the region, brightness saturation ratio is used to characterize whether there is high-brightness reflection or gray-scale saturation in the region, and neighborhood index concentration is used to characterize whether the focus source image numbers of the pixels surrounding the target pixel are concentrated.

[0056] The regional adjustment weights can be calculated as follows: ,in, Indicates the target pixel position Regional adjustment weights; Represents the neighborhood window of the target pixel Average texture energy after normalization of the inner diameter; Represents the neighborhood window of the target pixel Brightness saturation ratio after unified processing of the inner diameter range; Represents the neighborhood window of the target pixel Concentration of neighborhood indexes after unified processing of the inner interval; and This represents the weighting for regional correction calculations within the calibration parameter set. Among these, the brightness saturation ratio... The concentration of the neighborhood index can be determined by the ratio of the number of pixels within the neighborhood window that meet the brightness saturation condition to the total number of pixels in the neighborhood window. The average texture energy can be determined by the proportion of the most frequently occurring image source numbers within the neighborhood window. The lower the value, the more likely the area is to experience unstable weak texture focusing; the higher the brightness saturation ratio, the more likely the area is to be affected by reflections or saturation interference; the higher the neighborhood index concentration, the more consistent the focusing source selection of neighboring pixels.

[0057] After obtaining the region correction weights, for pixels with abnormal index jumps, the neighboring pixels within their neighborhood window that meet the focus confidence condition are read, and the focus source image numbers corresponding to these neighboring pixels are counted. If the region correction weight is greater than the correction threshold in the calibration parameter set, the focus index number of the pixel with abnormal index jumps is corrected to the focus source image number that appears most frequently in the neighborhood and meets the focus confidence condition; if the region correction weight is not greater than the correction threshold, the original focus index number of the pixel is retained.

[0058] After processing all anomalous index jump pixels, an optimized focus index matrix is ​​obtained. The optimized focus index matrix retains the focus source image number corresponding to each pixel position and performs index correction based on depth continuity constraints and region correction weights at anomalous jump positions.

[0059] S60. Generate pixel-level depth data and preliminary 3D reconstruction results based on the optimized focus index matrix, and perform reverse pixel-level source tracing for abnormal reconstruction areas and correct the focus index number, and output the final DOF 3D reconstruction result. In this embodiment, based on the optimized focus index matrix obtained in step S50, the optimal focus source image number corresponding to each pixel position is determined, and pixel-level depth data is generated according to the focal plane depth relationship recorded in the calibration parameter set. Subsequently, a preliminary 3D reconstruction result is formed based on the pixel-level depth data, and reverse pixel-level source tracing and index correction are performed on abnormal regions, finally outputting the DOF 3D reconstruction result. Specifically, the following sub-steps are included: S601. Map the optimal focus source image number of each pixel position in the optimized focus index matrix to the corresponding focal plane calibration depth, and combine the pixel coordinates with the corresponding focal plane calibration depth to generate pixel-level depth data; Specifically, the optimal focus source image number corresponding to each pixel position in the optimized focus index matrix is ​​read, and the corresponding focal plane number is found based on this image number. Since the focal plane number has already established a correspondence with the focal plane depth in the calibration parameter set, the calibration depth value of that pixel position can be determined based on the focal plane number.

[0060] After completing the above mapping, the x-coordinate, y-coordinate, and corresponding calibrated depth value of each pixel location are combined to form pixel-level depth data. For the same surface of the object under test, the pixel-level depth data are arranged according to the image spatial position to form a depth distribution result corresponding to the image coordinates.

[0061] After obtaining pixel-level depth data, the pixel coordinates and depth values ​​can be converted into three-dimensional spatial coordinates according to the camera calibration relationship, and a preliminary three-dimensional point cloud or a preliminary three-dimensional surface model can be generated. This preliminary three-dimensional reconstruction result reflects the depth distribution of the surface of the object under test at each pixel location.

[0062] S602. Determine the reconstruction anomaly region in the preliminary 3D reconstruction result based on the reconstruction anomaly conditions in the calibration parameter set, reverse map the reconstruction anomaly region to the corresponding pixel position, correct the focus index number based on the corresponding pixel position, update the pixel-level depth data, and output the final DOF 3D reconstruction result.

[0063] Specifically, anomaly detection is performed on the preliminary 3D reconstruction results. Detection targets include areas with abrupt changes in local height, isolated points, local faults, and surface continuity anomalies. During detection, based on the reconstruction anomaly conditions in the calibration parameter set, depth changes between adjacent pixels or adjacent 3D points are assessed, and anomaly regions are marked.

[0064] For the marked reconstructed anomalous regions, based on the correspondence between the 3D reconstruction results and pixel-level depth data, the anomalous region is back-mapped to the corresponding pixel position in the pixel-level focus index matrix. Subsequently, the original focus index number, candidate focus confidence score, and neighborhood focus index distribution of the corresponding pixel position are read, and it is determined whether the focus index number of the pixel position needs to be corrected.

[0065] When the original focus index number at the corresponding pixel location is inconsistent with the distribution of focus indices in the neighborhood, and there exists a candidate focal plane image that meets the focus confidence requirement, the focus index number at that pixel location is corrected to the image number corresponding to the candidate focal plane image. When there is no candidate focal plane image that meets the requirement, the original focus index number is retained.

[0066] After correcting the focus index number, the corresponding focal plane depth is read again based on the corrected focus index number, and the pixel-level depth data at that pixel location is updated. Finally, the 3D reconstruction result is reconstructed based on the updated pixel-level depth data, and the final DOF 3D reconstruction result is output.

[0067] Example 2 like Figure 4 As shown, this embodiment provides a multi-focus fusion DOF three-dimensional reconstruction system based on Siamese networks, including: The image sequence construction module is used to acquire multi-focal-plane images of the object under test using a DOF camera, and generate a standardized multi-focal-plane image sequence by combining a calibration parameter set containing focal plane depth, threshold parameters, and weight parameters. Specifically, it includes the following sub-modules: The calibration parameter construction submodule is used to read the acquisition trigger time, focal plane drive position and camera calibration parameters of each image according to the focal plane acquisition sequence, establish the correspondence between image number, focal plane number and focal plane depth, and write the focal plane depth, threshold parameter group and weight parameters into the calibration parameter set.

[0068] The standard sequence generation submodule is used to perform spatial registration, size unification, brightness normalization and noise suppression processing on multiple focal plane images based on the calibration parameter set, and to filter valid images according to the registration threshold to generate a standardized multi-focal plane image sequence.

[0069] The focused feature extraction module is used to input two adjacent multi-focal plane images into a Siamese network with a weight-sharing structure, and generate focused feature maps corresponding to the two images in the same feature space through CNN, FPN, and DCN. Specifically, it includes the following sub-modules: The dual-input configuration submodule is used to input two adjacent multi-focal plane images into the first and second feature extraction branches of the Siamese network, respectively, and to make the two branches use the same convolution kernel parameters, normalization parameters, and feature mapping parameters.

[0070] The feature enhancement output submodule is used to extract basic focusing features through CNN, fuse multi-scale focusing features through FPN, and adjust the sampling position according to the local structural response through DCN in the two feature extraction branches, and output the focusing feature maps corresponding to the two images.

[0071] The focus decision generation module calculates a focus confidence map characterizing the reliability of the depth source based on the focus feature map, and generates a focus decision map by combining focus response differences. Specifically, it includes the following sub-modules: The confidence evidence calculation submodule is used to process the feature response intensity, local texture energy, noise variance, and neighborhood feature consistency of the same pixel position in the focus feature maps of two images, and calculate the focus confidence of the pixel position according to the weight parameters in the calibration parameter set to generate a focus confidence map.

[0072] The decision map generation submodule is used to determine the focus source of a pixel location based on the focus response difference and response judgment conditions of two images at the same pixel location, and associate the focus source with the corresponding focus confidence to generate a focus decision map.

[0073] The index recursive update module performs pixel-level fusion based on the focus decision map and focus confidence map, recursively compares the fusion result with the next focal plane image, and updates the pixel-level focus index matrix. Specifically, it includes the following sub-modules: The initial index writing submodule is used to write the focus source image number corresponding to each pixel position in the focus decision map into the pixel-level focus index matrix in the first round of recursive comparison, and write the corresponding focus confidence into the historical confidence matrix. At the same time, it generates the first round of fusion results based on the focus decision map.

[0074] The recursive index update submodule is used to input the previous round of fusion results and the next focal plane image into the Siamese network in subsequent recursive comparisons, and update the focus index number of the target pixel according to the current focus confidence, historical confidence, neighborhood consistency condition and confidence update condition. If the update condition is not met, the original focus index number of the target pixel is retained.

[0075] The index constraint optimization module is used to construct depth continuity constraints and region correction weights based on the calibration parameter set, and to correct abnormal index jumps in the pixel-level focus index matrix to obtain an optimized focus index matrix. Specifically, it includes the following sub-modules: The jump pixel determination submodule is used to convert the focal plane number of the target pixel and the focal plane number of the neighboring pixels in the pixel-level focus index matrix into calibration depth values, and to determine the abnormal index jump pixels based on the calibration depth difference, boundary response value and jump determination conditions.

[0076] The region weight correction submodule is used to generate region correction weights based on the texture energy, brightness saturation ratio, and neighborhood index concentration of the region where the abnormal index jump pixel is located, and correct the focus index number of the abnormal index jump pixel according to the region correction weight to obtain the optimized focus index matrix.

[0077] The reconstruction feedback correction module generates pixel-level depth data and preliminary 3D reconstruction results based on the optimized focus index matrix. It then performs reverse pixel-level source tracing for reconstructed anomalies and corrects the focus index numbers, outputting the final DOF 3D reconstruction result. Specifically, it includes the following sub-modules: The depth data generation submodule is used to map the best focus source image number of each pixel position in the optimized focus index matrix to the corresponding focal plane calibration depth, and combine the pixel coordinates with the corresponding focal plane calibration depth to generate pixel-level depth data, thereby forming a preliminary 3D reconstruction result.

[0078] The source tracing and reconstruction correction submodule is used to determine the reconstruction anomaly region in the preliminary 3D reconstruction result based on the reconstruction anomaly conditions, reverse map the reconstruction anomaly region to the corresponding pixel position, correct the focus index number based on the corresponding pixel position, update the pixel-level depth data, and output the final DOF 3D reconstruction result.

[0079] Corresponding to the above embodiments, the present invention provides a computer storage medium, including: at least one memory and at least one processor; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute a multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks.

[0080] Corresponding to the above embodiments, this embodiment of the invention provides a computer-readable storage medium containing one or more program instructions, which are executed by a processor to provide a multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks.

[0081] The embodiments disclosed in this invention provide a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed on a computer, the computer performs the above-described multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks.

[0082] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0083] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.

[0084] The storage medium can be memory, such as volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0085] Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0086] Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0087] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0088] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using a combination of hardware and software. When applied as software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0089] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks, characterized in that, Includes the following steps: S10. Acquire multi-focal plane images of the object under test using a DOF camera, and generate a standardized multi-focal plane image sequence by combining a calibration parameter set containing focal plane depth, threshold parameter set, and weight parameters. S20. Input two adjacent multi-focal plane images into a Siamese network with a weight-sharing structure, and generate focal feature maps corresponding to the two images in the same feature space through CNN, FPN and DCN. S30. Calculate a focus confidence map to characterize the reliability of the depth source based on the focus feature map, and generate a focus decision map by combining the focus response differences; S40. Perform pixel-level fusion based on the focus decision map and focus confidence map, recursively compare the fusion result with the next focal plane image, and update the pixel-level focus index matrix; including the following sub-steps: S401. In the first round of recursive comparison, the focus source image number corresponding to each pixel position in the focus decision map is written into the pixel-level focus index matrix, and the focus confidence corresponding to the same pixel position is written into the historical confidence matrix. S402. In the subsequent recursive comparison, the previous fusion result and the next focal plane image are input into the Siamese network, and the focus index number of the target pixel is updated according to the current focus confidence, historical confidence, neighborhood consistency condition and confidence update condition in the calibration parameter set. If the update condition is not met, the original focus index number of the target pixel is retained. S50. Construct depth continuity constraints and region correction weights based on the calibration parameter set, and correct abnormal index jumps in the pixel-level focus index matrix to obtain the optimized focus index matrix. S60. Generate pixel-level depth data and preliminary 3D reconstruction results based on the optimized focus index matrix, perform reverse pixel-level source tracing for abnormal reconstruction areas and correct the focus index number, and output the final DOF 3D reconstruction results.

2. The multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks according to claim 1, characterized in that, Multi-focal-plane images of the object under test are acquired using a DOF camera, and a standardized multi-focal-plane image sequence is generated by combining a calibration parameter set containing focal plane depth, threshold parameter set, and weight parameters, including the following sub-steps: S101. Read the acquisition trigger time, focal plane drive position and camera calibration parameters of each image according to the focal plane acquisition order, and write the focal plane depth, threshold parameter group and weight parameters into the calibration parameter set. S102. Based on the calibration parameter set, perform spatial registration, size unification, brightness normalization and noise suppression processing on multiple images, and filter image data according to the registration threshold in the calibration parameter set to obtain a standardized multi-focal plane image sequence.

3. The multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks according to claim 1, characterized in that, Two adjacent multi-focal plane images are input into a Siamese network with a weight-sharing structure. Focus feature maps corresponding to the two images in the same feature space are generated through CNN, FPN, and DCN, including the following sub-steps: S201. Input two adjacent multi-focal plane images into the first feature extraction branch and the second feature extraction branch respectively. The first feature extraction branch and the second feature extraction branch use the same convolution kernel parameters, normalization parameters and feature mapping parameters. S202. In the first feature extraction branch and the second feature extraction branch, the basic focusing features are extracted by CNN, the multi-scale focusing features are fused by FPN, and the sampling offset is learned by DCN based on the local structural response to obtain the focusing feature maps corresponding to the two images.

4. The multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks according to claim 1, characterized in that, The process involves calculating a focus confidence map characterizing the reliability of the depth source based on the focus feature map, and generating a focus decision map by combining the focus response differences. This includes the following sub-steps: S301. Normalize the feature response intensity, local texture energy, noise variance and neighborhood feature consistency of the same pixel location in the focus feature maps of the two images, and calculate the focus confidence of the pixel location according to the weight parameters in the calibration parameter set. S302. Determine the focus source at the pixel location based on the focus response difference between the two images at the same pixel location and the response judgment conditions in the calibration parameter set, and associate the focus source with the corresponding focus confidence to generate a focus decision map.

5. The multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks according to claim 1, characterized in that, Based on the calibration parameter set, depth continuity constraints and region correction weights are constructed, and abnormal index jumps in the pixel-level focus index matrix are corrected to obtain the optimized focus index matrix, including the following sub-steps: S501. Convert the focal plane number of the target pixel and the focal plane number of the neighboring pixels in the pixel-level focus index matrix into calibration depth values ​​respectively, and determine the abnormal index jump pixels according to the calibration depth difference, boundary response value and jump judgment conditions in the calibration parameter set. S502. Generate region correction weights based on the texture energy, brightness saturation ratio, and neighborhood index concentration of the region where the abnormal index jump pixel is located, and correct the focus index number of the abnormal index jump pixel according to the region correction weights to obtain the optimized focus index matrix.

6. The multi-focus fusion DOF three-dimensional reconstruction method based on Siamese networks according to claim 1, characterized in that, Pixel-level depth data and preliminary 3D reconstruction results are generated based on the optimized focus index matrix. Anomalies in the reconstruction are then traced back to their source at the pixel level, and the focus index numbers are corrected. The final DOF 3D reconstruction result is output, including the following sub-steps: S601. Map the optimal focus source image number of each pixel position in the optimized focus index matrix to the corresponding focal plane calibration depth, and combine the pixel coordinates with the corresponding focal plane calibration depth to generate pixel-level depth data; S602. Determine the reconstruction anomaly region in the preliminary 3D reconstruction result based on the reconstruction anomaly conditions in the calibration parameter set, reverse map the reconstruction anomaly region to the corresponding pixel position, correct the focus index number based on the corresponding pixel position, update the pixel-level depth data, and output the final DOF 3D reconstruction result.

7. A multi-focus fusion DOF three-dimensional reconstruction system based on Siamese networks, characterized in that, include: The image sequence construction module is used to acquire multi-focal plane images of the object under test through a DOF camera, and combine them with a calibration parameter set containing focal plane depth, threshold parameter set and weight parameters to generate a standardized multi-focal plane image sequence. The focus feature extraction module is used to input two adjacent multi-focal plane images into a Siamese network with a weight-sharing structure, and generate focus feature maps corresponding to the two images in the same feature space through CNN, FPN and DCN; The focus decision generation module is used to calculate a focus confidence map that characterizes the reliability of the depth source based on the focus feature map, and to generate a focus decision map by combining the focus response differences. The index recursive update module is used to perform pixel-level fusion based on the focus decision map and focus confidence map, recursively compare the fusion result with the next focal plane image, and update the pixel-level focus index matrix. Specifically, in the first round of recursive alignment, the focus source image number corresponding to each pixel position in the focus decision map is written into the pixel-level focus index matrix, and the focus confidence corresponding to the same pixel position is written into the historical confidence matrix; in subsequent recursive alignments, the previous round of fusion results and the next focal plane image are input into the Siamese network, and the focus index number of the target pixel is updated according to the current focus confidence, historical confidence, neighborhood consistency condition and confidence update condition in the calibration parameter set. When the update condition is not met, the original focus index number of the target pixel is retained. The index constraint optimization module is used to construct depth continuity constraints and region correction weights based on the calibration parameter set, and to correct abnormal index jumps in the pixel-level focus index matrix to obtain an optimized focus index matrix. The reconstruction feedback correction module is used to generate pixel-level depth data and preliminary 3D reconstruction results based on the optimized focus index matrix, and to perform reverse pixel-level source tracing and correction of focus index numbers for abnormal reconstruction areas, and output the final DOF 3D reconstruction results.

Images