Micro-lens array centroid extraction method based on sensor and convolutional neural network

Abstract

The application discloses a microlens array centroid extraction method based on a sensor and a convolutional neural network, and comprises the following steps: collecting a microlens array light spot image and preprocessing to generate a standardized microlens array light spot image; performing array region division and local window interception to generate a light spot local image block set; inputting the light spot local image block set into a DCNv4 array constraint network to generate a light spot depth feature set; generating a light spot region response graph, an initial centroid coordinate set and a light spot boundary marker set based on the light spot depth feature set; generating an effective light spot region set through array constraint matching; generating a corrected centroid coordinate set based on the pixel gray value in the effective light spot region; and performing sorting, abnormality elimination and missing completion on the corrected centroid coordinate set to generate a microlens array centroid extraction result. The application can improve the stability and accuracy of centroid extraction under complex light spot conditions.

Description

Technical Field

[0001] This invention relates to the field of optical sensing and image processing technology, and in particular to a method for extracting the centroid of a microlens array based on sensors and convolutional neural networks. Background Technology

[0002] In recent years, with the development of micro-optical devices, wavefront sensing, light field imaging, and precision optical detection technologies, the application of microlens arrays in optical measurement systems has gradually increased. In existing technologies, image sensors are typically used to acquire images of the light spots formed by the microlens array on the imaging surface. These images undergo preprocessing such as dark field subtraction, flattening correction, noise suppression, and intensity normalization. Then, the light spot regions are divided according to the row and column arrangement of the microlens array. The center coordinates of each light spot are extracted using threshold segmentation, connected component analysis, gray-level weighted centroid method, or curve fitting. Some solutions also incorporate convolutional neural networks to identify light spot regions, assisting in spot localization and screening for abnormal spots.

[0003] However, existing methods for extracting the centroid of microlens arrays still suffer from insufficient localization stability under complex imaging conditions. On the one hand, traditional threshold segmentation and gray-level weighted centroid calculation rely on the integrity of the gray-level distribution of the light spot. When the light spot image has noise interference, uneven light intensity, blurred edges, light spot distortion, or local overlap of adjacent light spots, it is easy to cause incomplete response of the light spot region, inaccurate boundary localization, and centroid coordinate shift. On the other hand, ordinary convolutional neural networks focus more on the local morphology recognition of individual light spots, and do not make sufficient use of the row and column arrangement relationship of the microlens array, the position of the microlens unit, and the spatial continuity of adjacent light spots. The light spot region, center position, and boundary range output by the network are prone to mismatch with the actual microlens unit, and subsequent sub-pixel correction is also difficult to perform stably within the effective light spot area, affecting the accuracy and consistency of the microlens array centroid extraction results. Summary of the Invention

[0004] One objective of this invention is to propose a method for centroid extraction of microlens arrays based on sensors and convolutional neural networks. This invention fully utilizes image sensor imaging, DCNv4 array constraint network, and subpixel correction technology. It describes in detail a complete method for microlens array spot images, from standardization processing, array constraint feature extraction, spot region matching to fine correction of centroid coordinates. It has the advantages of strong adaptability to complex spots, high array matching stability, and high centroid extraction accuracy.

[0005] The method for extracting the centroid of a microlens array based on sensors and convolutional neural networks according to embodiments of the present invention includes the following steps: An image sensor is used to acquire images of the light spots formed by the microlens array on the imaging surface, and the light spot images are preprocessed to generate standardized microlens array light spot images. Based on the row and column arrangement of the microlens array, the standardized microlens array spot image is divided into array regions and local windowed to generate a set of local image blocks of the spot. The set of local image patches of light spots is input into the DCNv4 array constraint network to extract features of light spot morphology and array spatial relationship, and generate a set of light spot depth features. Based on the light spot depth feature set, light spot region identification, center response prediction and boundary response prediction are performed to generate light spot region response map, initial centroid coordinate set and light spot boundary marker set; Array constraint matching is performed on the spot region response map, the initial centroid coordinate set, and the spot boundary marker set to generate an effective spot region set; Based on the set of effective spot regions, local gray-level distribution extraction, light intensity peak localization, edge transition localization and sub-pixel correction are performed on the pixel gray-level values ​​corresponding to the effective spot regions in the standardized microlens array spot image to generate a set of corrected centroid coordinates. The modified centroid coordinate set is sorted, abnormal coordinates are removed, and missing coordinates are filled in according to the row and column order of the microlens array to generate the microlens array centroid extraction result.

[0006] Optionally, the preprocessing includes dark field subtraction, flat field correction, bad pixel repair, background light subtraction, noise suppression, light intensity normalization, geometric distortion correction, pixel coordinate unification, and image scale unification.

[0007] Optionally, the generation of the set of local image patches of the light spot includes: Based on the row and column arrangement of the microlens array, the row direction, column direction, and position of the microlens units are determined in the standardized microlens array spot image. Based on the position of the microlens unit, the standardized microlens array spot image is divided into multiple array regions, so that each array region covers the spot distribution range formed by a microlens unit. A local window is cropped based on the center position of each array region to generate a local spot image patch; The local light spot image patches are arranged according to the row and column order of the microlens array, and the local light spot image patches are associated with the positions of the microlens units to generate a set of local light spot image patches.

[0008] Optionally, the generation of the light spot depth feature set includes: The set of local image patches of light spots is input into the DCNv4 array constraint network. The DCNv4 array constraint network includes an array offset guidance layer, a DCNv4 sampling correction layer, an adjacent unit consistency aggregation layer, and a depth feature output layer. In the array offset guidance layer, array constraint offset is generated based on the window coordinates of the local spot image patch and the position of the microlens unit; In the DCNv4 sampling correction layer, the original sampling offset is generated based on the grayscale changes and edge changes of the local spot image patch. The original sampling offset is fused with the array constraint offset to generate the corrected sampling offset. The DCNv4 deformable convolution processing is performed on the local spot image patch according to the corrected sampling offset to generate spot morphology features. In the adjacent unit consistent aggregation layer, the feature differences between the current local spot image patch and the adjacent local spot image patches are aggregated according to the row and column order of the microlens array to generate array spatial relationship features; In the deep feature output layer, the spot morphology features and array spatial relationship features are fused to generate a set of spot depth features.

[0009] Optionally, the generation of the spot region response map, the initial centroid coordinate set, and the spot boundary marker set includes: The light spot depth feature set is normalized and convolutionally mapped to generate region recognition features, center response features, and boundary response features; Pixel-level response mapping is performed on the region recognition features to generate a spot region response map; In the response map of the spot area, a continuous response region is determined, and the corresponding center response features and boundary response features are retained according to the continuous response region; The retained central response features are mapped to determine the central response position in each continuous response region, and the central response position is converted into coordinates in the standardized microlens array spot image to generate an initial centroid coordinate set. The boundary position is mapped on the preserved boundary response features to determine the boundary range of each continuous response region. The boundary range is then associated with the initial centroid coordinate set according to the row and column order of the local image patch set of the light spot to generate a set of light spot boundary markers.

[0010] Optionally, the generation of the effective spot area set includes: Based on the position of the microlens units and the row and column order of the microlens array, unit matching reference data is generated; Pixel coordinates are extracted from the continuous response region in the spot region response map to generate the coordinate range of the continuous response region. The coordinate range of the continuous response region is correlated with the response value in the spot region response map to generate a set of candidate spot regions; The initial centroid coordinate set is registered with the unit matching reference data, and the lateral and longitudinal offsets of each initial centroid coordinate relative to the position of the microlens unit are calculated to generate matching offset data. The set of light spot boundary markers and the set of candidate light spot regions are overlapped to generate boundary overlap data. Following the row and column order of the microlens array, the adjacent cell continuity of the matching offset data and boundary coincidence data is processed to generate array continuity data. Based on the matching offset data, boundary coincidence data, and array continuity data, the effective spot regions are determined from the candidate spot region set, and the effective spot region set is generated.

[0011] Optionally, the generation of the modified centroid coordinate set includes: Based on the pixel range of each effective spot region in the effective spot region set, the pixel grayscale values ​​within the same pixel range are extracted from the standardized microlens array spot image to generate effective region grayscale data. Based on the horizontal and vertical coordinates of each pixel within the effective spot area, the pixel gray values ​​in the effective area gray data are written to the corresponding coordinate positions to generate a local gray matrix, and the local gray matrix is ​​determined as the local gray distribution. In the local grayscale distribution, the position of the pixel with the largest grayscale value is determined, and the position of the light intensity peak is generated based on the grayscale value changes of the neighboring pixels around the pixel position. Extract the positions where the gray values ​​change from the inside of the spot to the edge of the spot along the horizontal and vertical directions of the effective spot area to generate the edge transition positions; The initial centroid coordinates, peak light intensity position, and edge transition position are transformed to the same local coordinate range. The initial centroid coordinates are then sub-pixel corrected based on the local grayscale distribution to generate corrected centroid coordinates. Arrange the corrected centroid coordinates according to the row and column order of the microlens array to generate a set of corrected centroid coordinates.

[0012] Optionally, the generation of the microlens array centroid extraction result includes: The positions of each corrected centroid coordinate in the corrected centroid coordinate set are correlated with the positions of the microlens units to generate coordinate row and column correlation data. Arrange the corrected centroid coordinates according to the row and column indices in the coordinate row and column association data to generate centroid coordinate sorted data; For cases where multiple corrected centroid coordinates are associated with the same microlens unit position, the lateral and longitudinal offsets of each corrected centroid coordinate relative to the position of the microlens unit are calculated. The corrected centroid coordinate with the smallest offset value is retained and the remaining corrected centroid coordinates are removed to generate abnormal coordinate removal data. Based on the abnormal coordinate elimination data, the position of the microlens unit with unassociated corrected centroid coordinates is determined, and coordinate completion is performed based on adjacent corrected centroid coordinates in the same row and column to generate completed centroid coordinates. The retained corrected and completed centroid coordinates are arranged in the row and column order of the microlens array to generate the centroid extraction result of the microlens array.

[0013] The beneficial effects of this invention are: This invention acquires images of light spots formed by a microlens array on the imaging surface using an image sensor, and preprocesses these images to reduce the impact of dark current, background light, bad pixel noise, uneven light intensity, and geometric distortion on subsequent centroid extraction. This results in standardized microlens array light spot images with a more stable grayscale distribution and spatial coordinate basis. By dividing the array region and extracting local windows based on the row and column arrangement of the microlens array, each local light spot image block is clearly associated with the position of the microlens unit, reducing interference from local overlap of adjacent light spots, edge light spot offset, and non-target responses on the light spot localization process.

[0014] This invention inputs a set of local image patches of a light spot into a DCNv4 array constraint network. Through an array offset guidance layer, a DCNv4 sampling correction layer, and an adjacent unit consistency aggregation layer, joint feature extraction is performed on the light spot morphology and array spatial relationships. This enables the network to not only adapt to the irregular shapes of distorted, offset, and weak-response light spots, but also to constrain the sampling process using the microlens unit positions and the continuity relationships between adjacent light spots. The resulting light spot depth feature set can simultaneously support light spot region identification, center response prediction, and boundary response prediction, improving the stability of light spot regions, initial centroid coordinates, and light spot boundary markers under complex imaging conditions.

[0015] This invention further generates a set of effective spot regions through array constraint matching, and associates the spot region response map, the initial centroid coordinate set, and the spot boundary marker set with the microlens unit position, reducing the number of abnormal, repetitive, and mismatched responses entering the subsequent centroid correction process. Based on the set of effective spot regions, local grayscale distribution extraction, light intensity peak localization, edge transition localization, and sub-pixel correction are performed on the pixel grayscale values ​​corresponding to the effective spot regions in the standardized microlens array spot image. This further refines the centroid coordinates based on the initial network localization, reducing centroid shift caused by uneven light intensity and edge blurring. Through coordinate sorting, abnormal coordinate removal, and missing coordinate completion, a microlens array centroid extraction result with complete row and column structure and consistent coordinate relationships can be formed. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of the microlens array centroid extraction method based on sensors and convolutional neural networks proposed in this invention; Figure 2 This is a schematic diagram illustrating the generation of the effective spot region set in the microlens array centroid extraction method based on sensors and convolutional neural networks proposed in this invention. Figure 3 This is a schematic diagram illustrating the generation of the corrected centroid coordinate set in the microlens array centroid extraction method based on sensors and convolutional neural networks proposed in this invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figures 1-3 A method for extracting the centroid of a microlens array based on sensors and convolutional neural networks includes the following steps: An image sensor is used to acquire images of the light spots formed by the microlens array on the imaging surface, and the light spot images are preprocessed to generate standardized microlens array light spot images. Based on the row and column arrangement of the microlens array, the standardized microlens array spot image is divided into array regions and local windowed to generate a set of local image blocks of the spot. The set of local image patches of light spots is input into the DCNv4 array constraint network to extract features of light spot morphology and array spatial relationship, and generate a set of light spot depth features. Based on the light spot depth feature set, light spot region identification, center response prediction and boundary response prediction are performed to generate light spot region response map, initial centroid coordinate set and light spot boundary marker set; Array constraint matching is performed on the spot region response map, the initial centroid coordinate set, and the spot boundary marker set to generate an effective spot region set; Based on the set of effective spot regions, local gray-level distribution extraction, light intensity peak localization, edge transition localization and sub-pixel correction are performed on the pixel gray-level values ​​corresponding to the effective spot regions in the standardized microlens array spot image to generate a set of corrected centroid coordinates. The modified centroid coordinate set is sorted, abnormal coordinates are removed, and missing coordinates are filled in according to the row and column order of the microlens array to generate the microlens array centroid extraction result.

[0019] In this embodiment, preprocessing includes dark field subtraction, flat field correction, bad pixel repair, background light subtraction, noise suppression, light intensity normalization, geometric distortion correction, pixel coordinate unification, and image scale unification. The process of dark field subtraction is as follows: under the same exposure time, the incident light is blocked, a dark field image is acquired, and then the dark field image is subtracted from the actual acquired spot image to obtain the spot image after dark field subtraction. The flat field correction process is as follows: a flat field image under uniform illumination is acquired with the same exposure time and sensor operating conditions as the spot image; dark field subtraction is performed on the flat field image to generate a dark field subtracted flat field image; the average gray value of the effective pixel area in the dark field subtracted flat field image is calculated to generate a flat field reference gray value; the flat field reference gray value is divided by the gray value of each pixel in the dark field subtracted flat field image to generate a flat field compensation coefficient for each pixel position; the gray value of each pixel in the dark field subtracted spot image is multiplied by the flat field compensation coefficient at the same pixel position to generate a flat field corrected spot image.

[0020] In this embodiment, the generation of the set of local image patches of light spots includes: Based on the row and column arrangement of the microlens array, the row direction, column direction, and microlens unit position of the microlens array are determined in the standardized microlens array spot image; wherein, the row and column arrangement of the microlens array includes the number of rows, the number of columns, the row direction, the column direction, the spacing between adjacent microlens units, and the position of the microlens units. Based on the position of the microlens unit, the standardized microlens array spot image is divided into multiple array regions, so that each array region covers the spot distribution range formed by a microlens unit. A local window is cropped based on the center position of each array region to generate a local spot image patch; The local light spot image patches are arranged according to the row and column order of the microlens array, and the local light spot image patches are associated with the positions of the microlens units to generate a set of local light spot image patches.

[0021] In this embodiment, the generation of the light spot depth feature set includes: The set of local image patches of light spots is input into the DCNv4 array constraint network. The DCNv4 array constraint network includes an array offset guidance layer, a DCNv4 sampling correction layer, an adjacent unit consistency aggregation layer, and a depth feature output layer. The improvements of the DCNv4 array constraint network are specifically reflected in the following aspects: An array offset guidance layer is introduced into the DCNv4 deformable convolutional structure, using the window coordinates of the local spot image patch and the microlens unit position together to generate the array constraint offset. This ensures that the convolution sampling position is no longer automatically determined solely by local grayscale changes, but is simultaneously constrained by the row and column positions of the microlens array. In the DCNv4 sampling correction layer, the original sampling offset obtained based on spot grayscale and edge changes is fused with the array constraint offset to generate a corrected sampling offset. This allows the deformable convolution sampling points to be adjusted in a constrained manner around distorted spots, offset spots, and locally overlapping spots. In the adjacent unit consistent aggregation layer, the feature differences between the current local spot image patch and adjacent local spot image patches are aggregated according to the row and column order of the microlens array, enabling the network to extract the morphological features of individual spots while preserving the array spatial relationship. The training dataset consists of microlens array spot images acquired by an image sensor under different exposure times, light intensity distributions, microlens array orientations, and noise conditions, supplemented by spot images generated from microlens array optical simulations. The training dataset undergoes dark field subtraction, flat field correction, and light intensity normalization. Local spot image patches are extracted according to the row and column arrangement of the microlens array, and each local spot image patch is labeled with a spot region mask, centroid coordinates, spot boundary, and microlens unit position to generate a training sample set. This training sample set is then fed into a DCNv4 array constraint network. An array constraint offset is generated through an array offset guidance layer, a corrected sampling offset is generated through a DCNv4 sampling correction layer, array spatial relationship features are generated through an adjacent unit consistency aggregation layer, and a training spot depth feature set is generated through a depth feature output layer. Based on the training spot depth feature set, spot region identification, center response prediction, and boundary response prediction are performed to generate a training spot region response map, an initial set of centroid coordinates, and a set of training spot boundary markers. The training parameters include batch size, initial learning rate, learning rate decay coefficient, weight decay coefficient, number of training epochs, sampling offset constraint weights, and array consistency constraint weights. The AdamW optimizer is used to iteratively update the network parameters of the DCNv4 array constraint network. The loss function is composed of a weighted average of region loss, center coordinate loss, boundary loss, sampling offset constraint loss, and array consistency loss. When the loss function converges and the centroid coordinate error in the verification sample set satisfies the training termination condition, the trained DCNv4 array constraint network is obtained. In the array offset guidance layer, array constraint offset is generated based on the window coordinates of the local spot image patch and the position of the microlens unit; The array offset guiding layer processes the following: Using the starting pixel of the local spot image patch window as the local coordinate origin, the positions of each pixel within the local spot image patch are organized into a window coordinate matrix; based on the microlens unit positions, the center position of each microlens unit within the local spot image patch is determined, and the center position of each microlens unit is transformed from the standardized microlens array spot image coordinates to the local window coordinate system, generating the local coordinates of the unit center; the horizontal and vertical coordinate differences of each pixel position relative to the local coordinates of the unit center in the window coordinate matrix are calculated to generate a center offset matrix; the center offset is adjusted according to the window width and window height of the local spot image patch. The shift matrix is ​​scaled and normalized to generate a normalized center offset matrix. Based on the window boundaries of the local spot image patch, the normalized center offset matrix is ​​truncated to remove offset components exceeding the local window range, generating a boundary-constrained center offset matrix. The boundary-constrained center offset matrix is ​​channel-expanded according to the number of sampling positions in the DCNv4 deformable convolutional layer, generating a sampling position center offset matrix. The sampling position center offset matrix is ​​convolved to generate horizontal and vertical offset components for each sampling position. The horizontal and vertical offset components are nonlinearly compressed to generate an array-constrained offset. In the DCNv4 sampling correction layer, the original sampling offset is generated based on the grayscale changes and edge changes of the local spot image patch. The original sampling offset is fused with the array constraint offset to generate the corrected sampling offset. The DCNv4 deformable convolution processing is performed on the local spot image patch according to the corrected sampling offset to generate spot morphology features. The DCNv4 sampling correction layer processes the following steps: Local spot image patches are converted into single-channel grayscale matrices, and the grayscale matrices are normalized. A horizontal differential convolution kernel is used to convolve the normalized grayscale matrix, generating horizontal grayscale variation data. A vertical differential convolution kernel is used to convolve the normalized grayscale matrix, generating vertical grayscale variation data. The horizontal and vertical grayscale variation data are concatenated along the channel dimension to generate grayscale variation data. An edge detection convolution kernel is used to extract edge responses from the grayscale variation data, generating edge variation data. The local spot image patches, grayscale variation data, and edge variation data are concatenated along the channel dimension to generate a spot variation representation. A 3x3 convolution, a normalization process, and an activation process are sequentially applied to the spot variation representation to generate offset features. A 1x1 convolution is applied to the offset features, adjusting the number of channels to twice the number of sampling positions in the DCNv4 deformable convolutional layer, generating the original sampling offset. Each sampling position has a horizontal original offset and a vertical original offset. The original sampling offsets and array constraint offsets are arranged according to the same sampling position number to generate sampling offset alignment data. A one-to-one convolution and sigmoid activation process is performed on the sampling offset alignment data to generate offset fusion weights. The original sampling offsets and array constraint offsets are weighted position-by-position according to the offset fusion weights to generate corrected sampling offsets. The base sampling position of the DCNv4 deformable convolutional layer is added to the corrected sampling offset to generate corrected sampling positions. The corrected sampling positions that exceed the boundaries of the local spot image patch are truncated. Bilinear interpolation sampling is performed in the local spot image patch according to the corrected sampling positions to generate sampled pixel values. The sampled pixel values ​​are weighted and aggregated channel-by-channel with the convolution weights of the DCNv4 deformable convolutional layer to generate convolution aggregated features. Normalization and activation processing are performed on the convolution aggregated features to generate spot morphology features. In the adjacent unit consistent aggregation layer, the feature differences between the current local spot image patch and the adjacent local spot image patches are aggregated according to the row and column order of the microlens array to generate array spatial relationship features; The processing of the adjacent unit consistent aggregation layer is as follows: According to the row and column order of the local spot image patch set, row and column indices are established for each local spot image patch to generate local image patch index data; based on the local image patch index data, adjacent local spot image patches are determined to generate adjacent image patch groups; boundary marking is performed on positions located at the edge of the microlens array that lack adjacent local spot image patches to generate effective neighborhood markings; spot morphology features of the current local spot image patch are extracted, and spot morphology features of each adjacent local spot image patch in the adjacent image patch group are extracted to generate adjacent feature groups; the spot morphology features of the current local spot image patch are compared with those of each adjacent feature group. The light spot morphology features are aligned by channel, and the absolute value of the difference between them is calculated to generate directional feature differences. Effective neighborhood labels are used to filter the directional feature differences, removing invalid differences caused by missing neighborhoods at the edges of the microlens array. The filtered directional feature differences are then stitched together in the left, right, up, and down directional order to generate neighborhood difference features. A one-to-one convolution is performed on the neighborhood difference features for channel compression, followed by a three-to-three convolution for local aggregation, generating neighborhood difference aggregated features. The neighborhood difference aggregated features are then stitched together with the light spot morphology features of the current local light spot image patch, and after normalization and activation processing, array spatial relationship features are generated. In the deep feature output layer, the spot morphology features and array spatial relationship features are fused to generate a set of spot depth features.

[0022] In this embodiment, the generation of the spot region response map, the initial centroid coordinate set, and the spot boundary marker set includes: The light spot depth feature set is normalized and convolutionally mapped to generate region recognition features, center response features, and boundary response features; The generation of region recognition features, center response features, and boundary response features specifically includes: extracting the spot depth features of each local spot image patch in the spot depth feature set one by one; calculating the mean and variance of the spot depth features according to the channel dimension; and standardizing the features based on the mean and variance to generate normalized depth features; performing a one-by-one convolution on the normalized depth features to adjust the number of channels to a uniform feature dimension, generating uniform-dimensional depth features; performing a three-by-three convolution, a normalization process, and an activation process on the uniform-dimensional depth features to generate shared mapping features; and performing region convolution mapping, center convolution mapping, and boundary response mapping on the shared mapping features respectively. Boundary convolutional mapping is used, where region convolutional mapping uses a single-layer 1x1 convolution to generate region recognition features to distinguish between spot regions and background regions, center convolutional mapping uses a single-layer 1x1 convolution to generate center response features containing center confidence information, horizontal center offset information, and vertical center offset information, and boundary convolutional mapping uses a single-layer 1x1 convolution to generate boundary response features containing upper boundary information, lower boundary information, left boundary information, and right boundary information; the region recognition features, center response features, and boundary response features are arranged according to the row and column order of the spot local image patch set, maintaining the consistency of the positional relationship between the three types of features and each local spot image patch; Pixel-level response mapping is performed on the region recognition features to generate a spot region response map; The generation of the spot region response map specifically includes: performing a one-to-one convolution on the region recognition features to compress the number of channels of the region recognition features into a single channel, generating the original region response map; performing Sigmoid activation on the original region response map to transform the response value of each pixel position to between zero and one, generating a region response probability map; mapping each pixel position in the region response probability map to the pixel position within the local spot image patch according to the window coordinates of the local spot image patch, generating a local region response map; filling the local region response map into the coordinate range of the standardized microlens array spot image based on the window position of the local spot image patch in the standardized microlens array spot image; averaging and fusing the response values ​​in the overlapping areas of adjacent local spot image patches to generate a stitched region response map; and arranging the response values ​​in the stitched region response map according to the row and column order of the microlens array to generate the spot region response map. In the response map of the spot region, continuous response regions are identified, and the corresponding center response features and boundary response features are preserved according to the continuous response regions. Specifically, the response map of the spot region is divided into local region response maps corresponding to each local spot image patch according to the row and column order of the local image patch set. The pixel position with the largest response value is determined in each local region response map to generate the region center response position. Starting from the region center response position, the response values ​​in the local region response map are connected and labeled to generate continuous response regions. Each pixel position in the continuous response region is mapped to the same spatial position in the center response feature, and the center response features at the same spatial position are extracted to generate the preserved center response features. Each pixel position in the continuous response region is mapped to the same spatial position in the boundary response feature, and the boundary response features at the same spatial position are extracted to generate the preserved boundary response features. The continuous response regions, the preserved center response features, and the preserved boundary response features are associated and organized according to the row and column order of the local image patch set. The retained center response features are mapped to determine the center response position within each continuous response region, and the center response position is converted into coordinates in the standardized microlens array spot image to generate an initial centroid coordinate set. Specifically, the retained center response features are split into channels to obtain a center response channel, a horizontal center offset channel, and a vertical center offset channel. The center response channel is subjected to a one-to-one convolution and sigmoid activation to generate a center response map. The pixel position with the largest response value in the center response map within each continuous response region is selected to generate the center peak pixel position. The horizontal and vertical center offset values ​​of the center peak pixel position are extracted from the horizontal and vertical center offset channels. The coordinates of the center peak pixel position, the horizontal center offset value, and the vertical center offset value are summed to generate a local center response position. Based on the window coordinates of the local spot image patch, the local center response position is converted to the coordinate range of the standardized microlens array spot image to generate initial centroid coordinates. The initial centroid coordinates are arranged according to the row and column order of the spot local image patch set to generate an initial centroid coordinate set. The boundary position is mapped on the preserved boundary response features to determine the boundary range of each continuous response region. The boundary range is then associated with the initial centroid coordinate set according to the row and column order of the spot local image patch set to generate a spot boundary marker set. Boundary location mapping specifically includes: channel splitting of the preserved boundary response features to generate upper boundary distance data, lower boundary distance data, left boundary distance data, and right boundary distance data; extracting boundary distance values ​​with the same spatial location from the upper boundary distance data, lower boundary distance data, left boundary distance data, and right boundary distance data according to the pixel positions within the continuous response region; aggregating the upper boundary distance values, lower boundary distance values, left boundary distance values, and right boundary distance values ​​within the same continuous response region to generate upper boundary distance, lower boundary distance, left boundary distance, and right boundary distance; and transforming the initial centroid coordinates associated with the continuous response region in the initial centroid coordinate set to the coordinate range of the local spot image patch. Within the local area, generate local initial centroid coordinates; subtract the upper boundary distance from the vertical coordinate of the local initial centroid coordinates to generate the local upper boundary position; add the lower boundary distance to the vertical coordinate of the local initial centroid coordinates to generate the local lower boundary position; subtract the left boundary distance from the horizontal coordinate of the local initial centroid coordinates to generate the local left boundary position; add the right boundary distance to the horizontal coordinate of the local initial centroid coordinates to generate the local right boundary position; combine the local upper boundary position, local lower boundary position, local left boundary position, and local right boundary position to generate the local boundary range; based on the window coordinates of the local spot image patch, transform the local boundary range to the coordinate range of the standardized microlens array spot image to generate the boundary range.

[0023] In this embodiment, the generation of the effective spot area set includes: Based on the position of the microlens units and the row and column order of the microlens array, unit matching reference data is generated; The generation of unit matching reference data specifically includes: numbering the row and column positions of each microlens unit according to the row and column order of the microlens array to generate unit row and column index data; associating the horizontal and vertical coordinates of each microlens unit position with the unit row and column index data to generate unit center reference coordinate data; generating row-to-row unit spacing data based on the horizontal coordinate difference between adjacent microlens unit positions within the same row; generating column-to-column unit spacing data based on the vertical coordinate difference between adjacent microlens unit positions within the same column; determining the unit region range of each microlens unit in the standardized microlens array spot image based on the unit center reference coordinate data, row-to-row unit spacing data, and column-to-column unit spacing data; associating each microlens unit position with adjacent microlens unit positions according to the left, right, top, and bottom row and column adjacency relationships to generate adjacent unit position data; and organizing the unit row and column index data, unit center reference coordinate data, unit region range, and adjacent unit position data according to the row and column order of the microlens array to generate unit matching reference data. Pixel coordinates are extracted from the continuous response region in the spot region response map to generate the coordinate range of the continuous response region. The coordinate range of the continuous response region is correlated with the response value in the spot region response map to generate a set of candidate spot regions; The initial centroid coordinate set is registered with the unit matching reference data, and the lateral and longitudinal offsets of each initial centroid coordinate relative to the position of the microlens unit are calculated to generate matching offset data. The generation of matching offset data specifically includes: extracting the lateral and longitudinal coordinates of each initial centroid coordinate from the initial centroid coordinate set; extracting the position, center coordinate, and row / column index data of each microlens unit from the unit matching reference data; subtracting the lateral coordinate of each initial centroid coordinate from the lateral coordinate of each microlens unit position to generate the lateral coordinate difference; subtracting the longitudinal coordinate of the same initial centroid coordinate from the longitudinal coordinate of each microlens unit position to generate the longitudinal coordinate difference; squaring each set of lateral and longitudinal coordinate differences, and then squaring the same set of differences. The results are summed to generate registration distance values. These registration distance values ​​are then sorted from smallest to largest according to the same initial centroid coordinates. The microlens unit position corresponding to the first registration distance value is determined as the matched microlens unit position. The lateral coordinate difference between the initial centroid coordinates and the matched microlens unit position is determined as the lateral offset, and the longitudinal coordinate difference is determined as the longitudinal offset. The initial centroid coordinates, matched microlens unit positions, unit row and column index data, lateral offset, and longitudinal offset are recorded together to generate matching offset data. The set of light spot boundary markers and the set of candidate light spot regions are overlapped to generate boundary overlap data. The generation of boundary overlap data specifically includes: extracting the boundary range of each spot from the spot boundary marker set; combining the upper boundary, lower boundary, left boundary, and right boundary positions of the spot boundary range into a bounding box region; extracting the pixel coordinates contained in each candidate spot region from the candidate spot region set to generate the candidate region pixel range; unifying the bounding box region and the candidate region pixel range to the coordinate range of the standardized microlens array spot image; comparing the pixel coordinates of the bounding box region and the candidate region pixel range, identifying pixels that are simultaneously located within both the bounding box region and the candidate region pixel range as overlapping pixels, and generating an overlapping pixel set; counting the number of pixels within the bounding box region, the number of pixels within the candidate region pixel range, and the number of pixels within the overlapping pixel set; and associating and organizing the overlapping pixel set, the number of pixels within the bounding box region, the number of pixels within the candidate region pixel range, and the number of pixels within the overlapping pixel set to generate boundary overlap data. Following the row and column order of the microlens array, the adjacent cell continuity of the matching offset data and boundary coincidence data is processed to generate array continuity data. The generation of array continuity data specifically includes: based on the cell row and column index data, organizing the matching offset data and boundary coincidence data into a row and column structure; for adjacent microlens cell positions within the same row, extracting lateral offset, vertical offset, and boundary coincidence data, calculating the offset changes and boundary coincidence changes between adjacent positions, and generating row-oriented continuous records; for adjacent microlens cell positions within the same column, extracting lateral offset, vertical offset, and boundary coincidence data, calculating the offset changes and boundary coincidence changes between adjacent positions, and generating column-oriented continuous records; associating and organizing the row-oriented continuous records and column-oriented continuous records according to the row and column order of the microlens array to generate array continuity data; Based on matching offset data, boundary coincidence data, and array continuity data, effective spot regions are determined from the candidate spot region set, and an effective spot region set is generated. The generation of the effective spot region set specifically includes: extracting the lateral and longitudinal offsets associated with each candidate spot region from the matching offset data, taking the absolute values ​​of the lateral and longitudinal offsets respectively, and adding the two absolute values ​​to generate a position offset evaluation value; extracting the number of pixels in the overlapping pixel set, the number of pixels in the bounding box area, and the number of pixels within the candidate region pixel range from the boundary overlap data, calculating the proportion of the number of pixels in the overlapping pixel set within the bounding box area and the proportion within the candidate region pixel range respectively, and generating a boundary overlap evaluation value; extracting the continuous arrangement records between the candidate spot regions and adjacent candidate spot regions in the row and column directions from the array continuity data, and generating an array continuity evaluation value; sorting the candidate spot regions associated with the same microlens unit position according to the order of position offset evaluation value from small to large, boundary overlap evaluation value from large to small, and array continuity evaluation value from large to small; determining the candidate spot region at the top of the sorted list as the effective spot region at that microlens unit position; and arranging the effective spot regions at each microlens unit position according to the row and column order of the microlens array to generate an effective spot region set.

[0024] In this embodiment, the generation of the corrected centroid coordinate set includes: Based on the pixel range of each effective spot region in the effective spot region set, the pixel grayscale values ​​within the same pixel range are extracted from the standardized microlens array spot image to generate effective region grayscale data. Based on the horizontal and vertical coordinates of each pixel within the effective spot area, the pixel gray values ​​in the effective area gray data are written to the corresponding coordinate positions to generate a local gray matrix, and the local gray matrix is ​​determined as the local gray distribution. In the local grayscale distribution, the position of the pixel with the largest grayscale value is determined, and the position of the light intensity peak is generated based on the grayscale value changes of the neighboring pixels around the pixel position. The generation of the peak intensity position is specifically as follows: Traverse the grayscale values ​​of each pixel in the local grayscale distribution, and determine the pixel position with the largest grayscale value as the peak reference pixel position; Using the peak reference pixel position as the center, extract the grayscale values ​​of the left, right, top, and bottom adjacent pixels, as well as the four diagonally adjacent pixels, to generate peak neighborhood grayscale data; Based on the peak reference pixel position, the grayscale values ​​of the left and right adjacent pixels, generate horizontal grayscale variation data; Based on the peak reference pixel position, the grayscale values ​​of the top and bottom adjacent pixels, generate vertical grayscale variation data; Calculate the peak offset of the peak reference pixel position in the horizontal direction based on the horizontal grayscale variation data, and calculate the peak offset of the peak reference pixel position in the vertical direction based on the vertical grayscale variation data; Add the horizontal coordinate of the peak reference pixel position to the horizontal offset to generate the peak horizontal coordinate; Add the vertical coordinate of the peak reference pixel position to the vertical offset to generate the peak vertical coordinate; Combine the peak horizontal coordinate and peak vertical coordinate to generate the peak intensity position. Extract the positions where the gray values ​​change from the inside of the spot to the edge of the spot along the horizontal and vertical directions of the effective spot area to generate the edge transition positions; The generation of edge transition positions is specifically as follows: starting from the peak position of light intensity, extract the pixel grayscale values ​​in the local grayscale distribution pixel by pixel along the left, right, up, and down directions of the effective light spot area until the boundary of the effective light spot area is reached, generating left-side grayscale sequences, right-side grayscale sequences, up-side grayscale sequences, and down-side grayscale sequences; calculate the difference between the grayscale values ​​of two adjacent pixels in the left-side, right-side, up-side, and down-side grayscale sequences, and take the absolute value of the difference calculation result to generate left-side grayscale change sequences, right-side grayscale change sequences, up-side grayscale change sequences, and down-side grayscale change sequences. The process involves several steps: First, determining the position of maximum grayscale change in the left-direction grayscale change sequence to generate the left edge transition position. Then, determining the position of maximum grayscale change in the right-direction grayscale change sequence to generate the right edge transition position. Next, determining the position of maximum grayscale change in the upward-direction grayscale change sequence to generate the upper edge transition position. Finally, determining the position of maximum grayscale change in the downward-direction grayscale change sequence to generate the lower edge transition position. These left, right, upper, and lower edge transition positions are then combined according to their coordinates within the local grayscale distribution to generate the final edge transition position. The initial centroid coordinates, peak light intensity position, and edge transition position are transformed to the same local coordinate range. The initial centroid coordinates are then sub-pixel corrected based on the local grayscale distribution to generate corrected centroid coordinates. The sub-pixel correction process is as follows: Subtract the top-left pixel coordinates of the effective spot area from the initial centroid coordinates to generate the initial centroid coordinates within the area; subtract the top-left pixel coordinates of the effective spot area from the peak intensity position to generate the peak intensity position within the area; subtract the top-left pixel coordinates of the effective spot area from the edge transition position to generate the edge transition position within the area; define the grayscale correction range by the left, right, top, and bottom edge transition positions within the area; extract the coordinates and grayscale values ​​of each pixel within the grayscale correction range from the local grayscale distribution to generate the grayscale data for the correction range; extract the minimum grayscale value from the grayscale data for the correction range, subtract the minimum grayscale value from the grayscale values ​​of each pixel to generate the background subtraction grayscale value; multiply the horizontal coordinates of each pixel by the background subtraction grayscale value. The values ​​are summed and then divided by the sum of all background-subtracted grayscale values ​​to generate grayscale-weighted horizontal coordinates. The vertical coordinates of each pixel are multiplied by the background-subtracted grayscale values ​​and summed, then divided by the sum of all background-subtracted grayscale values ​​to generate grayscale-weighted vertical coordinates. The grayscale-weighted horizontal and vertical coordinates are combined to generate the grayscale-weighted center position. The edge horizontal and vertical center positions are calculated based on the edge transition positions within the region and combined to generate the edge constraint center position. The initial centroid coordinates, the light intensity peak position, the grayscale-weighted center position, and the edge constraint center position within the region are weighted and fused to generate the corrected centroid coordinates within the region. The corrected centroid coordinates within the region are added to the upper left corner pixel coordinates of the effective spot area and transformed to the coordinate range of the standardized microlens array spot image to generate the corrected centroid coordinates. Arrange the corrected centroid coordinates according to the row and column order of the microlens array to generate a set of corrected centroid coordinates.

[0025] In this embodiment, the generation of the microlens array centroid extraction result includes: The positions of each corrected centroid coordinate in the corrected centroid coordinate set are correlated with the positions of the microlens units to generate coordinate row and column correlation data. Arrange the corrected centroid coordinates according to the row and column indices in the coordinate row and column association data to generate centroid coordinate sorted data; For cases where multiple corrected centroid coordinates are associated with the same microlens unit position, the lateral and longitudinal offsets of each corrected centroid coordinate relative to the position of the microlens unit are calculated. The corrected centroid coordinate with the smallest offset value is retained and the remaining corrected centroid coordinates are removed to generate abnormal coordinate removal data. Based on the abnormal coordinate elimination data, the position of the microlens unit with unassociated corrected centroid coordinates is determined, and coordinate completion is performed based on adjacent corrected centroid coordinates in the same row and column to generate completed centroid coordinates. The retained corrected and completed centroid coordinates are arranged in the row and column order of the microlens array to generate the centroid extraction result of the microlens array.

[0026] Example 1: To verify the feasibility of this invention in practice, it was applied to a microlens array spot centroid extraction scenario in a micro-optical device testing unit. In this scenario, the object to be tested is a microlens array used for wavefront sensing and light field imaging calibration. The microlens array is mounted on an optical platform, and an image sensor is positioned at the imaging plane to acquire the spot images formed by each microlens unit. During actual testing, due to localized uneven illumination, dark current and dead pixel noise in the sensor, and slight attitude deviations after microlens array assembly, some spots exhibit blurred edges, weak brightness, distorted shapes, and localized overlap between adjacent spots. When using traditional fixed threshold segmentation and gray-scale weighted centroid calculation, the spot area is easily truncated or adheres to adjacent spots. Spots in edge areas are also prone to centroid coordinate shifts due to incomplete response, affecting the stability of subsequent wavefront reconstruction and array calibration.

[0027] In this scenario, on-site inspectors first use an image sensor to acquire images of the light spots formed by the microlens array on the imaging surface. These images are then preprocessed to obtain standardized microlens array light spot images. Subsequently, based on the row and column arrangement of the microlens array, the positions of the microlens units are determined within the standardized microlens array light spot images. The array region is then divided and local windows are extracted around each microlens unit position, forming a set of local light spot image blocks. This set of local light spot image blocks is input into the DCNv4 array constraint network. The array offset guidance layer uses the window coordinates of the local light spot image blocks and the positions of the microlens units to generate array constraint offsets. The DCNv4 sampling correction layer generates the original sampling offset based on grayscale and edge changes, and fuses it with the array constraint offset to obtain the corrected sampling offset. This ensures that the network sampling position can adapt to the morphological changes of distorted, offset, and locally overlapping light spots. The adjacent unit consistency aggregation layer aggregates the feature differences between adjacent local light spot image blocks according to the row and column order of the microlens array. The depth feature output layer fuses the light spot morphological features and array spatial relationship features to generate a set of light spot depth features. After generating a spot region response map, an initial centroid coordinate set, and a spot boundary marker set based on the spot depth feature set, an effective spot region set is obtained through array constraint matching. Based on the effective spot region set, the corresponding pixel gray values ​​in the standardized microlens array spot image are extracted, and local gray-level distribution extraction, light intensity peak localization, edge transition localization, and sub-pixel correction are performed to obtain a corrected centroid coordinate set. Finally, the coordinates are sorted, abnormal coordinates are removed, and missing coordinates are filled in according to the row and column order of the microlens array to form the microlens array centroid extraction result.

[0028] In practical applications, traditional methods tend to misjudge the edges of weak light spots as background, merge adjacent light spots into the same region when encountering overlapping light spots, and shift the centroid towards brighter areas when encountering uneven sensor responses. This invention reduces input errors by standardizing microlens array light spot images, simultaneously learns the light spot morphology and array spatial relationships through a DCNv4 array constraint network, filters out abnormal responses inconsistent with microlens unit positions through array constraint matching, and uses local grayscale distribution, peak light intensity position, and edge transition position to participate in sub-pixel correction, ensuring that the centroid coordinates depend not only on the network's initial prediction but also on the actual grayscale distribution and changes in the light spot boundaries.

[0029] Table 1. Performance Comparison of the Invention Method and Traditional Microlens Array Centroid Extraction Method

[0030] As shown in Table 1, the method of this invention provides a relatively stable improvement in centroid positioning accuracy compared to the traditional method. The average centroid positioning error of the traditional method is 0.186 pixels, while that of the method of this invention is reduced to 0.132 pixels; the root mean square error of centroid positioning of the traditional method is 0.242 pixels, while that of the method of this invention is reduced to 0.176 pixels. This result indicates that the present invention does not rely solely on a fixed threshold and gray-level weighted centroid calculation, but rather, based on the initial centroid coordinate set output by the DCNv4 array constraint network, it combines the local gray-level distribution, peak intensity position, and edge transition position within the effective spot area for sub-pixel correction, enabling the centroid coordinates to better match the actual spot gray-level distribution and reducing centroid shift caused by uneven intensity and local noise.

[0031] Regarding edge spot processing, the average offset of edge spots using traditional methods is 0.298 pixels, while the method of this invention reduces it to 0.219 pixels. Edge spots are often susceptible to distortion at the edge of the imaging field of view, brightness attenuation, and spot shape stretching. Traditional gray-level weighting methods tend to shift the centroid towards locally bright areas. This invention extracts spot morphology features and array spatial relationship features through a DCNv4 array constraint network, and combines this with a set of spot boundary markers for array constraint matching. This allows spot spots in edge regions to be identified and corrected under the constraints of microlens unit positions, resulting in more stable edge spot localization results.

[0032] Regarding the handling of weak light spots and interference from adjacent light spots, the traditional method has a false negative rate of 4.85% for weak light spots, which is reduced to 3.21% by the method of this invention; the traditional method has a false mismatch rate of 6.41% for adjacent light spots, which is reduced to 4.37% by the method of this invention. This improvement mainly comes from the correlation processing between the local image patch set of the light spot and the position of the microlens unit, as well as the array offset guidance and consistent aggregation of adjacent units in the DCNv4 array constraint network. The traditional method is more sensitive to weak response regions and locally overlapping regions, and is prone to false negatives or adhesion when the light spot boundary is unclear; the present invention performs array constraint matching based on the light spot region response map, the initial centroid coordinate set, and the light spot boundary marker set, which can reduce the mismatch between adjacent light spots.

[0033] In terms of missing coordinate completion and overlapping spot recognition, the method of this invention also shows good improvement. The missing coordinate completion error of the traditional method is 0.326 pixels, while that of the method of this invention is reduced to 0.251 pixels; the accuracy rate of overlapping spot region recognition of the traditional method is 91.36%, while that of the method of this invention is improved to 94.28%. This is because after generating the corrected centroid coordinate set, this invention does not directly output the result, but continues to sort the coordinates, remove abnormal coordinates, and complete missing coordinates according to the row and column order of the microlens array, so that the final microlens array centroid extraction result can maintain better row and column consistency. Overall, the performance improvement of the method of this invention is quite reasonable, mainly reflected in the stable recognition under complex spot conditions, array matching, and centroid refinement capabilities.

[0034] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for extracting the centroid of a microlens array based on sensors and convolutional neural networks, characterized in that, Includes the following steps: An image sensor is used to acquire images of the light spots formed by the microlens array on the imaging surface, and the light spot images are preprocessed to generate standardized microlens array light spot images. Based on the row and column arrangement of the microlens array, the standardized microlens array spot image is divided into array regions and local windowed to generate a set of local image blocks of the spot. The set of local image patches of light spots is input into the DCNv4 array constraint network to extract features of light spot morphology and array spatial relationship, and generate a set of light spot depth features. Based on the light spot depth feature set, light spot region identification, center response prediction and boundary response prediction are performed to generate light spot region response map, initial centroid coordinate set and light spot boundary marker set; Array constraint matching is performed on the spot region response map, the initial centroid coordinate set, and the spot boundary marker set to generate an effective spot region set; Based on the set of effective spot regions, local gray-level distribution extraction, light intensity peak localization, edge transition localization and sub-pixel correction are performed on the pixel gray-level values ​​corresponding to the effective spot regions in the standardized microlens array spot image to generate a set of corrected centroid coordinates. The modified centroid coordinate set is sorted, abnormal coordinates are removed, and missing coordinates are filled in according to the row and column order of the microlens array to generate the microlens array centroid extraction result.

2. The method for centroid extraction of microlens arrays based on sensors and convolutional neural networks according to claim 1, characterized in that, The preprocessing includes dark field subtraction, flat field correction, bad pixel repair, background light subtraction, noise suppression, light intensity normalization, geometric distortion correction, pixel coordinate unification, and image scale unification.

3. The method for extracting the centroid of a microlens array based on sensors and convolutional neural networks according to claim 1, characterized in that, The generation of the set of local image patches of light spots includes: Based on the row and column arrangement of the microlens array, the row direction, column direction, and position of the microlens units are determined in the standardized microlens array spot image. Based on the position of the microlens unit, the standardized microlens array spot image is divided into multiple array regions, so that each array region covers the spot distribution range formed by a microlens unit. A local window is cropped based on the center position of each array region to generate a local spot image patch; The local light spot image patches are arranged according to the row and column order of the microlens array, and the local light spot image patches are associated with the positions of the microlens units to generate a set of local light spot image patches.

4. The method for extracting the centroid of a microlens array based on sensors and convolutional neural networks according to claim 1, characterized in that, The generation of the light spot depth feature set includes: The set of local image patches of light spots is input into the DCNv4 array constraint network. The DCNv4 array constraint network includes an array offset guidance layer, a DCNv4 sampling correction layer, an adjacent unit consistency aggregation layer, and a depth feature output layer. In the array offset guidance layer, array constraint offset is generated based on the window coordinates of the local spot image patch and the position of the microlens unit; In the DCNv4 sampling correction layer, the original sampling offset is generated based on the grayscale changes and edge changes of the local spot image patch. The original sampling offset is fused with the array constraint offset to generate the corrected sampling offset. The DCNv4 deformable convolution processing is performed on the local spot image patch according to the corrected sampling offset to generate spot morphology features. In the adjacent unit consistent aggregation layer, the feature differences between the current local spot image patch and the adjacent local spot image patches are aggregated according to the row and column order of the microlens array to generate array spatial relationship features; In the deep feature output layer, the spot morphology features and array spatial relationship features are fused to generate a set of spot depth features.

5. The method for centroid extraction of microlens arrays based on sensors and convolutional neural networks according to claim 1, characterized in that, The generation of the spot region response map, the initial centroid coordinate set, and the spot boundary marker set includes: The light spot depth feature set is normalized and convolutionally mapped to generate region recognition features, center response features, and boundary response features; Pixel-level response mapping is performed on the region recognition features to generate a spot region response map; In the response map of the spot area, a continuous response region is determined, and the corresponding center response features and boundary response features are retained according to the continuous response region; The retained central response features are mapped to determine the central response position in each continuous response region, and the central response position is converted into coordinates in the standardized microlens array spot image to generate an initial centroid coordinate set. The boundary position is mapped on the preserved boundary response features to determine the boundary range of each continuous response region. The boundary range is then associated with the initial centroid coordinate set according to the row and column order of the local image patch set of the light spot to generate a set of light spot boundary markers.

6. The method for centroid extraction of microlens arrays based on sensors and convolutional neural networks according to claim 1, characterized in that, The generation of the effective spot region set includes: Based on the position of the microlens units and the row and column order of the microlens array, unit matching reference data is generated; Pixel coordinates are extracted from the continuous response region in the spot region response map to generate the coordinate range of the continuous response region. The coordinate range of the continuous response region is correlated with the response value in the spot region response map to generate a set of candidate spot regions; The initial centroid coordinate set is registered with the unit matching reference data, and the lateral and longitudinal offsets of each initial centroid coordinate relative to the position of the microlens unit are calculated to generate matching offset data. The set of light spot boundary markers and the set of candidate light spot regions are overlapped to generate boundary overlap data. Following the row and column order of the microlens array, the adjacent cell continuity of the matching offset data and boundary coincidence data is processed to generate array continuity data. Based on the matching offset data, boundary coincidence data, and array continuity data, the effective spot regions are determined from the candidate spot region set, and the effective spot region set is generated.

7. The method for extracting the centroid of a microlens array based on sensors and convolutional neural networks according to claim 1, characterized in that, The generation of the modified centroid coordinate set includes: Based on the pixel range of each effective spot region in the effective spot region set, the pixel grayscale values ​​within the same pixel range are extracted from the standardized microlens array spot image to generate effective region grayscale data. Based on the horizontal and vertical coordinates of each pixel within the effective spot area, the pixel gray values ​​in the effective area gray data are written to the corresponding coordinate positions to generate a local gray matrix, and the local gray matrix is ​​determined as the local gray distribution. In the local grayscale distribution, the position of the pixel with the largest grayscale value is determined, and the position of the light intensity peak is generated based on the grayscale value changes of the neighboring pixels around the pixel position. Extract the positions where the gray values ​​change from the inside of the spot to the edge of the spot along the horizontal and vertical directions of the effective spot area to generate the edge transition positions; The initial centroid coordinates, peak light intensity position, and edge transition position are transformed to the same local coordinate range. The initial centroid coordinates are then sub-pixel corrected based on the local grayscale distribution to generate corrected centroid coordinates. Arrange the corrected centroid coordinates according to the row and column order of the microlens array to generate a set of corrected centroid coordinates.

8. The method for extracting the centroid of a microlens array based on sensors and convolutional neural networks according to claim 1, characterized in that, The generation of the centroid extraction result of the microlens array includes: The positions of each corrected centroid coordinate in the corrected centroid coordinate set are correlated with the positions of the microlens units to generate coordinate row and column correlation data. Arrange the corrected centroid coordinates according to the row and column indices in the coordinate row and column association data to generate centroid coordinate sorted data; For cases where multiple corrected centroid coordinates are associated with the same microlens unit position, the lateral and longitudinal offsets of each corrected centroid coordinate relative to the position of the microlens unit are calculated. The corrected centroid coordinate with the smallest offset value is retained and the remaining corrected centroid coordinates are removed to generate abnormal coordinate removal data. Based on the abnormal coordinate elimination data, the position of the microlens unit with unassociated corrected centroid coordinates is determined, and coordinate completion is performed based on adjacent corrected centroid coordinates in the same row and column to generate completed centroid coordinates. The retained corrected and completed centroid coordinates are arranged in the row and column order of the microlens array to generate the centroid extraction result of the microlens array.

Images