A method for calculating relative depth of handwriting based on reference surface reconstruction

Abstract

The application discloses a handwriting relative depth calculation method based on reference surface reconstruction, comprising the following steps: obtaining two-dimensional brightness data and two-dimensional depth data of the same handwriting sample, aligning the two-dimensional brightness data and the two-dimensional depth data to a unified imaging coordinate reference, performing feature segmentation detection based on the two-dimensional brightness data to obtain a handwriting mask and an uncertain band mask; performing abnormality identification and repair on the two-dimensional depth data to obtain preprocessed depth data, constructing a depth data set of a non-handwriting area according to the handwriting mask, performing consistency screening on the depth data set, and forming depth data of a reference surface area; performing boundary sampling on depth data of the uncertain band mask to obtain boundary depth data, and constructing a reference surface depth model based on the depth data of the reference surface area and the boundary depth data; calculating relative depth data based on the preprocessed depth data and the reference surface depth model, performing threshold stretching on the relative depth data, and then converting the relative depth data into a plurality of pseudo-color maps and outputting the pseudo-color maps.

Description

Technical Field

[0001] This invention relates to the field of handwriting relative depth calculation technology, and in particular to a method for calculating handwriting relative depth based on reference surface reconstruction. Background Technology

[0002] In scenarios such as document examination, forensic evidence collection, handwriting identification, and digital forensics, handwritten text is often accompanied by localized indentations, grooves, and near-surface morphological changes on the paper. Since the text area typically exhibits a more pronounced difference in apparent brightness compared to the background, existing technologies often acquire brightness images and perform threshold segmentation or feature detection to obtain a "text area mask / contour." Based on this, the mask is then mapped onto depth data to extract depth statistics, construct depth feature maps, or perform visualization rendering within the text area.

[0003] However, the aforementioned common technical approaches still face significant systemic constraints in engineering implementation. Specifically, depth and brightness data are often generated from different acquisition callback chains, different post-processing steps, or different coordinate constraints. Even with good mask segmentation, pixel-level offsets or boundary mismatches can still occur when mapping them to the depth matrix. This can lead to situations where the same location is identified as "handwriting" from a brightness perspective, but corresponds to the background or a misaligned area from a depth perspective. Such positioning errors are continuously propagated in subsequent processing chains, affecting the features calculated based on the depth mask (such as depth amplitude distribution, edge transition depth, and local morphological differences) due to positioning offsets, thereby reducing the reliability and interpretability of the depth feature extraction results.

[0004] Furthermore, even if a certain degree of handwriting depth features is obtained under the luminance-guided framework, the existing depth data itself is usually not a "stable quantity that can be directly analyzed." The depth scan output may contain invalid depth values, breakpoints, abnormal reflection points, and inconsistencies in numerical scales due to non-uniform units. If invalid value repair, outlier suppression, and physical quantity conversion steps for the depth matrix are lacking, the depth feature extraction will be superimposed with the effects of noise and outliers; moreover, since the correspondence between the mask and the depth inherently has potential mismatches, the aforementioned noise and mismatches will amplify each other, making it difficult to maintain consistency in results under different samples, different devices, or different scanning parameters.

[0005] Meanwhile, paper is not an ideal plane, and background undulations and creases can easily be mistaken for changes in indentation depth. Furthermore, the reference surface has fewer usable background points and more complex constraints near the handwriting outline, amplifying fitting errors during difference calculations. This leads to a shift in relative depth at the stroke start / end points and line boundaries, resulting in edge depth distortion. Additionally, when multiple characters, dense strokes, or non-convex shapes exist, existing methods relying solely on simple mask neighborhood expansion to construct reference samples can easily introduce non-background information near boundaries, causing local contamination of the reference surface and leading to boundary misconnections. This allows local errors to spread and affect subsequent feature interpretation.

[0006] The purpose of this invention is to design a method for calculating the relative depth of handwriting based on reference surface reconstruction, which addresses the problems existing in the prior art. Summary of the Invention

[0007] In view of this, the purpose of this invention is to propose a method for calculating the relative depth of handwriting based on reference surface reconstruction, which can solve the above-mentioned problems.

[0008] This invention provides a method for calculating the relative depth of handwriting based on reference surface reconstruction, comprising: S1 acquires two-dimensional brightness data and two-dimensional depth data of the same handwriting sample, aligns the two to a unified imaging coordinate reference, and performs feature segmentation detection based on the two-dimensional brightness data to obtain the handwriting mask and the uncertain band mask; S2 performs anomaly identification and repair on 2D depth data to obtain preprocessed depth data. Based on the handwriting mask, it constructs a set of depth data for non-handwriting areas, performs consistency screening on it, and forms depth data for the reference surface area. S3 performs boundary sampling on uncertain masked depth data to obtain boundary depth data, and constructs a reference surface depth model based on the depth data of the reference surface region and the boundary depth data; S4 calculates relative depth data based on preprocessed depth data and a reference surface depth model. After thresholding the relative depth data, it converts it into several pseudo-color images for output.

[0009] The beneficial effects of this invention are: First, by using alignment confidence based on geometric and boundary consistency, the propagation of errors caused by brightness mask boundary misalignment and missed / false detections is reduced. The alignment confidence map quantifies uncertainty into usable weights, thereby improving overall robustness.

[0010] Secondly, abnormal depths such as flying spots, reflective jumps, and occlusion mismatches are identified and repaired to prevent them from contaminating the reference surface model. By filtering and clustering based on the updated non-handwriting set, the probability of handwriting-related depth residues entering the reference surface model is reduced. Spatial clustering and local plane consistency scoring are used to select candidate regions that better match the paper / reference surface.

[0011] Third, candidate sampling is performed along the normal direction in the uncertain zone, and adaptive offset is performed based on the alignment confidence to reduce the reference surface reconstruction error caused by mask boundary deviation. The reference surface is divided into sub-regions, and the mesh step size is adjusted according to the confidence to balance efficiency and accuracy. High-confidence points are retained in the same mesh cell to reduce fitting / interpolation instability caused by depth jumps or repeated mapping. Different modeling algorithms are switched according to the total number of points and the proportion of uncertain zones to improve cross-scene stability.

[0012] Fourth, relative depth is used to highlight the raised / lower information of the handwriting relative to the paper surface, rather than being interfered with by the overall height difference of the paper. Abnormal relative depths are eliminated in the handwriting mask area to avoid extreme values ​​from compromising the dynamic range. A preset pseudo-color mapping is used to stably convert relative depth into visual semantics, facilitating comparison, review, or subsequent feature analysis. This reduces the interference of paper background undulations on handwriting depth analysis, improves the depth recovery accuracy of handwriting edge areas and the two-dimensional display effect, and is suitable for document examination, handwriting identification, and digital forensics scenarios. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings required in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of the method in this embodiment. Detailed Implementation

[0015] To facilitate understanding by those skilled in the art, the structure of the present invention will now be described in further detail with reference to the accompanying drawings. It should be understood that, unless otherwise specified, the order of the steps mentioned in this embodiment can be adjusted according to actual needs, and they can even be executed simultaneously or partially simultaneously.

[0016] like Figure 1 As shown, this embodiment of the invention provides a method for calculating the relative depth of handwriting based on reference surface reconstruction, including: S1 acquires two-dimensional brightness data and two-dimensional depth data of the same handwriting sample, aligns the two to a unified imaging coordinate reference, and performs feature segmentation detection based on the two-dimensional brightness data to obtain the handwriting mask and the uncertain band mask; In this step, handwriting often exhibits depth differences due to variations in ink / indentation or reflection, as well as unevenness on the paper. Furthermore, handwriting boundaries are most susceptible to distortion due to lighting, texture similarity, or threshold segmentation. Abrupt depth changes exist at handwriting edges, while luminance edges do not always correspond to the true spatial edges of the handwriting. Depth reprojection and local geometry / boundary consistency can be used to determine whether the boundary is truly credible in space; however, the handwriting depth data still determines the reliability of the authentication.

[0017] The brightness image can be acquired by any imaging device capable of outputting brightness information, including but not limited to RGB cameras, grayscale cameras, infrared cameras, or combinations thereof. The depth image can be acquired by any device capable of outputting depth information, including but not limited to ToF depth cameras, structured light depth cameras, binocular stereo vision systems, laser scanning, or depth maps reconstructed from multi-view images.

[0018] S101 reprojects the depth data onto the pixel grid of the brightness data according to the calibration parameters to obtain aligned depth data; During the S102 alignment process, pixel-level alignment confidence is calculated based on geometric consistency confidence and boundary consistency confidence to form an alignment confidence map and output alignment brightness data; In the S1021 aligned depth data, the neighborhood of any pixel is selected, and an effective depth set is extracted from the neighborhood. If the number of effective depth sets is less than a preset threshold, the geometric consistency confidence is set to 0; otherwise, the discrete variance of the effective depth set is calculated using the following formula: , in, Represents the effective depth set, This represents the discrete variance of the effective depth set. This represents the variance operation function for a set of samples; S1022 maps the local depth discrete variance to geometric consistency confidence, and the calculation formula is as follows: , in, Indicates the confidence level of geometric consistency. Indicates the scale parameter. Represents an exponential function; In this step, the alignment yields the depth. If the depth in the neighborhood roughly originates from the same object surface, the set variance is small. However, if the region contains occluded edges, mismatched reprojection edges, or depth noise, the neighborhood variance will be large. For example, there might be ink on paper, but the edges may reflect light, or the edges of the writing might correspond to a "depth discontinuity" (e.g., strong gradients caused by ink indentations or sensor errors). You might have detected a "pixel as a writing boundary" in terms of brightness (binary edges are 1), but when reprojected onto the vicinity of that pixel, the depth value fluctuates greatly in the neighborhood (large variance), reducing the confidence in geometric consistency, meaning the depth evidence is unreliable.

[0019] S1023 calculates the gradient magnitude of the aligned brightness data. When the gradient magnitude exceeds the brightness gradient magnitude threshold, a binary brightness edge response map is obtained. S1024 calculates the gradient magnitude of the alignment depth data. When the gradient magnitude exceeds the depth gradient magnitude threshold, a binary depth edge response map is obtained. S1025 calculates the local matching ratio within the neighborhood of a pixel based on the binary brightness edge response map and the binary depth edge response map, and maps the local matching ratio to obtain the boundary consistency confidence. The calculation formula is as follows: , in, Indicates the confidence level of boundary consistency. This represents the binary brightness edge response map. Represents a binary depth edge response map. Represents the row and column coordinates of the neighboring pixel positions. Represented in pixels A local domain set centered on; S1026 performs a weighted fusion of geometric consistency confidence and boundary consistency confidence for each pixel to obtain pixel-level alignment confidence, forming an alignment confidence map.

[0020] In this step, the binary brightness edge response map reflects whether brightness edges appear, and the binary depth edge response map reflects whether depth edges appear. If "brightness edges" and "depth edges" frequently appear simultaneously in the neighborhood, it indicates that the boundary semantics of brightness segmentation are also valid in space.

[0021] For example, paper texture, shadows, and broken strokes cause brightness to detect edges, but depth is actually smooth in that area (there is no obvious spatial transition on the paper surface). , If this happens, the confidence level for boundary consistency will decrease.

[0022] S103 performs threshold segmentation on the aligned brightness data after denoising to obtain the initial mask. Then, it performs connected component filtering, morphological opening and closing operations and hole filling on the initial mask to obtain the handwriting mask. S104 extracts the boundary pixel set from the handwriting mask, calculates the distance from each pixel to the boundary, and defines the expansion radius as increasing as the alignment confidence decreases. When the distance from each pixel to the boundary is less than the expansion radius, an uncertain band mask is generated. The calculation formula is as follows: , in, This represents the expansion radius of each pixel. Indicates minimum bandwidth. Indicates the expansion factor. This represents the alignment confidence level for each pixel. Represents pixel coordinates.

[0023] In this step, the text area at the brightness segmentation point may experience missed / false detections at the edges (due to threshold, lighting, and variations in stroke thickness), or even edge offsets caused by morphological operations. In reality, boundaries are not necessarily ideal hard edges; ink feathering and paper fiber penetration occur. Depth sensing errors and scattering are more pronounced at the boundaries. If the alignment is reliable, the expansion radius is small; if the alignment is unreliable, the expansion radius is large, converting the error risk into a geometric buffer width.

[0024] S2 performs anomaly identification and repair on 2D depth data to obtain preprocessed depth data. Based on the handwriting mask, it constructs a set of depth data for non-handwriting areas, performs consistency screening on it, and forms depth data for the reference surface area. In this step, due to potential issues such as depth noise, flying spots, occlusion, and reflection deviation in the imaging of text and non-text areas, directly fitting a reference surface to the full image depth would cause the reference surface model to be contaminated by text or outliers, thereby reducing the accuracy of subsequent relative depth calculations and pseudo-color image generation. The specific steps are as follows: S201 For any pixel, calculate the median of the effective depth in its local neighborhood. If the difference between the pixel's depth and the median of the neighborhood depth exceeds a preset median threshold, it is marked as an abnormal depth. For pixels with abnormal depth, S202 collects a set of valid depth pixels in their neighborhood, uses the depth data of the region with a confidence level higher than the confidence threshold for interpolation repair, and outputs preprocessed depth data. S203 updates the handwriting mask based on the preprocessed depth data and uses the depth data outside the handwriting mask as the depth data set of the non-handwriting area. In this step, the median is more robust to flypoint / extreme error depths than the mean, preventing abnormal depths from directly participating in reference surface clustering and fitting.

[0025] Some pixels exhibit depth errors, often associated with low confidence levels. Repairing this issue by interpolating only high-confidence pixels can reduce the spread of depth shifts caused by reflections into the reference surface model. Noise is dense at edges; direct interpolation may introduce noise into the fitting of non-text areas. Confidence gating can mitigate this risk.

[0026] S204 performs structured clustering on the depth data set of non-handwriting areas, calculates a consistency score for each cluster, and obtains reference surface depth data.

[0027] In this step, a piece of paper may have locally curved / segmented planes, requiring geometrically similar points to be clustered together; the reference plane may not be a single global plane, and structured clustering allows finding the set of local planes most likely belonging to the reference plane; clusters inconsistent with the reference plane are filtered out using a consistency score. The specific steps are as follows: S2041 back-projects the depth data of the non-text area onto the plane to obtain the corresponding three-dimensional coordinates; S2042 For any two points with three-dimensional coordinates, calculate their spatial proximity distance and depth difference. If any two points satisfy the condition that the spatial proximity distance is less than the spatial proximity threshold and the depth difference is less than the depth difference threshold, then the two points belong to the same candidate cluster and are divided into several candidate clusters. For any cluster, S2043 obtains a fitted local plane by fitting the three-dimensional coordinate points within the cluster based on the least squares plane, calculates the Euclidean distance from each point within the cluster to the fitted plane, and calculates the variance of the Euclidean distance as a consistency score. S2044 If the consistency score of the current cluster meets the preset threshold, the points within the cluster are used as the depth data of the reference surface region; otherwise, the cluster is discarded.

[0028] In this step, the camera's perspective can cause the same depth value to correspond to different spatial locations in the image; only after backprojection does the clustering have true spatial meaning. At the same time, spatial proximity and depth difference constraints ensure that the clusters are both close and share the same depth trend, thus being closer to the same planar region.

[0029] The edges of the paper may have depth variations, and the background may also exist. 2D connected components alone may string together unrelated points, but adding 3D proximity distance and depth difference can prevent incorrect clustering.

[0030] Non-reference areas (such as areas with handwriting indentations or localized bends) result in a more dispersed distribution of distances from points to the fitted plane, leading to greater variance; reference plane areas, on the other hand, have more concentrated points and less variance. This consistency score allows for the early removal of these clusters, preventing them from affecting the accuracy of subsequent reference plane reconstruction.

[0031] S3 performs boundary sampling on uncertain masked depth data to obtain boundary depth data, and constructs a reference surface depth model based on the depth data of the reference surface region and the boundary depth data; In this step, due to issues such as brightness-depth misalignment, mask boundary offset, depth noise, and spatial transition band at the character boundary, directly reconstructing the reference surface using a fixed density / fixed sampling strategy can easily introduce character influence points or depth anomalies at the boundary into the reference model, resulting in amplification of relative depth errors.

[0032] S301 acquires an uncertain zone mask, and on the contour corresponding to the uncertain zone, the sampling step size is adaptively adjusted according to the alignment confidence map to obtain effective boundary depth data; S3011 extracts the uncertain outer contour line with a mask. For each contour point on the outer contour line, it calculates the unit normal vector and generates candidate pixel coordinates along the unit normal vector. The calculation formula is as follows: , in, Indicates the coordinates of the candidate pixel. Represents the pixel coordinates on the outer contour line. This represents the offset distance of the profile normal. Indicates the unit normal direction; For each contour point, S3012 reads the confidence level of the corresponding pixel position in the alignment confidence map, and sets the offset distance based on the confidence level. The calculation formula is as follows: , in, This indicates the maximum allowable offset distance in the low confidence region. This represents the minimum allowable offset distance in the high confidence region. Indicates the alignment confidence map at pixel position Confidence level at the location; S3013 takes the depth data of the retained candidate pixel coordinates as the effective boundary depth data.

[0033] In this step, the normal direction aligns with the local geometric structure of the boundary, limiting the search to the "most likely reference / non-reference boundary direction crossing the boundary." This avoids introducing numerous irrelevant points due to tangential sampling, reducing computational and artifact risks. High confidence indicates more reliable brightness-depth alignment of the pixel, requiring a smaller offset range to minimize the risk of crossing into the handwriting side. Low confidence allows for a wider search within the normal direction to compensate for boundary offsets caused by alignment errors.

[0034] S302 acquires the reference surface region mask, and performs adaptive mesh sampling on the sampling density based on the alignment confidence map to obtain effective reference surface depth data; S3021 obtains the reference surface region mask, divides it into several sub-reference surface regions, and selects the confidence level of the center pixel of the sub-reference surface region as the confidence level of the current sub-reference surface region; S3022 calculates the mesh step size of the current sub-reference surface region based on the confidence level of the current sub-reference surface region, using the following formula: , in, This indicates the range of values ​​for the adaptive grid step size. Indicates the grid step size. This indicates the confidence level of the current sub-reference surface region. Indicates the nonlinear adjustment parameter; S3023 starts from the current sub-reference surface region and generates candidate sampling coordinates according to the grid step size of the current position; S3024 If multiple candidate sampling coordinates are mapped to the same grid cell / same sampling coordinates, the depth data at the pixel coordinates with higher confidence is retained to obtain the effective depth data at each grid cell, which is used as the effective reference surface depth data.

[0035] In this step, the reference surface is not necessarily globally flat and uniform: the paper may have local curvature; using a fixed sampling step size will lead to: undersampling in areas with high local curvature (increasing reconstruction error) and oversampling in areas with low local curvature but high confidence (decreasing efficiency). Sub-region division reduces the complexity of confidence modeling, and the confidence of the center pixel can usually represent the alignment reliability of that local area.

[0036] Nonlinear power To adjust the sensitivity of confidence level changes to sampling density, the step size is reduced when the confidence level is low, thus giving the reference surface model stronger constraints in the boundary neighborhood. Generating sampling points according to the step size ensures that the distribution of sampling points conforms to the adaptive strategy, rather than uniformly scattering points, ensuring that computational resources are concentrated in the required areas.

[0037] S303 calculates the total number of sampling points and the proportion of sampling points with uncertain masking. Based on the total number of sampling points and the proportion of sampling points with uncertain masking, the effective reference surface depth data and the effective boundary depth data are reconstructed on a grid coordinate system of the same scale as the preprocessed depth data to obtain the reference surface depth model.

[0038] S3031 If it is uncertain whether the proportion of masked sampling points is greater than or equal to the proportion threshold and the total number of sampling points is greater than or equal to the total number of sampling points threshold, then the Kriging algorithm is used to reconstruct the reference surface. S3032 If it is uncertain whether the proportion of masked sampling points is greater than or equal to the proportion threshold and the total number of sampling points is less than the total number of sampling points threshold, then the neighborhood enhancement fitting algorithm is used to reconstruct the reference surface. S3033 If it is uncertain whether the proportion of masked sampling points is less than the proportion threshold and the total number of sampling points is greater than or equal to the total number of sampling points threshold, then the triangulation interpolation algorithm is used to reconstruct the reference surface. If it is uncertain whether the proportion of masked sampling points is less than the proportion threshold and the total number of sampling points is less than the total number of sampling points threshold, then a linear interpolation algorithm is used to reconstruct the reference surface.

[0039] In this step, different reconstruction methods are suitable for different "sampling quality / spatial coverage conditions": the Kriging algorithm relies on statistical smoothing and spatial correlation, and is suitable for medium to high data density; the neighborhood enhancement fitting algorithm improves fitting stability through local enhancement when data is insufficient or unevenly distributed; the triangulation interpolation algorithm relies on geometric connectivity and local neighborhoods, and can use triangulation networks for local interpolation when data is scarce; the linear interpolation algorithm provides usable baseline results when data is minimal or coverage is extremely sparse. All of the above algorithms are existing technologies, and will not be elaborated upon in this invention.

[0040] For example, if the handwriting is thick and the proportion of uncertain boundaries is high, the boundary sampling ratio is high, and the data distribution may be more biased towards the boundaries and transition regions. In this case, a more conservative algorithm or one that relies more on local fitting is needed to avoid being dominated by boundary noise when using global statistical methods. In addition, if the shooting distance is far and increases depth noise, it means that the total number of effective points may decrease. After triggering the threshold condition, switch to an interpolation / fitting strategy that is more suitable for sparse data.

[0041] S4 calculates relative depth data based on preprocessed depth data and a reference surface depth model. After thresholding the relative depth data, it converts it into several pseudo-color images for output.

[0042] In this step, due to the noise, outliers, and residual errors between the reference surface model and the actual paper surface in depth sensing, directly linearly stretching or visualizing the relative depth of the entire image can easily lead to local saturation, insufficient contrast, and outliers dominating the dynamic range, thereby reducing the ability and stability of the pseudo-color image to express the text layers.

[0043] S401 subtracts the depth data of the reference surface depth model from the preprocessed depth data to calculate the relative depth data, extracts the relative depth data of the handwriting mask area, and removes the relative depth data that exceeds the outlier threshold. In this step, relative depth removes the overall height difference of the paper surface, retaining only the degree of protrusion of the text relative to the paper. For example, near the edges of the text, a few pixels may experience abrupt depth jumps due to ToF reflection. After relative depth calculation, these pixels will exhibit extreme positive / negative values. After removal, the pseudocolor image will not have isolated bright / dark spots dominating the overall color gradation range. If there is local occlusion at the strokes of the text causing depth errors, the relative depth anomaly will deviate significantly. Removal prevents erroneous depth from stretching the entire image.

[0044] S402 calculates the maximum and minimum values ​​of the relative depth data after removing outliers, and maps the relative depth data to the complete display grayscale range based on the maximum and minimum values ​​to obtain the grayscale matrix of the entire image. S403 If the grayscale matrix is ​​less than the first grayscale threshold, then compress and map it to the lower display range; If the grayscale matrix is ​​greater than the second grayscale threshold, then S404 will compress and map it to the high-end display range; S405 If the first grayscale threshold ≤ grayscale matrix ≤ second grayscale threshold, then stretch and map to the complete display grayscale range; S406 generates a pseudo-color image by mapping the grayscale matrix after secondary stretching according to a preset pseudo-color table.

[0045] This step aims to improve effective contrast and suppress saturation. Essentially, it's a robust contrast enhancement and outlier suppression strategy. For excessively dark / negatively abnormal or near-background depths, its grayscale proportion is limited to prevent noise from occupying a large visual space at the low end. For excessively bright / positively extreme areas (potentially caused by residual outliers or local depth abrupt changes), their dominance in the display dynamic range is limited to prevent the subject's hierarchy from being compressed. The main distribution (usually corresponding to the true text hierarchy) is maximized, thereby enhancing visual sensitivity. In the pseudocolor image, different relative depths correspond to different color segments, making the convex / concave layering of the text more intuitive, facilitating subsequent text identification and display.

[0046] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0047] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0048] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0049] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0050] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The words first, second, and third, etc., do not indicate any order. These words can be interpreted as names.

[0051] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0052] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0053] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0054] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

Claims

1. A method for calculating the relative depth of handwriting based on reference surface reconstruction, characterized in that, include: S1 acquires two-dimensional brightness data and two-dimensional depth data of the same handwriting sample, aligns the two to a unified imaging coordinate reference, and performs feature segmentation detection based on the two-dimensional brightness data to obtain the handwriting mask and the uncertain band mask; S2 performs anomaly identification and repair on 2D depth data to obtain preprocessed depth data. Based on the handwriting mask, it constructs a set of depth data for non-handwriting areas, performs consistency screening on it, and forms depth data for the reference surface area. S3 performs boundary sampling on uncertain masked depth data to obtain boundary depth data, and constructs a reference surface depth model based on the depth data of the reference surface region and the boundary depth data; S4 calculates relative depth data based on preprocessed depth data and a reference surface depth model. After thresholding the relative depth data, it converts it into several pseudo-color images for output.

2. The method for calculating the relative depth of handwriting based on reference surface reconstruction according to claim 1, characterized in that, The process of acquiring two-dimensional brightness data and two-dimensional depth data of the same handwriting sample, aligning them to a unified imaging coordinate reference, and performing feature segmentation detection based on the two-dimensional brightness data to obtain a handwriting mask and an uncertain band mask includes: S101 reprojects the depth data onto the pixel grid of the brightness data according to the calibration parameters to obtain aligned depth data; During the S102 alignment process, pixel-level alignment confidence is calculated based on geometric consistency confidence and boundary consistency confidence to form an alignment confidence map and output alignment brightness data; S103 performs threshold segmentation on the aligned brightness data after denoising to obtain the initial mask. Then, it performs connected component filtering, morphological opening and closing operations and hole filling on the initial mask to obtain the handwriting mask. S104 extracts the boundary pixel set from the handwriting mask, calculates the distance from each pixel to the boundary, and defines the expansion radius as increasing as the alignment confidence decreases. When the distance from each pixel to the boundary is less than the expansion radius, an uncertain band mask is generated. The calculation formula is as follows: ， in, This represents the expansion radius of each pixel. Indicates minimum bandwidth. Indicates the expansion factor. This represents the alignment confidence level for each pixel. Represents pixel coordinates.

3. The method for calculating the relative depth of handwriting based on reference surface reconstruction according to claim 2, characterized in that, During the alignment process, pixel-level alignment confidence is calculated based on geometric consistency confidence and boundary consistency confidence to form an alignment confidence map. The output alignment brightness data includes: In the S1021 aligned depth data, the neighborhood of any pixel is selected, and an effective depth set is extracted from the neighborhood. If the number of effective depth sets is less than a preset threshold, the geometric consistency confidence is set to 0; otherwise, the discrete variance of the effective depth set is calculated using the following formula: ， in, Represents the effective depth set, This represents the discrete variance of the effective depth set. This represents the variance operation function for a set of samples; S1022 maps the local depth discrete variance to geometric consistency confidence, and the calculation formula is as follows: ， in, Indicates the confidence level of geometric consistency. Indicates the scale parameter. Represents an exponential function; S1023 calculates the gradient magnitude of the aligned brightness data. When the gradient magnitude exceeds the brightness gradient magnitude threshold, a binary brightness edge response map is obtained. S1024 calculates the gradient magnitude of the alignment depth data. When the gradient magnitude exceeds the depth gradient magnitude threshold, a binary depth edge response map is obtained. S1025 calculates the local matching ratio within the neighborhood of a pixel based on the binary brightness edge response map and the binary depth edge response map, and maps the local matching ratio to obtain the boundary consistency confidence. The calculation formula is as follows: ， in, Indicates the confidence level of boundary consistency. This represents the binary brightness edge response map. Represents a binary depth edge response map. Represents the row and column coordinates of the neighboring pixel positions. Represented in pixels A local domain set centered on; S1026 performs a weighted fusion of geometric consistency confidence and boundary consistency confidence for each pixel to obtain pixel-level alignment confidence, forming an alignment confidence map.

4. The method for calculating the relative depth of handwriting based on reference surface reconstruction according to claim 1, characterized in that, The process of anomaly identification and repair of two-dimensional depth data to obtain preprocessed depth data, constructing a depth data set of non-text areas based on a text mask, performing consistency screening on the preprocessed depth data to form the depth data of the reference surface area includes: S201 For any pixel, calculate the median of the effective depth in its local neighborhood. If the difference between the pixel's depth and the median of the neighborhood depth exceeds a preset median threshold, it is marked as an abnormal depth. For pixels with abnormal depth, S202 collects a set of valid depth pixels in their neighborhood, uses the depth data of the region with a confidence level higher than the confidence threshold for interpolation repair, and outputs preprocessed depth data. S203 updates the handwriting mask based on the preprocessed depth data and uses the depth data outside the handwriting mask as the depth data set of the non-handwriting area. S204 performs structured clustering on the depth data set of non-handwriting areas, calculates a consistency score for each cluster, and obtains reference surface depth data.

5. The method for calculating the relative depth of handwriting based on reference surface reconstruction according to claim 4, characterized in that, The structured clustering of the depth data set of non-handwriting areas, and the calculation of a consistency score for each cluster, yields the reference surface depth data, including: S2041 back-projects the depth data of the non-text area onto the plane to obtain the corresponding three-dimensional coordinates; S2042 For any two points with three-dimensional coordinates, calculate their spatial proximity distance and depth difference. If any two points satisfy the condition that the spatial proximity distance is less than the spatial proximity threshold and the depth difference is less than the depth difference threshold, then the two points belong to the same candidate cluster and are divided into several candidate clusters. For any cluster, S2043 obtains a fitted local plane by fitting the three-dimensional coordinate points within the cluster based on the least squares plane, calculates the Euclidean distance from each point within the cluster to the fitted plane, and calculates the variance of the Euclidean distance as a consistency score. S2044 If the consistency score of the current cluster meets the preset threshold, the points within the cluster are used as the depth data of the reference surface region; otherwise, the cluster is discarded.

6. The method for calculating the relative depth of handwriting based on reference surface reconstruction according to claim 1, characterized in that, The step of sampling the uncertain masked depth data at the boundary to obtain boundary depth data, and constructing a reference surface depth model based on the depth data of the reference surface region and the boundary depth data includes: S301 acquires an uncertain zone mask, and on the contour corresponding to the uncertain zone, the sampling step size is adaptively adjusted according to the alignment confidence map to obtain effective boundary depth data; S302 acquires the reference surface region mask, and performs adaptive mesh sampling on the sampling density based on the alignment confidence map to obtain effective reference surface depth data; S303 calculates the total number of sampling points and the proportion of sampling points with uncertain masking. Based on the total number of sampling points and the proportion of sampling points with uncertain masking, the effective reference surface depth data and the effective boundary depth data are reconstructed on a grid coordinate system of the same scale as the preprocessed depth data to obtain the reference surface depth model.

7. The method for calculating the relative depth of handwriting based on reference surface reconstruction according to claim 6, characterized in that, The process of obtaining the uncertain band mask involves adaptively adjusting the sampling step size based on the alignment confidence map on the contour corresponding to the uncertain band to obtain effective boundary depth data, including: S3011 extracts the uncertain outer contour line with a mask. For each contour point on the outer contour line, it calculates the unit normal vector and generates candidate pixel coordinates along the unit normal vector. The calculation formula is as follows: ， in, Indicates the coordinates of the candidate pixel. Represents the pixel coordinates on the outer contour line. This represents the offset distance of the profile normal. Indicates the unit normal direction; For each contour point, S3012 reads the confidence level of the corresponding pixel position in the alignment confidence map, and sets the offset distance based on the confidence level. The calculation formula is as follows: ， in, This indicates the maximum allowable offset distance in the low confidence region. This represents the minimum allowable offset distance in the high confidence region. Indicates the alignment confidence map at pixel position Confidence level at the location; S3013 takes the depth data of the retained candidate pixel coordinates as the effective boundary depth data.

8. The method for calculating the relative depth of handwriting based on reference surface reconstruction according to claim 6, characterized in that, The process of obtaining the reference surface region mask, and adaptively sampling the sampling density based on the alignment confidence map to obtain effective reference surface depth data, includes: S3021 obtains the reference surface region mask, divides it into several sub-reference surface regions, and selects the confidence level of the center pixel of the sub-reference surface region as the confidence level of the current sub-reference surface region; S3022 calculates the mesh step size of the current sub-reference surface region based on the confidence level of the current sub-reference surface region, using the following formula: ， in, This indicates the range of values ​​for the adaptive grid step size. Indicates the grid step size. This indicates the confidence level of the current sub-reference surface region. Indicates the nonlinear adjustment parameter; S3023 starts from the current sub-reference surface region and generates candidate sampling coordinates according to the grid step size of the current position; S3024 If multiple candidate sampling coordinates are mapped to the same grid cell / same sampling coordinates, the depth data at the pixel coordinates with higher confidence is retained to obtain the effective depth data at each grid cell, which is used as the effective reference surface depth data.

9. The method for calculating the relative depth of handwriting based on reference surface reconstruction according to claim 6, characterized in that, The calculation of the total number of sampling points and the proportion of sampling points with uncertain masks, based on the total number of sampling points and the proportion of sampling points with uncertain masks, involves reconstructing the effective reference surface depth data and the effective boundary depth data on a grid coordinate system of the same scale as the preprocessed depth data, resulting in a reference surface depth model including: S3031 If it is uncertain whether the proportion of masked sampling points is greater than or equal to the proportion threshold and the total number of sampling points is greater than or equal to the total number of sampling points threshold, then the Kriging algorithm is used to reconstruct the reference surface. S3032 If it is uncertain whether the proportion of masked sampling points is greater than or equal to the proportion threshold and the total number of sampling points is less than the total number of sampling points threshold, then the neighborhood enhancement fitting algorithm is used to reconstruct the reference surface. S3033 If it is uncertain whether the proportion of masked sampling points is less than the proportion threshold and the total number of sampling points is greater than or equal to the total number of sampling points threshold, then the triangulation interpolation algorithm is used to reconstruct the reference surface. If it is uncertain whether the proportion of masked sampling points is less than the proportion threshold and the total number of sampling points is less than the total number of sampling points threshold, then a linear interpolation algorithm is used to reconstruct the reference surface.

10. The method for calculating the relative depth of handwriting based on reference surface reconstruction according to claim 1, characterized in that, The process of calculating relative depth data based on preprocessed depth data and a reference surface depth model, and then performing threshold stretching on the relative depth data to convert it into several pseudo-color images for output includes: S401 subtracts the depth data of the reference surface depth model from the preprocessed depth data to calculate the relative depth data, extracts the relative depth data of the handwriting mask area, and removes the relative depth data that exceeds the outlier threshold. S402 calculates the maximum and minimum values ​​of the relative depth data after removing outliers, and maps the relative depth data to the complete display grayscale range based on the maximum and minimum values ​​to obtain the grayscale matrix of the entire image. S403 If the grayscale matrix is ​​less than the first grayscale threshold, then compress and map it to the lower display range; If the grayscale matrix is ​​greater than the second grayscale threshold, then S404 will compress and map it to the high-end display range; S405 If the first grayscale threshold ≤ grayscale matrix ≤ second grayscale threshold, then stretch and map to the complete display grayscale range; S406 generates a pseudo-color image by mapping the grayscale matrix after secondary stretching according to a preset pseudo-color table.

Images