A curved surface image correction method and device based on multi-modal data fusion

Abstract

The application provides a curved surface image correction method and device based on multi-modal data fusion, comprising the following steps: S1, projecting a sparse grid laser dot array to a to-be-corrected object, collecting an image of the to-be-corrected object with laser marks, and determining whether it is a high-curvature region according to the interval change of adjacent laser dots; S2, projecting a high-density grid laser dot array to the high-curvature region, collecting the image, obtaining fine depth information of each laser dot to form laser point cloud data; S3, constructing and solving an energy minimization model to generate an inverse distortion mapping field of the corresponding curved surface to a two-dimensional flat plane; and S4, inputting the inverse distortion mapping field to a correction model and outputting a corrected image according to the correction model. The application has the beneficial effects that the laser projection is not affected by the surface texture of the to-be-corrected object and environmental light, the precision and strong robustness are improved, the high-curvature region is pre-screened by using the sparse grid laser, and the working efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of image data processing technology, and in particular relates to a method and apparatus for correcting curved images based on multimodal data fusion. Background Technology

[0002] In scenarios such as ancient book digitization, educational scanning, and mobile office, users often need to photograph open book pages. Due to binding tension and paper flexibility, the pages naturally form non-planar curved surfaces, resulting in severe geometric distortions (such as curved text and uneven line spacing) and texture degradation (such as blurred edges and uneven lighting) in the photographed images. Traditional flatbed scanners cannot handle such scenarios, and existing curved surface image correction methods mostly use direct image correction, which is affected by the surface texture of the object to be corrected and the ambient lighting. Pure visual correction methods are prone to feature extraction failure and inaccurate distortion calculation in scenarios with no texture and low contrast curved surfaces. Summary of the Invention

[0003] In view of this, the present invention aims to propose a method and apparatus for surface image correction based on multimodal data fusion, in order to solve at least one of the above-mentioned technical problems.

[0004] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0005] The first aspect of this invention provides a surface image correction method based on multimodal data fusion, comprising the following steps:

[0006] S1. Project a sparse grid of laser dots onto the object to be corrected, acquire an image of the object marked with laser dots, and determine whether it is a high curvature region based on the change in the spacing between adjacent laser dots:

[0007] S2. Project a high-density grid of laser dots onto the high curvature region, and acquire fine depth information of each laser dot after image acquisition to form laser dot cloud data.

[0008] S3. Collect image data of the object to be corrected and align it with laser point cloud data. Using the three-dimensional elevation surface as the solution target, construct and solve the energy minimization model to obtain the optimal three-dimensional elevation surface and generate the inverse distortion mapping field from the corresponding surface to the two-dimensional flat plane.

[0009] S4. Input the inverse distortion mapping field into the correction model, and output the corrected image according to the correction model.

[0010] Furthermore, S1 includes the following steps:

[0011] S11. Project a sparse grid of laser dots onto the object to be corrected using a laser.

[0012] S12. Acquire images of the object to be corrected with laser markings using an industrial camera, perform grayscale conversion, adaptive threshold segmentation, and morphological noise reduction processing, and extract the ROI region of each laser spot.

[0013] S13. The center coordinates of each laser spot are determined using the gray-scale centroid method, as shown in the following formula:

[0014] ;

[0015] in This represents the grayscale value at pixel (x, y) within the ROI region. , These are the center coordinates of the laser spot;

[0016] S14. Calculate the rate of change of the distance between adjacent laser spots, using the following formula:

[0017] ;

[0018] in, The rate of change of the distance between adjacent laser spots. This represents the change in spacing between adjacent laser beams. This represents the actual distance interval along the laser scanning direction;

[0019] S15. If the rate of change of the distance between adjacent laser spots is greater than the distance threshold, then the region where the laser spot is located is determined to be a high curvature region.

[0020] Furthermore, S1 includes the following steps:

[0021] S11. Project a sparse grid of laser dots onto the object to be corrected using a laser.

[0022] S12. Acquire images of the object to be corrected with laser markings using an industrial camera, perform grayscale conversion, adaptive threshold segmentation, and morphological noise reduction processing, and extract the coordinate data of each laser spot.

[0023] S13. Based on the pre-calibrated laser and camera triangulation model, calculate the three-dimensional spatial coordinates of each laser point, calculate the elevation gradient of adjacent laser points, and if the elevation gradient exceeds the preset curvature threshold, determine that the area is a high curvature area and mark its coordinate range.

[0024] The formula for calculating the elevation gradient is:

[0025] ;

[0026] in, For elevation gradient, The elevation values ​​are those of two adjacent laser points. The X-axis coordinates of two adjacent laser points. The Y-axis coordinates of two adjacent laser points.

[0027] Furthermore, the high curvature region in S2 is the region with the laser point as the center and the side length as the distance between adjacent laser points;

[0028] The sparse grid laser dot matrix is ​​a uniform dot matrix with a size of 5×5.

[0029] The high-density dot matrix is ​​a uniform dot matrix with a size of 20×20.

[0030] Furthermore, S3 includes the following steps:

[0031] S31. Acquire high-resolution image data of the object to be calibrated and align it with the laser point cloud data;

[0032] S32. Construct an energy minimization model with physical prior constraints, including spine continuity constraints and page edge straightness constraints.

[0033] S33. Solve the energy minimization model to obtain the optimal three-dimensional elevation surface, and generate the inverse distortion mapping field from the corresponding surface to the two-dimensional flat plane;

[0034] The energy minimization model is as follows:

[0035] ;

[0036] in, The objective of solving the total energy function model is Take the minimum value. The weights corresponding to the spine continuity constraint, The weights for the page margin straightness constraint, The three-dimensional elevation surface to be solved is the elevation of each point within the region to be corrected. The measured elevation value of the laser point cloud obtained by solving the high-density laser point array for S2 is shown. For spine continuity constraints, This is a constraint on the straightness of page margins.

[0037] Furthermore, the correction model structure in step S4 is as follows:

[0038] The correction model is a pre-trained dual-branch end-to-end neural network, which includes a geometric correction branch, a texture enhancement branch, and a feature fusion module.

[0039] The geometric correction branch takes the inverse distortion mapping field as input, learns and outputs a pixel-level distortion correction function;

[0040] The texture enhancement branch takes the registered high-resolution natural light image as input and performs image deblurring, illumination equalization, and detail enhancement.

[0041] The features output from the two branches are fused across branches through the feature fusion module to output the corrected image.

[0042] A second aspect of the present invention provides a surface image correction device based on multimodal data fusion, comprising:

[0043] An image acquisition module, configured to acquire images of the object to be corrected with laser markings;

[0044] A laser emitting module, configured to project a sparse grid of laser dots toward the object to be corrected;

[0045] The high curvature determination module is configured to determine whether a region is a high curvature region based on the change in the spacing between adjacent laser points. The control module controls the laser emission module to project a high-density grid laser dot array onto the high curvature region, and the control module controls the image acquisition module to acquire the image after the high-density grid laser dot array is projected.

[0046] The data processing module is configured to acquire fine depth information of each laser point to form laser point cloud data, collect image data of the object to be corrected and align it with the laser point cloud data, construct and solve an energy minimization model with a three-dimensional elevation surface as the solution target, obtain the optimal three-dimensional elevation surface, generate the inverse distortion mapping field from the corresponding surface to the two-dimensional flat plane, input the inverse distortion mapping field into the correction model, and output the corrected image according to the correction model.

[0047] A third aspect of the present invention provides an electronic device including a processor and a memory communicatively connected to the processor and used to store processor-executable instructions, the processor being used to perform the method described in the first aspect above.

[0048] A fourth aspect of the present invention provides a server including at least one processor and a memory communicatively connected to the processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the processor to cause the at least one processor to perform the method as described in the first aspect.

[0049] The fifth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0050] Compared with existing technologies, the surface image correction method and apparatus based on multimodal data fusion described in this invention have the following advantages:

[0051] (1) The surface image correction method based on multimodal data fusion described in this invention is not affected by the surface texture of the object to be corrected or the ambient light, thus improving accuracy and robustness. The method uses sparse grid laser to pre-screen high curvature areas, thereby improving work efficiency. Attached Figure Description

[0052] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0053] Figure 1 This is a schematic diagram of the correction method described in an embodiment of the present invention. Detailed Implementation

[0054] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0055] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0056] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0057] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0058] like Figure 1 As shown, a surface image correction method based on multimodal data fusion includes the following steps:

[0059] S1. Project a sparse grid of laser dots onto the object to be corrected, acquire an image of the object marked with laser dots, and determine whether it is a high curvature region based on the change in the spacing between adjacent laser dots:

[0060] S2. Project a high-density grid of laser dots onto the high curvature region, and acquire fine depth information of each laser dot after image acquisition to form laser dot cloud data.

[0061] S3. Collect image data of the object to be corrected and align it with laser point cloud data. Using the three-dimensional elevation surface as the solution target, construct and solve the energy minimization model to obtain the optimal three-dimensional elevation surface and generate the inverse distortion mapping field from the corresponding surface to the two-dimensional flat plane.

[0062] S4. Input the inverse distortion mapping field into the correction model, and output the corrected image according to the correction model.

[0063] In some embodiments, S1 includes the following steps:

[0064] S11. Project a sparse grid of laser dots onto the object to be corrected using a laser.

[0065] S12. Acquire images of the object to be corrected with laser markings using an industrial camera, perform grayscale conversion, adaptive threshold segmentation, and morphological noise reduction processing, and extract the ROI region of each laser spot.

[0066] S13. The center coordinates of each laser spot are determined using the gray-scale centroid method, as shown in the following formula:

[0067] ;

[0068] in This represents the grayscale value at pixel (x, y) within the ROI region. , These are the center coordinates of the laser spot;

[0069] S14. Calculate the rate of change of the distance between adjacent laser spots, using the following formula:

[0070] ;

[0071] in, The rate of change of the distance between adjacent laser spots. This represents the change in spacing between adjacent laser spots, specifically the difference between the measured spacing and the actual distance between two adjacent coordinates in S13. This represents the actual distance interval along the laser scanning direction;

[0072] S15. If the rate of change of the distance between adjacent laser spots is greater than the distance threshold, which ranges from 0.1 to 0.3, then the region where the laser spot is located is determined to be a high curvature region. Using distance calculation for rapid pre-screening is highly efficient.

[0073] k<0.1: The page is basically flat with slight curvature;

[0074] 0.1≤k≤0.2: Normal opening, moderate curvature;

[0075] k>0.2: Significant curvature, large curvature;

[0076] k>0.3: Pages are severely curled or folded at the corners;

[0077] Each laser point has at least two adjacent laser points in four directions. If the rate of change of the distance between laser points in any direction exceeds the threshold, the region where the laser point is located is determined to be a high curvature region.

[0078] The method for determining the range of high curvature regions is as follows:

[0079] Taking the high curvature region in S2 as the laser point whose spacing change rate is greater than the distance threshold as the center, the region is extended by a specified length in the direction it points according to its k value. The formula for extending the specified length is as follows:

[0080] / 2;

[0081] in, To extend to a specified length, The basic calibration value is generally taken as 0.8.

[0082] By extending different lengths based on the rate of change of the spacing between adjacent laser points in different directions, high curvature regions can be defined. When the rate of change of the spacing in a certain direction is large, the deformation in that direction is more severe. Expanding the extension length in that direction not only ensures the accuracy of curvature detection but also controls the detection range and improves work efficiency.

[0083] In other embodiments, S1 includes the following steps:

[0084] S11. Project a sparse grid of laser dots onto the object to be corrected using a laser.

[0085] S12. Acquire images of the object to be corrected with laser markings using an industrial camera, perform grayscale conversion, adaptive threshold segmentation, and morphological noise reduction processing, and extract the coordinate data of each laser spot.

[0086] S13. Based on the pre-calibrated laser and camera triangulation model, calculate the three-dimensional spatial coordinates of each laser point, calculate the elevation gradient of adjacent laser points, and if the elevation gradient exceeds the preset curvature threshold, determine that the area is a high curvature area and mark its coordinate range; the curvature threshold range is 0.1-0.3.

[0087] The formula for calculating the elevation gradient is:

[0088] ;

[0089] in, For elevation gradient, The elevation values ​​are those of two adjacent laser points. The X-axis coordinates of two adjacent laser points. Using the Y-axis coordinates of two adjacent laser points, the three-dimensional elevation gradient enables sub-millimeter-level precise area positioning.

[0090] The sparse grid laser dot matrix is ​​a uniform dot matrix with a size of 5×5;

[0091] The high-density dot matrix is ​​a uniform dot matrix with a size of 20×20 or 50×50.

[0092] In other embodiments, if the elevation gradient exceeds a preset curvature threshold, or the rate of change of the distance between adjacent laser spots is greater than a distance threshold, the region where the laser point is located is determined to be a high curvature region.

[0093] A method of high-density laser scanning in high-curvature areas using sparse grid laser pre-screening was adopted. High-density point cloud acquisition was performed only in the high-curvature areas with the most severe distortion, while low-computing-power sparse sampling was used in non-core distortion areas. This method reduced the amount of laser point cloud data and improved work efficiency while ensuring the sampling accuracy of key distortion areas.

[0094] S3 includes the following steps:

[0095] S31. Acquire high-resolution image data of the object to be calibrated and align it with the laser point cloud data;

[0096] S32. Construct an energy minimization model with physical prior constraints, including spine continuity constraints and page edge straightness constraints.

[0097] S33. Solve the energy minimization model to obtain the optimal three-dimensional elevation surface, and generate the inverse distortion mapping field from the corresponding surface to the two-dimensional flat plane.

[0098] The energy minimization model is as follows:

[0099] ;

[0100] in, The objective of solving the total energy function model is Take the minimum value. The weights corresponding to the spine continuity constraint, The weights for the page margin straightness constraint, The three-dimensional elevation surface to be solved is the elevation of each point within the region to be corrected. The measured elevation value of the laser point cloud obtained by solving the high-density laser point array for S2 is shown. For spine continuity constraints, This is a constraint on the straightness of page margins.

[0101] In some embodiments, The value of is related to the range and number of high curvature regions, and the calculation formula is as follows:

[0102] ;

[0103] in, The basic weight ranges from 0.5 to 1.5, and n is the correlation coefficient for the number of high curvature regions. If there is only one high curvature region, then n is 1. If there is more than one high curvature region, then n = 1 + f / 4, where f is the proportion of high curvature regions. The calculation formula is f = S1 / S2, where S1 is the area of ​​the high curvature region and S2 is the total area of ​​the object to be corrected.

[0104] In other embodiments, The calculation formula is as follows:

[0105] ;

[0106] in, The basic weights range from 0.5 to 1.5, K1 is the distortion coefficient, K2 is the spatial location coefficient, and K3 is the dispersion coefficient.

[0107] The formula for calculating K1 is as follows:

[0108] ;

[0109] The average rate of change of spacing for all high curvature regions. The high curvature distance threshold is set to 0.2. The maximum rate of change of spacing; The amplitude coefficient is set to 1, and β is the nonlinear exponent with a value of 2.

[0110] The more severe the distortion, the stronger the spine continuity constraint becomes, providing a core anchor point for surface fitting in severely curled scenarios and preventing fitting divergence and breakage.

[0111] The formula for calculating K2 is as follows:

[0112] ;

[0113] The normalized average distance from the center of gravity of all high curvature regions to the spine (the maximum distance from the spine to the edge of the page is 1, and the distance at the spine is 0). The position weighting coefficient is set to 1.2.

[0114] The closer the high curvature area is to the spine, the stronger the constraint, which conforms to the physical nature of the rigid bending of the page around the spine. The deformation core near the spine is corrected by continuous constraint, and the constraint weight is automatically reduced when the page edge curls locally.

[0115] The formula for calculating K3 is as follows:

[0116] ;

[0117] The number of high curvature regions. The total number of points in the sparse laser array; The coefficient of variation of the distance from the centroid to the spine in all high curvature regions; The dispersion coefficient is set to 0.8.

[0118] The more numerous and dispersed the high curvature regions are, the stronger the constraint, providing a globally unified physical anchor point for multi-regional dispersed distortion.

[0119] The adaptive weight field can automatically adapt to different spine positions, page sizes, and opening angles, improving the accuracy of the inverse distortion mapping field and thus ensuring the accuracy of subsequent corrected images.

[0120] The structure of the calibration model in step S4 is as follows:

[0121] The correction model is a pre-trained two-branch end-to-end neural network, which includes a geometric correction branch, a texture enhancement branch, and a feature fusion module.

[0122] The geometric correction branch takes the inverse distortion map field as input, learns and outputs a pixel-level distortion correction function;

[0123] The texture enhancement branch takes the registered high-resolution natural light image as input and performs image deblurring, illumination equalization, and detail enhancement.

[0124] The features output from the two branches are fused across branches through the feature fusion module to output the corrected image.

[0125] A surface image correction device based on multimodal data fusion includes:

[0126] An image acquisition module, configured to acquire images of the object to be corrected with laser markings;

[0127] A laser emitting module, configured to project a sparse grid of laser dots toward the object to be corrected;

[0128] The high curvature determination module is configured to determine whether a region is a high curvature region based on the change in the spacing between adjacent laser points. The control module controls the laser emission module to project a high-density grid laser dot array onto the high curvature region, and the control module controls the image acquisition module to acquire the image after the high-density grid laser dot array is projected.

[0129] The data processing module is configured to acquire fine depth information of each laser point to form laser point cloud data, collect image data of the object to be corrected and align it with the laser point cloud data, construct and solve an energy minimization model with a three-dimensional elevation surface as the solution target, obtain the optimal three-dimensional elevation surface, generate the inverse distortion mapping field from the corresponding surface to the two-dimensional flat plane, input the inverse distortion mapping field into the correction model, and output the corrected image according to the correction model.

[0130] The system employs a triangulation optical path that combines oblique laser projection with a camera's vertical downward view.

[0131] The projection optical axis of the laser and the DMD digital micromirror device is at a fixed tilt angle of 15° to 30° with the normal of the page plane. The imaging optical axis of the industrial camera coincides with the normal of the page plane. A narrowband filter matching the wavelength of the laser is installed in front of the industrial camera lens. The main control unit also includes a synchronous trigger module, which is used to output microsecond-level hard trigger pulses to realize synchronous control of laser projection and industrial camera capture.

[0132] Beneficial effects:

[0133] A method of high-density laser scanning in high-curvature areas using sparse grid laser pre-screening was adopted. High-density point cloud acquisition was performed only in the high-curvature areas with the most severe distortion, while low-computing-power sparse sampling was used in non-core distortion areas. This method reduced the amount of laser point cloud data and improved work efficiency while ensuring the sampling accuracy of key distortion areas.

[0134] Two high curvature region determination schemes (pixel spacing change rate / elevation gradient threshold determination) can be flexibly configured according to hardware computing power and scene requirements, thus broadening the scope of application.

[0135] Laser projection is unaffected by the surface texture of the object to be corrected or ambient lighting, solving the industry problem of feature extraction failure and inaccurate distortion calculation in textureless and low-contrast curved surface scenes where pure vision correction schemes fail. It achieves sub-pixel-level center positioning of the laser spot through the gray-scale centroid method, and solves high-precision 3D point cloud with the help of triangulation model, providing reliable basic data for surface solution.

[0136] For core application scenarios such as document scanning, the system innovatively incorporates spine continuity constraints and page edge straightness constraints. This not only forces the surface to conform to the laser-measured elevation through data fidelity terms, but also constrains the surface to conform to the physical curvature characteristics of the book page through regularization terms.

[0137] A dual-branch end-to-end neural network correction architecture is adopted to achieve deep synergy between geometric correction and texture optimization. The geometric correction branch completes pixel-level distortion correction based on a precise inverse distortion mapping field, while the texture enhancement branch simultaneously completes image deblurring, illumination equalization, and detail enhancement. Finally, cross-branch feature fusion is achieved.

[0138] Example 2:

[0139] An electronic device includes a processor and a memory communicatively connected to the processor and used to store processor-executable instructions, the processor being used to execute the method described in Embodiment 1 above.

[0140] Example 3:

[0141] A server includes at least one processor and a memory communicatively connected to the processor, the memory storing instructions executable by the at least one processor to cause the at least one processor to perform the method as described in Embodiment 1.

[0142] Example 4:

[0143] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A surface image correction method based on multimodal data fusion, characterized in that, Includes the following steps: S1. Project a sparse grid of laser dots onto the object to be corrected, acquire an image of the object marked with laser dots, and determine whether it is a high curvature region based on the change in the spacing between adjacent laser dots: S2. Project a high-density grid of laser dots onto the high curvature region, and acquire fine depth information of each laser dot after image acquisition to form laser dot cloud data. S3. Collect image data of the object to be corrected and align it with laser point cloud data. Using the three-dimensional elevation surface as the solution target, construct and solve the energy minimization model to obtain the optimal three-dimensional elevation surface and generate the inverse distortion mapping field from the corresponding surface to the two-dimensional flat plane. S4. Input the inverse distortion mapping field into the correction model, and output the corrected image according to the correction model.

2. The surface image correction method based on multimodal data fusion according to claim 1, characterized in that, S1 includes the following steps: S11. Project a sparse grid of laser dots onto the object to be corrected using a laser. S12. Acquire images of the object to be corrected with laser markings using an industrial camera, perform grayscale conversion, adaptive threshold segmentation, and morphological noise reduction processing, and extract the ROI region of each laser spot. S13. The center coordinates of each laser spot are determined using the gray-scale centroid method, as shown in the following formula: ； in This represents the grayscale value at pixel (x, y) within the ROI region. , These are the center coordinates of the laser spot; S14. Calculate the rate of change of the distance between adjacent laser spots, using the following formula: ； in, The rate of change of the distance between adjacent laser spots. Adjacent , This represents the actual distance interval along the laser scanning direction; S15. If the rate of change of the distance between adjacent laser spots is greater than the distance threshold, then the region where the laser spot is located is determined to be a high curvature region.

3. The surface image correction method based on multimodal data fusion according to claim 1, characterized in that, S1 includes the following steps: S11. Project a sparse grid of laser dots onto the object to be corrected using a laser. S12. Acquire images of the object to be corrected with laser markings using an industrial camera, perform grayscale conversion, adaptive threshold segmentation, and morphological noise reduction processing, and extract the coordinate data of each laser spot. S13. Based on the pre-calibrated laser and camera triangulation model, calculate the three-dimensional spatial coordinates of each laser point, calculate the elevation gradient of adjacent laser points, and if the elevation gradient exceeds the preset curvature threshold, determine that the area is a high curvature area and mark its coordinate range. The formula for calculating the elevation gradient is: ； in, For elevation gradient, The elevation values ​​are those of two adjacent laser points. The X-axis coordinates of two adjacent laser points. The Y-axis coordinates of two adjacent laser points.

4. The surface image correction method based on multimodal data fusion according to claim 1, characterized in that: The high curvature region in S2 is the region with the laser point as the center and the side length as the distance between adjacent laser points; The sparse grid laser dot matrix is ​​a uniform dot matrix with a size of 5×5. The high-density dot matrix is ​​a uniform dot matrix with a size of 20×20.

5. The surface image correction method based on multimodal data fusion according to claim 1, characterized in that: S3 includes the following steps: S31. Acquire high-resolution image data of the object to be calibrated and align it with the laser point cloud data; S32. Construct an energy minimization model with physical prior constraints, including spine continuity constraints and page edge straightness constraints. S33. Solve the energy minimization model to obtain the optimal three-dimensional elevation surface, and generate the inverse distortion mapping field from the corresponding surface to the two-dimensional flat plane; The energy minimization model is as follows: ； in, The objective of solving the total energy function model is Take the minimum value. The weights corresponding to the spine continuity constraint, The weights for the page margin straightness constraint, The three-dimensional elevation surface to be solved is the elevation of each point within the region to be corrected. The measured elevation value of the laser point cloud obtained by solving the high-density laser point array for S2 is shown. For spine continuity constraints, This is a constraint on the straightness of page margins.

6. The surface image correction method based on multimodal data fusion according to claim 1, characterized in that, The structure of the correction model in step S4 is as follows: The correction model is a pre-trained dual-branch end-to-end neural network, which includes a geometric correction branch, a texture enhancement branch, and a feature fusion module. The geometric correction branch takes the inverse distortion mapping field as input, learns and outputs a pixel-level distortion correction function; The texture enhancement branch takes the registered high-resolution natural light image as input and performs image deblurring, illumination equalization, and detail enhancement. The features output from the two branches are fused across branches through the feature fusion module to output the corrected image.

7. A surface image correction device based on multimodal data fusion, characterized in that, include: An image acquisition module, configured to acquire images of the object to be corrected with laser markings; A laser emitting module, configured to project a sparse grid of laser dots toward the object to be corrected; The high curvature determination module is configured to determine whether a region is a high curvature region based on the change in the spacing between adjacent laser points. The control module controls the laser emission module to project a high-density grid laser dot array onto the high curvature region, and the control module controls the image acquisition module to acquire the image after the high-density grid laser dot array is projected. The data processing module is configured to acquire fine depth information of each laser point to form laser point cloud data, collect image data of the object to be corrected and align it with the laser point cloud data, construct and solve an energy minimization model with a three-dimensional elevation surface as the solution target, obtain the optimal three-dimensional elevation surface, generate the inverse distortion mapping field from the corresponding surface to the two-dimensional flat plane, input the inverse distortion mapping field into the correction model, and output the corrected image according to the correction model.

8. An electronic device comprising a processor and a memory communicatively connected to the processor and used for storing processor-executable instructions, characterized in that: The processor is used to execute the method described in any one of claims 1-6.

9. A server, characterized in that: The method includes at least one processor and a memory communicatively connected to the processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the processor to cause the at least one processor to perform the method as described in any one of claims 1-6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

Images