Projector-side projection trapezoidal correction method, electronic device, and storage medium

Abstract

This application relates to the field of projector technology, and discloses a method, electronic device, and storage medium for trapezoidal correction of side projection in a projector. The method includes: acquiring an coded light image reflected from the projection surface; reconstructing the three-dimensional shape data of the projection surface based on the coded light image and determining an index of the surface deviation; solving for inverse transformation parameters based on the mapping relationship between the ideal image contour and the physical image contour projected onto the projection surface; determining the basic mapping position of each pixel in the original projection image based on the inverse transformation parameters; constructing a nonlinear distortion compensation model to generate a local region offset when the surface deviation index exceeds a preset threshold; superimposing the local region offset onto the basic mapping position, and sampling the original projection image using the superimposed position to obtain a corrected projection image. The embodiments of this application can eliminate residual trapezoidal distortion caused by large-angle side projection or non-planar projection surfaces.

Description

Technical Field

[0001] This application relates to the field of projector technology, and in particular to a method for side projection trapezoidal correction in a projector, an electronic device, and a storage medium. Background Technology

[0002] Automatic keystone correction technology is a practical function designed to automatically square the image after side projection from a projector, making it easier for users to watch movies. In related technologies, automatic keystone correction for side projection typically assumes that the projection surface is an ideal plane and uses perspective transformation based on the four corner coordinates to achieve keystone correction.

[0003] However, this trapezoidal correction method ignores the curvature and roughness of the projection surface, as well as the nonlinear distortion caused by excessive projection angle. When projecting at a large angle (horizontal deflection angle greater than 20°), the corrected image still has obvious residual trapezoidal distortion. Summary of the Invention

[0004] The purpose of this application is to provide a method, electronic device and storage medium for side projection trapezoidal correction of a projector, which aims to solve the technical problem that the image still has obvious residual trapezoidal distortion after correction due to ignoring the curvature and roughness of the projection surface and the nonlinear distortion caused by excessive projection angle.

[0005] This application provides a method for correcting trapezoidal distortion in side projection of a projector, including: Acquire the image of the coded light reflected from the projection surface after the projector projects a coded light pattern onto the projection surface; Based on the coded light image, the three-dimensional topography data of the projection surface is reconstructed; Based on the three-dimensional topographic data, the surface deviation index of the projection surface is determined; Based on the mapping relationship between the ideal image contour and the physical image contour projected by the projector onto the projection surface, the inverse transformation parameters are solved; Based on the inverse transform parameters, the basic mapping position of each pixel in the original projected image is determined; When the surface deviation index exceeds a preset threshold, a nonlinear distortion compensation model is constructed based on the three-dimensional topography data. The image coordinates of each pixel in the original projected image are input into the nonlinear distortion compensation model to obtain the corresponding local region offset. The local region offset is superimposed onto the base mapping position, and the original projection image is sampled at the superimposed position to obtain the corrected projection image.

[0006] In some embodiments, reconstructing the three-dimensional topographic data of the projection surface based on the coded light image includes: Feature points are extracted from the coded light image; Determine the correspondence between the feature points in the image coordinate system of the visual sensor that acquires the coded light image and the image coordinate system of the projector; Based on the correspondence and the calibration parameters between the visual sensor and the projector, the three-dimensional coordinates of the feature points in space are reconstructed to obtain the three-dimensional shape data of the projection surface.

[0007] In some embodiments, determining the surface deviation index of the projection surface based on the three-dimensional topography data includes: Based on the three-dimensional coordinates of the feature points in the three-dimensional topography data, fit the equation of the projection plane space. Determine the degree of deviation of the feature point relative to the spatial plane equation of the projection surface; Statistical values ​​are calculated based on the degree of deviation of the feature points, and these values ​​serve as an index of the degree of surface deviation.

[0008] In some embodiments, solving for the inverse transformation parameters based on the mapping relationship between the ideal image contour and the physical image contour projected by the projector onto the projection surface includes: An optimization objective function is constructed using the reprojection error of feature points in the three-dimensional topography data under the mapping relationship, the rectangular constraint term used to constrain the four corners of the corrected image to be right angles, and the aspect ratio constraint term used to constrain the corrected image to maintain the original aspect ratio as optimization objectives. The objective function is minimized, and the objective function is iteratively optimized to obtain the inverse transform parameters.

[0009] In some embodiments, constructing a nonlinear distortion compensation model based on the three-dimensional topography data includes: Select a subset of feature points from the three-dimensional topography data as the center points of the radial basis function; Determine the spatial distance between the center point and the feature points in the three-dimensional topography data, and construct a kernel function matrix based on the actual mapping deviation at each feature point; The kernel function matrix is ​​solved using the regularized least squares method to obtain the weight coefficients corresponding to each center point. The weight coefficients are then used as parameters of the radial basis function to obtain the nonlinear distortion compensation model.

[0010] In some embodiments, the projector side projection trapezoidal correction method further includes: Based on the local transformation characteristics of the inverse transform parameters, the scaling factor of each local region in the corrected projected image is determined; the scaling factor is used to evaluate the degree of resolution loss in the local region. Based on the scaling factor, the compensation intensity of the local region is determined; In the frequency domain, high-frequency component enhancement processing is performed on the resolution loss region to obtain the enhanced resolution loss region, and / or when sampling the original projected image in the spatial domain, a sharpening interpolation kernel is used to sample the resolution loss region, and a smoothing interpolation kernel is used to sample other local regions; the resolution loss region is the local region where the local scaling factor is less than a preset reference value.

[0011] In some embodiments, the projector side projection trapezoidal correction method further includes: Extract the actual edge contour of the corrected projection image and the target edge contour of the projection surface; Calculate the Hausdorff distance between the actual edge contour and the target edge contour, and use the Hausdorff distance as the alignment deviation between the actual edge contour and the target edge contour; When the alignment deviation exceeds a preset deviation threshold, the translation, rotation, and scaling are used as variables to be optimized. The distance between corresponding points between the actual edge contour and the target edge contour is minimized by the iterative nearest point method to solve for the target fine-tuning amount. The target fine-tuning amount is superimposed on the inverse transform parameter.

[0012] In some embodiments, the projector side projection trapezoidal correction method further includes: During the projection output process, the pose change of the projector is acquired; When the translation amount in the pose change exceeds a preset translation threshold or the rotation amount exceeds a preset rotation threshold, an incremental transformation matrix is ​​derived based on the pose change. The incremental transformation matrix and the inverse transformation parameters are combined to obtain the updated inverse transformation parameters.

[0013] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described projector side projection keystone correction method.

[0014] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described projector side projection keystone correction method.

[0015] The beneficial effects of this application are as follows: By reconstructing the three-dimensional topography data of the projection surface, when the surface deviation index of the projection surface determined based on the three-dimensional topography data exceeds a preset threshold, a nonlinear distortion compensation model is constructed based on the three-dimensional topography data, and the local region offset of each pixel is calculated. The local region offset is then superimposed onto the base mapping position, and the original projection image is precisely sampled to achieve trapezoidal correction for side projection of the projector. Thus, by reconstructing the three-dimensional topography data of the projection surface, the geometric characteristics of the projection surface can be truly perceived, rather than relying solely on the planar assumption. By constructing a nonlinear distortion compensation model and calculating the local region offset of each pixel, and then superimposing the local region offset onto the base mapping position, fine-grained pixel-level adjustments can be made for the curvature differences of different regions on the projection surface, generating a visually perfectly square, straight-edge, and locally distortion-free corrected projection image, eliminating residual trapezoidal distortion under large-angle side projection or non-planar projection surfaces. Attached Figure Description

[0016] Figure 1 This is a flowchart of the projector side projection trapezoidal correction method provided in the embodiments of this application.

[0017] Figure 2 This is a flowchart of a method for reconstructing the three-dimensional topographic data of a projection surface provided in an embodiment of this application.

[0018] Figure 3 This is a flowchart of a method for determining the surface deviation index of a projection surface, as provided in an embodiment of this application.

[0019] Figure 4 This is a flowchart of a method for solving inverse transform parameters provided in an embodiment of this application.

[0020] Figure 5 This is a flowchart of a method for constructing a nonlinear distortion compensation model provided in an embodiment of this application.

[0021] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and drawings are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application. Furthermore, the information, data, and signals involved in the embodiments of this application are all authorized by relevant parties or have been fully authorized by all parties, and the collection, use, and processing of related data comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0025] In traditional automatic keystone correction technology, when the projection surface is not an ideal plane and the projection angle is too large, the correction algorithm cannot fully compensate for the curvature and roughness of the projection surface, as well as the nonlinear distortion caused by the excessive projection angle, resulting in residual keystone distortion in the corrected image. This problem stems from the simplified assumptions about the geometric characteristics of the projection surface, namely, perspective transformation based only on the four-corner coordinates, without considering the three-dimensional shape characteristics of the actual projection surface. This makes it impossible to accurately establish the mapping relationship between the physical image outline and the ideal image outline under large-angle side projection conditions, thus affecting the geometric fidelity and visual consistency of the image. For example, in a home theater application scenario, the projector is installed on the side wall of the room and projects an image onto the plasterboard wall with a horizontal deflection angle of 25°. Due to the construction process, the wall has local curvature and slight unevenness. When the coded light pattern is reflected by the wall, the traditional correction method only solves the inverse transformation parameters based on the four corner coordinates. However, the curvature of the wall causes the physical image outline to be unevenly distributed in the horizontal direction, resulting in the horizontal line bending and vertical line tilting in the central area of ​​the corrected image. This manifests as the image not being completely square and the edge area having local distortion.

[0026] If the above problems are not resolved, residual trapezoidal distortion will cause persistent geometric distortion in the image, reducing the user's viewing experience and potentially causing visual discomfort. In the professional display field, this problem will limit the applicability of projection technology in non-ideal projection surface environments, affecting the technical reliability and market competitiveness of the product.

[0027] Based on this, embodiments of this application provide a projector side projection trapezoidal correction method, electronic device, and storage medium. By reconstructing the three-dimensional topography data of the projection surface and dynamically constructing a nonlinear distortion compensation model, the residual trapezoidal distortion problem caused by the deviation of the projection surface is effectively handled. It can effectively compensate for the nonlinear distortion caused by the non-ideal projection surface, reduce residual trapezoidal distortion, and improve image clarity and visual comfort.

[0028] like Figure 1 As shown, in one embodiment, a projector side projection trapezoidal correction method is provided, which includes, but is not limited to, the following steps S101 to S107.

[0029] Step S101: Obtain the image of the coded light reflected from the projection surface after the projector projects the coded light pattern onto the projection surface.

[0030] A coded optical pattern is a pre-designed optical pattern that typically contains specific coded information, such as a stripe, dot matrix, or grid structure. This pattern is projected onto a projection surface by a projector to provide spatial reference information for subsequent image acquisition and processing, facilitating the reconstruction of the projection surface's geometry.

[0031] A coded light image is an image captured from a specific viewing angle by a vision sensor (such as a camera) after a projector projects a coded light pattern onto a projection surface. This image contains distortion information of the coded light pattern on the projection surface, which reflects the geometric characteristics of the projection surface and the influence of the projection angle.

[0032] In one implementation, an external vision sensor, such as a separate camera, can be manually positioned and focused to capture the coded light image on the projection surface as the projector projects the coded light pattern. Another implementation involves a vision sensor integrated within the projector that automatically activates and acquires the coded light image when the user triggers a correction function. For example, the projector can project a checkerboard pattern, and the vision sensor can then capture the coded light image of that pattern on the projection surface.

[0033] Step S102: Based on the coded light image, reconstruct the three-dimensional topographic data of the projection surface.

[0034] Three-dimensional topographic data refers to the geometric shape information of a projection surface in three-dimensional space, reconstructed by analyzing and processing coded light images. This data is usually represented in the form of point clouds, meshes, or parametric surfaces, and can accurately describe the undulations, curvature, and irregularities of the projection surface.

[0035] In one implementation, specific marker points in the coded light image are manually identified, and their two-dimensional coordinates in the image are manually input. Then, using a pre-defined geometric model and triangulation principles, the coordinates of these marker points in three-dimensional space are calculated, thus obtaining the three-dimensional topographic data of the projection surface. Another implementation can employ a method based on structured light principles. By analyzing the deformation of the coded light pattern on the projection surface, such as by calculating the phase information of the stripe pattern, the depth information of each point on the projection surface can be derived, thereby constructing the three-dimensional topographic data of the projection surface.

[0036] Step S103: Based on the three-dimensional topography data, determine the surface deviation index of the projection surface.

[0037] The surface deviation index is a numerical value used to quantify the degree of deviation between the projected surface and the ideal plane. This index is calculated by analyzing three-dimensional topographic data and can reflect the curvature, unevenness, or roughness of the projected surface, providing a basis for subsequent judgment on whether nonlinear distortion compensation is needed.

[0038] In one implementation, several representative points in the 3D topography data can be selected, and the distances from these points to their mean plane can be calculated. The average of these distances can then be used as an indicator of surface deviation. For example, several points in the center region of the projection plane can be manually selected, and their average distances to the plane fitted through these points can be calculated. Another implementation involves performing preliminary surface fitting on the 3D topography data, such as fitting a quadratic surface, and then calculating the average distance from the original 3D topography data points to the fitted surface as an indicator of surface deviation.

[0039] Step S104: Based on the mapping relationship between the ideal image outline and the physical image outline projected onto the projection surface by the projector, solve for the inverse transformation parameters.

[0040] The ideal image outline refers to the shape of the image boundary that is expected to appear on the projection surface without distortion; it is usually a standard rectangle. This outline serves as the target for correction and is compared with the actual physical image outline projected to determine the transformation relationship required for correction.

[0041] The physical image outline refers to the actual shape of the image boundary projected onto the projection surface at the current side projection angle. Due to the influence of the side projection angle and the geometry of the projection surface, this outline usually exhibits trapezoidal or other irregular distortions.

[0042] Inverse transform parameters are a set of mathematical parameters calculated by analyzing the mapping relationship between the ideal image contour and the physical image contour. These parameters are used to correct distorted images to the ideal image. They typically include perspective transformation, affine transformation, or more complex geometric transformation parameters, which are used to determine the mapping rules for the pixels of the original image.

[0043] In one implementation, the four corner points of the ideal image can be marked on the projection surface, and the positions of the four corner points of the actual projected image in the visual sensor image are simultaneously recorded. Then, a perspective transformation matrix is ​​directly calculated as the inverse transformation parameters using these manually marked corresponding points. For example, the user can use a laser pointer to indicate the desired image boundary on the projection surface, and the system records these indicated points. Another implementation involves pre-setting a standard rectangle as the ideal image outline, extracting the edge contour of the actual projected image from the coded light image using image processing techniques, and then using optimization algorithms such as least squares to solve for the inverse transformation parameters that map the actual contour to the ideal contour.

[0044] Step S105: Determine the basic mapping position of each pixel in the original projected image based on the inverse transform parameters.

[0045] The base mapping position refers to the corresponding position of each pixel in the original projected image in the corrected projected image, considering only linear distortion (such as trapezoidal distortion). This position is calculated by applying inverse transform parameters and is the preliminary correction result before performing nonlinear distortion compensation.

[0046] In one implementation, the coordinates of each pixel in the original projected image are directly substituted into the transformation formula represented by the inverse transformation parameters obtained above to calculate the corresponding position of that pixel in the corrected image, which is the base mapping position. For example, if the inverse transformation parameters are a 3x3 perspective transformation matrix, multiplying the homogeneous coordinates of the original pixel by this matrix yields the transformed homogeneous coordinates, thereby determining the base mapping position.

[0047] Step S106: When the surface deviation index exceeds the preset threshold, a nonlinear distortion compensation model is constructed based on the three-dimensional topography data. The image coordinates of each pixel in the original projected image are input into the nonlinear distortion compensation model to obtain the corresponding local region offset.

[0048] A nonlinear distortion compensation model is a mathematical model used to correct image distortion caused by non-planar characteristics when the projection surface has a predetermined degree of curvature or roughness. This model is typically built based on three-dimensional topographic data and can calculate the additional offset of each pixel due to nonlinear factors.

[0049] Local offset refers to the tiny displacement that each pixel needs to be adjusted from its base mapping position, calculated by a nonlinear distortion compensation model. This offset is used to accurately compensate for local image distortion caused by the non-planar characteristics of the projection surface.

[0050] In one implementation, when the surface deviation index reaches a preset level, a basic polynomial model can be used as a nonlinear distortion compensation model. This model estimates the nonlinear distortion of the entire projection surface by fitting the local deviations of a few key points in the 3D topography data. For example, several high-curvature regions on the projection surface can be selected, the residuals between the actual distortion of these regions and the linear correction results can be calculated, and then a low-order polynomial can be used to fit these residuals as a nonlinear distortion compensation model.

[0051] Step S107: The local region offset is superimposed onto the base mapping position, and the original projection image is sampled at the superimposed position to obtain the corrected projection image.

[0052] In one implementation, the calculated local region offset for each pixel is directly added to its base mapping position to obtain the final sampling position. Then, a nearest neighbor interpolation algorithm is used to obtain the pixel value at the corresponding position from the original projected image and fill it into the corrected projected image. For example, if the base mapping position of a pixel is (x, y) and the local offset is (dx, dy), then the final sampling position is (x+dx, y+dy), and the pixel value is then taken from (x+dx, y+dy) in the original projected image.

[0053] The following example will provide a more detailed explanation of the above technical solution: Suppose user A is in a room and places a projector in a corner of the room, projecting onto a wall with a certain curvature at a large side projection angle (e.g., a horizontal deflection angle of 30 degrees). Because the wall is not an ideal plane and the projection angle is large, traditional correction methods based solely on four-corner perspective transformation will not be able to completely eliminate image distortion, resulting in curved distortion and local blurring at the edges of the projected image.

[0054] To address this issue, firstly, the projector projects a pre-defined coded light pattern, such as a dot matrix pattern with a specific density, onto the curved wall. Simultaneously, a vision sensor integrated within the projector captures the reflected image of this dot matrix pattern on the wall—the coded light image. This coded light image contains information about the deformation of the dot matrix pattern due to the wall's curvature and the projection angle.

[0055] Next, based on the coded light image, feature points in the dot matrix pattern are extracted using image processing algorithms, and the three-dimensional topographic data of the wall surface is reconstructed using the principle of triangulation. This three-dimensional topographic data accurately describes the actual geometry of the wall in the form of a point cloud, including its subtle curvature and irregularities.

[0056] Subsequently, based on the 3D topography data, the surface deviation index of the wall is calculated. Specifically, an optimal plane is fitted to approximate the wall surface, and the standard deviation of the distance from each feature point in the 3D topography data to the fitted plane is calculated. If this standard deviation, i.e., the surface deviation index, reaches a preset threshold, it indicates that nonlinear distortion compensation is required.

[0057] Simultaneously, the mapping relationship between the ideal rectangular image outline and the physical image outline actually projected onto the wall is determined. By analyzing feature points extracted from the coded light image, the boundary of the actual projected image is identified and matched with a standard rectangular target outline, thereby solving for the inverse transformation parameters used for preliminary correction of trapezoidal distortion. These parameters describe the geometric transformation required to linearly transform the distorted physical image outline into the ideal rectangular outline.

[0058] After obtaining the inverse transform parameters, the base mapping position of each pixel in the original projected image is calculated based on these parameters. This means that if the wall is perfectly flat and has only linear trapezoidal distortion, then mapping the pixels of the original image to these base mapping positions will yield a preliminarily corrected image.

[0059] However, since the wall's curvature deviation index has been confirmed to have reached a preset threshold, a nonlinear distortion compensation model is constructed based on the previously reconstructed 3D topographic data. This model can accurately describe the local nonlinear distortion caused by the wall curvature to the projected image. The image coordinates of each pixel in the original projected image are input into this nonlinear distortion compensation model, and the model outputs the local region offset of each pixel due to the wall curvature. For example, for protruding or recessed areas on the wall, the model calculates the distance that the corresponding pixel needs to be finely adjusted outward or inward.

[0060] Finally, the calculated local region offsets are superimposed onto the base mapping position of each pixel to obtain the final sampling position. Then, the original projected image is precisely sampled using these superimposed positions to generate the final corrected projected image. When this corrected projected image is projected onto a curved wall, user A will observe a visually perfectly square image with straight edges and no local distortion, achieving a high-quality viewing experience even on side projections and curved projection surfaces. Therefore, this method effectively solves the image distortion problem of traditional methods on non-planar projection surfaces and large-angle side projection scenes by combining 3D topographic data to compensate for nonlinear distortion.

[0061] Based on the above examples, the technical solution of this embodiment demonstrates significant technical contributions. Traditional projector side projection trapezoidal correction methods, such as those relying solely on four-corner perspective transformation, will produce residual distortion when faced with curved walls and large-angle side projection scenes encountered by User A in the above examples, because they ignore the actual three-dimensional shape of the projection surface. Their core drawback lies in assuming the projection surface is an ideal plane, making it unable to handle nonlinear distortions caused by curvature, roughness, or extreme projection angles.

[0062] In contrast, the method in this embodiment achieves precise compensation for nonlinear distortion by introducing the reconstruction and analysis of the three-dimensional topographic data of the projection surface. Specifically, by acquiring coded light images and reconstructing the three-dimensional topographic data of the projection surface, this method can realistically perceive the geometric characteristics of the projection surface, rather than relying solely on the planar assumption. For example, in the above example, it can identify the actual curvature of the wall and quantify its surface deviation index. This step is not available in the prior art, and it lays the foundation for subsequent nonlinear compensation.

[0063] Furthermore, when the surface deviation index reaches a preset threshold, this method can construct a nonlinear distortion compensation model based on the 3D topography data and calculate the local offset of each pixel. This mechanism can perform fine-grained pixel-level adjustments for the curvature differences in different regions of the projection surface. For example, at protrusions in the wall, pixels are fine-tuned to counteract the image compression caused by the protrusions; at depressions, pixels are fine-tuned to counteract the image stretching caused by the depressions. This nonlinear compensation capability based on the actual 3D topography is significantly superior to existing schemes that only perform linear perspective transformations, which cannot effectively correct local nonlinear distortions.

[0064] Ultimately, by superimposing local offsets onto the base mapping position and performing precise sampling, this method can generate a visually perfectly square, straight-edge, and locally distortion-free corrected projection image. This not only solves the residual trapezoidal distortion problem of traditional methods under large-angle side projection and non-planar projection surfaces, but also improves the overall quality of the projected image and the user experience. The technical concept of this embodiment, by introducing three-dimensional shape perception and nonlinear compensation mechanisms, overcomes the limitations of existing technologies in complex projection environments, bringing substantial progress to projector side projection trapezoidal correction technology.

[0065] like Figure 2 As shown, in one embodiment, the method for reconstructing the three-dimensional topographic data of the projection plane includes, but is not limited to, the following steps S201 to S203.

[0066] Step S201: Extract feature points from the coded light image.

[0067] Step S202: Determine the correspondence between the feature points in the image coordinate system of the visual sensor that acquires the encoded light image and the image coordinate system of the projector.

[0068] Step S203: Based on the correspondence and the calibration parameters between the visual sensor and the projector, the three-dimensional coordinates of the feature points in space are reconstructed to obtain the three-dimensional shape data of the projection surface.

[0069] Extracting feature points from coded light images can be implemented using corner detection algorithms based on grayscale changes or gradient information, such as the Harris corner detector or the Shi-Tomasi corner detector, or using feature descriptor algorithms based on scale-space extrema, such as scale-invariant feature transformation or accelerated robust features.

[0070] Determining the correspondence between feature points in the image coordinate system of the visual sensor acquiring the coded light image and the image coordinate system of the projector can be achieved by analyzing the structure of the coded light pattern (e.g., stripe coding, Gray code coding, etc.) to resolve the original position of each feature point in the projector image; or, if the coded light pattern contains identifiable unique markers, an image matching algorithm (such as feature descriptor matching, template matching) can be used to establish the feature point correspondence between the visual sensor image and the projector image.

[0071] Reconstructing the 3D coordinates of feature points in space can be achieved by backprojecting the coordinates of a 2D image into 3D space and using the projection information of corresponding points from different viewpoints to accurately calculate their 3D positions. Reconstruction methods can include, but are not limited to: triangulation based on binocular vision principles, where the vision sensor and projector are treated as a pair of "virtual" binocular cameras; or, using structured light 3D measurement technology, calculating 3D coordinates by analyzing the deformation of the coded light pattern on the projection surface. Ultimately, the set of reconstructed 3D coordinates of these feature points constitutes the 3D topographic data of the projection surface.

[0072] This application's solution achieves accurate reconstruction of the 3D topographic data of the projection surface from coded light images through a series of orderly steps. First, salient and distinguishable feature points are extracted from the coded light images captured by the vision sensor. Then, by analyzing the characteristics of the coded light pattern, a precise correspondence is established between these extracted feature points in the image coordinate system of the vision sensor and the image coordinate system of the projector. Based on this, using pre-acquired calibration parameters between the vision sensor and the projector, these feature points with established correspondences are back-projected from the 2D image space to the 3D space. Furthermore, using geometric principles such as triangulation, their 3D coordinates in the real world are calculated, obtaining the 3D topographic data of the projection surface. In this way, this solution overcomes the limitations of relying solely on 2D image information for correction, providing a precise 3D geometric basis for subsequent surface deviation assessment and nonlinear distortion compensation, thereby significantly improving the accuracy and adaptability of the projector's side-projection trapezoidal correction.

[0073] In one specific embodiment, the projector projects a set of coded light patterns (including a checkerboard pattern and a Gray code sequence) over a normal projection screen, and the vision sensor simultaneously acquires the coded light image reflected from the projection surface. Let the projected coded light pattern be P_s(x,y) and the coded light image acquired by the vision sensor be I_c(u,v), and establish the correspondence between the projector pixel coordinates (x,y) and the vision sensor pixel coordinates (u,v).

[0074] Feature points are extracted from the encoded light image using the Harris corner response function: R(x,y)=det(M) - k×(trace(M))^2, Where M is the structure tensor matrix, M=sum_{w}[Ix^2,Ix×Iy;Ix×Iy,Iy^2], Ix and Iy are the gradients of the encoded light image in the x and y directions, respectively, w is the local window, and k is an empirical constant (usually taken as 0.04-0.06). When R(x,y) is greater than the preset threshold and is a local maximum, it is determined to be a valid feature point.

[0075] After extracting feature points from the coded light image, 3D coordinate reconstruction is performed on each valid feature point based on the correspondence between the image coordinate systems of the projector and the calibration parameters between the vision sensor and the projector. Let the intrinsic parameter matrix of the vision sensor be K_c, and the intrinsic parameter matrix of the projector be K_p. The extrinsic parameters between them (rotation matrix R_cp and translation vector t_cp) are obtained through offline calibration. For the i-th feature point, its 3D coordinates in the vision sensor coordinate system are solved using triangulation. Z_ci×[ui,vi,1]^T=K_c×[X_i,Y_i,Z_i,1]^T, Z_pi×[xi,yi,1]^T=K_p× [X_i,Y_i,Z_i,1]^T, Where Z_ci is the scale factor of the feature point in the camera coordinate system, Z_pi is the scale factor of the feature point in the projector coordinate system, ui and vi are the coordinates of the i-th feature point in the visual sensor pixel coordinate system, xi and yi are the coordinates of the i-th feature point in the projector pixel coordinate system, X_i, Y_i and Z_i are the coordinates of the i-th feature point in space, and T represents the transpose. By solving the above system of equations, the three-dimensional coordinates [X_i, Y_i, Z_i] of each feature point in space can be obtained.

[0076] like Figure 3 As shown, in one embodiment, the method for determining the surface deviation index of the projection surface includes, but is not limited to, the following steps S301 to S303.

[0077] Step S301: Fit the equation of the projection plane space based on the three-dimensional coordinates of the feature points in the three-dimensional topography data.

[0078] Step S302: Determine the degree of deviation of the feature points from the equation of the projection plane space.

[0079] Step S303: Calculate statistical values ​​based on the degree of deviation of feature points, which serve as indicators of surface deviation.

[0080] Fitting the equation of the projection plane space can be done by using the least squares method, which determines the plane parameters by minimizing the sum of squared distances from all feature points to the plane, or by using the RANSAC algorithm.

[0081] The deviation of a feature point from the equation of the projection plane refers to the perpendicular distance of each feature point to the fitted plane. By calculating these deviations, a statistic can be obtained that reflects the average dispersion of all feature points from the fitted plane. The larger the standard deviation, the more uneven the projection plane is, and the greater its curvature; conversely, the smaller the standard deviation, the closer the projection plane is to the ideal plane.

[0082] The proposed solution further analyzes the aforementioned 3D topographic data to quantify the degree of surface deviation of the projection surface. First, using the 3D coordinates of feature points reconstructed from the coded light image, a spatial plane equation representing the overall trend of the projection surface is fitted using mathematical methods. This fitted plane serves as a benchmark for evaluating the flatness of the projection surface. Next, the vertical distance from each feature point to this fitted plane is calculated as the degree of deviation of the feature point relative to the spatial plane equation of the projection surface; these distances directly reflect the degree of deviation of each local region from the ideal plane. To comprehensively evaluate the flatness of the entire projection surface, the standard deviation of these vertical distances is calculated. The standard deviation, as a statistical value, effectively characterizes the dispersion of all feature points relative to the fitted plane, thus objectively reflecting the overall surface deviation of the projection surface. This calculated standard deviation, or other statistical value, is defined as a surface deviation index. In this way, the above method can extract key geometric features, namely the flatness information of the projection surface, from the reconstructed 3D topographic data, thereby providing a clear basis for subsequent correction processes. When this indicator exceeds a preset threshold, it can accurately identify significant curvature on the projection surface, thereby triggering the construction and application of a nonlinear distortion compensation model to ensure high-quality image correction under different projection surface conditions.

[0083] In one specific embodiment, the three-dimensional coordinates of all feature points are fitted to a plane. After outliers are removed using the RANSAC algorithm, the optimal projection plane equation is fitted, i.e., the projection plane space equation: a×X + b×Y + c×Z + d=0, Where [a,b,c] is the plane normal vector, and d is the distance from the plane to the origin. From this, the angle between the projector's optical axis and the normal vector of the projection plane (i.e., the side projection deflection angle) can be calculated: theta_h=arccos(|n_plane×v_optical| / (|n_plane|×|v_optical|)), Where n_plane is the normal vector of the projection surface, v_optical is the direction vector of the projector's optical axis, and theta_h is the horizontal side projection deflection angle.

[0084] Simultaneously, the fitting residuals of the feature points relative to the spatial plane equation of the projection surface are calculated as the degree of deviation of the feature points from the spatial plane equation of the projection surface. Furthermore, the standard deviation of the fitting residuals of the feature points relative to the spatial plane equation of the projection surface is calculated as an indicator of the degree of surface deviation. Statistical analysis is then performed on the fitting residuals. sigma_surf=sqrt((1 / N)×sum_{i=1}^{N} (n_plane×P_i + d)^2 / |n_plane|^2), Where sigma_surf is the standard deviation of the fitting residual, N is the number of feature points, and P_i is the three-dimensional spatial coordinate vector [X_i, Y_i, Z_i]^T of the i-th feature point. When sigma_surf is greater than the preset threshold T_curve, the projection surface is determined to be a non-ideal plane.

[0085] like Figure 4 As shown, in one embodiment, the method for solving the inverse transform parameters includes, but is not limited to, the following steps S401 to S402.

[0086] Step S401: The optimization objective function is constructed using the reprojection error of feature points in the 3D topography data under the mapping relationship, the rectangular constraint term used to constrain the four corners of the corrected image to be right angles, and the aspect ratio constraint term used to constrain the corrected image to maintain the original aspect ratio as optimization objectives.

[0087] Step S402: With the goal of minimizing the objective function, the objective function is iteratively optimized to obtain the inverse transformation parameters.

[0088] Reprojection error, in fields such as 3D reconstruction and camera calibration, refers to the distance between a reprojected point and the original image point after reprojecting a 3D spatial point onto a 2D image plane using a projection model. In projector side-projection trapezoidal correction methods, reprojection error measures the deviation between the projected point calculated using the current mapping relationship and the actually observed projected point, serving as a crucial indicator for evaluating the accuracy of the mapping relationship. Its calculation typically involves mapping feature points in the 3D topography data to a 2D image plane using inverse transform parameters to be solved, and then comparing these parameters with ideal 2D image points.

[0089] The rectangular constraint term is a component of the optimization objective function. Its purpose is to ensure that the corrected projected image presents a standard rectangular shape, meaning that the four corners of the corrected image should be right angles. In side projection trapezoidal correction, due to the influence of the projection angle and the shape of the projection surface, the edges of the unoptimized image may appear irregular or non-right angled.

[0090] The aspect ratio constraint is another component of the optimization objective function, aiming to maintain the inherent aspect ratio of the original projected image during the correction process. Without constraint, trapezoidal correction can result in stretching or compression of the corrected image, leading to content distortion. The aspect ratio constraint forces the corrected image to maintain the same width and height ratio as the original image, thus preventing content distortion. This constraint can be constructed by calculating the width-to-height ratio of the corrected image and ensuring it matches the aspect ratio of the original image; for example, by calculating the difference between the ratio of the longer side to the shorter side of the corrected image and the original aspect ratio.

[0091] This application further proposes a method based on an optimization objective function. This method first constructs a comprehensive optimization objective function, which incorporates the reprojection error of feature points in the 3D topography data under the mapping relationship, a rectangular constraint term to ensure the four corners of the corrected image are right angles, and an aspect ratio constraint term to ensure the corrected image maintains its original aspect ratio. The reprojection error term ensures that the solved inverse transform parameters make the physical image contour of the projector match the ideal image contour as closely as possible in space, meaning the corrected image accurately covers the target area. Simultaneously, the rectangular and aspect ratio constraint terms strictly limit the correction results from both geometric and proportional dimensions. The rectangular constraint term penalizes non-right-angled corners of the corrected image, forcing regular image edges and avoiding the distortion or deformation problems associated with trapezoidal correction. The aspect ratio constraint term ensures that the corrected image content is not stretched or compressed, maintaining the visual proportions of the original image and thus avoiding image content distortion. By integrating these interrelated errors and constraints into a single optimization objective function, this method can comprehensively evaluate the quality of the inverse transform parameters. Subsequently, the objective function is iteratively optimized to minimize it. During the iteration process, the algorithm continuously adjusts the inverse transform parameters to reduce reprojection error while better satisfying the rectangle and aspect ratio constraints. This iterative optimization mechanism effectively finds a balance point among multiple mutually constraining objectives, resulting in a set of optimal inverse transform parameters. These parameters not only achieve accurate trapezoidal correction but also ensure that the corrected image has good geometry and a correct aspect ratio, significantly improving the quality of the correction effect and the visual experience.

[0092] In a specific embodiment, the expression for the optimization objective function is: J(H)=sum_{i=1}^{N_pts} ||Q'_i - f_H(Q_i)||^2 + lambda_1×E_rect +lambda_2×E_aspect, Where J(H) is the output of the objective function, sum_{i=1}^{N_pts} ||Q'_i - f_H(Q_i)||^2 is the reprojection error of the feature points under the mapping relationship, E_rect is the rectangle constraint term, E_aspect is the aspect ratio constraint term, and lambda_1 and lambda_2 are the weights of the constraint terms, respectively.

[0093] The expression for the rectangular constraint term is: E_rect=sum_{j=1}^{4} |dot(v_{j},v_{j+1})|, Where v_j is the direction vector of the j-th edge, v_{j+1} is the direction vector of the adjacent edge (subscript modulo), and dot is the vector dot product. When all four corners are right angles, E_rect = 0.

[0094] The expression for the aspect ratio constraint is: E_aspect=(W_out / W_in - H_out / H_in)^2, Where W_in and H_in are the width and height of the original image, respectively, and W_out and H_out are the width and height of the corrected image, respectively.

[0095] The above objective function is optimized by nonlinear minimization: H_{k+1}=H_k - (J^T×J + mu×I)^(-1)×J^T×r, Where J is the Jacobian matrix, r is the residual vector, mu is the damping factor, and I is the identity matrix. Iteration until convergence yields the optimal homography matrix H_opt.

[0096] like Figure 5 As shown, in one embodiment, the method for constructing a nonlinear distortion compensation model includes, but is not limited to, the following steps S501 to S503.

[0097] Step S501: Select some feature points from the three-dimensional topography data as the center points of the radial basis function.

[0098] Step S502: Determine the spatial distance between the center point and the feature points in the three-dimensional topography data, and construct the kernel function matrix based on the actual mapping deviation at each feature point.

[0099] Step S503: The kernel function matrix is ​​solved using the regularized least squares method to obtain the weight coefficients corresponding to each center point. The weight coefficients are used as parameters of the radial basis function to obtain the nonlinear distortion compensation model.

[0100] When constructing a nonlinear distortion compensation model, representative points need to be selected from the reconstructed 3D topography data as the center points of the radial basis functions. These center points can be uniformly distributed on the projection surface, for example, obtained through grid sampling or random sampling; or they can be adaptively selected based on regions with large curvature changes on the projection surface to better capture local distortion features.

[0101] The kernel function matrix is ​​a matrix composed of radial basis function values, where each element represents the radial basis function value between a feature point and a center point. This matrix, combined with the actual mapping deviation, is used to subsequently solve for the parameters of the radial basis functions.

[0102] This application's scheme utilizes the powerful nonlinear fitting capability of the Radial Basis Function (RBF) combined with the regularized least squares method to construct a robust and accurate nonlinear distortion compensation model. Specifically, when the surface deviation of the projection plane is significant, a representative subset of feature points is strategically selected from the reconstructed 3D topography data as the center points of the RBF, serving as the foundation of the model. Subsequently, the spatial distances between these center points and all feature points in the 3D topography data are calculated; these distances are the inputs to the RBF kernel. Simultaneously, training data is provided for the model based on the actual mapping deviation of each feature point on the projection plane—the difference between the ideal projection position and the actual observation position. Based on these spatial distances and actual mapping deviations, a kernel function matrix is ​​constructed, reflecting the nonlinear relationship between feature points and center points, as well as the corresponding distortion information. To ensure the model's stability and generalization ability, the regularized least squares method is used to solve for the kernel function matrix, yielding the weight coefficients corresponding to each center point. These weight coefficients, as parameters of the RBF, collectively define the final nonlinear distortion compensation model. This model can calculate the local offset of any pixel on the projection surface based on the image coordinates of any pixel in the original projected image, thereby achieving accurate compensation for nonlinear distortions of complex surfaces. This method combines 3D topographic data with actual mapping deviations, effectively solving the challenge of modeling nonlinear distortions of complex surfaces through the local approximation characteristics of RBF and the optimization capability of regularized least squares, ensuring that the corrected projected image has higher accuracy and visual quality.

[0103] In one specific embodiment, when the surface deviation index exceeds a preset threshold, nonlinear distortion compensation is superimposed on the inverse transform parameters. A radial basis function (RBF) interpolation model is used to model the residual distortion. delta_x(u,v)=sum_{i=1}^{N_rbf} w_i×phi(||[u,v] - [u_i,v_i]||), delta_y(u,v)=sum_{i=1}^{N_rbf} w'_i×phi(||[u,v] - [u_i,v_i]||), Where phi(||[u,v] - [u_i,v_i]||) is the Gaussian radial basis function, [u_i,v_i] is the coordinate of the RBF center point, w_i and w'_i are the weight coefficients to be determined, sigma_rbf is the shape parameter, delta_x(u,v) and delta_y(u,v) are the spatial distances between the center point and the feature points in the 3D topography data, respectively, and N_rbf is the number of center points selected in the RBF interpolation model.

[0104] The weighting coefficients are solved using the regularized least squares method: [w_1,...,w_N]^T=(Phi^T×Phi + lambda_reg×I)^(-1)×Phi^T×delta_obs, Where Phi is the RBF kernel matrix, Phi_{ij}=phi(||c_i - c_j||), delta_obs is the observed distortion bias vector, and lambda_reg is the regularization coefficient.

[0105] The final nonlinear distortion compensation model is as follows: [u',v']^T=H_opt×[u,v,1]^T + [delta_x(u,v),delta_y(u,v)]^T.

[0106] In some embodiments, the projector side projection trapezoidal correction method further includes: determining the scaling factor of each local region in the corrected projection image based on the local transformation characteristics of the inverse transform parameters; determining the compensation intensity of the local region based on the scaling factor; performing high-frequency component enhancement processing on the resolution loss region in the frequency domain to obtain the enhanced resolution loss region; and / or sampling the original projection image in the spatial domain using a sharpening interpolation kernel to sample the resolution loss region and a smoothing interpolation kernel to sample other local regions; the resolution loss region is a local region where the local scaling factor is less than a preset reference value.

[0107] The scaling factor is used to assess the degree of resolution loss in a local region. The scaling factor reflects the density change of image pixels during the mapping process from the original image to the corrected image. For example, the local scaling factor can be calculated by analyzing the Jacobian matrix of the inverse transform parameters in the local region; the determinant of the Jacobian matrix characterizes the scaling ratio of the local area. Another approach is to divide the original image into a grid, calculate the positions of the grid vertices in the corrected image using the inverse transform parameters, and then compare the ratio of the original grid area to the transformed grid area to determine the local scaling factor. When the scaling factor is less than 1, it indicates that the pixels in that local region are compressed, which may lead to loss of detail, i.e., resolution loss; when the scaling factor is greater than 1, it indicates that the pixels in that local region are stretched, which may lead to pixelation or blurring. By setting a preset reference value, regions with scaling factors less than this reference value can be identified as resolution loss regions.

[0108] This application's solution, building upon the completion of projector side projection keystone correction and nonlinear distortion compensation, further addresses the potential resolution loss during the correction process. The solution first analyzes the local transformation characteristics of the inverse transform parameters to accurately calculate the scaling factors for each local region in the corrected projected image. These scaling factors directly reflect the degree of stretching or compression of the image in different regions, thus enabling the assessment of the resolution loss in each local region. Specifically, when a local scaling factor is less than a preset reference value, it indicates that the pixels in that region are compressed, resulting in resolution loss. To compensate for this resolution loss, this application's solution dynamically determines the required compensation intensity for each local region based on the assessed scaling factors. For regions with resolution loss, two complementary or optional enhancement strategies can be employed. One strategy involves high-frequency component enhancement in the frequency domain. This method obtains the frequency domain coefficients by performing a time-frequency transform on the resolution-loss region, then selectively applies high-frequency boost gain to these coefficients to restore image details and sharpness, and finally obtains the enhanced region through an inverse time-frequency transform. Another strategy involves using a sharpening interpolation kernel for sampling regions with resolution loss during spatial sampling to enhance edges and details, while using a smoothing interpolation kernel for other non-resolution loss regions to maintain a natural image transition. Through this mechanism, the proposed solution can identify and specifically compensate for local resolution loss during the correction process, avoiding the overall image quality degradation that may result from traditional correction methods. This localized, adaptive enhancement processing ensures that the corrected projected image not only has a regular geometry but also effectively preserves visual details, thereby significantly improving the overall display effect of the projected image.

[0109] The following is a concrete example to illustrate this. After the projector performs side projection keystone correction, it first calculates the mapping position of each pixel in the original projected image in the corrected image based on the solved inverse transform parameters. During this process, these mapping relationships can be further analyzed. For example, by calculating the Jacobian matrix of each small region (e.g., a 16x16 pixel block) and extracting its determinant value as the local scaling factor for that region. Assuming a preset reference value of 0.95, when the scaling factor of a local region is less than 0.95, that region is identified as a resolution loss region. For these resolution loss regions, the compensation intensity is determined based on the scaling factor. For example, the smaller the scaling factor, the higher the compensation intensity. In terms of frequency domain enhancement, a short-time Fourier transform is performed on the image data of the resolution loss region to obtain its coefficients at different times and frequencies. Then, a high-frequency boosting filter, such as a Gaussian high-pass filter, can be designed, with its gain correlated with the previously determined compensation intensity, weighting the frequency domain coefficients. For example, for the high-frequency part, the gain can be set to 1.5 to 2.0, while for the low-frequency part, the gain can remain at 1. After weighting, an inverse short-time Fourier transform is performed to convert the enhanced frequency domain coefficients back to the spatial domain, resulting in image regions with richer details. As an alternative or supplementary implementation, when sampling the original projected image to generate the corrected projected image, the interpolation kernel is dynamically selected based on the previously identified resolution loss regions. For resolution loss regions, the Lanczos interpolation kernel can be used for sampling, as it has good sharpening effects and can effectively restore image details. For other non-resolution loss regions, a bicubic interpolation kernel can be used to ensure smooth image transitions and avoid introducing unnecessary artifacts. In this way, the system can specifically improve the visual quality of resolution loss regions while maintaining good display effects in other regions.

[0110] In some embodiments, the projector side projection keystone correction method further includes: extracting the actual edge contour of the corrected projection image and the target edge contour of the projection surface; calculating the Hausdorff distance between the actual edge contour and the target edge contour, and using the Hausdorff distance as the alignment deviation between the actual edge contour and the target edge contour; when the alignment deviation exceeds a preset deviation threshold, using translation, rotation, and scaling as variables to be optimized, minimizing the distance between corresponding points between the actual edge contour and the target edge contour through an iterative nearest point method to solve for the target fine-tuning amount; and superimposing the target fine-tuning amount onto the inverse transform parameters.

[0111] After initially completing the trapezoidal correction for side projection, this application introduces a fine-tuning mechanism to further improve the alignment accuracy of the projected image. First, the actual edge contour of the currently corrected projected image is identified from the image captured by the visual sensor and compared with the pre-defined target edge contour of the projection surface. To quantify the degree of mismatch, the Hausdorff distance is used as a measure of alignment deviation, which robustly reflects the maximum difference between the two contours. When the calculated alignment deviation exceeds a preset tolerance threshold, further fine-tuning is required. At this point, an optimization process is initiated, using translation, rotation, and scaling as adjustable variables, and employing the Iterative Closest Point (ICP) method to find an optimal geometric transformation. This ICP method repeatedly searches for the closest correspondence between the actual contour points and the target contour points, and calculates the transformation parameters that minimize the distance between these corresponding points, thereby solving for the target fine-tuning amount that can accurately align the actual contour to the target contour. Finally, these solved target fine-tuning amounts are superimposed on the previously calculated inverse transformation parameters in a composite operation. In this way, subsequent projection image sampling will be based on finely adjusted inverse transform parameters, thereby enabling the projected image to be more accurately aligned with the target area of ​​the projection surface.

[0112] The following is a concrete example to illustrate this. After solving the inverse transformation parameters based on the 3D topographic data and mapping relationship, the following steps can be performed. First, the corrected projection image projected onto the projection surface by the projector is captured using a vision sensor connected to the projector. The Canny edge detection algorithm is applied to this image to extract the actual edge contour of the corrected projection image, which can be represented as a series of ordered pixel coordinates. Simultaneously, based on the geometric model of the projection surface or a user preset, an ideal rectangle or polygon can be generated as the target edge contour of the projection surface, also represented as a series of point coordinates. Then, the Hausdorff distance between these two point sets is calculated, for example, using functions provided in an image processing library. If the calculated Hausdorff distance is greater than a preset threshold of 0.5 pixels, a fine-tuning process is initiated. At this point, the actual edge contour point set and the target edge contour point set are used as input, and the Iterative Closest Point (ICP) algorithm is used for registration. In the ICP algorithm, the maximum number of iterations can be set to 50, and the convergence threshold to 0.001. The ICP algorithm outputs a transformation matrix containing translation, rotation, and scaling parameters, which is the target fine-tuning amount. Finally, the geometric transformation represented by this target fine-tuning amount is combined with the previously solved inverse transformation parameters. For example, if the inverse transformation parameters are represented as a 3x3 homogeneous transformation matrix M_inv, and the target fine-tuning amount is represented as a 3x3 transformation matrix M_fine_tune, then the updated inverse transformation parameters M_updated = M_fine_tune × M_inv. The projector will then use M_updated to generate the final corrected projected image.

[0113] In some embodiments, the projector side projection trapezoidal correction method further includes: during the projection output process, acquiring the pose change of the projector; when the translation amount in the pose change exceeds a preset translation threshold or the rotation amount exceeds a preset rotation threshold, deriving an incremental transformation matrix based on the pose change; and performing a composite operation on the incremental transformation matrix and the inverse transformation parameters to obtain the updated inverse transformation parameters.

[0114] During projection output, the pose changes of the projector are continuously acquired. When the translation or rotation of these pose changes exceeds a preset threshold, it indicates that the projector's pose has changed significantly and the correction parameters need to be updated. At this point, based on the detected pose changes, an incremental transformation matrix is ​​derived, which accurately describes the spatial transformation of the projector from the last calibrated or updated pose to the current new pose. Subsequently, this incremental transformation matrix is ​​combined with the currently used inverse transformation parameters. This combination operation, such as through matrix multiplication, incorporates the latest pose information of the projector into the original inverse transformation parameters, thereby obtaining an updated inverse transformation parameter. In this way, the time-consuming and complex encoded light projection and 3D reconstruction process can be repeated, enabling rapid and real-time adjustment of correction parameters. This ensures that even if the projector's pose changes during projection, the projected image still maintains accurate keystone correction and nonlinear distortion compensation effects.

[0115] The following is a concrete example. A miniature inertial measurement unit (IMU) can be integrated inside the projector, such as a six-axis or nine-axis sensor like the STMicroelectronics LSM6DSOX or the Bosil BMI270. This IMU continuously acquires acceleration and angular velocity data during projector operation and estimates the projector's pose change relative to its initial calibration time in real time using Kalman filtering or complementary filtering algorithms. This includes three-dimensional translation vectors and three-dimensional rotation quaternions. A translation threshold, such as 5 mm, and a rotation threshold, such as 0.5 degrees, are preset. When the cumulative translation component in any direction exceeds 5 mm, or the cumulative rotation angle in any axis exceeds 0.5 degrees, the inverse transformation parameter update process is triggered. Once the update is triggered, a 4x4 homogeneous incremental transformation matrix is ​​constructed using the currently detected pose change. This matrix transforms the projector coordinate system from the old pose to the new pose. Assuming the current inverse transform parameters can be represented as a 4x4 homogeneous transform matrix, the updated inverse transform parameters will be calculated through matrix multiplication. Through this composite operation, the new `T_inverse_updated` matrix accurately reflects the projector's correction requirements at its latest pose, allowing for precise pixel mapping using these updated parameters directly in subsequent projected image sampling.

[0116] This application also provides an electronic device. Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. For example... Figure 6As shown, the electronic device in this embodiment mainly includes a processor 601 and a memory 602. The memory 602 can be configured to store a program for executing the projector side-projection keystone correction method of the above-described method embodiments. The processor 601 can be configured to execute the program in the memory 602, which includes, but is not limited to, a program for executing the projector side-projection keystone correction method of the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application.

[0117] In some embodiments, the electronic device may further include multiple processors 601 and multiple memories 602. The program executing the projector side-projection keystone correction method of the above method embodiments can be divided into multiple subroutines. Each subroutine can be loaded and run by a processor 601 to execute different steps of the projector side-projection keystone correction method of the above method embodiments. Specifically, each subroutine can be stored in a different memory 602, and each processor 601 can be configured to execute programs in one or more memories 602 to jointly implement the projector side-projection keystone correction method of the above method embodiments. That is, each processor 601 executes different steps of the projector side-projection keystone correction method of the above method embodiments to jointly implement the projector side-projection keystone correction method of the above method embodiments.

[0118] The aforementioned multiple processors 601 can be processors deployed on the same device. For example, the aforementioned electronic device can be a high-performance device composed of multiple processors, and the aforementioned multiple processors 601 can be processors configured on that high-performance device. Alternatively, the aforementioned multiple processors 601 can also be processors deployed on different devices. For example, the aforementioned electronic device can be a server cluster, and the aforementioned multiple processors 601 can be processors on different servers within the server cluster.

[0119] This application also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the projector side-projection keystone correction method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described projector side-projection keystone correction method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a memory formed by various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0120] The projector side projection trapezoidal correction method, electronic device, and storage medium provided in this application reconstruct the three-dimensional topography data of the projection surface. When the surface deviation index of the projection surface determined based on the three-dimensional topography data exceeds a preset threshold, a nonlinear distortion compensation model is constructed based on the three-dimensional topography data, and the local region offset of each pixel is calculated. The local region offset is superimposed to the base mapping position, and the original projection image is precisely sampled to achieve projector side projection trapezoidal correction. Thus, by reconstructing the three-dimensional topography data of the projection surface, the geometric characteristics of the projection surface can be truly perceived, rather than relying solely on the planar assumption. By constructing a nonlinear distortion compensation model and calculating the local region offset of each pixel, and then superimposing the local region offset to the base mapping position, fine-grained pixel-level adjustments can be made for the curvature differences of different regions on the projection surface, generating a visually perfectly square, straight-edge, and locally distortion-free corrected projection image, eliminating residual trapezoidal distortion under large-angle side projection or non-planar projection surfaces.

[0121] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for correcting trapezoidal distortion in a side-projection projector, characterized in that, include: Acquire the image of the coded light reflected from the projection surface after the projector projects a coded light pattern onto the projection surface; Based on the coded light image, the three-dimensional topography data of the projection surface is reconstructed; Based on the three-dimensional topographic data, the surface deviation index of the projection surface is determined; Based on the mapping relationship between the ideal image contour and the physical image contour projected by the projector onto the projection surface, the inverse transformation parameters are solved; Based on the inverse transform parameters, the basic mapping position of each pixel in the original projected image is determined; When the surface deviation index exceeds a preset threshold, a nonlinear distortion compensation model is constructed based on the three-dimensional topography data. The image coordinates of each pixel in the original projected image are input into the nonlinear distortion compensation model to obtain the corresponding local region offset. The local region offset is superimposed onto the base mapping position, and the original projection image is sampled at the superimposed position to obtain the corrected projection image.

2. The projector side projection trapezoidal correction method according to claim 1, characterized in that, The process of reconstructing the three-dimensional topography data of the projection surface based on the coded light image includes: Feature points are extracted from the coded light image; Determine the correspondence between the feature points in the image coordinate system of the visual sensor that acquires the coded light image and the image coordinate system of the projector; Based on the correspondence and the calibration parameters between the visual sensor and the projector, the three-dimensional coordinates of the feature points in space are reconstructed to obtain the three-dimensional shape data of the projection surface.

3. The projector side projection trapezoidal correction method according to claim 1, characterized in that, The determination of the surface deviation index of the projection surface based on the three-dimensional topography data includes: Based on the three-dimensional coordinates of the feature points in the three-dimensional topography data, fit the equation of the projection plane space. Determine the degree of deviation of the feature point relative to the spatial plane equation of the projection surface; Statistical values ​​are calculated based on the degree of deviation of the feature points, and these values ​​serve as an index of the degree of surface deviation.

4. The projector side projection trapezoidal correction method according to claim 1, characterized in that, The inverse transformation parameters are calculated based on the mapping relationship between the ideal image contour and the physical image contour projected onto the projection surface by the projector, including: An optimization objective function is constructed using the reprojection error of feature points in the three-dimensional topography data under the mapping relationship, the rectangular constraint term used to constrain the four corners of the corrected image to be right angles, and the aspect ratio constraint term used to constrain the corrected image to maintain the original aspect ratio as optimization objectives. The objective function is minimized, and the objective function is iteratively optimized to obtain the inverse transform parameters.

5. The projector side projection trapezoidal correction method according to claim 1, characterized in that, The construction of a nonlinear distortion compensation model based on the three-dimensional topography data includes: Select a subset of feature points from the three-dimensional topography data as the center points of the radial basis function; Determine the spatial distance between the center point and the feature points in the three-dimensional topography data, and construct a kernel function matrix based on the actual mapping deviation at each feature point; The kernel function matrix is ​​solved using the regularized least squares method to obtain the weight coefficients corresponding to each center point. The weight coefficients are then used as parameters of the radial basis function to obtain the nonlinear distortion compensation model.

6. The projector side projection trapezoidal correction method according to claim 1, characterized in that, The projector side projection trapezoidal correction method further includes: Based on the local transformation characteristics of the inverse transform parameters, the scaling factor of each local region in the corrected projected image is determined; the scaling factor is used to evaluate the degree of resolution loss in the local region. Based on the scaling factor, the compensation intensity of the local region is determined; In the frequency domain, high-frequency component enhancement processing is performed on the resolution loss region to obtain the enhanced resolution loss region, and / or when sampling the original projected image in the spatial domain, a sharpening interpolation kernel is used to sample the resolution loss region, and a smoothing interpolation kernel is used to sample other local regions; the resolution loss region is the local region where the local scaling factor is less than a preset reference value.

7. The projector side projection trapezoidal correction method according to claim 1, characterized in that, The projector side projection trapezoidal correction method further includes: Extract the actual edge contour of the corrected projection image and the target edge contour of the projection surface; Calculate the Hausdorff distance between the actual edge contour and the target edge contour, and use the Hausdorff distance as the alignment deviation between the actual edge contour and the target edge contour; When the alignment deviation exceeds a preset deviation threshold, the translation, rotation, and scaling are used as variables to be optimized. The distance between corresponding points between the actual edge contour and the target edge contour is minimized by the iterative nearest point method to solve for the target fine-tuning amount. The target fine-tuning amount is superimposed on the inverse transform parameter.

8. The projector side projection trapezoidal correction method according to claim 1, characterized in that, The projector side projection trapezoidal correction method further includes: During the projection output process, the pose change of the projector is acquired; When the translation amount in the pose change exceeds a preset translation threshold or the rotation amount exceeds a preset rotation threshold, an incremental transformation matrix is ​​derived based on the pose change. The incremental transformation matrix and the inverse transformation parameters are combined to obtain the updated inverse transformation parameters.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the projector side projection keystone correction method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the projector side projection keystone correction method according to any one of claims 1 to 8.

Images