Projection screen image reconstruction method, system and medium combined with distortion correction

Abstract

The application discloses a projection screen image reconstruction method and system combining distortion correction, and a medium, and relates to the technical field of image processing.The method comprises the following steps: a target projection device projects a calibration image to a screen, and a perception device group collects the projection result; geometric distortion is identified based on a calibration pattern library, and the screen distortion state is obtained; the distortion state is input into a multi-channel distortion correction model, and geometric, color and brightness correction decisions are made; the real-time projection image is reconstructed accordingly, and the reconstructed image is projected to the screen again. The technical effect of real-time, accurate and comprehensive reconstruction of the projection image is achieved, and the image quality and display stability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a projection screen image reconstruction method and system combined with distortion correction and a medium. BACKGROUND

[0002] Generally, a projection device projects an image onto a projection screen for viewers to watch. However, in the actual projection process, due to the complexity of the projection environment, such as changes in the projection angle, unevenness of the screen surface, and limitations of optical elements, the actual projected image often has problems such as geometric distortion, color deviation, and uneven brightness. These distortions not only affect the visual effect of the image, but also seriously affect the visual effect of the projected image and the accuracy of information transmission, reducing the viewing experience and understanding effect of the audience on the projected content.

[0003] Existing projection distortion correction techniques mostly only correct a single type of distortion, such as only performing simple linear transformation correction on geometric distortion, or only performing rough adjustment on the uniformity of color or brightness. These techniques cannot comprehensively and effectively correct the geometric distortion, color distortion, and brightness distortion of the projection screen. In actual application, even if the geometric distortion is corrected, the color of the image may deviate, and the brightness may not be uniform, and vice versa. Moreover, the correction methods used in existing technologies often require manual adjustment of parameters, which is tedious and difficult to accurately control, and cannot adapt to the diversified needs under different projection environments and screen conditions, resulting in unsatisfactory correction effect and failing to meet the user's requirements for high-quality projected images. SUMMARY

[0004] The present application provides a projection screen image reconstruction method combined with distortion correction, a system, and a medium to solve the technical problems of single correction dimension, tedious operation, and unsatisfactory correction effect in the prior art, realize real-time, accurate, and comprehensive reconstruction of projected images, and improve the image quality and display stability.

[0005] In a first aspect, the present application provides a projection screen image reconstruction method combined with distortion correction, wherein the projection screen image reconstruction method combined with distortion correction comprises:

[0006] A target projection device projects a calibration image onto a target projection screen, and a perception device group collects the calibration image projection result.

[0007] Based on a preset calibration pattern library, the calibration image projection result is analyzed to identify geometric distortion and obtain projection screen geometric distortion state information.

[0008] inputting the distortion state information into a pre-constructed multi-channel distortion correction model to make a distortion correction decision, wherein the multi-channel distortion correction model includes a geometric correction channel, a color correction channel, and a brightness correction channel.

[0009] Based on the distortion correction decision result, the received real-time projection image is reconstructed, and the real-time reconstruction image obtained through image reconstruction is projected to the target projection screen through the target projection device.

[0010] In a feasible implementation, projecting the calibration image to the target projection screen through the target projection device includes:

[0011] Obtaining the physical shape parameters of the target projection screen, the pixel density parameters of the target projection device, and the size parameters of the target projection area.

[0012] Accessing a pre-set calibration pattern library, and extracting a corresponding standard calibration pattern from the calibration pattern library as an index condition of the screen shape parameters, pixel density parameters, and projection size parameters.

[0013] Constructing a calibration image in combination with the standard calibration pattern, wherein the calibration image includes structured pattern information for geometric calibration.

[0014] Projecting the calibration image to the target projection screen through the target projection device.

[0015] In a feasible implementation, acquiring the calibration image projection result through the perception device group includes:

[0016] Deploying a multi-modal perception device in the target projection environment, wherein the multi-modal perception device includes a brightness test device for brightness measurement, a colorimetric test device for color detection, and a high-resolution image acquisition device for image acquisition.

[0017] Activating the multi-modal perception device to respectively acquire geometric image information, color distribution information, and brightness distribution information of the projection calibration image projected on the target projection screen.

[0018] Performing spatial remapping processing on the acquired geometric image information, color distribution information, and brightness distribution information, and uniformly converting them into an ideal projection screen spatial coordinate system to form the calibration image projection result of the calibration image.

[0019] In a feasible implementation, based on the pre-set calibration pattern library, the calibration image projection result is analyzed to identify geometric distortion, and the projection screen geometric distortion state information is obtained, including:

[0020] Performing pattern recognition matching in the calibration image projection result, obtaining a matched image pair set, wherein the matched image pair includes the standard calibration pattern and the corresponding distortion calibration pattern.

[0021] Performing geometric deformation recognition on the matched image pair set, including scaling, rotation, translation and perspective distortion, to obtain a first recognition result.

[0022] Performing brightness consistency evaluation and chrominance deviation evaluation on the matched image pair set to obtain a second recognition result.

[0023] Respectively performing feature extraction on the first recognition result and the second recognition result, and fusing the feature extraction results to generate the projection screen geometric distortion state information.

[0024] In a feasible implementation, respectively performing feature extraction on the first recognition result and the second recognition result, and fusing the feature extraction results to generate the geometric distortion parameters of the target projection screen further includes:

[0025] Performing primary geometric distortion recognition on the first recognition result, including extracting scaling features, rotation features, translation features and perspective deformation features related to the flatness of the screen.

[0026] Based on the second recognition result, performing deep correction on the primary geometric distortion recognition result to obtain distortion depth features. Wherein, the deep correction includes inferring the directionality of the physical distortion of the target projection screen according to the brightness and chrominance deviation information.

[0027] Performing feature fusion on the distortion depth features and the distortion depth features, and outputting the feature fusion result as the geometric distortion parameters of the target projection screen.

[0028] In a feasible implementation, obtaining a pre-constructed multi-channel distortion correction model includes:

[0029] Based on the inverse transformation algorithm, defining and constructing the geometric correction channel in matrix form.

[0030] Obtaining sample color deviation state information as training data, constructing and training a color mapping network based on a machine learning model, and outputting the color correction channel, wherein the sample color deviation state information includes sample projection device characteristic parameters, sample light incidence angle, sample projection color value, and sample color deviation index.

[0031] Obtaining sample brightness deviation state information and constructing a brightness mapping model by function fitting method, wherein the sample brightness deviation state information includes sample calibration viewing position, sample light incidence angle offset, and sample brightness attenuation amount.

[0032] The geometry correction channel, the color correction channel and the brightness correction channel are integrated to obtain the multi-channel distortion correction model.

[0033] In an available implementation, the distortion state information is input to a pre-constructed multi-channel distortion correction model to make a distortion correction decision, which includes:

[0034] The distortion state information is input to the geometry correction channel to initialize the channel, and a geometry correction vector is calculated based on the initialized geometry correction channel to obtain a geometry correction decision matrix.

[0035] According to the geometry correction decision matrix and the distortion state information, a derived distortion point is located to obtain a derived distortion point set.

[0036] A mapping relationship between the derived distortion point set and the distortion state information is established to extract a derived distortion state parameter.

[0037] The derived distortion state parameter is input to the color correction channel and the brightness correction channel respectively, a color correction vector and a brightness correction vector are calculated, and a color correction decision matrix and a brightness correction decision matrix are constructed correspondingly.

[0038] The geometry correction decision matrix, the color correction decision matrix and the brightness correction decision matrix are output as the distortion correction decision result.

[0039] In an available implementation, the inverse transformation algorithm includes at least one of a homography inverse transformation, an affine inverse transformation and a perspective inverse transformation.

[0040] In a second aspect, the application further provides a projection screen image reconstruction system combined with distortion correction, which includes:

[0041] A calibration image projection and acquisition module is configured to project a calibration image to a target projection screen through a target projection device, and acquire a calibration image projection result through a perception device group.

[0042] A geometric distortion identification module is configured to analyze the calibration image projection result based on a pre-set calibration pattern library to identify geometric distortion and obtain geometric distortion state information of the projection screen.

[0043] A distortion correction decision module is configured to input the distortion state information to a pre-constructed multi-channel distortion correction model to make a distortion correction decision, wherein the multi-channel distortion correction model includes a geometry correction channel, a color correction channel and a brightness correction channel.

[0044] The image reconstruction and real-time projection module is used for image reconstruction on the received real-time projection image based on the distortion correction decision result, and projecting the real-time reconstruction image obtained through the image reconstruction to the target projection screen through the target projection device.

[0045] In a third aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the projection screen image reconstruction method combined with distortion correction.

[0046] The present application discloses a projection screen image reconstruction method combined with distortion correction, a system and a medium, which comprises the following steps: projecting a calibration image to a target projection screen through a target projection device, and collecting the projection effect of the image by using a perception device group; analyzing the collected image in combination with a preset calibration pattern library, identifying and extracting the geometric distortion state information of the projection screen; inputting the distortion state information into a pre-constructed multi-channel distortion correction model, which comprises three channels of geometric correction, color correction and brightness correction, for generating a corresponding distortion correction decision; performing image reconstruction processing on the received real-time projection image according to the correction decision, and projecting the reconstructed image to the target screen through the target projection device again to realize dynamic optimization of the projection effect. The projection screen image reconstruction method combined with distortion correction disclosed in the present application solves the technical problems of single correction dimension, complicated operation and unsatisfactory correction effect, realizes real-time, accurate and comprehensive reconstruction of the projection image, and improves the technical effects of image quality and display stability. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 It is a flowchart of the projection screen image reconstruction method combined with distortion correction of the present application.

[0048] Figure 2 It is a structure diagram of the projection screen image reconstruction system combined with distortion correction of the present application.

[0049] Mark explanation: calibration image projection and collection module 11, geometric distortion identification module 12, distortion correction decision module 13, image reconstruction and real-time projection module 14. DETAILED DESCRIPTION

[0050] The above technical solutions will be described in detail below in combination with the accompanying drawings and specific embodiments, so that the above technical solutions can be better understood. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments for explaining the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, not all.

[0051] Embodiment one, as Figure 1 The flowchart of the projection screen image reconstruction method combined with distortion correction of the present application, wherein the projection screen image reconstruction method combined with distortion correction comprises:

[0052] S100: projecting a calibration image to a target projection screen by a target projection device, and collecting the projection result of the calibration image by a perception device group.

[0053] Specifically, first, a preset calibration image is projected to a target projection screen by a target projection device (such as a DLP projector, an LCD projection device, etc.), which is used for subsequent distortion detection and modeling. The calibration image can include: a regular grid pattern (such as a rectangular grid, an orthogonal grid); a diagonal cross pattern; a multi-scale checkerboard pattern; a specific coding pattern (such as Gray coding, structured light coding pattern), etc.

[0054] Specifically, the perception device group deployed at the predetermined position synchronously collects the calibration image on the projection screen, and the acquired image data should include: the distortion characteristics caused by the deformation information of the projected calibration image, the local concave-convex, bending, texture disturbance, etc. of the projection screen surface.

[0055] The role of this step is to provide accurate raw data for subsequent geometric distortion identification. By projecting a calibration image and collecting the projection result, the actual image information presented on the projection screen can be obtained, which is compared and analyzed with the standard calibration pattern to identify the geometric distortion state information, laying a foundation for subsequent distortion correction decision.

[0056] In the collection process, in some embodiments, the target projection device projects a calibration image to the target projection screen, comprising:

[0057] Obtaining a physical shape parameter of a target projection screen, a pixel density parameter of a target projection device, and a size parameter of a target projection area; accessing a preset calibration pattern library, and extracting a corresponding standard calibration pattern from the calibration pattern library as an index condition of the screen shape parameter, the pixel density parameter, and the projection size parameter; constructing a calibration image in combination with the standard calibration pattern, wherein the calibration image includes structured pattern information for geometric calibration; and projecting the calibration image to the target projection screen through the target projection device.

[0058] Specifically, the calibration pattern library is a database that stores a plurality of standard calibration patterns, which have known geometric shapes, sizes, and color characteristics, and are used as a reference for identifying distortion.

[0059] Specifically, first, parameter information related to the projection environment is obtained, including: a physical shape parameter of a target projection screen, such as the curvature radius, bending angle, surface fluctuation characteristics, etc. of the screen; a pixel density parameter of a target projection device, such as the number of pixels per unit area (dpi) or the actual physical size corresponding to a single pixel; and a size parameter of a target projection area, such as the width, height, diagonal size, etc. of the projection area. Then, a preset calibration pattern library is accessed, which stores a plurality of standard calibration patterns optimized for different screen shapes, pixel densities, and size specifications. Each standard calibration pattern corresponds to a set of index parameters, including: the applicable screen shape type; the recommended pixel density range; and the supported projection size range. Next, the screen physical shape parameter, the pixel density parameter, and the projection size parameter are used as index conditions to extract the most suitable standard calibration pattern from the calibration pattern library that matches the current projection environment. The matching strategy can include exact matching and nearest neighbor matching based on minimum distance.

[0060] Further, a calibration image suitable for the current projection environment is constructed in combination with the extracted standard calibration pattern. The calibration image includes at least structured pattern information for geometric calibration, which can include regular grids, checkerboards, coded lines, feature point arrays, etc. Optionally, the pattern is scaled, stretched, or adaptively transformed according to the actual projection size when necessary to ensure that the scale and distribution of the calibration pattern on the projection screen meet the expected design. The generated calibration image is projected onto the surface of the target projection screen by the target projection device for subsequent collection and processing by the perception device group.

[0061] Through the above implementation, the most suitable calibration pattern is dynamically selected according to the actual projection environment, which helps to improve the matching degree of the calibration image and the screen surface characteristics, ensure the structural integrity and geometric accuracy of the calibration pattern in the projection area, and improve the subsequent distortion modeling and correction accuracy.

[0062] In some embodiments, the calibration image projection result is collected by the group of sensing devices, including:

[0063] A multi-modal sensing device is deployed in a target projection environment, including a luminance testing device for luminance measurement, a chrominance testing device for color detection, and a high-resolution image acquisition device for image acquisition; the multi-modal sensing device is activated to collect geometric image information, color distribution information, and luminance distribution information of a projection calibration image projected on a target projection screen; the collected geometric image information, color distribution information, and luminance distribution information are spatially remapped and converted to an ideal projection screen spatial coordinate system to form the calibration image projection result of the calibration image.

[0064] Specifically, the multi-modal sensing device refers to a combination of devices integrating multiple different types of sensors for simultaneously collecting multiple types of image information. The luminance testing device is used to measure the luminance distribution of the projection image, i.e., the luminous intensity of each region of the image; the chrominance testing device is used to detect the color distribution of the image, i.e., the types and saturation of colors; and the high-resolution image acquisition device is used to obtain high-definition geometric image information to capture the details and shape features of the image.

[0065] Specifically, the ideal projection screen spatial coordinate system is a virtual three-dimensional coordinate system used for reference, which represents the spatial model of the projection screen under perfect projection conditions. In this coordinate system, the projection screen is assumed to be a completely flat ideal plane located at a specific position and orientation, and the projection image is displayed perfectly on this plane without any distortion. The origin is usually set at a specific position of the screen (e.g., the upper left corner or the center point), and the coordinate axes (e.g., X-axis, Y-axis, and Z-axis) correspond to the horizontal direction, vertical direction, and depth direction of the screen, respectively.

[0066] Specifically, first, a multi-modal sensing device is deployed in a target projection environment, including: a luminance testing device for measuring the luminance distribution of the projected image in different regions; a chrominance testing device for detecting the color distribution characteristics of the projected image, such as color temperature, color gamut coverage, color deviation, etc.; and a high-resolution image acquisition device for acquiring geometric shape information of the projected image, including pattern outline, feature point position, etc. The above sensing devices can exist in the form of a single device integration (such as a multi-modal sensor module) or work collaboratively as multiple independent devices.

[0067] Specifically, the multi-modal sensing device deployed above is activated to collect different modal information of the calibration image projected on the target projection screen, including:

[0068] Geometric image information: including the actual projection shape of the calibration pattern, edge profile, feature point coordinates, etc.; color distribution information: including the color values of each region (such as RGB, Lab or XYZ color space data); brightness distribution information: including the brightness values of each region (for example, measured data in units of cd / m² (candela per square meter)).

[0069] Optionally, multiple sampling and data fusion are performed during the acquisition process to improve the accuracy and stability of the data.

[0070] Further, spatial remapping processing is performed on the obtained geometric image information, color distribution information and brightness distribution information, that is, each modality data is uniformly converted to a preset ideal projection screen space coordinate system, and spatial distortion caused by factors such as sensing device position offset, viewing angle difference, screen curvature, etc. is corrected, thereby ensuring the spatial alignment relationship between different modalities of information, ensuring that the geometric, color and brightness information at the same spatial position correspond one-to-one, forming a calibration image projection result data set corresponding to the ideal coordinate system, which is used as the basis data for subsequent distortion modeling, color correction and brightness compensation.

[0071] S200: Based on the preset calibration pattern library, analyze the calibration image projection result to identify geometric distortion and obtain projection screen geometric distortion state information.

[0072] Specifically, the calibration pattern library is a database that stores a plurality of standard geometric patterns and their corresponding feature information. By comparing the differences between the calibration image projection result and the corresponding calibration pattern in the standard calibration pattern, the deformation of the image on the projection screen in terms of geometric shape, such as stretching, rotation, translation, perspective distortion, etc. can be detected. The obtained projection screen geometric distortion state information refers to specific parameters and features that describe the distortion of the image on the projection screen, including the type, degree, position, etc. of the distortion, which is used for subsequent distortion correction decision-making.

[0073] This step can accurately and comprehensively identify the geometric distortion on the projection screen, providing reliable data support for subsequent correction. Using the calibration pattern library as a reference benchmark, it can effectively detect various types of geometric distortion, improving the accuracy and comprehensiveness of distortion identification; the fusion of brightness and chroma identification results further enhances the richness and reliability of the distortion state information, which helps the subsequent correction model to make more accurate correction decisions.

[0074] In some embodiments, based on the preset calibration pattern library, the calibration image projection result is analyzed to identify geometric distortion, and the projection screen geometric distortion state information is obtained, including:

[0075] In the calibration image projection result, pattern recognition and matching are performed to obtain a set of matched image pairs, wherein each matched image pair includes a standard calibration pattern and a corresponding distorted calibration pattern; geometric deformation recognition is performed on the set of matched image pairs, including stretching, rotation, translation, and perspective distortion, to obtain a first recognition result; brightness consistency evaluation and chrominance deviation evaluation are performed on the set of matched image pairs to obtain a second recognition result; feature extraction is performed on the first recognition result and the second recognition result respectively, and the feature extraction results are fused to generate the projection screen geometric distortion state information.

[0076] Specifically, first, in the calibration image projection result, pattern recognition and matching operations are performed, that is, based on a pre-set standard calibration pattern, a corresponding distorted calibration pattern is extracted from the projection result to form a set of matched image pairs, wherein each matched image pair includes a standard calibration pattern and its corresponding distorted calibration pattern; the matching process can use feature point matching (such as SIFT, ORB), template matching or deep learning assisted recognition method.

[0077] Then, the set of matched image pairs is traversed, and geometric deformation analysis is performed on each matched image pair to identify deformations including but not limited to the following types: stretching deformation, rotation deformation, translation deformation, and perspective distortion, to generate corresponding first recognition results to describe the spatial distribution and degree of each type of geometric distortion. Optionally, geometric deformation recognition is implemented using methods such as Homography Estimation and Deformation Field Modeling.

[0078] At the same time, the set of matched image pairs is traversed to perform brightness consistency evaluation (compare the brightness difference of the corresponding regions of the standard calibration pattern and the distorted calibration pattern) and chrominance deviation evaluation (analyze the color deviation degree of the corresponding regions of the standard calibration pattern and the distorted calibration pattern) on each matched image pair, because the deformation of the screen surface will cause refraction and scattering of light, and then affect the accuracy of different colors and the brightness of different positions. Based on the above evaluation, the corresponding second recognition result is generated to describe the spatial distribution of brightness and color consistency. Among them, the brightness and chrominance evaluation can be realized based on the mean square error, structural similarity, color difference formula (such as CIEDE2000) and other indicators.

[0079] Further, the first recognition result and the second recognition result are respectively subjected to feature extraction, and the extracted geometric deformation features, brightness consistency features and chroma deviation features are fused to generate geometric distortion state information comprehensively describing the geometric distortion state of the projection screen. The geometric deformation features are main information, and the brightness consistency features and the chroma deviation features are auxiliary information, which are used to supplement the deformation information that cannot be clearly determined by the geometric deformation features, thereby helping to accurately identify various geometric distortion phenomena of the projection screen in the actual projection process and improving the adaptability and calibration accuracy in complex projection environments.

[0080] In some implementations, the feature extraction of the first recognition result and the second recognition result and the fusion of the feature extraction results to generate the geometric distortion parameters of the target projection screen further include:

[0081] The primary geometric distortion identification of the first recognition result includes extracting stretching features, rotating features, translation features and perspective deformation features related to the flatness of the screen. The primary geometric distortion identification result is subjected to deep correction based on the second recognition result to obtain distortion depth features. The deep correction includes inferring the directionality of the physical distortion of the target projection screen according to the brightness and chroma deviation information. The distortion depth features are subjected to feature fusion, and the feature fusion result is output as the geometric distortion parameters of the target projection screen.

[0082] Specifically, first, the primary geometric distortion identification of the first recognition result is performed to extract geometric features related to the flatness of the screen, such as stretching features, rotating features, translation features and perspective deformation features, to determine whether the projection screen has geometric distortion and roughly determine the type and degree of distortion. Then, the primary geometric distortion identification result is subjected to deep correction based on the second recognition result, i.e., the brightness inconsistency evaluation and the chroma deviation evaluation, i.e., the brightness inconsistency evaluation and the chroma deviation evaluation information in the second recognition result are used to further correct the primary geometric distortion identification result to obtain more accurate distortion depth features to infer the directionality of the physical distortion and obtain the distortion depth features, thereby further improving the description of the geometric distortion. For example, according to the brightness deviation information, the local area physical concave or convex trend is inferred; according to the chroma deviation information, the change direction of the screen surface normal vector (i.e., the color deviation may indicate the change of reflected light at different angles) is inferred to form the distortion depth features describing the directionality and degree of the physical distortion of the projection screen.

[0083] Optionally, the deep correction can be inferred in combination with a light model (such as a Lambertian reflection model) and color space analysis.

[0084] Further, the distortion depth features are fused to output the feature fusion result as the geometric distortion parameter of the target projection curtain. First, the primary geometric distortion features and the distortion depth features (such as the concave-convex direction) are normalized to eliminate the dimensional difference and facilitate subsequent feature fusion. Then, the two types of features are aligned according to the spatial position (pixel coordinates, region block index, etc.) to ensure that the features at the same position can be correspondingly fused. Next, the feature fusion is performed by using the feature splicing (directly splicing the primary feature vector and the depth feature vector to form a new fusion feature vector), weighted summation (weighted summation after assigning different weights to different features, and the weights can be set according to experience or obtained by training) or feature coding (coding the spliced features by using a small neural network (such as MLP) to extract higher-level fusion features), and the fused features are mapped to the final geometric distortion parameter.

[0085] Optionally, the geometric distortion parameter can include: curtain local scaling rate distribution, curtain local rotation angle distribution, curtain local translation amount distribution, curtain local perspective distortion intensity, curtain surface normal vector offset distribution (direction and degree), and curtain local concave-convex depth estimation value.

[0086] The above steps are to more accurately identify and quantify the geometric distortion by comprehensively utilizing the geometric, luminance and chrominance information, to provide more comprehensive and accurate parameter basis for the correction decision of the subsequent multi-channel distortion correction model, so as to realize higher quality image reconstruction effect. Among them, the primary geometric distortion identification can preliminarily determine the type and degree of geometric distortion; the depth correction further utilizes the luminance and chrominance information to accurately determine the directionality of the distortion, making up for the possible shortcomings of pure geometric identification, so that the distortion features are more complete and accurate. The final feature fusion process integrates multiple features into unified geometric distortion parameters, which is helpful to realize more accurate geometric correction, and thus improves the quality and effect of the projection image reconstruction.

[0087] S300: input the distortion state information to a pre-constructed multi-channel distortion correction model to make a distortion correction decision, wherein the multi-channel distortion correction model includes a geometric correction channel, a color correction channel and a luminance correction channel.

[0088] Specifically, the distortion state information (including geometric distortion parameters, color offset information and luminance non-uniformity information) obtained in the above step is input as an input into a multi-channel distortion correction model, and is processed in the geometric correction, color correction and luminance correction three different correction channels respectively. The final distortion compensation instruction or correction configuration is generated by comprehensively considering the correction decisions of the three channels.

[0089] The significance of the above is that different types of distortion (deformation, color deviation, uneven brightness) are independently modeled by special channels, which helps to improve the correction effect and ensure the pertinence and controllability of the correction decision. For example, the geometric correction channel can solve the shape distortion of the image, the color correction channel can restore the true color of the image, and the brightness correction channel can uniform the brightness distribution of the image. The multi-channel correction method avoids other distortion problems that may be caused by single correction, and improves the stability and reliability of the correction effect.

[0090] In some embodiments, a pre-constructed multi-channel distortion correction model is obtained, comprising:

[0091] Based on the inverse transformation algorithm, the geometric correction channel in the form of a matrix is defined and constructed; sample color deviation state information is obtained as training data, a color mapping network based on a machine learning model is constructed and trained, and the output is the color correction channel, wherein the sample color deviation state information includes sample projection device characteristic parameters, sample light incidence angle, sample projection color value, and sample color deviation index; sample brightness deviation state information is obtained, and a brightness mapping model is constructed by a function fitting method, wherein the sample brightness deviation state information includes sample calibration viewing position, sample light incidence angle offset, and sample brightness attenuation amount; the geometric correction channel, the color correction channel, and the brightness correction channel are integrated to obtain the multi-channel distortion correction model.

[0092] Specifically, the geometric correction channel refers to a module that corrects the geometric deformation of the projection image based on the inverse transformation algorithm and uses matrix transformation. The color correction channel refers to a module that corrects color deviation based on a color mapping network trained by a machine learning method. The brightness correction channel refers to a module that compensates for brightness attenuation or unevenness based on a brightness mapping model established by a function fitting method.

[0093] Specifically, the sample color deviation state information includes optical characteristic parameters of the sample projection device, sample light incidence angle, sample projection color value, and sample color deviation index (such as ΔE, color temperature drift, etc.). The sample brightness deviation state information includes sample calibration viewing position (such as a specific area or angle on the screen), sample light incidence angle offset, and sample brightness attenuation amount (such as the percentage of decline relative to the center brightness).

[0094] Specifically, first, based on the inverse transformation algorithm, a mathematical model of geometric distortion is defined, such as affine transformation, perspective transformation, or free deformation model, and the distortion and correction relationship is represented in the form of a matrix, i.e. an inverse transformation matrix from the distorted image coordinates to the ideal image coordinates is established as the core output of the geometric correction channel.

[0095] Specifically, record the projection color values under different projection devices and different light incidence angle conditions, measure the color deviation index (such as ΔE value) between the actual projection result and the standard color, obtain a large amount of sample color deviation state information, and use the sample color deviation state information as input and the standard color value as label to construct and train a color mapping network based on machine learning.

[0096] For example, a shallow or deep fully connected neural network (MLP) is used, and the input feature vector is, for example, [device parameters, incidence angle, projection color value, color deviation index]; and the output is a color correction parameter (such as a color gain matrix or a gamut mapping matrix).

[0097] Specifically, using the same idea, record the brightness values corresponding to different viewing positions (such as center, edge, corner), and measure the corresponding relationship between the light incidence angle offset and the brightness attenuation; then, based on the collected data, a function fitting method (such as polynomial fitting, spline interpolation) is used to establish a brightness mapping model as a brightness correction channel.

[0098] Further, the above-constructed geometric correction channel, color correction channel and brightness correction channel are integrated to form a complete multi-channel distortion correction model, wherein the integration method uses inter-channel dependency modeling, that is, geometric correction is performed first, and then color and brightness compensation are performed.

[0099] Through the above process, the geometric deformation, color deviation and brightness non-uniformity problems are independently modeled in a modular manner, so that the multi-channel distortion correction model formed has good scalability and real-time performance, and the overall image quality and user experience are improved.

[0100] In some embodiments, the distortion state information is input to the pre-constructed multi-channel distortion correction model to make a distortion correction decision, including:

[0101] The distortion state information is input to the geometric correction channel for channel initialization, and a geometric correction vector is calculated based on the initialized geometric correction channel to obtain a geometric correction decision matrix; the derived distortion points are located according to the geometric correction decision matrix and the distortion state information to obtain a derived distortion point set; a mapping relationship between the derived distortion point set and the distortion state information is established to extract derived distortion state parameters; the derived distortion state parameters are input to the color correction channel and the brightness correction channel respectively to calculate color correction vectors and brightness correction vectors, and corresponding color correction decision matrices and brightness correction decision matrices are constructed; and the geometric correction decision matrix, the color correction decision matrix and the brightness correction decision matrix are output as the distortion correction decision result.

[0102] Specifically, the geometric correction decision matrix refers to a transformation matrix used to restore the distorted image to a standard geometric shape, which is usually an affine matrix, a perspective matrix or a free deformation grid matrix. The derived distortion points are image points that will be mapped to the positions where the distortion originally existed in the distorted region after preliminary geometric correction. These points need to be pre-corrected in color and brightness to overcome potential abnormalities. The derived distortion state parameters are color deviation and brightness offset parameters extracted based on the derived distortion points, which are used for subsequent color and brightness correction.

[0103] Specifically, the color correction decision matrix refers to a mapping matrix or gain matrix used to correct color deviation. The brightness correction decision matrix refers to a gain matrix or compensation map used to compensate for brightness unevenness or brightness attenuation.

[0104] Specifically, first, the distortion state information is input into the geometric correction channel, and the channel is initialized. In the initialization process, the geometric features (such as grid deformation, angle offset, translation, etc.) in the distortion state information are analyzed, the geometric correction vector is calculated based on the inverse transformation algorithm, and the geometric correction decision matrix is generated. The geometric correction decision matrix is used to restore the standard geometric shape of the image.

[0105] Specifically, then, according to the geometric correction decision matrix and the distortion state information, an inverse mapping relationship is established, that is, the image points that will be mapped to the positions where the distortion originally existed on the original curtain after geometric correction are identified as derived distortion points, and all identified derived distortion points form a derived distortion point set. Further, based on the derived distortion point set, the color deviation parameter and the brightness offset parameter corresponding to each point are extracted to form a derived distortion state parameter set.

[0106] Further, the derived distortion state parameters are input into the color correction channel and the brightness correction channel respectively; the color correction vector is calculated in the color correction channel, and the color correction decision matrix is generated; the brightness correction vector is calculated in the brightness correction channel, and the brightness correction decision matrix is generated, and the generated geometric correction decision matrix, color correction decision matrix and brightness correction decision matrix are output as the distortion correction decision result.

[0107] In the above method steps, through the initialization and vector calculation of the geometric correction channel, the geometric distortion can be accurately positioned and corrected. At the same time, by extracting the derived distortion state parameters and inputting them into the color and brightness correction channels respectively, accurate correction of color and brightness distortion is realized. This multi-channel cooperative processing method ensures the comprehensiveness and accuracy of the correction decision, and lays a foundation for realizing high-quality image reconstruction.

[0108] In some embodiments, the inverse transformation algorithm includes at least one of a homography inverse transformation, an affine inverse transformation and a perspective inverse transformation.

[0109] S400: based on the distortion correction decision result, performing image reconstruction on the received real-time projection image, and projecting the real-time reconstructed image obtained through image reconstruction to the target projection screen through the target projection device.

[0110] Specifically, according to the distortion correction decision result, the real-time projection image with distortion is processed to generate new image data, so that the reconstructed image meets the expected effect in terms of geometric shape, color and brightness.

[0111] Specifically, first, the real-time projection image frame output from the image source (such as an image server, a camera, a sensor or a graphics processing unit GPU) is received as an input image; then, the pre-generated distortion correction decision result is called to sequentially perform geometric correction, color correction and brightness correction on the image; further, after the image reconstruction is completed, the reconstructed image is projected onto the target projection screen through the target projection device to be displayed to the audience.

[0112] Optionally, during the image reconstruction process, if there is a sparse area (i.e. pixel missing or mapping hole) in the reconstructed image due to geometric transformation, a fast interpolation processing is performed, including but not limited to nearest neighbor interpolation, bilinear interpolation, weighted average interpolation and convolution padding interpolation, to fill the sparse area and improve the image continuity and visual quality.

[0113] In summary, the projection screen image reconstruction method provided by the present application has the following technical effects:

[0114] The calibration image is projected onto the target projection screen through the target projection device, and the projection effect of the image is collected by the perception device group; the collected image is analyzed in combination with the pre-set calibration pattern library to identify and extract the geometric distortion state information of the projection screen; the distortion state information is input into the pre-constructed multi-channel distortion correction model, which includes three channels of geometric correction, color correction and brightness correction, to generate the corresponding distortion correction decision; according to the correction decision, the received real-time projection image is subjected to image reconstruction processing, and the reconstructed image is projected onto the target screen through the target projection device again to realize dynamic optimization of the projection effect, thereby realizing real-time, accurate and comprehensive reconstruction of the projection image and improving the image quality and display stability.

[0115] Embodiment two, as Figure 2 is a structural schematic diagram of the projection screen image reconstruction system combined with distortion correction of the present application. For example, Figure 1 The flowchart of the projection screen image reconstruction method combined with distortion correction of the present application can be implemented by the structure as Figure 2 shown.

[0116] Based on the same concept as the embodiment of the projection screen image reconstruction method combined with distortion correction, the application also provides a projection screen image reconstruction system combined with distortion correction, which comprises:

[0117] A calibration image projection and collection module 11 is configured to project a calibration image to a target projection screen through a target projection device and collect a projection result of the calibration image through a group of perception devices.

[0118] A geometric distortion identification module 12 is configured to analyze the projection result of the calibration image based on a preset calibration pattern library to identify geometric distortion and obtain geometric distortion state information of the projection screen.

[0119] A distortion correction decision module 13 is configured to input the distortion state information to a pre-constructed multi-channel distortion correction model to make a distortion correction decision, wherein the multi-channel distortion correction model comprises a geometric correction channel, a color correction channel and a brightness correction channel.

[0120] An image reconstruction and real-time projection module 14 is configured to reconstruct a real-time projection image based on a distortion correction decision result, and project a real-time reconstruction image obtained through image reconstruction to a target projection screen through a target projection device.

[0121] In some embodiments, the calibration image projection and collection module 11 comprises:

[0122] A parameter acquisition unit is configured to acquire a physical shape parameter of a target projection screen, a pixel density parameter of a target projection device and a size parameter of a target projection area.

[0123] A standard calibration pattern extraction unit is configured to access a preset calibration pattern library and extract a corresponding standard calibration pattern from the calibration pattern library as an index condition of the screen shape parameter, the pixel density parameter and the projection size parameter.

[0124] A calibration image construction unit is configured to construct a calibration image in combination with the standard calibration pattern, wherein the calibration image comprises structured pattern information for geometric calibration.

[0125] A calibration image projection unit is configured to project the calibration image to a target projection screen through a target projection device.

[0126] In some embodiments, the calibration image projection and collection module 11 further comprises:

[0127] A multi-modal perception device deployment unit is configured to deploy a multi-modal perception device in a target projection environment, wherein the multi-modal perception device comprises a brightness test device for brightness measurement, a colorimetric test device for color detection and a high-resolution image acquisition device for image acquisition.

[0128] The calibration image information acquisition unit is configured to activate the multi-modal perception device, and acquire geometric image information, color distribution information, and brightness distribution information of a projection calibration image projected on the target projection screen.

[0129] The spatial remapping processing unit is configured to perform spatial remapping processing on the acquired geometric image information, color distribution information, and brightness distribution information, and convert the spatial remapping processing results into an ideal projection screen spatial coordinate system to form the calibration image projection result of the calibration image.

[0130] In some embodiments, the geometric distortion identification module 12 includes:

[0131] The pattern recognition matching unit is configured to perform pattern recognition matching in the calibration image projection result to obtain a set of matched image pairs, wherein each matched image pair includes the standard calibration pattern and a corresponding distorted calibration pattern.

[0132] The geometric deformation identification unit is configured to perform geometric deformation identification on the set of matched image pairs, including stretching, rotating, translating, and perspective distortion, to obtain a first identification result.

[0133] The brightness and chrominance evaluation unit is configured to perform brightness consistency evaluation and chrominance deviation evaluation on the set of matched image pairs to obtain a second identification result.

[0134] The feature extraction and fusion unit is configured to perform feature extraction on the first identification result and the second identification result, respectively, and fuse the feature extraction results to generate the projection screen geometric distortion state information.

[0135] In some implementations, the feature extraction and fusion unit in the geometric distortion identification module 12 includes:

[0136] The primary geometric distortion identification unit is configured to perform primary geometric distortion identification on the first identification result, including extracting stretching features, rotating features, translating features, and perspective deformation features related to the flatness of the screen.

[0137] The depth correction unit is configured to perform depth correction on the primary geometric distortion identification result based on the second identification result to obtain distortion depth features; wherein the depth correction includes inferring the directionality of the physical distortion of the target projection screen according to the brightness and chrominance deviation information.

[0138] The feature fusion and distortion parameter output unit is configured to perform feature fusion on the distortion depth features and the distortion depth features, and output the feature fusion result as the geometric distortion parameters of the target projection screen.

[0139] In some embodiments, the distortion correction decision module 13 includes:

[0140] a geometric correction channel construction unit configured to define and construct the geometric correction channel in a matrix form based on an inverse transformation algorithm.

[0141] a color correction channel training unit configured to obtain sample color deviation state information as training data, construct and train a color mapping network based on a machine learning model, and output the color correction channel, wherein the sample color deviation state information includes sample projection device characteristic parameters, sample light ray incidence angles, sample projection color values, and sample color deviation indicators.

[0142] a brightness correction channel construction unit configured to obtain sample brightness deviation state information and construct a brightness mapping model through a function fitting method, wherein the sample brightness deviation state information includes sample calibration viewing positions, sample light ray incidence angle offsets, and sample brightness attenuation amounts.

[0143] a multi-channel distortion correction model integration unit configured to integrate the geometric correction channel, the color correction channel, and the brightness correction channel, and obtain the multi-channel distortion correction model.

[0144] In some embodiments, the distortion correction decision module 13 includes:

[0145] a geometric correction channel initialization and decision matrix obtaining unit configured to input the distortion state information to the geometric correction channel for channel initialization, and obtain a geometric correction decision matrix based on the initialized geometric correction channel for geometric correction vector calculation.

[0146] a derived distortion point positioning and point set obtaining unit configured to position derived distortion points and obtain a derived distortion point set based on the geometric correction decision matrix and the distortion state information.

[0147] a mapping relationship establishment and derived distortion state parameter extraction unit configured to establish a mapping relationship between the derived distortion point set and the distortion state information, and extract derived distortion state parameters.

[0148] a color and brightness correction decision matrix construction unit configured to input the derived distortion state parameters to the color correction channel and the brightness correction channel respectively, calculate color correction vectors and brightness correction vectors, and correspondingly construct a color correction decision matrix and a brightness correction decision matrix.

[0149] a distortion correction decision result output unit configured to output the geometric correction decision matrix, the color correction decision matrix, and the brightness correction decision matrix as the distortion correction decision result.

[0150] In some embodiments, the inverse transformation algorithm includes at least one of a homography inverse transformation, an affine inverse transformation, and a perspective inverse transformation.

[0151] In one embodiment, the present application provides a computer readable storage medium for storing a software program, computer executable program and modules, such as the program instructions / modules corresponding to the projection screen image reconstruction method with distortion correction according to the embodiments of the present application, so as to realize the projection screen image reconstruction method with distortion correction.

[0152] It should be understood that the embodiments mentioned in the specification are focused on their differences from other embodiments, and the specific embodiments in the foregoing embodiment one are also applicable to the projection screen image reconstruction system with distortion correction according to the embodiment two, and for the sake of brevity of the specification, no further expansion is made here.

[0153] It should be understood that the embodiments disclosed in the present application and the above description can enable those skilled in the art to implement the present application. Meanwhile, the present application is not limited to the above-mentioned part of the embodiments, and it should be understood that those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method of reconstruction of a projection screen image with distortion correction, characterized in that, The method comprises the steps of: projecting a calibration image to a target projection screen through a target projection device, and collecting the projection result of the calibration image through a perception device group; based on a pre-set calibration pattern library, analyzing the calibration image projection result to identify geometric distortion and obtain projection screen geometric distortion state information; inputting the distortion state information into a pre-constructed multi-channel distortion correction model to make a distortion correction decision, wherein the multi-channel distortion correction model includes a geometric correction channel, a color correction channel, and a brightness correction channel; based on the distortion correction decision result, reconstructing the received real-time projection image, and projecting the real-time reconstruction image obtained through image reconstruction to the target projection screen through the target projection device; inputting the distortion state information into a pre-constructed multi-channel distortion correction model to make a distortion correction decision, comprising: inputting the distortion state information into the geometric correction channel for channel initialization, and performing geometric correction vector calculation based on the initialized geometric correction channel to obtain a geometric correction decision matrix; locating derivative distortion points according to the geometric correction decision matrix and the distortion state information to obtain a set of derivative distortion points; establishing a mapping relationship between the set of derivative distortion points and the distortion state information, and extracting derivative distortion state parameters, including color deviation and brightness deviation based on the derivative distortion points; respectively inputting the derivative distortion state parameters into the color correction channel and the brightness correction channel, calculating color correction vectors and brightness correction vectors, and correspondingly constructing a color correction decision matrix and a brightness correction decision matrix; outputting the geometric correction decision matrix, the color correction decision matrix, and the brightness correction decision matrix as the distortion correction decision result; projecting a calibration image to a target projection screen through a target projection device, comprising: obtaining the physical shape parameters of the target projection screen, the pixel density parameters of the target projection device, and the size parameters of the target projection area; accessing a pre-set calibration pattern library, and extracting a corresponding standard calibration pattern from the calibration pattern library as an index condition based on the screen shape parameters, pixel density parameters, and projection size parameters; constructing a calibration image based on the standard calibration pattern, wherein the calibration image includes structured pattern information for geometric calibration; projecting the calibration image to the target projection screen through the target projection device; based on a pre-set calibration pattern library, analyzing the calibration image projection result to identify geometric distortion and obtain projection screen geometric distortion state information, comprising: based on the standard calibration pattern, performing pattern recognition and matching on the calibration image projection result, extracting a distortion calibration pattern corresponding to the standard calibration pattern from the projection result, and obtaining a set of matched image pairs, wherein the matched image pairs include the standard calibration pattern and the corresponding distortion calibration pattern; iterating through the set of matched image pairs to identify geometric deformation, including stretching, rotating, translating, and perspective distortion, to obtain a first identification result; iterating through the set of matched image pairs to evaluate brightness consistency and chrominance deviation, to obtain a second identification result; The first recognition result and the second recognition result are respectively subjected to feature extraction, and the feature extraction results are fused to generate the projection screen geometric distortion state information.

2. The projection screen image reconstruction method incorporating distortion correction of claim 1, wherein, The calibration image projection result is collected by the perception device group, including: A multi-modal perception device is deployed in a target projection environment, which includes a brightness test device for brightness measurement, a colorimetric test device for color detection, and a high-resolution image acquisition device for image acquisition; The multi-modal perception device is activated to collect geometric image information, color distribution information, and brightness distribution information of the projected calibration image on the target projection screen, respectively; The collected geometric image information, color distribution information, and brightness distribution information are subjected to spatial remapping processing and unified conversion to an ideal projection screen spatial coordinate system to form the calibration image projection result.

3. The projection screen image reconstruction method incorporating distortion correction of claim 1, wherein, The feature extraction of the first recognition result and the second recognition result is also included in generating the geometric distortion parameters of the target projection screen, which includes: The first recognition result is subjected to primary geometric distortion recognition, including extracting stretching features, rotation features, translation features, and perspective deformation features related to the flatness of the screen; The primary geometric distortion recognition result is subjected to deep correction based on the second recognition result to obtain distortion depth features, wherein the deep correction includes inferring the directionality of the physical distortion of the target projection screen according to the brightness and color deviation information; The primary geometric distortion features and the distortion depth features are subjected to feature fusion, and the feature fusion result is output as the geometric distortion parameters of the target projection screen.

4. The projection screen image reconstruction method incorporating distortion correction of claim 1, wherein, The inverse transformation algorithm at least includes one of a homography inverse transformation, an affine inverse transformation, and a perspective inverse transformation.

5. A projection screen image reconstruction system incorporating distortion correction, characterized by, The projection screen image reconstruction method with distortion correction according to any one of claims 1 to 4 is implemented, including: A calibration image projection and acquisition module is configured to project a calibration image onto a target projection screen by a target projection device and collect a calibration image projection result by a perception device group; A geometric distortion recognition module is configured to analyze the calibration image projection result based on a preset calibration pattern library to perform geometric distortion recognition and obtain projection screen geometric distortion state information; A distortion correction decision module is configured to input the distortion state information into a pre-constructed multi-channel distortion correction model to make a distortion correction decision, wherein the multi-channel distortion correction model includes a geometric correction channel, a color correction channel, and a brightness correction channel; An image reconstruction and real-time projection module is configured to reconstruct an image based on the distortion correction decision result and project a real-time reconstructed image obtained by image reconstruction onto the target projection screen by the target projection device.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the projection screen image reconstruction method with distortion correction according to any one of claims 1 to 4.

Images