Intelligent projector control method based on color correction and depth correction

Abstract

The present application relates to the technical field of projectors, and specifically provides a kind of intelligent projector control method based on color correction and depth correction, image and depth information of projection plane are collected by optical perception device, and color deviation tensor and depth deviation tensor between it and ideal plane are calculated, color correction and depth correction are carried out on pre-projection image according to color deviation tensor and depth deviation tensor, so that projected image will not be affected by color and flatness on projection plane.The present application avoids color interference and flatness influence of projection plane, ensures that projector can realize flat, achromatic projection under visual effect in extreme conditions such as plane and space irregularity, color stain and the like, and solves the problems such as precision limitation, single scene and high cost in existing projection mapping application.

Description

Technical Field

[0001] This invention relates to the field of projector technology, and specifically provides a smart projector control method based on color correction and depth correction. Background Technology

[0002] Projectors are widely used in meetings, teaching, presentations, and home entertainment. The quality of their projection directly affects the audience's understanding and acceptance of the content. However, in actual use, due to environmental factors, the characteristics of the wall or screen, and other factors, the projection effect varies, and problems such as color difference, blurriness, and insufficient brightness may occur. In particular, when the projection plane is uneven or the background color is uneven, traditional projectors cannot find a regular projection plane, resulting in the distortion or stretching of the projected image. At the same time, uneven color on the projection plane can also cause color distortion in the projected image, affecting the user's viewing experience.

[0003] Therefore, there is a need for an intelligent projector system and control method that can accurately and quickly measure the reflectivity and flatness of the projection plane and adaptively adjust the image to achieve better projection results. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a smart projector control method based on color correction and depth correction. It acquires image and depth information of the projection plane using optical sensing devices, analyzes and obtains color interference and flatness issues on the projection plane, and improves the projection effect through color correction and depth correction. This ensures that the projector achieves a flat and color-difference-free projection even under extreme conditions such as irregularities in the plane and space, and color stains.

[0005] The intelligent projector control method based on color correction and depth correction provided by this invention includes the following steps:

[0006] S1. Acquire images and depth information of the projection plane using optical sensing devices;

[0007] S2. The controller performs color processing on the projection plane based on the image to obtain the color deviation tensor between the image and the global color fitting plane; it also performs depth analysis on the projection plane based on the depth information to obtain the depth deviation tensor between the depth information and the global depth fitting plane.

[0008] S3. Correct the color of the pre-projected image based on the color deviation tensor. Obtain a polynomial model reflecting color information by fitting the RGB data of each point on the image. Correct the color of the pre-projected image by adjusting the coefficients in the polynomial model until the color deviation vector is eliminated.

[0009] For any line segment AB on the global depth fitting plane, there exists a corresponding line segment A′B′ in the viewer's visual space, satisfying the vector similarity constraint:

[0010]

[0011] Where S represents the distance between any line segment AB and the viewer in the depth direction;

[0012] The measured ratio of any line segment on the projected area to the corresponding line segment in the viewer's field of vision is calculated based on the depth deviation tensor. When the measured ratio is less than... At that time, the size of the corresponding region on the pre-projected image is reduced based on the depth deviation tensor; when the measured ratio is greater than At that time, the size of the corresponding region on the preprojected image is magnified based on the depth deviation tensor;

[0013] S4. Project the pre-projected image with completed depth correction onto the projection area obtained by color avoidance.

[0014] Preferably, the optical sensing device uses a binocular camera, including an RGB channel and a Depth channel.

[0015] Preferably, the color processing is as follows:

[0016] The image's R, G, and B color channels, along with the height and width of the projection plane, form a five-dimensional feature space. H represents the height of the projection plane, and W represents its width. Any point in the projection plane is represented as [r, g, b]. (h,w) r represents the specific value of the point in dimension R, g represents the specific value of the point in dimension G, b represents the specific value of the point in dimension B, h represents the specific value of the point in dimension H, and w represents the specific value of the point in dimension W.

[0017] In the five-dimensional feature space, the R, G, and B color values ​​of each point on the image are fitted to obtain the global color fitting plane. The [r, g, b] values ​​at any point on the projection plane are then calculated. (h,w) The color deviation tensor of corresponding points is fitted to the global color under the three color channels R, G, and B.

[0018] Preferably, the in-depth analysis specifically includes:

[0019] The depth information, along with the height and width of the projection plane, forms a three-dimensional feature space. Any point in the projection plane is represented as d. (h,w) d is the specific value of the depth of the point on the projection plane;

[0020] In the three-dimensional feature space, the depth information is fitted to obtain the global depth fitting plane, and the depth d at any point on the projection plane is calculated. (h,w)The depth deviation tensor of the corresponding point on the global depth fitting plane.

[0021] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0022] This invention collects image and depth information of the projection plane through optical sensing devices, analyzes the color interference and flatness of the projection plane, and performs color correction and depth correction. This avoids the influence of color interference and flatness, and ensures that the projector can achieve flat and color-difference-free projection under extreme conditions such as irregular planes and spaces and color stains. It solves the problems of limited accuracy, limited scenarios and high cost in existing projection mapping applications.

[0023] Furthermore, this invention eliminates the need for adjustments and avoidance of the projection area, making it more suitable for scenarios where the projection plane is constantly changing. Attached Figure Description

[0024] Figure 1 This is an overall framework diagram of the intelligent projector system control method provided according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of the global analysis of an intelligent projector provided according to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of a depth correction scenario provided by an embodiment of the present invention. Detailed Implementation

[0027] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0029] like Figure 1 As shown in the figure, the intelligent projector control method based on color correction and depth correction provided in this embodiment of the invention mainly includes the following steps:

[0030] S1. A binocular camera is selected as the optical sensing device. The binocular camera includes an RGB channel and a Depth channel, which correspond to the image acquisition module and the depth acquisition module, respectively, to acquire visual images and depth information.

[0031] Before data collection, the positions of the viewer, projector, and binocular camera need to be calibrated using coordinates. The relative positions of the viewer, projector, and binocular camera are measured, and their respective coordinate systems with their origins are obtained. This is for accurate modeling and precise control later. A viewer coordinate system U is established with the viewer as the origin, a camera coordinate system C is established with the binocular camera as the origin, and a projection coordinate system P is established with the projector as the origin. During calibration, measuring tools are used to measure the six-dimensional pose of the binocular camera and viewer relative to the projector in detail, obtaining the pose transformation matrix of the binocular camera relative to the projector. and the pose transformation matrix of the viewer relative to the projector The two pose transformation matrices described above describe the position and orientation of the binocular camera and the viewer relative to the projector.

[0032] like Figure 2 As shown, S2, the controller performs color processing on the projection plane based on the image, combining the R, G, and B color channels of the image with the height and width of the projection plane to form a five-dimensional feature. This feature represents the color space information R (red), G (green), and B (blue) of the projection plane, and the geometric space information H (height) and W (width). The plane formed by (H, W) represents the projection plane. Therefore, any point on this plane can be represented as [r, g, b]. (h,w) Here, r represents the specific value of the point in dimension R, g represents the specific value of the point in dimension G, b represents the specific value of the point in dimension B, h represents the specific value of the point in dimension H, and w represents the specific value of the point in dimension W. Color processing aims to obtain the five-dimensional deviation tensor between the data in each dimension of the planar scene and the ideal projection plane. Therefore, based on the collected information, the color gamut values ​​in the three dimensions of r, g, and b are used to fit each color gamut plane, and outlier points are continuously removed to fit a global color fitting plane, which is used as the ideal plane. The color information of any point on this plane is denoted as [r0, g0, b0]. Any point [r, g, b] within the known global range (H, W) is considered a color fitting plane. (h,w) By analyzing the value distribution of the values, we can obtain the color deviation tensor of the pixels in the image at the corresponding points on the global color fitting plane under the R, G, and B color channels:

[0033] ΔC(h,w)=[ΔR(h,w),ΔG(h,w),ΔB(h,w)].

[0034] The controller performs depth analysis on the projection plane based on depth information, constructing a three-dimensional feature by combining the depth D of the projection plane with its height H and width W. Different depth analysis algorithms are developed for binocular cameras of varying precision, with the precision ranging from simple to complex. The baseline dimension calculation methods include: pixel method, plane fitting method, spot method, and ray method. Any point on the projection plane is represented as d. (h,w)Let d represent the specific depth value of the point on the projection plane. In the three-dimensional feature space, depth analysis first fuses and tensors the depth information with spatial geometric data. A plane fitting algorithm is selected according to the accuracy level of the stereo camera capturing the depth information. Through fitting and continuous outlier removal, a global depth fitting plane is obtained and used as the ideal plane. For any point within the global range (H, W), the depth information d... (h,w) This allows us to calculate the depth deviation tensor of the spatial location containing depth information relative to the global depth fitting plane.

[0035] S3. Due to color interference on the projection plane, color correction of the pre-projected image is required based on the color deviation tensor. Considering the non-linear relationship between the color values ​​of the pre-projected image and the color compensation values ​​to be corrected, a polynomial model is selected to fit the dataset of compensation values. A quadratic polynomial model is established, and its expression is as follows:

[0036] R′=a0+a1x+a2y+a3x 2 +a4y 2 +a5xy;

[0037] G′=b0+b1x+b2y+b3x 2 +b4y 2 +b5xy;

[0038] B′=c0+c1x+c2y+c3x 2 +c4y 2 +c5xy;

[0039] Where R', G', and B' are the compensated values ​​of the red, green, and blue channels, x and y are the position coordinates in the projection plane, and a0, a1, ..., a5, b0, b1, ..., b5, c0, c1, ..., c5 are all coefficients of the polynomial.

[0040] Based on the color deviation tensor, the coefficients of the quadratic polynomial model can be determined using the least squares method or other curve fitting methods. The most suitable coefficient values ​​are selected according to the visual image to ensure the polynomial model accurately fits the data and compensates for color distortion. A two-dimensional color compensation matrix P is constructed from the compensation values ​​of each pixel in the pre-projected image. During projection, the coefficients in the two-dimensional color compensation matrix P are used to compensate for the pre-projected image, eliminating color shift and preventing distortion in the projected image.

[0041] Depth correction of the projection area mainly includes the following steps:

[0042] pose transformation matrix of binocular camera relative to projector and the pose transformation matrix of the viewer relative to the projector Perform a coordinate transformation on any point R, and then calibrate the points in the three coordinate systems: the projected coordinate system P, the viewer's coordinate system U, and the camera's coordinate system C. The calibration formula is as follows:

[0043]

[0044]

[0045] Among them, R P This indicates that any point R in the projected coordinate system P, R C This indicates that any point R is in the camera coordinate system C, R U This indicates that any point R lies in the viewer's coordinate system U.

[0046] like Figure 3 As shown, l1+l2 is the actual projection plane. The l2 region is flat, and the deviation in the projection effect mainly comes from the l1 region. Using D(h,w) U The function represents the depth information in the projection plane l1 region, which is incorporated into the viewer's coordinate system U as (h,w,D(h,w)). U In the viewer's coordinate system U, a viewer's visual space is obtained. For any line segment AB on the ideal depth plane, there exists a corresponding line segment A′B′ in the viewer's visual space, satisfying vector similarity constraints:

[0047]

[0048] Where S represents the distance between any line segment AB and the viewer in the depth direction; the measured ratio of any line segment on the projection plane to the corresponding line segment in the viewer's field of vision is calculated based on the depth deviation tensor. When the measured ratio is less than... When the measured ratio is greater than a certain value, it indicates that the projection plane is concave, and the size of the corresponding area on the pre-projected image needs to be reduced according to the depth deviation tensor, that is, the light in that part is contracted to reduce the scattering angle; when the measured ratio is greater than a certain value, it indicates that the projection plane is concave, and the size of the corresponding area on the pre-projected image needs to be reduced according to the depth deviation tensor, that is, the light in that part is contracted to When the projection plane is convex, the size of the corresponding area on the pre-projected image is magnified according to the depth deviation tensor. This means that the light in that part is expanded, increasing the scattering angle. This ensures that the image seen by the viewer, whether on the convex plane or in the depression, is the same size as the image on the ideal depth plane, avoiding overlapping or tearing of the projected image at concave and convex locations. The scaling factors for reduction and magnification are calibrated with the coordinate positions to obtain the global depth compensation matrix Q. The depth of the pre-projected image is then corrected based on the depth compensation matrix Q.

[0049] The processing logic for the pre-projected image (img) is as follows:

[0050] simg = P·Q(img);

[0051] Where simg represents the final projected image obtained by processing the pre-projected image img.

[0052] S5. Send the projected image simg to the projector for projection onto the projection plane. If the projected image is projected onto a changing projection plane, the frequency of acquiring image and depth information of the projection plane needs to be adaptively adjusted using a cyclic decision detection algorithm.

[0053] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0054] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A smart projector control method based on color correction and depth correction, characterized in that, Includes the following steps: S1. Acquire images and depth information of the projection plane using optical sensing devices; S2. The controller performs color processing on the projection plane based on the image to obtain the color deviation tensor between the image and the global color fitting plane. The specific color processing is as follows: The image's R, G, and B color channels are combined with the height and width of the projection plane to form a five-dimensional feature space, where H represents the height of the projection plane and W represents its width. Any point in the projection plane is represented as... , This represents the specific value of the point in dimension R. This represents the specific value of the point in dimension G. This indicates the specific value of the point in dimension B. This indicates the specific value of the point in dimension H. This indicates the specific value of the point in dimension W; In the five-dimensional feature space, the R, G, and B color values ​​of each point on the image are fitted to obtain the global color fitting plane, and the color fitting plane of any point on the projection plane is calculated. The color deviation tensor of the corresponding points on the global color fitting plane under the three color channels R, G, and B; Based on the depth information, a depth analysis is performed on the projection plane to obtain the depth deviation tensor between the depth information and the global depth fitting plane; In-depth analysis specifically includes: The depth information, along with the height and width of the projection plane, forms a three-dimensional feature space. Any point in the projection plane is represented as... , This represents the specific depth of the point on the projection plane. In a three-dimensional feature space, depth information is fitted to obtain a global depth fitting plane, and the depth of any point on the projection plane is calculated. The depth deviation tensor of the corresponding point on the global depth fitting plane; S3. Correct the color of the pre-projected image based on the color deviation tensor. Obtain a polynomial model reflecting color information by fitting the RGB data of each point on the image. Correct the color of the pre-projected image by adjusting the coefficients in the polynomial model until the color deviation vector is eliminated. For any line segment on the global depth-fitting plane AB There are corresponding line segments within the viewer's field of vision. And it satisfies the vector similarity constraint: ； Where S represents the distance between any line segment AB and the viewer in the depth direction; Calculate the measured ratio of any line segment on the projection area to the corresponding line segment in the viewer's field of vision. When the measured ratio is less than... At that time, the size of the corresponding region on the pre-projected image is reduced based on the depth deviation tensor; when the measured ratio is greater than At that time, the size of the corresponding region on the preprojected image is magnified based on the depth deviation tensor; S4. Process the pre-projected image to obtain the final projected image, and send the final projected image to the projector for projection.

2. The intelligent projector control method based on color correction and depth correction as described in claim 1, characterized in that, The optical sensing device uses a binocular camera, including RGB channels and Depth channels.

Images