Intelligent projector control method based on color correction and depth correction

Abstract

The invention relates to the technical field of projectors, in particular to an intelligent projector control method based on color correction and depth correction, and the method comprises the steps: collecting image and depth information of a projection plane through optical sensing equipment, and calculating a color deviation tensor and a depth deviation tensor between the projection plane and an ideal plane; and performing color correction and depth correction on the pre-projection image according to the color deviation tensor and the depth deviation tensor, so that the projected image is not influenced by the color and flatness on the projection plane. According to the invention, the color interference and flatness influence of the projection plane are avoided, the projector is ensured to realize the flat and chromatic aberration-free projection under the visual effect under the extreme conditions of irregular plane and space, color stains and the like, and the problems of limited precision, single scene, high cost and the like in the existing projection mapping application are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of projectors, and specifically provides an intelligent projector control method based on color correction and depth correction. Background Art

[0002] Projectors are widely used in conferences, teaching, exhibitions, and home entertainment. The quality of their projection effects directly affects the audience's understanding and acceptance of the content. However, in actual use, due to environmental factors, wall or screen characteristics and other factors, the projection effects are different, and there may be problems such as color difference, blur, and insufficient brightness. Especially when the projection plane is uneven in shape or the background color is uneven, traditional projectors cannot find a regular projection plane, resulting in distortion or stretching of the projected image. At the same time, the uneven color of the projection plane will also cause color distortion in the projected image, affecting the user's viewing experience.

[0003] Therefore, there is a need for an intelligent projector system and control method that can accurately and quickly measure the reflectivity and flatness of the projection plane and adaptively adjust the image to achieve better projection effects. Summary of the invention

[0004] In order to solve the above problems, the present invention provides an intelligent projector control method based on color correction and depth correction. The method collects the image and depth information of the projection plane through an optical sensing device, analyzes the color interference and flatness problems of the projection plane, and improves the projection effect through color correction and depth correction to ensure that the projector can achieve smooth and color-free projection under extreme conditions such as plane and space irregularities and color stains.

[0005] The intelligent projector control method based on color correction and depth correction provided by the present invention comprises the following steps:

[0006] S1. Collecting the image and depth information of the projection plane through an optical sensing device;

[0007] S2, the controller performs color processing on the projection plane according to the image to obtain a color deviation tensor between the image and the global color fitting plane; performs depth analysis on the projection plane according to the depth information to obtain a depth deviation tensor between the depth information and the global depth fitting plane;

[0008] S3, performing color correction on the pre-projected image according to the color deviation tensor, obtaining a polynomial model reflecting color information according to the RGB data of each point on the image, and correcting the color of the pre-projected image by adjusting the coefficients in the polynomial model until the color deviation vector is eliminated;

[0009] For any line segment AB on the global depth fitting plane, there is a corresponding line segment A′B′ in the viewer’s field of view, and the vector similarity constraint is satisfied:

[0010]

[0011] Where S represents the distance between any line segment AB and the viewer in the depth direction;

[0012] The measured ratio of any line segment on the projection area to the corresponding line segment in the viewer's field of view is calculated based on the depth deviation tensor. When , the size of the corresponding area on the pre-projected image is reduced according to the depth deviation tensor; when the measured ratio is greater than When , the size of the corresponding area on the pre-projected image is enlarged according to the depth deviation tensor;

[0013] S4, projecting the pre-projected image that has completed depth correction onto the projection area obtained by color avoidance division.

[0014] Preferably, the optical sensing device adopts a binocular camera including an RGB channel and a Depth channel.

[0015] Preferably, the color processing is specifically as follows:

[0016] The three color channels of the image, R, G, and B, and the height and width of the projection plane form a five-dimensional feature space, where H represents the height of the projection plane and W represents the width of the projection plane. Any point in the projection plane is represented as [r, g, b] (h,) , r represents the specific value of the point in the R dimension, g represents the specific value of the point in the G dimension, b represents the specific value of the point in the B dimension, h represents the specific value of the point in the H dimension, and w represents the specific value of the point in the W dimension;

[0017] In the five-dimensional feature space, the R, G, and B color values ​​of each point on the image are fitted to obtain the global color fitting plane, and the [r, g, b] of any point on the projection plane is calculated. (h,) The color deviation tensor of the corresponding points on the global color fitting under the three color channels of R, G, and B.

[0018] Preferably, the in-depth analysis is specifically as follows:

[0019] The depth information and the height and width of the projection plane form a three-dimensional feature space, and any point in the projection plane is represented as d (h,) , d is the specific value of the depth of the point on the projection plane;

[0020] In the three-dimensional feature space, the depth information is fitted to obtain the global depth fitting plane, and the d (h,)The depth deviation tensor of the corresponding points on the global depth fitting plane.

[0021] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0022] The present invention collects the image and depth information of the projection plane through an optical sensing device, analyzes the color interference and flatness of the projection plane, and performs color correction and depth correction, thereby avoiding the influence of color interference and flatness, and ensuring that the projector can achieve smooth and color-free projection under extreme conditions such as plane and space irregularities and color stains under visual effects, thereby solving the problems of limited accuracy, single scene and high cost in existing projection mapping applications.

[0023] In addition, the present invention does not need to adjust or avoid the projection area, and is more suitable for scenes where the projection plane is constantly changing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is an overall framework diagram of a smart projector system control method provided according to an embodiment of the present invention;

[0025] Figure 2 is a global analysis flow chart of a smart projector provided according to an embodiment of the present invention;

[0026] Figure 3 4 is a schematic diagram of a scene of depth correction provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0028] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0029] like Figure 1 As shown, the intelligent projector control method based on color correction and depth correction provided by the embodiment of the present invention mainly includes the following steps:

[0030] S1. A binocular camera is selected as an optical perception device. The binocular camera includes RGB channels and Depth channels, which correspond to the image acquisition module and the depth acquisition module respectively, to acquire visual images and depth information.

[0031] Before information collection, it is first necessary to calibrate the positions of the viewer, projector, and binocular camera, measure the relative position relationship between the viewer, projector, and binocular camera, and obtain their coordinate systems with their respective positions as the origin. This is done for subsequent accurate modeling and precise control. The viewer coordinate system U is established with the viewer as the origin, the camera coordinate system C is established with the binocular camera as the origin, and the projection coordinate system P is established with the projector as the origin. During the calibration process, the measurement tool is used to measure the six-dimensional posture state of the binocular camera and the viewer relative to the projector in detail to obtain the posture transformation matrix of the binocular camera relative to the projector. And the viewer's position transformation matrix relative to the projector The above two pose transformation matrices describe the position and posture of the binocular camera and the viewer relative to the projector.

[0032] like Figure 2 As shown, S2, the controller processes the color of the projection plane according to the image, and combines the three color channels of the image, R, G, and B, with the height and width of the projection plane to form a five-dimensional feature, which respectively represents the color space information R (red), G (green), and B (blue) of the projection plane, and the geometric space information H (height) and W (width). The plane composed of (H, W) represents the projection plane, so any point on this plane can be represented as [r, g, b] (h,w) , where r represents the specific value of the point in the R dimension, g represents the specific value of the point in the G dimension, b represents the specific value of the point in the B dimension, h represents the specific value of the point in the H dimension, and w represents the specific value of the point in the W dimension. Color processing is to obtain the five-dimensional deviation tensor of the data of each dimension in the plane scene and the ideal projection plane. Therefore, the color gamut values ​​of the three dimensions of r, g, and b are fitted to each color gamut plane based on the collected information, and the outliers are continuously stripped to fit the global color fitting plane. As the ideal plane, the color information of any point on it is recorded as [r0, g0, b0]. At any point [r, g, b] within the known global range (H, W) (h,w) The value distribution of can be used to obtain the color deviation tensor of the corresponding point on the global color fitting plane under the three color channels of R, G, and B:

[0033] ΔC(h,w)=[ΔR(h,w),ΔG(h,w),ΔB(h,w)].

[0034] The controller performs depth analysis on the projection plane based on the depth information, and combines the depth D of the projection plane with the height H and width W of the projection plane to form a three-dimensional feature. The corresponding depth analysis algorithm is formulated according to the binocular cameras with different precisions. The precision ranges from simple to complex. The base dimension calculation methods can be selected: pixel method, plane fitting method, spot method, and light method. Any point in the projection plane is represented by d (h,w), d represents the specific value of the depth of the point on the projection plane. In the three-dimensional feature space, the depth analysis first fuses the depth information with the spatial geometric data and quantizes them. The plane fitting algorithm is selected according to the accuracy level of the binocular camera captured by the depth information. The global depth fitting plane is obtained by fitting and continuously removing outliers. It is used as the ideal plane. For any point in the global range (H, W), the depth information d (h,w) , the depth deviation tensor of the spatial position containing the depth information relative to the global depth fitting plane can be obtained.

[0035] S3. Due to the color interference in the projection plane, the pre-projected image needs to be color corrected according to the color deviation tensor. Considering the nonlinear relationship between the conversion between the color value of the pre-projected image and the color compensation value to be corrected, a polynomial model is selected to fit the compensation value data set, and a quadratic polynomial model is established, which is expressed as:

[0036] R′=a0+a1x+a2y+a3x 2 +a4y 2 +a5xy;

[0037] G′=b0+b1x+b2y+b3x 2 +b4y 2 +b5xy;

[0038] B′=c0+c1x+c2y+c3x 2 +c4y 2 +c5xy;

[0039] Among them, R', G', B' are the values ​​of the compensated red, green and blue channels, x and y are the position coordinates in the projection plane, and a0, a1, ..., a5, b0, b1, ..., b5, c0, c1, ..., c5 are all coefficients of the polynomial.

[0040] Based on the color deviation tensor, the coefficients of the quadratic polynomial model can be determined by the least squares method or other curve fitting methods. The most suitable coefficient value is selected according to the visual image so that the polynomial model can accurately fit the data and compensate the color. The compensation values ​​of each pixel of the pre-projected image form a two-dimensional color compensation matrix P. During the projection process, the coefficients in the two-dimensional color compensation matrix P are used to compensate the pre-projected image to eliminate the color deviation problem and avoid distortion of the projected image display.

[0041] Depth correction for the projection area mainly includes the following steps:

[0042] The pose transformation matrix of the binocular camera relative to the projector And the viewer's position transformation matrix relative to the projector Transform the coordinates of any point R, and calibrate the points in the projection coordinate system P, the viewer coordinate system U, and the camera coordinate system C. The calibration formula is:

[0043]

[0044]

[0045] Among them, R P Represents any point R in the projected coordinate system P, R C Indicates that any point R is in the camera coordinate system C, R U Represents an arbitrary point R in the viewer's coordinate system U.

[0046] like Figure 3 As shown in the figure, l1+l2 is the actual projection plane, the l2 area is flat, and the deviation of the projection effect mainly comes from the l1 area. Use D(h,w) U The function represents the depth information of the projection plane l1 area, which is incorporated into the viewer coordinate system U as (h,w,D(h,w) U ). A viewer's field of view is obtained by dividing in the viewer coordinate system U. For any line segment AB on the depth ideal plane, there is a corresponding line segment A′B′ in the viewer's field of view, and the vector similarity constraint is satisfied:

[0047]

[0048] Where S represents the distance between any line segment AB and the viewer in the depth direction; the measured ratio of any line segment on the projection plane to the corresponding line segment in the viewer's field of view is calculated based on the depth deviation tensor. When the measured ratio is less than When , it indicates that the projection plane is concave, and it is necessary to reduce the size of the corresponding area on the pre-projection image according to the depth deviation tensor, that is, to shrink the light in this part and reduce the scattering angle; when the measured ratio is greater than When , it indicates that the projection plane is convex. The size of the corresponding area on the pre-projected image is enlarged according to the depth deviation tensor, that is, the light of this part is expanded and the scattering angle is increased, that is, the image size of the viewer on the convex plane or the concave inside is the same as the image size on the ideal depth plane, avoiding overlapping or tearing of the projected image at the concave and convex positions. The scale coefficients of reduction and enlargement are calibrated with the coordinate position to obtain the global depth compensation matrix Q, and the depth correction of the pre-projected image is performed according to the depth compensation matrix Q.

[0049] The processing logic of the pre-projected image img is:

[0050] simg = P·Q(img);

[0051] Wherein simg represents the final projection image obtained by processing the pre-projection image img.

[0052] S5, sending the projection image simg to the projector for projection, and projecting the projection image on the projection plane. If the projection image is projected on a changing projection plane, the frequency of collecting the image and depth information of the projection plane needs to be adaptively adjusted through a cyclic decision detection algorithm.

[0053] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0054] The above specific implementations of the present invention do not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An intelligent projector control method based on color correction and depth correction, characterized in that: The following steps are involved: S1. Collecting the image and depth information of the projection plane through an optical sensing device; S2, the controller performs color processing on the projection plane according to the image to obtain a color deviation tensor between the image and the global color fitting plane; performs depth analysis on the projection plane according to the depth information to obtain a depth deviation tensor between the depth information and the global depth fitting plane; S3, performing color correction on the pre-projected image according to the color deviation tensor, obtaining a polynomial model reflecting color information according to the RGB data of each point on the image, and correcting the color of the pre-projected image by adjusting the coefficients in the polynomial model until the color deviation vector is eliminated; For any line segment AB on the global depth fitting plane, there is a corresponding line segment A′B′ in the viewer’s field of view, and the vector similarity constraint is satisfied: Where S represents the distance between any line segment AB and the viewer in the depth direction; The measured ratio of any line segment on the projection area to the corresponding line segment in the viewer's field of view is calculated based on the depth deviation tensor. When , the size of the corresponding area on the pre-projected image is reduced according to the depth deviation tensor; when the measured ratio is greater than When , the size of the corresponding area on the pre-projected image is enlarged according to the depth deviation tensor; S4, projecting the pre-projected image that has completed depth correction onto the projection area obtained by color avoidance division.

2. The intelligent projector control method based on color correction and depth correction as claimed in claim 1, characterized in that: The optical perception device uses a binocular camera, including RGB channels and Depth channels.

3. The intelligent projector control method based on color correction and depth correction as claimed in claim 1, characterized in that: The color processing is as follows: The three color channels of the image, R, G, and B, and the height and width of the projection plane form a five-dimensional feature space, where H represents the height of the projection plane and W represents the width of the projection plane. Any point in the projection plane is represented as [r, g, b] (h,w) , r represents the specific value of the point in the R dimension, g represents the specific value of the point in the G dimension, b represents the specific value of the point in the B dimension, h represents the specific value of the point in the H dimension, and w represents the specific value of the point in the W dimension; In the five-dimensional feature space, the R, G, and B color values ​​of each point on the image are fitted to obtain the global color fitting plane, and the [r, g, b] of any point on the projection plane is calculated. (h,w) The color deviation tensor of the corresponding point on the global color fitting under the three color channels of R, G, and B.

4. The intelligent projector control method based on color correction and depth correction as claimed in claim 1, characterized in that: The in-depth analysis is as follows: The depth information and the height and width of the projection plane form a three-dimensional feature space, and any point in the projection plane is represented as d (h,w) , d is the specific value of the depth of the point on the projection plane; In the three-dimensional feature space, the depth information is fitted to obtain the global depth fitting plane, and the d (h,w) The depth deviation tensor of the corresponding points on the global depth fitting plane.

Images