Intelligent projector control method based on color avoidance and depth correction

Abstract

The invention relates to the technical field of projectors, and particularly provides an intelligent projector control method based on color avoidance and depth correction, which comprises the following steps of: determining a candidate region on a pre-projection image according to color deviation tensors of a projection plane and a global color fitting plane, dividing a projection region in the candidate region according to the shape of the pre-projection image, and performing depth correction on the projection region. And performing depth correction on the projection area according to the depth deviation tensor of the projection area and the global depth fitting plane, and projecting the pre-projection image subjected to depth correction on the projection area obtained through color evasion division. According to the invention, the influence of color interference and flatness is avoided, the projector is ensured to realize smooth and chromatic aberration-free projection under a visual effect under extreme conditions of plane and space irregularity, color stains and the like, and the problems of limited precision, single scene, high cost and the like in the existing projection mapping application are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of projectors, and specifically provides an intelligent projector control method based on color avoidance and depth correction. Background Art

[0002] Projectors are widely used in conferences, teaching, exhibitions, and home entertainment. The quality of their projection effects directly affects the audience's understanding and acceptance of the content. However, in actual use, due to environmental factors, wall or screen characteristics and other factors, the projection effects are different, and there may be problems such as color difference, blur, and insufficient brightness. Especially when the projection plane is uneven in shape or the background color is uneven, traditional projectors cannot find a regular projection plane, resulting in distortion or stretching of the projected image. At the same time, the uneven color of the projection plane will also cause color distortion in the projected image, affecting the user's viewing experience.

[0003] Therefore, there is a need for an intelligent projector system and control method that can accurately and quickly measure the reflectivity and flatness of the projection plane and adaptively adjust the image to achieve better projection effects. Summary of the invention

[0004] In order to solve the above problems, the present invention provides an intelligent projector control method based on color avoidance and depth correction. By finding the non-heterochromatic area and performing depth compensation on the non-heterochromatic area, automatic finding and compensation of flatness defects in the projection area are achieved, thereby greatly improving the projection effect.

[0005] The intelligent projector control method based on color avoidance and depth correction provided by the present invention comprises the following steps:

[0006] S1. Collecting the image and depth information of the projection plane through an optical sensing device;

[0007] S2, the controller performs color processing on the projection plane according to the image to obtain a color deviation tensor between the image and the global color fitting plane; performs depth analysis on the projection plane according to the depth information to obtain a depth deviation tensor between the depth information and the global depth fitting plane;

[0008] S3, perform color avoidance on the pre-projection image according to the color deviation tensor, traverse all pixels on the image, and calculate the color deviation tensor ΔC of any pixel, compare the color deviation tensor ΔC with the color channel threshold ΔT, divide the area where the color deviation tensor ΔC is less than the color channel threshold ΔT, and divide the area into candidate areas according to the shape of the pre-projection image. The pre-projection image is scaled according to the geometric size of the projection area, and the angle and focal length of the projection light are adjusted according to the scaling ratio of the pre-projection image and the spatial position of the projection area;

[0009] For any line segment AB on the global depth fitting plane, there is a corresponding line segment A′B′ in the viewer’s field of view, and the vector similarity constraint is satisfied:

[0010]

[0011] Where S represents the distance between any line segment AB and the viewer in the depth direction;

[0012] The measured ratio of any line segment on the projection area to the corresponding line segment in the viewer's field of view is calculated based on the depth deviation tensor. When , the size of the corresponding area on the pre-projected image is reduced according to the depth deviation tensor; when the measured ratio is greater than When , the size of the corresponding area on the pre-projected image is enlarged according to the depth deviation tensor;

[0013] S4, projecting the pre-projected image that has completed depth correction onto the projection area obtained by color avoidance division.

[0014] Preferably, the color processing is specifically as follows:

[0015] The three color channels of the image, R, G, and B, and the height and width of the projection plane form a five-dimensional feature space, where H represents the height of the projection plane and W represents the width of the projection plane. Any point in the projection plane is represented as [r, g, b] (h,w) , r represents the specific value of the point in the R dimension, g represents the specific value of the point in the G dimension, b represents the specific value of the point in the B dimension, h represents the specific value of the point in the H dimension, and w represents the specific value of the point in the W dimension;

[0016] In the five-dimensional feature space, the R, G, and B color values ​​of each point on the image are fitted to obtain the global color fitting plane, and the [r, g, b] of any point on the projection plane is calculated. (h,w) The color deviation tensor of the corresponding points on the global color fitting under the three color channels of R, G, and B.

[0017] Preferably, the in-depth analysis is specifically as follows:

[0018] The depth information and the height and width of the projection plane form a three-dimensional feature space, and any point in the projection plane is represented as d (h,w) , d is the specific value of the depth of the point on the projection plane;

[0019] In the three-dimensional feature space, the depth information is fitted to obtain the global depth fitting plane, and the d (h,w) The depth deviation tensor of the corresponding points on the global depth fitting plane.

[0020] Preferably, the process of color avoidance is:

[0021] Traverse all pixels of any frame of pre-projection image, and calculate the color deviation tensor ΔC(x,y) of each pixel of the pre-projection image according to the obtained color deviation tensor of the projection plane, where x and y represent the coordinates of the pixel point on the pre-projection image;

[0022] Set the color channel threshold to ΔT. When |ΔC(x,y)|>ΔT, take the pixel as the starting point and expand the rectangular box in the four directions of upward, downward, left and right. When the color deviation tensor of the pixel is less than or equal to the color channel threshold, stop expanding. The obtained rectangular box is a heterochromatic area, which is avoided.

[0023] Preferably, the starting pixel is (x1, y1) and the pixel currently expanded to is (x i ,y i ), the judgment process of different color areas is:

[0024] If |ΔC(x2,y i-1 )|≤ΔT, then (x2,y i-1 ) is the upper boundary of the rectangular box;

[0025] If |ΔC(x2,y i+1 )|≤ΔT, then (x2,t i+1 ) is the lower border of the rectangular box;

[0026] If |ΔC(x i-1 ,y2)|≤ΔT, then (x i-1 ,y2) is the left boundary of the rectangular box;

[0027] If |ΔC(x i+1 ,y2)|≤ΔT, then (x i+1 ,y2) is the right border of the rectangular box.

[0028] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0029] The present invention obtains the color deviation tensor of the projection plane through the image, and can automatically divide the different color areas, obtain the projection area in other areas, and perform depth correction on the projection area, thereby avoiding the influence of color interference and flatness, and ensuring that the projector can achieve flat and color-free projection under extreme conditions such as plane and space irregularities and color stains under visual effects, thereby solving the problems of limited accuracy, single scene and high cost in existing projection mapping applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is an overall framework diagram of a smart projector system control method provided according to an embodiment of the present invention;

[0031] Figure 2 is a global analysis flow chart of a smart projector provided according to an embodiment of the present invention;

[0032] Figure 3 4 is a schematic diagram of a scene of depth correction provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0034] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0035] like Figure 1 As shown, the intelligent projector control method based on color avoidance and depth correction provided by the embodiment of the present invention mainly includes the following steps:

[0036] S1. Use a binocular camera as an optical perception device to collect image and depth information of the projection plane. Before information collection, it is first necessary to calibrate the positions of the viewer, projector and binocular camera, measure the relative position relationship between the viewer, projector and binocular camera, and obtain their coordinate systems with their respective positions as the origin. The purpose of this is to facilitate subsequent accurate modeling and precise control. Establish the viewer coordinate system U with the viewer as the origin, establish the camera coordinate system C with the binocular camera as the origin, and establish the projection coordinate system P with the projector as the origin. During the calibration process, use measurement tools to measure in detail the six-dimensional posture state of the binocular camera and the viewer relative to the projector, and obtain the posture transformation matrix of the binocular camera relative to the projector. And the viewer's position transformation matrix relative to the projector The above two pose transformation matrices describe the position and posture of the binocular camera and the viewer relative to the projector.

[0037] like Figure 2 As shown, S2, the controller processes the color of the projection plane according to the image, and combines the three color channels of the image, R, G, and B, with the height and width of the projection plane to form a five-dimensional feature, which respectively represents the color space information R (red), G (green), and B (blue) of the projection plane, and the geometric space information H (height) and W (width). The plane composed of (H, W) represents the projection plane, so any point on this plane can be represented as [r, g, b] (h,w), where r represents the specific value of the point in the R dimension, g represents the specific value of the point in the G dimension, b represents the specific value of the point in the B dimension, h represents the specific value of the point in the H dimension, and w represents the specific value of the point in the W dimension. Color processing is to obtain the five-dimensional deviation tensor of the data of each dimension in the plane scene and the ideal projection plane. Therefore, the color gamut values ​​of the three dimensions of r, g, and b are fitted to each color gamut plane based on the collected information, and the outliers are continuously stripped to fit the global color fitting plane. As the ideal plane, the color information of any point on it is recorded as [r0, g0, b0]. At any point [r, g, b] within the known global range (H, W) (h,w) The value distribution of can be used to obtain the color deviation tensor of the corresponding point on the global color fitting plane under the three color channels of R, G, and B:

[0038] ΔC(h,w)=[ΔR(h,w),ΔG(h,w),ΔB(h,w)].

[0039] The controller performs depth analysis on the projection plane based on the depth information, and combines the depth D of the projection plane with the height H and width W of the projection plane to form a three-dimensional feature. The corresponding depth analysis algorithm is formulated according to the binocular cameras with different precisions. The precision ranges from simple to complex. The base dimension calculation methods can be selected: pixel method, plane fitting method, spot method, and light method. Any point in the projection plane is represented by d (h,w) , d represents the specific value of the depth of the point on the projection plane. In the three-dimensional feature space, the depth analysis first fuses the depth information with the spatial geometric data and quantizes them. The plane fitting algorithm is selected according to the accuracy level of the binocular camera captured by the depth information. The global depth fitting plane is obtained by fitting and continuously removing outliers. It is used as the ideal plane. For any point in the global range (H, W), the depth information d (h,) , the depth deviation tensor of the spatial position containing the depth information relative to the global depth fitting plane can be obtained.

[0040] S3. Perform color avoidance on the pre-projected image according to the color deviation tensor, traverse each pixel position on the image, and calculate the color deviation tensor ΔC(x, y) of each pixel position, where (x, y) represents the position coordinates of the point on the image, and corresponds one-to-one to the point on the projection plane (H, W).

[0041] A color channel threshold ΔT is set to determine whether the color deviation of a pixel exceeds the threshold. The color deviation tensor ΔC is compared with the color channel threshold ΔT. When |ΔC(x,y)|≥ΔT, it means that the deviation of the pixel on a certain color channel exceeds the threshold. The area is a heterochromatic area and needs to be avoided. When |ΔC(x,y)|<ΔT, the area is a candidate area.

[0042] When |ΔC(x,y)|>ΔT, the rectangular box is expanded in the four directions of upward, downward, left and right with the pixel as the starting point. When the color deviation tensor of the pixel is less than or equal to the color channel threshold, the expansion stops and the obtained rectangular box is a heterochromatic area, which is avoided.

[0043] The following is a specific example of the rectangular frame expansion process: Assume that the starting pixel is (x1, y1) and the pixel currently expanded to is (x i ,y i ), determine whether to continue expanding based on the following example:

[0044] If |ΔC(x2,y i-1 )|≤ΔT, then (x2,t i-1 ) is the upper boundary of the rectangular box;

[0045] If |ΔC(x2,y i+1 )|≤ΔT, then (x2,y i+1 ) is the lower border of the rectangular box;

[0046] If |ΔC(x i-1 ,y2)|≤ΔT, then (x i-1 ,y2) is the left boundary of the rectangular box;

[0047] If |ΔC(x i+1 ,y2)|≤ΔT, then (x i+1 ,y2) is the right border of the rectangular box.

[0048] Through the above process, a rectangular box that does not meet the color channel threshold can be determined, and the rectangular box is represented as a heterochromatic area. This process can be used to identify areas with large color deviations in the pre-projected image. Once the heterochromatic area is identified, directional projection measures can be taken, that is, the projection light avoids the heterochromatic area or adjusts the optical parameters, thereby achieving the effect of color avoidance.

[0049] After removing the heterochromatic area on the projection plane, a candidate area is obtained, and a projection area is divided in the candidate area according to the shape of the pre-projection image. The pre-projection image is scaled according to the geometric size of the projection area, and the angle and focal length of the projection light are adjusted according to the scaling ratio of the pre-projection image and the spatial position of the projection area.

[0050] Depth correction for the projection area mainly includes the following steps:

[0051] The pose transformation matrix of the binocular camera relative to the projector And the viewer's position transformation matrix relative to the projector Transform the coordinates of any point R, and calibrate the points in the projection coordinate system P, the viewer coordinate system U, and the camera coordinate system C. The calibration formula is:

[0052]

[0053]

[0054] Among them, R p Represents any point R in the projected coordinate system P, R C Indicates that any point R is in the camera coordinate system C, R U Represents an arbitrary point R in the viewer's coordinate system U.

[0055] like Figure 3 As shown in the figure, l1+l2 is the actual projection plane, the l2 area is flat, and the deviation of the projection effect mainly comes from the l1 area. Use D(h,w) U The function represents the depth information of the projection plane l1 area, which is incorporated into the viewer coordinate system U as (h,w,D(h,w) U ). A viewer's field of view is obtained by dividing in the viewer coordinate system U. For any line segment AB on the depth ideal plane, there is a corresponding line segment A′B′ in the viewer's field of view, and the vector similarity constraint is satisfied:

[0056]

[0057] Where S represents the distance between any line segment AB and the viewer in the depth direction; the measured ratio of any line segment on the projection plane to the corresponding line segment in the viewer's field of view is calculated based on the depth deviation tensor. When the measured ratio is less than When , it indicates that the projection plane is concave, and it is necessary to reduce the size of the corresponding area on the pre-projection image according to the depth deviation tensor, that is, to shrink the light in this part and reduce the scattering angle; when the measured ratio is greater than When , it indicates that the projection plane is convex. The size of the corresponding area on the pre-projected image is enlarged according to the depth deviation tensor, that is, the light of this part is expanded and the scattering angle is increased, that is, the image size of the viewer on the convex plane or the concave inside is the same as the image size on the ideal depth plane, avoiding overlapping or tearing of the projected image at the concave and convex positions. The scale coefficients of reduction and enlargement are calibrated with the coordinate position to obtain the global depth compensation matrix Q, and the depth correction of the pre-projected image is performed according to the depth compensation matrix Q.

[0058] S4, projecting the pre-projected image after depth correction onto the projection area obtained by color avoidance division. If the projection image is projected onto a changing projection plane, the frequency of collecting the image and depth information of the projection plane needs to be adaptively adjusted through a cyclic decision detection algorithm.

[0059] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0060] The above specific implementations of the present invention do not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An intelligent projector control method based on color avoidance and depth correction, characterized in that: The following steps are involved: S1. Collecting the image and depth information of the projection plane through an optical sensing device; S2, the controller performs color processing on the projection plane according to the image to obtain a color deviation tensor between the image and the global color fitting plane; performs depth analysis on the projection plane according to the depth information to obtain a depth deviation tensor between the depth information and the global depth fitting plane; S3, perform color avoidance on the pre-projection image according to the color deviation tensor, traverse all pixels on the image, and calculate the color deviation tensor ΔC of any pixel, compare the color deviation tensor ΔC with the color channel threshold ΔT, divide the area where the color deviation tensor ΔC is less than the color channel threshold ΔT, and divide the area into candidate areas according to the shape of the pre-projection image. The pre-projection image is scaled according to the geometric size of the projection area, and the angle and focal length of the projection light are adjusted according to the scaling ratio of the pre-projection image and the spatial position of the projection area; For any line segment AB on the global depth fitting plane, there is a corresponding line segment A′B′ in the viewer’s field of view, and the vector similarity constraint is satisfied: Where S represents the distance between any line segment AB and the viewer in the depth direction; The measured ratio of any line segment on the projection area to the corresponding line segment in the viewer's field of view is calculated based on the depth deviation tensor. When , the size of the corresponding area on the pre-projected image is reduced according to the depth deviation tensor; when the measured ratio is greater than When , the size of the corresponding area on the pre-projected image is enlarged according to the depth deviation tensor; S4, projecting the pre-projected image that has completed depth correction onto the projection area obtained by color avoidance division.

2. The intelligent projector control method based on color avoidance and depth correction as claimed in claim 1, characterized in that: The color processing is as follows: The three color channels of the image, R, G, and B, and the height and width of the projection plane form a five-dimensional feature space, where H represents the height of the projection plane and W represents the width of the projection plane. Any point in the projection plane is represented as [r, g, b] (h,w) , r represents the specific value of the point in the R dimension, g represents the specific value of the point in the G dimension, b represents the specific value of the point in the B dimension, h represents the specific value of the point in the H dimension, and w represents the specific value of the point in the W dimension; In the five-dimensional feature space, the R, G, and B color values ​​of each point on the image are fitted to obtain the global color fitting plane, and the [r, g, b] of any point on the projection plane is calculated. (h,w) The color deviation tensor of the corresponding points on the global color fitting under the three color channels of R, G, and B.

3. The intelligent projector control method based on color avoidance and depth correction as claimed in claim 1, characterized in that: The in-depth analysis is as follows: The depth information and the height and width of the projection plane form a three-dimensional feature space, and any point in the projection plane is represented as d (h,w) , d is the specific value of the depth of the point on the projection plane; In the three-dimensional feature space, the depth information is fitted to obtain the global depth fitting plane, and the d (h,w) The depth deviation tensor of the corresponding points on the global depth fitting plane.

4. The intelligent projector control method based on color avoidance and depth correction as claimed in claim 1, characterized in that: The process of color avoidance is: Traverse all pixels of any frame of pre-projection image, and calculate the color deviation tensor ΔC(x,y) of each pixel of the pre-projection image according to the obtained color deviation tensor of the projection plane, where x and y represent the coordinates of the pixel point on the pre-projection image; Set the color channel threshold to ΔT. When |ΔC(x,y)|>ΔT, take the pixel as the starting point and expand the rectangular box in the four directions of upward, downward, left and right. When the color deviation tensor of the pixel is less than or equal to the color channel threshold, stop expanding. The obtained rectangular box is a heterochromatic area, which is avoided.

5. The intelligent projector control method based on color avoidance and depth correction as claimed in claim 4, characterized in that: The starting pixel is (x1, y1), and the pixel currently expanded to is (x i ,y i ), the judgment process of different color areas is: If |ΔC(x2,y i-1 )|≤ΔT, then (x2,t i-1 ) is the upper boundary of the rectangular box; If |ΔC(x2,y i+1 )|≤ΔT, then (x2,y i+1 ) is the lower border of the rectangular box; If |ΔC(x i-1 ,y2)|≤ΔT, then (x i-1 ,y2) is the left boundary of the rectangular box; If |ΔC(x i+1 ,y2)|≤ΔT, then (x i+1 ,y2) is the right border of the rectangular box.

Images