Intelligent projector control method based on color avoidance and depth correction

Abstract

The application relates to the technical field of projectors, and particularly provides an intelligent projector control method based on color avoidance and depth correction, candidate regions on a pre-projection image are determined according to a color deviation tensor of a projection plane and a global color fitting plane, a projection region is divided in the candidate regions according to the shape of the pre-projection image, depth correction is performed on the projection region according to a depth deviation tensor of the projection region and a global depth fitting plane, and the pre-projection image after the depth correction is projected on the projection region obtained through color avoidance division. The application avoids the influence of color interference and flatness, guarantees that the projector can realize flat and achromatic projection under visual effect in the case of plane and space irregularity, color stains and other extreme conditions, and solves the problems of precision limitation, single scene and high cost in the existing projection mapping application.

Description

Technical Field

[0001] This invention relates to the field of projector technology, and specifically provides a smart projector control method based on color avoidance and depth correction. Background Technology

[0002] Projectors are widely used in meetings, teaching, presentations, and home entertainment. The quality of their projection directly affects the audience's understanding and acceptance of the content. However, in actual use, due to environmental factors, the characteristics of the wall or screen, and other factors, the projection effect varies, and problems such as color difference, blurriness, and insufficient brightness may occur. In particular, when the projection plane is uneven or the background color is uneven, traditional projectors cannot find a regular projection plane, resulting in the distortion or stretching of the projected image. At the same time, uneven color on the projection plane can also cause color distortion in the projected image, affecting the user's viewing experience.

[0003] Therefore, there is a need for an intelligent projector system and control method that can accurately and quickly measure the reflectivity and flatness of the projection plane and adaptively adjust the image to achieve better projection results. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a smart projector control method based on color avoidance and depth correction. By finding non-different color areas and performing depth compensation on these areas, the method achieves automatic finding and compensation for flatness defects in the projection area, greatly improving the projection effect.

[0005] The intelligent projector control method based on color avoidance and depth correction provided by this invention includes the following steps:

[0006] S1. Acquire images and depth information of the projection plane using optical sensing devices;

[0007] S2. The controller performs color processing on the projection plane based on the image to obtain the color deviation tensor between the image and the global color fitting plane; it also performs depth analysis on the projection plane based on the depth information to obtain the depth deviation tensor between the depth information and the global depth fitting plane.

[0008] S3. Perform color avoidance on the pre-projected image based on the color deviation tensor. Traverse all pixels on the image and calculate the color deviation tensor ΔC for any pixel. Compare the color deviation tensor ΔC with the color channel threshold ΔT and divide the region where the color deviation tensor ΔC is less than the color channel threshold ΔT. This region is the candidate region. Divide the projection region in the candidate region according to the shape of the pre-projected image. Scale the pre-projected image according to the geometric size of the projection region. Adjust the angle and focal length of the projection light according to the scaling ratio of the pre-projected image and the spatial position of the projection region.

[0009] For any line segment AB on the global depth fitting plane, there exists a corresponding line segment A′B′ in the viewer's visual space, satisfying the vector similarity constraint:

[0010]

[0011] Where S represents the distance between any line segment AB and the viewer in the depth direction;

[0012] The measured ratio of any line segment on the projected area to the corresponding line segment in the viewer's field of vision is calculated based on the depth deviation tensor. When the measured ratio is less than... At that time, the size of the corresponding region on the pre-projected image is reduced based on the depth deviation tensor; when the measured ratio is greater than At that time, the size of the corresponding region on the preprojected image is magnified based on the depth deviation tensor;

[0013] S4. Project the pre-projected image with completed depth correction onto the projection area obtained by color avoidance.

[0014] Preferably, the color processing is as follows:

[0015] The image's R, G, and B color channels, along with the height and width of the projection plane, form a five-dimensional feature space. H represents the height of the projection plane, and W represents its width. Any point in the projection plane is represented as [r, g, b]. (h,w) r represents the specific value of the point in dimension R, g represents the specific value of the point in dimension G, b represents the specific value of the point in dimension B, h represents the specific value of the point in dimension H, and w represents the specific value of the point in dimension W.

[0016] In the five-dimensional feature space, the R, G, and B color values ​​of each point on the image are fitted to obtain the global color fitting plane. The [r, g, b] values ​​at any point on the projection plane are then calculated. (h,w) The color deviation tensor of corresponding points is fitted to the global color under the three color channels R, G, and B.

[0017] Preferably, the in-depth analysis specifically includes:

[0018] The depth information, along with the height and width of the projection plane, forms a three-dimensional feature space. Any point in the projection plane is represented as d. (h,w) d is the specific value of the depth of the point on the projection plane;

[0019] In the three-dimensional feature space, the depth information is fitted to obtain the global depth fitting plane, and the depth d at any point on the projection plane is calculated. (h,w) The depth deviation tensor of the corresponding point on the global depth fitting plane.

[0020] Preferably, the color avoidance process is as follows:

[0021] Iterate through all pixels of any frame of the pre-projected image, and calculate the color deviation tensor ΔC(x,y) of each pixel in the pre-projected image based on the obtained color deviation tensor of the projection plane, where x and y represent the coordinates of the pixel in the pre-projected image.

[0022] Set the color channel threshold to ΔT. When |ΔC(x,y)|>ΔT, expand the rectangle in four directions (up, down, left, and right) starting from that pixel. Stop expanding when the color deviation tensor of the pixel is less than or equal to the color channel threshold. The resulting rectangle is the heterochromatic region, and heterochromatic regions are avoided.

[0023] Preferably, the starting pixel is (x1, y1), and the currently expanded pixel is (x1, y1). i ,y i The process for determining the different colored regions is as follows:

[0024] If |ΔC(x2,y i-1 If | ≤ ΔT, then (x2, y i-1 () represents the upper boundary of the rectangle;

[0025] If |ΔC(x2,y i+1 If | ≤ ΔT, then (x2, t) i+1 () represents the lower boundary of the rectangle;

[0026] If |ΔC(x) i-1 If ,y2)|≤ΔT, then (x i-1 y2) is the left boundary of the rectangle;

[0027] If |ΔC(x) i+1 If ,y2)|≤ΔT, then (x i+1 y2) is the right boundary of the rectangle.

[0028] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0029] This invention obtains the color deviation tensor of the projection plane through an image, and can automatically divide the heterochromatic areas, obtain the projection area in other areas, and perform depth correction on the projection area, avoiding the influence of color interference and flatness. It ensures that the projector can achieve flat and color-difference-free projection under extreme conditions such as irregular plane and space, color stains, etc., and solves the problems of limited accuracy, single scene and high cost in existing projection mapping applications. Attached Figure Description

[0030] Figure 1 This is an overall framework diagram of the intelligent projector system control method provided according to an embodiment of the present invention;

[0031] Figure 2 This is a flowchart of the global analysis of an intelligent projector provided according to an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of a depth correction scenario provided by an embodiment of the present invention. Detailed Implementation

[0033] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0035] like Figure 1 As shown in the figure, the intelligent projector control method based on color avoidance and depth correction provided in this embodiment of the invention mainly includes the following steps:

[0036] S1. A binocular camera is selected as the optical sensing device to acquire image and depth information of the projection plane. Before information acquisition, the positions of the viewer, projector, and binocular camera need to be calibrated. The relative positional relationships between the viewer, projector, and binocular camera are measured, and their respective coordinate systems with their origins are obtained. This is done for accurate modeling and precise control later. A viewer coordinate system U is established with the viewer as the origin, a camera coordinate system C is established with the binocular camera as the origin, and a projection coordinate system P is established with the projector as the origin. During the calibration process, measuring tools are used to measure the six-dimensional pose states of the binocular camera and the viewer relative to the projector in detail, obtaining the pose transformation matrix of the binocular camera relative to the projector. and the pose transformation matrix of the viewer relative to the projector The two pose transformation matrices described above describe the position and orientation of the binocular camera and the viewer relative to the projector.

[0037] like Figure 2 As shown, S2, the controller performs color processing on the projection plane based on the image, combining the R, G, and B color channels of the image with the height and width of the projection plane to form a five-dimensional feature. This feature represents the color space information R (red), G (green), and B (blue) of the projection plane, and the geometric space information H (height) and W (width). The plane formed by (H, W) represents the projection plane. Therefore, any point on this plane can be represented as [r, g, b]. (h,w)Here, r represents the specific value of the point in dimension R, g represents the specific value of the point in dimension G, b represents the specific value of the point in dimension B, h represents the specific value of the point in dimension H, and w represents the specific value of the point in dimension W. Color processing aims to obtain the five-dimensional deviation tensor between the data in each dimension of the planar scene and the ideal projection plane. Therefore, based on the collected information, the color gamut values ​​in the three dimensions of r, g, and b are used to fit each color gamut plane, and outlier points are continuously removed to fit a global color fitting plane, which is used as the ideal plane. The color information of any point on this plane is denoted as [r0, g0, b0]. Any point [r, g, b] within the known global range (H, W) is considered a color fitting plane. (h,w) By analyzing the value distribution of the values, we can obtain the color deviation tensor of the pixels in the image at the corresponding points on the global color fitting plane under the R, G, and B color channels:

[0038] ΔC(h,w)=[ΔR(h,w),ΔG(h,w),ΔB(h,w)].

[0039] The controller performs depth analysis on the projection plane based on depth information, constructing a three-dimensional feature by combining the depth D of the projection plane with its height H and width W. Different depth analysis algorithms are developed for binocular cameras of varying precision, with the precision ranging from simple to complex. The baseline dimension calculation methods include: pixel method, plane fitting method, spot method, and ray method. Any point on the projection plane is represented as d. (h,w) Let d represent the specific depth value of the point on the projection plane. In the three-dimensional feature space, depth analysis first fuses and tensors the depth information with spatial geometric data. A plane fitting algorithm is selected according to the accuracy level of the stereo camera capturing the depth information. Through fitting and continuous outlier removal, a global depth fitting plane is obtained and used as the ideal plane. For any point within the global range (H, W), the depth information d... (h,) This allows us to calculate the depth deviation tensor of the spatial location containing depth information relative to the global depth fitting plane.

[0040] S3. Perform color avoidance on the pre-projected image based on the color deviation tensor. Traverse each pixel position on the image and calculate the color deviation tensor ΔC(x,y) for each pixel position. (x,y) represents the position coordinates of the point on the image, which corresponds one-to-one with the points on the projection plane (H,W).

[0041] A color channel threshold ΔT is set to determine whether the color deviation of a pixel exceeds the threshold. The color deviation tensor ΔC is compared with the color channel threshold ΔT. When |ΔC(x,y)|≥ΔT, it means that the deviation of the pixel in a certain color channel exceeds the threshold. This area is a heterochromatic area and needs to be avoided. When |ΔC(x,y)|<ΔT, this area is a candidate area.

[0042] When |ΔC(x,y)|>ΔT, starting from that pixel, the rectangle is expanded in four directions: up, down, left, and right. When the color deviation tensor of the pixel is less than or equal to the color channel threshold, the expansion stops. The resulting rectangle is a heterochromatic region, and heterochromatic regions are avoided.

[0043] A specific example of the rectangle expansion process is as follows: Assume the starting pixel is (x1, y1), and the currently expanded pixel is (x1, y1). i ,y i Based on the following example, determine whether to continue expanding:

[0044] If |ΔC(x2,y i-1 If | ≤ ΔT, then (x2, t) i-1 () represents the upper boundary of the rectangle;

[0045] If |ΔC(x2,y i+1 If | ≤ ΔT, then (x2, y i+1 () represents the lower boundary of the rectangle;

[0046] If |ΔC(x) i-1 If ,y2)|≤ΔT, then (x i-1 y2) is the left boundary of the rectangle;

[0047] If |ΔC(x) i+1 If ,y2)|≤ΔT, then (x i+1 y2) is the right boundary of the rectangle.

[0048] Through the above process, bounding boxes that do not meet the color channel thresholds can be identified; these bounding boxes are represented as dissimilar color regions. This process can be used to identify areas with significant color deviations in the pre-projected image. Once dissimilar color regions are identified, directional projection measures can be taken, that is, the projected light rays can avoid the dissimilar color regions or the optical parameters can be adjusted to achieve the effect of color avoidance.

[0049] After removing the discolored areas on the projection plane, candidate areas are obtained. The projection area is divided into the candidate areas according to the shape of the pre-projected image. The pre-projected image is scaled according to the geometric dimensions of the projection area. The angle and focal length of the projection light are adjusted according to the scaling ratio of the pre-projected image and the spatial position of the projection area.

[0050] Depth correction of the projection area mainly includes the following steps:

[0051] pose transformation matrix of binocular camera relative to projector and the pose transformation matrix of the viewer relative to the projector Perform a coordinate transformation on any point R, and then calibrate the points in the three coordinate systems: the projected coordinate system P, the viewer's coordinate system U, and the camera's coordinate system C. The calibration formula is as follows:

[0052]

[0053]

[0054] Among them, R p This indicates that any point R in the projected coordinate system P, R C This indicates that any point R is in the camera coordinate system C, R U This indicates that any point R lies in the viewer's coordinate system U.

[0055] like Figure 3 As shown, l1+l2 is the actual projection plane. The l2 region is flat, and the deviation in the projection effect mainly comes from the l1 region. Using D(h,w) U The function represents the depth information in the projection plane l1 region, which is incorporated into the viewer's coordinate system U as (h,w,D(h,w)). U In the viewer's coordinate system U, a viewer's visual space is obtained. For any line segment AB on the ideal depth plane, there exists a corresponding line segment A′B′ in the viewer's visual space, satisfying vector similarity constraints:

[0056]

[0057] Where S represents the distance between any line segment AB and the viewer in the depth direction; the measured ratio of any line segment on the projection plane to the corresponding line segment in the viewer's field of vision is calculated based on the depth deviation tensor. When the measured ratio is less than... When the measured ratio is greater than a certain value, it indicates that the projection plane is concave, and the size of the corresponding area on the pre-projected image needs to be reduced according to the depth deviation tensor, that is, the light in that part is contracted to reduce the scattering angle; when the measured ratio is greater than a certain value, it indicates that the projection plane is concave, and the size of the corresponding area on the pre-projected image needs to be reduced according to the depth deviation tensor, that is, the light in that part is contracted to When the projection plane is convex, the size of the corresponding area on the pre-projected image is magnified according to the depth deviation tensor. This means that the light in that part is expanded, increasing the scattering angle. This ensures that the image seen by the viewer, whether on the convex plane or in the depression, is the same size as the image on the ideal depth plane, avoiding overlapping or tearing of the projected image at concave and convex locations. The scaling factors for reduction and magnification are calibrated with the coordinate positions to obtain the global depth compensation matrix Q. The depth of the pre-projected image is then corrected based on the depth compensation matrix Q.

[0058] S4. Project the depth-corrected pre-projected image onto the projection area obtained through color avoidance segmentation. If the projected image is projected onto a changing projection plane, the frequency of acquiring image and depth information of the projection plane needs to be adaptively adjusted through a cyclic decision detection algorithm.

[0059] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0060] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A smart projector control method based on color avoidance and depth correction, characterized in that, Includes the following steps: S1. Acquire images and depth information of the projection plane using optical sensing devices; S2. The controller performs color processing on the projection plane based on the image to obtain the color deviation tensor between the image and the global color fitting plane. The specific color processing is as follows: The image's R, G, and B color channels are combined with the height and width of the projection plane to form a five-dimensional feature space, where H represents the height of the projection plane and W represents its width. Any point in the projection plane is represented as... , This represents the specific value of the point in dimension R. This represents the specific value of the point in dimension G. This indicates the specific value of the point in dimension B. This indicates the specific value of the point in dimension H. This indicates the specific value of the point in dimension W; In the five-dimensional feature space, the R, G, and B color values ​​of each point on the image are fitted to obtain the global color fitting plane, and the color fitting plane of any point on the projection plane is calculated. The color deviation tensor of the corresponding points on the global color fitting plane under the three color channels R, G, and B; Based on the depth information, a depth analysis is performed on the projection plane to obtain the depth deviation tensor between the depth information and the global depth fitting plane; In-depth analysis specifically includes: The depth information, along with the height and width of the projection plane, forms a three-dimensional feature space. Any point in the projection plane is represented as... , This represents the specific depth of the point on the projection plane. In a three-dimensional feature space, depth information is fitted to obtain a global depth fitting plane, and the depth of any point on the projection plane is calculated. The depth deviation tensor of the corresponding point on the global depth fitting plane; S3. Perform color avoidance on the pre-projected image based on the color deviation tensor. Traverse all pixels in the image and calculate the color deviation tensor for any pixel. Transform the color deviation tensor Values ​​and color channel thresholds ΔT Compare and divide the color deviation tensor Less than or equal to the color channel threshold ΔT The area is the candidate area. The projection area is divided into the candidate area according to the shape of the pre-projected image. The pre-projected image is scaled according to the geometric size of the projection area. The angle and focal length of the projection light are adjusted according to the scaling ratio of the pre-projected image and the spatial position of the projection area. For any line segment on the global depth-fitting plane AB There are corresponding line segments within the viewer's field of vision. And it satisfies the vector similarity constraint: ； Where S represents the distance between any line segment AB and the viewer in the depth direction; Calculate the measured ratio of any line segment on the projection area to the corresponding line segment in the viewer's field of vision. When the measured ratio is less than... At that time, the size of the corresponding region on the pre-projected image is reduced based on the depth deviation tensor; when the measured ratio is greater than At that time, the size of the corresponding region on the preprojected image is magnified based on the depth deviation tensor; S4. Project the pre-projected image with completed depth correction onto the projection area obtained by color avoidance.

2. The intelligent projector control method based on color avoidance and depth correction as described in claim 1, characterized in that, The process of color avoidance is as follows: Iterate through all pixels of any frame of the pre-projected image, and calculate the color deviation tensor of each pixel of the pre-projected image based on the obtained color deviation tensor of the projection plane. , Indicates the coordinates of a pixel on the pre-projected image; Set the color channel threshold to ,when At that time, starting from the pixel, the rectangle is expanded in four directions: upward, downward, left, and right. When the color deviation tensor of the pixel is less than or equal to the color channel threshold, the expansion stops. The resulting rectangle is a heterochromatic region, and heterochromatic regions are avoided.

3. The intelligent projector control method based on color avoidance and depth correction as described in claim 2, characterized in that, Starting pixel is The currently expanded pixels are The process of expanding the rectangle is as follows: like ,but It is the upper boundary of the rectangle; like ,but It is the lower boundary of the rectangle; like ,but It is the left boundary of the rectangle; like ,but It is the right boundary of the rectangle.

Images