Projection device and image processing method thereof

Abstract

The present invention discloses a projection device and an image processing method thereof. A user can set the single-pixel area and brightness of the projection. The projection device automatically identifies its own spatial information, selects a portion of the pixel area of ​​an image source according to the spatial information for projection with the set single-pixel area, and dynamically adjusts the pixel area, number of pixels, resolution and / or brightness of the projected display when the spatial information changes, thereby eliminating keystone distortion and automatically focusing the image. This ensures that the single-pixel area and brightness of the projected image on the screen remain unchanged regardless of how the distance between the projection device and the screen changes or how the projection device moves relative to the screen, thereby achieving stable projection under any movement and bringing about a stable and good adaptive display effect. The user does not need to consider the position of the projection device and can arbitrarily move the projection device to change the image source area and number of pixels of the image source projected on the screen, thereby getting rid of the problems of traditional projectors such as fixed position, limited projection area, projection brightness, resolution, and poor application effects in dynamic virtual reality and augmented reality fields.

Description

Technical Field

[0001] The present invention belongs to the technical field of projection display and image processing, and in particular relates to a projection device capable of adaptively projecting and displaying images and an image processing method thereof. Background Art

[0002] Existing projection devices are generally developed based on the thinking and usage scope of traditional displays. They are used in the same way as ordinary displays. They only need to be turned on and connected to the video source to fixedly project the output video source content to the projection screen.

[0003] However, this projection method has several defects and technical problems:

[0004] 1. The projected image resolution is fixed, subject to brightness limitations and optimal projection distance and position requirements. Frequent movement of the projector or its placement outside the optimal projection distance and position will severely impact the display quality, even if the projected image is optically focused. When the projector is moved closer to the screen, the projection resolution remains unchanged because the image source remains unchanged. However, the projected area decreases, resulting in smaller pixels and overlapping, making it difficult to see details. Even shifting the eye position closer to the screen will not achieve optimal results. The projected brightness remains unchanged, but the smaller area results in higher unit brightness, causing glare for the user, requiring manual brightness adjustment for more comfortable viewing. When the projector is moved away from the screen, the projected image loses its detail and brightness. The projection resolution remains unchanged, but the larger display area results in larger pixels, noticeable pixel outlines and jagged edges, and lower image brightness, resulting in poor projection quality and visual perception. Furthermore, large projection areas result in lower image brightness, exacerbating blur and impacting viewing experience. When projecting at an angle, a trapezoidal projection phenomenon occurs, causing the image to stretch and degrade. Most projectors use keystone correction to improve this, but this also affects the display quality.

[0005] 2. When applied to scenes with large areas, changing display areas, and high brightness and precision requirements such as exhibition halls and augmented reality, a single projector cannot meet the needs. Multiple projectors are required to form a large projection array (system). The images of each projector are stitched together to cover the entire display area. This leads to problems such as difficulty in image stitching, interference between adjacent images, low brightness, and high cost.

[0006] 3. The rapid development of virtual reality (VR), augmented reality (AR), and mixed reality (XR) technologies and applications in recent years has placed higher demands on display and projection technologies. Traditional projection technology cannot achieve high-precision projection with dynamic position changes. It is not suitable for VR, AR, and XR scenarios where the projected image can be freely changed. It can only change the projected image by changing the input image source. It does not support dynamic resolution, brightness, or image content adjustment, and cannot meet the application requirements of VR, AR, and XR. Summary of the Invention

[0007] In response to the above technical problems, the present invention proposes a projection device and an image processing method thereof. The projection device can automatically identify its own spatial state, and dynamically adjust the pixel area, pixel number, projection resolution and projection brightness of the image source projected and displayed when the spatial state changes, thereby realizing adaptive projection and completing keystone distortion elimination and image automatic focusing, so as to realize that the content and pixel number of the projected image change with the change of the spatial state of the projection device, and realize that the single-pixel area and brightness of the projected image on the screen remain unchanged when the distance and movement between the projection device and the screen are arbitrary, that is, when the spatial state of the projection device changes arbitrarily, thereby bringing a stable and good adaptive display effect.

[0008] The present invention is achieved by adopting the following technical solutions:

[0009] A projection device image processing method is proposed, which is applied to the projection device and includes:

[0010] Input image source and configure single pixel display area;

[0011] Monitor its own spatial information and select a target pixel area where the image source needs to be projected based on the real-time spatial information;

[0012] Setting a projection resolution based on the number of pixels in the target pixel area;

[0013] The target pixel area is projected at the projection resolution and the single-pixel display area.

[0014] In some embodiments of the present invention, the spatial information includes information about the position of the projection device in a direction perpendicular to the screen and the distance over which the projection device moves. Selecting a target pixel area where the image source needs to be projected based on the real-time spatial information includes:

[0015] Calculate the number of pixels of the projected image based on the distance between the projection device and the screen and the single-pixel display area:

[0016] according to and Calculate the horizontal and vertical resolutions of the projected image; where, is the target display area of ​​the projected image, u / v is the display ratio of the projected image, D is the distance between the projection device and the screen, T is the projection ratio of the projection device, and m is the single pixel display area.

[0017] In some embodiments of the present invention, the spatial information includes the position of the projection device on a plane parallel to the screen and the motion data occurring therein. Selecting the target pixel area where the image source needs to be projected based on the real-time spatial information specifically includes:

[0018] Determine the projection center coordinates of the next frame of projection image in the screen coordinate system according to the real-time position, displacement and motion acceleration;

[0019] Determine the pixel center coordinates of the projected image in the image source pixel coordinate system based on the correspondence between the screen coordinate system and the image source pixel coordinate system;

[0020] The target pixel area where the image source needs to be projected is determined based on the pixel center coordinates.

[0021] In some embodiments of the present invention, the spatial information includes angle information of the projection device due to left-right rotation and / or vertical pitch. Selecting a target pixel area where the image source needs to be projected based on the real-time spatial information specifically includes:

[0022] Determine the projection center coordinates of the next frame of projection image in the screen coordinate system according to the real-time angle, angular velocity and angular acceleration;

[0023] Determine the pixel center coordinates of the projected image in the image source pixel coordinate system based on the correspondence between the screen coordinate system and the image source pixel coordinate system;

[0024] Determine the pixel area where the image source needs to be projected based on the pixel center coordinates;

[0025] When projecting the target pixel area, a trapezoidal correction is performed on the projected image.

[0026] In some embodiments of the present invention, the spatial information includes information about a rotation angle of the projection device caused by the rotation of the projection device around the optical axis of the projection device. Selecting a target pixel area where the image source needs to be projected based on the real-time spatial information specifically includes:

[0027] Obtaining inclination angle data γ between the projection device, its optical axis and the horizontal plane;

[0028] The projected image is controlled to rotate synchronously around its center point at an inclination angle γ.

[0029] In some embodiments of the present invention, the method further comprises:

[0030] Measure the actual display area and actual display length / width data of the projected image;

[0031] When the error between the actual display area and the target display area exceeds the set ratio, and the error between the actual display length / width and the target display length / width exceeds the set value, calculate the number of pixels ΔP that need to be adjusted on the horizontal and vertical axes respectively. x and ΔP y ;

[0032] Based on the number of pixels to be adjusted ΔP x and ΔP yThe target projection resolution is corrected, and the target pixel area of ​​the projection is adjusted according to the corrected target projection resolution.

[0033] In some embodiments of the present invention, the method is based on M′=N×W x ×W y Measure the actual display area of ​​the projected image; where N is the total number of pixels of the projected image on the camera sensor plane, The actual display length is a single pixel. The actual display width is a single pixel; w x is the length of a single photosensitive pixel of the camera sensor, w y is the width of a single photosensitive pixel of the camera sensor, and f is the focal length of the camera lens.

[0034] In some embodiments of the present invention, the number of pixels ΔP that need to be adjusted on the horizontal and vertical axes is calculated. x and ΔP y , specifically including:

[0035] Based on L x =N x W x and L y =N y W y Calculate the actual display width of the horizontal axis and the actual display width of the vertical axis of the projected image;

[0036] Calculate the difference ΔL between the actual horizontal axis width and the target horizontal axis width x , and the difference between the actual vertical axis width and the target vertical axis width ΔL y ;

[0037] according to and Calculate the number of pixels ΔP that need to be adjusted on the horizontal and vertical axes respectively x and ΔP y .

[0038] In some embodiments of the present invention, the method further comprises:

[0039] Configure target projection brightness;

[0040] The projection brightness is adjusted in real time according to the distance between the projection device and the screen to achieve the target projection brightness.

[0041] The present invention also proposes a projection device, comprising a light source, a digital light projection module, a data processing module, an image acquisition module and a sensor module; wherein the light source provides a projection light source; the digital light projection module projects an image based on an input signal from the data processing module; the image acquisition module acquires the projected image; and the sensor module acquires spatial information of the projection device; and is characterized in that the data processing module completes the projection of a target pixel area of ​​the image source based on the above-described projection device image processing method.

[0042] Compared with the prior art, the advantages and positive effects of the present invention are as follows: in the projection device and image processing method thereof proposed by the present invention, the user can set the projected single pixel area or the corresponding display size of a certain resolution image. After the projection device is started, it automatically identifies its own spatial information based on various sensors, including but not limited to its own three-dimensional spatial coordinates, angular coordinates, acceleration, angular acceleration, posture, distance from the screen and movement, etc., selects a part of the pixel area of ​​the image source according to the spatial information to project with the set single pixel area, and dynamically adjusts the projected pixel area, pixel number, resolution and / or brightness when the spatial information changes, completing the trapezoidal Distortion elimination and image autofocus ensure that the single-pixel area and / or brightness of the projected image on the screen remain unchanged regardless of how the distance between the projection device and the screen changes or how the projection device moves relative to the screen, thereby achieving stable projection under any movement and bringing stable and good adaptive display effects. When using the projection device provided by the present invention, the user does not need to consider the fixed position of the projection device, and can move the projection device at will to change the pixel content and number of pixels of the image source projected on the screen, thus getting rid of the problems of fixed position, projection area, projection brightness, and resolution limitations of traditional projectors, and realizing dynamic changes and pixel-level adjustment of the display area of ​​the projection space.

[0043] Moreover, the system structure and image processing method of the present invention can be widely applied to various projection systems such as LCD, LCOS and DLP to achieve stable and good adaptive display effects. It can be applied to virtual reality, augmented reality, mixed reality, and projection scenes of various arbitrary areas and dynamically changing display areas in large spaces. One projection device can replace the traditional splicing of multiple projection devices to display large-area image sources.

[0044] Other features and advantages of the present invention will become more apparent after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is a modular block diagram of the projection device proposed in the present invention;

[0047] Figure 2 This is a block diagram of the laser light source module of the projection device proposed by the present invention;

[0048] Figure 3 This is a block diagram of the digital light projection module of the projection device proposed by the present invention;

[0049] Figure 4 This is a functional block diagram of the data processing module of the projection device proposed by the present invention;

[0050] Figure 5 The image processing method steps of the projection device proposed by the present invention;

[0051] Figure 6 Schematic diagram of the geometry of the method for measuring distance by angle rotation according to an embodiment of the present invention;

[0052] Figure 7 Schematic diagram of the correspondence between the screen coordinate system and the image source pixel coordinate system in an embodiment of the present invention;

[0053] Figure 8 Schematic diagram of coordinate determination for two-dimensional movement of a projection device according to an embodiment of the present invention;

[0054] Figure 9 Schematic diagram of the region where the projection device rotates around the optical axis according to an embodiment of the present invention;

[0055] Figure 10 Schematic diagram of the principle of measuring image area using pixel area in an embodiment of the present invention;

[0056] Figure 11 Schematic diagram of the operation logic of adjusting the number of pixels by actual area feedback in an embodiment of the present invention;

[0057] Figure 12 Schematic diagram (right) of the low-resolution sub-frame superposition method of the galvanometer scanning trajectory (left) in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] The image processing method proposed in the present invention is applicable to various types of projection devices. In a specific embodiment of the present invention, a DLP (Digital Light Processing) projection system with a laser light source is used as a research and development verification platform, and is implemented through the following technical ideas: after the projection device is started, it automatically identifies its own space and motion information, selects the pixel area of ​​the image source to be projected and displayed according to the user's settings and image processing method, and dynamically adjusts the pixel area, number of pixels, projection resolution, and projection brightness of the image source to be projected and displayed, completes keystone distortion correction and image automatic focus, realizes that the content of the projected image changes with the movement of the projection device, and realizes that the single pixel area and brightness of the projected image on the screen remain basically unchanged at any distance and movement between the projection device and the screen.

[0060] The laser light source module includes a three-color semiconductor laser, a laser radiator, a laser light combining lens group, and a laser beam expanding collimating lens group. It is controlled by the data processing module to generate and transmit the laser light source required by the digital light projection module. The module composition and laser transmission path are as follows: Figure 2 As shown, the dotted line is the signal data direction and the solid line is the laser transmission direction.

[0061] The core of the digital light projection module is the spatial light modulator. The research and development verification platform of the present invention uses a digital micromirror device as a spatial light modulator. Other spatial light modulators can also be used. The digital light projection module includes a digital micromirror and its circuit board, a digital micromirror heat sink, an illumination optical path, a projection optical path, a four-phase galvanometer, a projection lens, and a lens focus motor. The laser light generated by the laser light source module is controlled by the data processing module, receives and projects the displayed image by driving the spatial light modulator, and realizes dynamic focus through the lens focus motor. The module composition and projection process are as follows: Figure 3 As shown, the dotted line is the signal data direction, and the solid line is the laser and projection direction.

[0062] The data processing module includes a system power supply circuit board, a laser driver circuit board, a digital micromirror driver circuit board, and a system control and image processing circuit board. It processes the input image source according to the user settings and the spatial information obtained by the sensor, then controls the laser light source module to generate laser light and the digital light projection module to project and display the image. It receives data from the image acquisition module to realize image area measurement, dynamic focus, and projection effect self-test. The module composition is as follows: Figure 4 As shown, the dotted line is signal data and the solid line is system power supply.

[0063] The image acquisition module is mainly a monocular camera, binocular camera or computer vision camera equipped with an imaging lens, which is installed parallel to the projection lens and located above the projection lens. It collects the image projected by the digital light projection module in real time and sends it to the data processing module. It cooperates with the front laser ranging sensor to measure the area of ​​the projected image, assists the data processing module to achieve dynamic focus, and completes self-test of the projection display effect.

[0064] The sensor module includes a phase-type laser rangefinder sensor, a three-axis acceleration sensor, a flight attitude sensor, a brightness sensor, and an angle servo. The front laser rangefinder and brightness sensor are mounted parallel to the side of the projection lens. The left and right laser rangefinders are mounted on the angle servo, located on the left and right housings of the device, respectively. The bottom and top laser rangefinders are directly mounted on their respective housings. The three-axis acceleration sensor and flight attitude sensor are integrated into the system control and image processing circuit board.

[0065] Phase-type laser ranging sensors also include other sensors such as pulsed laser ranging sensors, ultrasonic ranging, monocular / binocular ranging, etc. Three-axis acceleration sensors also include multi-axis gravity sensors, gravity angular velocity sensors, accelerometers, etc. Flight attitude sensors also include level sensors, horizontal gimbals, angle sensors, attitude sensors, etc.

[0066] Based on the above laser device structure, the image processing implemented in this embodiment is as follows: Figure 5 As shown, the following steps are included:

[0067] S1: Input the image source and configure the single-pixel display area and target projection brightness.

[0068] 1) The laser projection device is turned on, and the user inputs the image source, sets the projection display area corresponding to the required resolution or the area of ​​the projected display single pixel, inputs the three-dimensional data of the projection space, or selects the device to measure it independently.

[0069] After the laser projection device is started, the user needs to input the image source. According to actual needs, this image source can be a flat projection image or a multi-plane spliced ​​image for 360-degree full-space projection.

[0070] The user can set the length and width of the projection display area corresponding to a certain required resolution, or set the single pixel display area (pixel side length) of the projected display image.

[0071] S2: Determine the target pixel area where the image source is projected based on its own spatial information.

[0072] The user can input the three-dimensional data of the projection space and the center point of the screen in which the system is working, or select the distance sensor equipped by this device to independently measure the three-dimensional data of the projection space where the device is located. The device obtains the height information H of the projection space through the top and bottom laser distance sensors. The height is the sum of the distance from the top, the distance from the bottom, and the height of the device itself. The depth information D of the projection space, that is, the distance between the projector and the projection screen, is obtained through the front laser distance sensor. Figure 6 As shown, the device obtains the width information L of the projected space through the left and right laser ranging sensors A. To avoid measurement errors caused by objects blocking the side, the left and right laser ranging sensors are mounted on corresponding angle servos. During operation, the angle servo B is located at the 0-degree position perpendicular to the projection optical axis, and then rotates plus or minus 45 degrees in 5-degree steps to obtain the distance data of a straight line. The trigonometric geometry of angles and distances is used to verify whether the width information of the three-dimensional space is correctly measured or whether there is an obstructing object C on the side. If there is no obstruction, the measured data of each angle can be normalized to a fixed value by multiplying the cosine of the angle. If the device is tilted during measurement, the height, depth, and width information can be compensated for angles using the data from the level's inclination sensor to obtain correct spatial data.

[0073] 2) Establish the correspondence between the image source coordinates and the screen coordinate system.

[0074] Before the device works, the projection screen coordinate system and the image source pixel coordinate system will be calibrated and merged. Figure 7 As shown, the device can extract the length and width data of the projection screen from the three-dimensional spatial data input by the user to establish the screen real coordinate system OXY, or analyze this coordinate system through the data measured by the device sensor. After the coordinate system is established, the device will locate the center point of the screen to match the center point of the image source. By default, the center point of the screen is the center of this coordinate plane, and the user can also customize the coordinates of the screen center point. The device reads the image source corresponding to this screen coordinate system OXY input by the user and establishes the image source pixel coordinate system O according to the pixels. p X p Y p , extract the coordinates of the center point or center pixel of the image and make them the same as the coordinates of the center point of the screen, so as to realize the screen coordinate system OXY and the image source pixel coordinate system O p X p Y p Calibration and fusion, as shown in the figure, OXY is the screen coordinate system, O p X p Y p is the image source pixel coordinate system, O p is the image source pixel coordinate origin position, P centre is the common center of the screen and image source pixels.

[0075] Formula (1) is the correspondence between the screen coordinate system and the image source pixel coordinate system. (x, y) is the coordinate of the device on the screen coordinate system. (x p ,y p ) is the coordinate of the device in the image source pixel coordinate system, (b x , b y ) is the coordinate of the image source pixel origin in the screen coordinate system.

[0076] (x, y) = (x p +b x ,y p +b y ) (1).

[0077] 3) Each module performs self-check and formally enters projection working mode after confirming no errors.

[0078] 4) The data processing module obtains the data from each sensor, calculates and analyzes the three-dimensional spatial coordinate information and posture information of the device, and calculates and selects the pixel area where the image source needs to be projected based on the projection screen distance and user settings.

[0079] 41) First, determine the center point (pixel) of the projected image at the position of the image source. Based on the relationship between the calibrated and fused image source pixel coordinate system and the screen coordinate system, the device uses the data obtained by the sensor to determine its own position coordinates on the two-dimensional plane parallel to the screen. Through the correspondence between the screen coordinate system, the image source pixel coordinate system, and the device's own two-dimensional plane coordinates, the center of the initial image is determined, as shown in the following example: Figure 7 As shown, P is the position of the device, and the dotted box is the projected image at a resolution corresponding to a certain number of pixels.

[0080] S3: Setting the projection resolution based on the number of pixels in the target pixel area.

[0081] 42) After determining the center point of the projected image, the device will determine the number of pixels of the projected image, i.e., the projection resolution, based on its distance from the screen and the user-set data. The projection area is determined by the number of projected pixels and the resolution. Figure 7 dashed box).

[0082] The device first analyzes the projection display length and width data or pixel area data corresponding to the resolution set by the user, uniformly analyzes it into a single pixel area and assigns it to the variable m, where m is the target area of ​​a single pixel on the projection screen; the relationship between projection resolution and projection area is as shown in formula (5), where M is the display area of ​​the current resolution image, and P x is the horizontal resolution of the image to be displayed, P y The vertical resolution of the image to be displayed.

[0083] M=P x ·Py ·m (5);

[0084] The device uses a projection ratio calculation method to quickly determine the area of ​​the projected display at the current distance D, and then determine the appropriate projection resolution. The calculation formula for the projection area is as shown in formula (6), where u:v is the projection image ratio of the device, which can be 16:10, 16:9, 4:3, etc., and T is the equivalent projection ratio of the projection lens. After the device is calibrated, the distance to the screen measured by the laser ranging sensor and the area of ​​the projected image measured by the monocular camera are used to obtain T of approximately 1.2 in this device.

[0085]

[0086] When the system is operating, the current projection area can be obtained using formula (6) based on the distance D from the screen measured by the front laser ranging sensor. Since the projected image ratio is known, the approximate resolution can be calculated. For example, the resolution solution when the image ratio is u:v is shown in formula group (7). After obtaining the resolution corresponding to the current distance D from the screen, the position (center coordinates) and size (number of pixels, resolution) of the pixel area that the device needs to project can be determined based on the center coordinates of the image area obtained above.

[0087] Formula group (7):

[0088]

[0089]

[0090] S4: Project the target pixel area at the projection resolution and single-pixel display area.

[0091] 5) The data processing module drives the laser light source module and the digital light projection module to generate laser light and project the image of the target pixel area onto the screen with a single pixel area and projection resolution, and completes the image dynamic focus and keystone distortion correction.

[0092] Among them, the projection brightness is adjusted in real time according to the distance D between the projection device and the screen so that the projection brightness reaches the target projection brightness set by the user, ensuring that no matter how the display area changes, the projection brightness is adjusted to the target projection brightness in real time, avoiding the problem of higher unit brightness due to reduced display area and lower unit brightness due to increased display area.

[0093] S5: Monitor its own spatial information and select a target pixel area where the image source needs to be projected based on the real-time spatial information.

[0094] The device uses data from a three-axis acceleration sensor and various laser ranging sensors to comprehensively analyze its three-dimensional spatial position, movement direction, velocity, and acceleration data. A posture sensor acquires all inclination angles between the device and the horizontal plane, and digital image processing methods are used to achieve trapezoidal correction of the angled projection. A brightness sensor acquires the brightness and illumination information of the projected image to achieve dynamic brightness adjustment.

[0095] In an embodiment of the present invention, the three-dimensional movement of the device is decomposed into two components: a two-dimensional movement parallel to the screen and a one-dimensional movement perpendicular to the screen. Through a real-time monitoring method, when the device moves in three dimensions, the following are achieved: the pixel area where the image source is projected is changed, the projection resolution and projection brightness of the projected image are adjusted, so that the projected image content changes as the device moves. Moreover, the single-pixel area of ​​the image actually displayed on the screen remains basically unchanged at any distance and movement between the device and the screen.

[0096] By analyzing data from laser ranging sensors and level inclination sensors, the device can obtain information about a three-dimensional projection space with the screen as the front boundary and the device itself as the rear boundary, as well as the position of the device on a two-dimensional plane parallel to the screen and two-dimensional movement data on this plane, and the distance and depth that the device needs to project.

[0097] 6) If the device moves parallel to the projection screen, the pixel area where the image source needs to be projected is changed in real time based on the movement direction, speed, and acceleration data obtained by the sensor.

[0098] When moving, the device will calculate the two-dimensional movement of the device parallel to the screen plane in real time based on the laser ranging sensors and acceleration sensors on the top, bottom and sides. The real-time position of the device is obtained through high-speed laser ranging, and the movement tendency of the device is predicted through acceleration sensor data analysis to determine the center point coordinates of the next frame of the projected image. Figure 8 As shown, P1 is the first frame position, P2 is the second frame position, v x is the projection of two-dimensional motion on the X axis, v y Projection of two-dimensional motion on the Y axis.

[0099] The coordinates of the center of the next frame of the device at any time in the screen coordinate system can be obtained by formula groups (2) and (3).

[0100] Formula group (2): Δv=at≤0.05v;

[0101]

[0102]

[0103] Formula group (3): at>0.05v;

[0104]

[0105]

[0106] Formula group (2) is the coordinate solution for the device when it moves at a constant speed, low acceleration, or floating acceleration. The device considers the velocity increment Δv caused by the acceleration change to be less than or equal to 5% as low / floating acceleration motion. and is the reduced average speed of the device's movement projected on the X and Y axes, and t is a constant equal to 1 / 60s.

[0107] Formula group (3) is used to solve the coordinates of the device during acceleration motion and variable acceleration motion. The coordinates of the next frame are obtained by accumulating the discrete quantities collected by the acceleration sensor. In the formula, v x and v y is the reduced initial velocity of the device moving on the X and Y axes, a x and a y is the acceleration data measured by the acceleration sensor, and the accumulated amount is the number of valid acceleration data measured by the sensor within time t. After obtaining the coordinates of the center of the next frame in the screen coordinate system, substitute them into formula (1) to obtain the center coordinates of the projected image area in the image source pixel coordinate system.

[0108] After obtaining the center coordinates of the new image source pixel coordinate system after the device moves, the pixel area that needs to be updated can be determined, and the pixel area can be projected with a set single-pixel area; in terms of visual effect, the next frame of image is displayed on the area of ​​the screen centered on P2, and the next frame of image is the pixel area repositioned from the image source.

[0109] 7) If the device has angular and rotational motion, the pixel area where the image source needs to be projected is changed in real time based on the angle and angular velocity data obtained by the sensor.

[0110] When the device undergoes pitch or tilt angular motion, the projected image content will also change along with the motion, that is, the projected image area and its center coordinates will change along with the motion.

[0111] The device will obtain its own inclination data through the level inclination sensor and the three-axis acceleration sensor, and obtain the angular velocity of the device movement through the three-axis acceleration sensor to predict and adjust the center coordinate position of the next frame projection image area.

[0112] The device decomposes any angular motion into two components: left and right rotation and vertical pitch, and reads them from the sensor data, namely the X-axis component θ of the OXY screen coordinate system. x and the Y-axis component θ y, and because the front laser ranging sensor has obtained the depth information of the projection space, that is, the distance D between the device and the screen, the center coordinates of the next frame image can be solved using trigonometric function formulas, see formula group (4), ω x and β x is the angular velocity and angular acceleration of the device X axis, ω y and β y are the angular velocity and angular acceleration of the device along the Y axis.

[0113] Formula group (4):

[0114]

[0115]

[0116] When the device has an angular motion around the projection optical axis, or an inclination motion with respect to the horizontal plane, that is, a motion that is not related to pitch and left-right rotation but is related to the left-right height change of the device itself, the attitude sensor will obtain the inclination angle data γ between the device and the optical axis and the horizontal plane, and control the projection image area to rotate synchronously at an angle of γ around the center point P of the area, adapt to the new projection rotation angle, and project the corresponding rotated image, such as Figure 9 shown.

[0117] Since tilted projection can cause trapezoidal distortion, causing the displayed image to be stretched and the pixel area to be non-uniform, this device uses digital keystone correction to ensure good projection effects and precise pixel area control. Projection keystone correction includes optical keystone correction and digital keystone correction. Optical keystone correction adjusts the keystone by adjusting the physical position of the lens. It has no effect on the physical pixels of the image and has high accuracy, but the adjustment range is limited and implementation is difficult. It requires an additional two-axis electric displacement platform, which adds extra weight, volume, and cost to the device. Digital keystone correction uses a software interpolation algorithm to perform a geometric transformation on the original image to produce an inversely compensated trapezoidal image to compensate for the image's keystone distortion. The adjustment range is larger than that of optical keystone correction and does not require additional devices or structures.

[0118] 8) If the device moves perpendicular to the projection screen, the resolution and pixel area of ​​the projected image are adjusted in real time based on the distance data obtained by the sensor and the method of calculating the throw ratio.

[0119] The resolution and pixel area of ​​the projected image can be adjusted in real time using the method shown in step 42).

[0120] Through the above steps, no matter how the distance between the projection device and the screen changes or how the projection device moves relative to the screen, the single-pixel area of ​​the projected image on the screen does not change, thereby achieving stable projection under any movement and bringing a stable and good adaptive display effect. When using the projection device provided by the present invention, the user does not need to consider the fixed position of the projection device, and can move the projection device at will to change the pixel content and number of pixels of the image source projected on the screen. This gets rid of the problems of fixed position, projection area, projection brightness, and resolution limitations of traditional projectors, and realizes dynamic changes and pixel-level adjustment of the display area of ​​the projection space.

[0121] In one embodiment of the present invention, the device uses the distance D from the screen measured by the laser ranging sensor to assist the monocular camera in measuring the projected display area, so as to avoid the large ranging error of the binocular camera and thus the increase of the area measurement error. The steps are as follows:

[0122] 1. Image preprocessing: sequentially perform contrast enhancement, histogram correction, edge detection, and connection on the image acquired by the monocular camera. 2. Count the pixel area (number) of the projected area on the camera sensor plane. 3. Introduce the distance D to solve for the actual area of ​​the projected image.

[0123] The solution formula is shown in formula group (8):

[0124] M′=N×W x ×W y ;

[0125]

[0126]

[0127] M′ is the projected image area measured by the monocular camera after image processing, N is the total number of pixels of the projected image on the camera sensor plane, and W x The actual object length in the X direction on the plane with a single pixel width corresponding to the object distance D, W y The actual object length in the Y direction on the plane with a single pixel width corresponding to the object distance D, w x is the length of a single photosensitive pixel of the camera sensor, w y is the width of a single photosensitive pixel of the camera sensor, f is the focal length of the camera lens, and the schematic diagram of the projection display area measurement principle is as follows Figure 10 shown.

[0128] The device also adjusts the projection brightness in real time based on the brightness sensor to display images suitable for human eyes.

[0129] 9) The image area is measured in real time while the device is moving to provide feedback for adjusting the number of projected pixels and resolution, and to perform dynamic image focus and keystone correction in real time.

[0130] Specifically, the device selects the center point of the projected area of ​​the image source through position data and movement data, dynamically adjusts the number of pixels and projection resolution of the image projection area through projection distance and depth data, measures the real-time projection area through the monocular camera of the image acquisition module, and uses the feedback method to further adjust the number of pixels and projection resolution of the image projection area to achieve the unchanged single-pixel area of ​​the projected image.

[0131] After the device measures the current actual projection area M', it will compare the area data to verify whether it meets the user's settings. If the error between the actual projection area and the target area is less than 2% and the length and width errors are less than 10mm, it is considered to be a correct projection. If the error between the actual projection area and the target area is greater than or equal to 2% and the length and width errors are greater than or equal to 10mm, the feedback method is used to adjust P x and P y To correctly project the required area, run the logic such as the process Figure 11 shown.

[0132] Such as process Figure 11 As shown, when the error between the actual projected area and the target area is greater than or equal to 2% and the length and width errors are greater than or equal to 10mm, the device passes W x and the number of pixels N on the horizontal axis of the image x Get the actual display image horizontal axis width L x , through W y and the number of pixels N on the vertical axis of the image y Get the vertical axis width L y , and then obtain the target horizontal axis width and vertical axis width The difference ΔL x and ΔL y , according to the difference, obtain the number of pixels ΔP that need to be adjusted on the horizontal and vertical axes respectively x and ΔP y , thereby obtaining the new horizontal and vertical pixel number P′ x and P′ y , as shown in formula group (9):

[0133] L x =N x W x ;

[0134] L y =N y W y ;

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0141] During projection, the device also adjusts the laser power in real time based on the distance D from the screen measured by the laser rangefinder to adjust the projection brightness, keeping the image brightness within a reasonable and fixed range to ensure good display and viewing quality. The device uses the real-time image captured by the monocular camera to adjust the lens focus motor to achieve dynamic focus of the projected image.

[0142] Repeat the above steps to continuously project the image until the user finishes using the projection device and turns off the projection device.

[0143] In some embodiments of the present invention, the laser projection device is based on a digital micromirror device with a physical resolution of 1080p (1920×1080). By reconstructing a high-resolution image based on a four-phase galvanometer and a low-resolution subframe, it supports a maximum resolution of 4k, with horizontal and vertical pixels of 4096×2160, a maximum image refresh rate of 60Hz, and a supported color bit depth of 8 bits and 12 bits.

[0144] The realization of ultra-high-definition projection with 4K resolution and 60Hz refresh rate is mainly based on the characteristics of human vision (human eye integration effect). The display resolution can be improved by superimposing low-resolution subframes. The principle is the mature method in the field of image processing to reconstruct high-resolution images using low-resolution image sequences. The principle is shown in formula (10):

[0145]

[0146] Where f(x) is the reconstructed high-resolution image, R represents the number of low-resolution images, and y r Represents a single low-resolution image, h r A filter representing a sequence of sampled signals.

[0147] Reconstructing a high-resolution image from a low-resolution image sequence means reading a high-resolution image and using convolutional image processing to downsample a high-resolution image to obtain multiple low-resolution sub-frame images based on its resolution, the total number of effective pixels, and the required number of sub-frames, resolution, and the total number of effective pixels. Most of the time, 2 or 4 sub-frames are obtained. This device uses a four-phase galvanometer to obtain 4 sub-frames.

[0148] The number of pixels corresponding to a certain resolution can be calculated by multiplying the horizontal and vertical pixels. This device uses a digital micromirror device with a physical resolution of 1080p, and its total effective pixels are 2,073,600. The effective pixels of 4k resolution are 8,847,360, and the effective pixels of 4 times the 1080p resolution of 8,294,400 are approximately equal to the effective pixels of 4k resolution of 8,847,360. This uses an image processing method that uses 4 1080p resolution sub-frame images to reconstruct a high-resolution image through the human eye integration effect and low-resolution image sequence, so that the human eye can visually observe a high-resolution image of 4k resolution, thereby achieving the maximum resolution supported by 4k.

[0149] The digital light projection module projects these four sub-frame images in sequence at 240Hz, and completes the four-point scanning by driving the four-phase galvanometer along a square track. Each point is a sub-frame image. After the integration effect of the human eye, a 4K resolution image will be reconstructed visually. The refresh rate of this 4K image is 60Hz. The schematic diagram of the galvanometer scanning track and the low-resolution sub-frame superposition method to improve the display resolution is shown below. Figure 11 shown.

[0150] Dynamic resolution adjustment of the device: For dynamic resolution adjustment of image resolution, the subframe resolution can be changed to output images of various resolutions. When the image ratio is u:V, the subframe horizontal and vertical resolutions required by different resolution images can be obtained by formula group (11), p x is the subframe horizontal axis resolution, p y is the subframe vertical axis resolution, P x is the horizontal resolution of the image to be displayed, P y The vertical resolution of the image to be displayed.

[0151] Formula group (11):

[0152]

[0153]

[0154] It should be noted that, in the specific implementation process, part of the above-mentioned method can be implemented by a hardware processor executing computer execution instructions in software form stored in the memory, which will not be elaborated here, and the programs corresponding to the executed actions can be stored in the system's computer-readable storage medium in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0155] The computer-readable storage medium mentioned above may include volatile memory, such as random access memory; may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; may also include a combination of the above types of memory.

[0156] The processor mentioned above can also be a collective term for multiple processing elements. For example, the processor can be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices (PLDs), discrete gate or transistor logic devices (LDDs), discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor, etc., and can also be a special-purpose processor.

[0157] It should be pointed out that the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A method for processing an image of a projection device, applied to a projection device, characterized in that: include: Input image source and configure single pixel display area; Monitor its own spatial information and select a target pixel area where the image source needs to be projected based on the real-time spatial information; Setting a projection resolution based on the number of pixels in the target pixel area; Projecting the target pixel area at the projection resolution and the single-pixel display area; The spatial information includes the position of the projection device in a direction perpendicular to the screen and the distance information of the movement; Then, based on the real-time spatial information, a target pixel area where the image source needs to be projected is selected, and a projection resolution is set based on the number of pixels in the target pixel area, including: Calculate the number of pixels of the projected image based on the distance between the projection device and the screen and the single-pixel display area: according to and Calculate the horizontal and vertical resolutions of the projected image; where, is the target display area of ​​the projected image, u / v is the display ratio of the projected image, is the distance between the projection device and the screen, is the throw ratio of the projection device, is the single pixel display area; The method further comprises: Configure target projection brightness; The projection brightness is adjusted in real time according to the distance between the projection device and the screen to achieve the target projection brightness.

2. The projection device image processing method according to claim 1, wherein: The spatial information includes the position of the projection device on a plane parallel to the screen and the motion data generated; selecting the target pixel area where the image source needs to be projected based on the real-time spatial information specifically includes: Determine the projection center coordinates of the next frame of projection image in the screen coordinate system according to the real-time position, displacement and motion acceleration; Determine the pixel center coordinates of the projected image in the image source pixel coordinate system based on the correspondence between the screen coordinate system and the image source pixel coordinate system; The target pixel area where the image source needs to be projected is determined based on the pixel center coordinates.

3. The projection device image processing method according to claim 1, wherein: The spatial information includes angle information of the projection device caused by left-right rotation and / or vertical pitch. Selecting a target pixel area where the image source needs to be projected based on the real-time spatial information specifically includes: Determine the projection center coordinates of the next frame of projection image in the screen coordinate system according to the real-time angle, angular velocity and angular acceleration; Determine the pixel center coordinates of the projected image in the image source pixel coordinate system based on the correspondence between the screen coordinate system and the image source pixel coordinate system; Determine the pixel area where the image source needs to be projected based on the pixel center coordinates; When projecting the target pixel area, a trapezoidal correction is performed on the projected image.

4. The projection device image processing method according to claim 1, wherein: The spatial information includes information about a rotation angle of the projection device caused by the rotation of the projection device around the optical axis of the projection device. Selecting a target pixel area where the image source needs to be projected based on the real-time spatial information specifically includes: Obtaining inclination angle data γ between the projection device, its optical axis and the horizontal plane; The projected image is controlled to rotate synchronously around its center point at an inclination angle γ.

5. The projection device image processing method according to claim 1, wherein: The method further comprises: Measure the actual display area and actual display length / width data of the projected image; When the error between the actual display area and the target display area exceeds the set ratio, and / or the error between the actual display length / width and the target display length / width exceeds the set value, calculate the number of pixels that need to be adjusted for the horizontal and vertical axes respectively. and ; Based on the number of pixels to be adjusted and The target projection resolution is corrected, and the target pixel area is adjusted according to the corrected target projection resolution.

6. The image processing method for a projection device according to claim 5, wherein: The method is based on Measures the actual display area of ​​the projected image; where, is the total number of pixels of the projected image on the camera sensor plane, The actual projected display length for a single pixel, The actual projected display width for a single pixel; is the length of a single photosensitive pixel of the camera sensor, is the width of a single photosensitive pixel of the camera sensor, is the focal length of the camera lens.

7. The image processing method for a projection device according to claim 6, wherein: Calculate the number of pixels that need to be adjusted for the horizontal and vertical axes and , specifically including: based on and Calculate the actual display width of the horizontal axis and the actual display width of the vertical axis of the projected image; Calculate the difference between the actual displayed horizontal axis width and the target horizontal axis width , and the difference between the actual displayed vertical axis width and the target vertical axis width ; according to and Calculate the number of pixels that need to be adjusted for the horizontal and vertical axes and , Nx is the number of pixels on the horizontal axis of the image, and Ny is the number of pixels on the vertical axis of the image.

8. A projection device, comprising a light source, a digital light projection module, a data processing module, an image acquisition module and a sensor module; wherein: The light source provides a projection light source; the digital light projection module projects an image according to the input signal of the data processing module; the image acquisition module acquires the projection image; the sensor module obtains spatial information of the projection device; it is characterized in that the data processing module completes the projection of the target pixel area of ​​the image source based on the projection device image processing method according to any one of claims 1 to 7.

Images