Interactive projection system and method based on structured light perception and image anti-distortion

Abstract

The invention discloses an interactive projection system and method based on structured light perception and image anti-distortion, and relates to the technical field of interactive projection, and the method comprises a structured light perception module which is used for obtaining a projection plane depth image and an object image in a projection area; the projection modeling and object recognition module is used for constructing a change relation from a plane coordinate to a three-dimensional world coordinate according to the projection plane depth image, and recognizing object state data according to the object image; the image control and projection display module is used for determining an original image according to the object state data; the image anti-distortion processing module is used for performing pre-transformation processing on the original image based on the grid model and the projection device parameters to generate an anti-distortion image; and the image control and projection display module is also used for projecting the anti-distortion image to the projection area in real time. The interactive experience and the projection effect can be remarkably improved.

Description

Technical Field

[0001] This application relates to the field of interactive projection technology, and in particular to an interactive projection system and method based on structured light perception and image distortion correction. Background Technology

[0002] With the rapid development of projection technology and interactive systems, welcome projection systems are increasingly widely used in areas such as car welcomes, smart displays, and retail interactions. However, existing welcome projection systems still have significant shortcomings in terms of interactive and projection effects. On the one hand, most welcome projections use static logos or simple dynamic projections, lacking real-time interactive capabilities and personalized experiences. Although some systems have attempted to introduce technologies such as millimeter-wave radar or ToF cameras to achieve interactive functions, problems such as low resolution, inaccurate motion capture, and the inability to model the projection surface still exist, resulting in unsatisfactory actual interactive effects. On the other hand, traditional ground projection technology is difficult to adapt to complex environments. When used on uneven surfaces (including but not limited to wrinkled fabrics, mud, gravel, steps, etc.), the projected image is prone to distortion and drift, making it difficult to present the preset image effects, further restricting the practical application effect of the projection system. Therefore, there is an urgent need for a solution that can simultaneously optimize the interactive experience and projection adaptability. Summary of the Invention

[0003] The purpose of this application is to provide an interactive projection system and method based on structured light perception and image distortion correction, which can significantly improve the interactive experience and projection effect.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] In a first aspect, this application provides an interactive projection system based on structured light sensing and image distortion correction, comprising:

[0006] The structured light sensing module is used to acquire depth images of the projection surface and object images within the projection area;

[0007] The projection modeling and object recognition module is used to construct the transformation relationship from the planar coordinates of the structured light sensing module to the three-dimensional world coordinates based on the projection surface depth image, and to recognize the object state data based on the object image.

[0008] The image control and projection display module is used to determine the original image based on the object state data, and transmit the original image and preset projection device parameters to the image anti-distortion processing module;

[0009] The image anti-distortion processing module is used to perform pre-transformation processing on the original image based on the change relationship and the projection device parameters to generate an anti-distortion image;

[0010] The image control and projection display module is also used to project the anti-distortion image onto the projection area in real time.

[0011] Optionally, the structured light sensing module includes a structured light projector and a camera; the structured light projector is used to emit a structured light coded pattern into the projection area; the camera is used to capture a projection surface depth image and an object image within the projection area when the structured light projector is working.

[0012] Optionally, the transformation relationship includes a mesh model. Specifically, in constructing the transformation relationship from the planar coordinates of the structured light perception module to the three-dimensional world coordinates based on the projection surface depth image, the projection modeling and object recognition module includes:

[0013] The projection plane depth image is subjected to coordinate transformation to obtain a three-dimensional point cloud;

[0014] The three-dimensional point cloud is converted into a mesh model using a point cloud reconstruction algorithm.

[0015] Optionally, the transformation relationship further includes a texture mapping matrix. After converting the 3D point cloud into a mesh model using the point cloud reconstruction algorithm, the projection modeling and object recognition module further includes:

[0016] Construct a texture mapping matrix; the texture mapping matrix represents the correspondence between texture coordinates and 3D point clouds;

[0017] The texture mapping matrix is ​​then mapped onto the surface of the mesh model.

[0018] Optionally, in the process of performing coordinate transformation on the projection surface depth image to obtain a 3D point cloud, the projection modeling and object recognition module specifically includes:

[0019]

[0020]

[0021] Where (X,Y,Z) are the 3D point cloud coordinates; D(x,y) is the projection plane depth image; s is the scale factor of the projection plane depth image; f x f is the focal length of the camera along the x-axis. y c is the focal length of the camera along the y-axis. x The x-coordinate of the principal point in the image coordinate system, c y The y-coordinate of the principal point in the image coordinate system.

[0022] Optionally, the object state data includes the positions of key points of the object and its dynamic behavior; in terms of recognizing the object state data based on the object image, the projection modeling and object recognition module specifically includes:

[0023] The image of the object is processed using an image recognition model to obtain the positions of the object's key points;

[0024] The dynamic behavior is determined based on the position of the object's key points at each moment.

[0025] Optionally, in the step of pre-transforming the original image based on the change relationship and the projection device parameters to generate an anti-distortion image, the image anti-distortion processing module specifically includes:

[0026] Based on the aforementioned transformation relationship and the parameters of the projection device, a projection transformation matrix is ​​constructed;

[0027] The original image is inversely transformed using the inverse of the projection transformation matrix to obtain an anti-distortion image.

[0028] Optionally, in constructing the projection transformation matrix based on the change relationship and the projection device parameters, the image distortion correction module specifically includes:

[0029] H = K·[R|t]·M;

[0030] Where H is the projection transformation matrix; K is the intrinsic parameter matrix of the projection device; [R|t] is the extrinsic parameter of the projection device, R is the rotation matrix, t is the translation vector; M is the transformation relationship from the planar coordinates of the structured light sensing module to the projection plane of the three-dimensional world coordinates.

[0031] Optionally, the original image is subjected to an inverse transformation using the inverse of the projection transformation matrix to obtain an anti-distortion image, specifically as follows:

[0032] I ' =I°H -1 ;

[0033] Where I represents the original image frame; I′ represents the anti-distortion image frame; H represents the original image frame. -1 is the inverse of the projection transformation matrix; ° is the composite function.

[0034] Secondly, this application provides an interactive projection method based on structured light sensing and image distortion correction, including:

[0035] The structured light sensing module acquires the projection surface depth image and object image within the projection area;

[0036] The projection modeling and object recognition module constructs the relationship between the planar coordinates of the structured light sensing module and the three-dimensional world coordinates based on the projection surface depth image, and identifies the object state data based on the object image.

[0037] The image control and projection display module determines the original image based on the object state data, and transmits the original image and preset projection device parameters to the image distortion correction module;

[0038] The image distortion correction processing module performs pre-transformation processing on the original image based on the change relationship and the projection device parameters to generate an distortion correction image;

[0039] The image control and projection display module projects the anti-distortion image onto the projection area in real time.

[0040] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0041] This application provides an interactive projection system and method based on structured light sensing and image distortion correction. The system acquires depth and object images through a structured light sensing module, enabling accurate perception of the projection surface and object information. Furthermore, the image distortion correction module performs pre-transformation processing based on changes in the projection surface and projection device parameters, effectively reducing image distortion caused by hardware devices or uneven surfaces, resulting in a more precise and natural projection effect. Simultaneously, the image control and projection display module adjusts the projected image in real time based on object state data and displays the optimized distortion-corrected image through the projection device, significantly improving the interactive experience and projection effect. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A schematic diagram of an interactive projection system based on structured light sensing and image distortion correction, provided for another embodiment of this application;

[0044] Figure 2 This is a schematic diagram of an interactive projection based on structured light perception and image distortion correction, provided as an embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] In existing technologies, structured light sensing technology is widely used in the field of 3D measurement due to its high-precision depth reconstruction capabilities. However, traditional solutions typically rely on multiple sensors or complex optical systems, resulting in high costs and low integration. Meanwhile, surface material identification and modeling still face challenges, especially under complex lighting or non-uniform reflection conditions, where traditional methods struggle to simultaneously extract accurate depth information and texture features. Therefore, achieving high-precision surface modeling and dynamic projection adaptation has become a key technical challenge in this field.

[0047] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] Example 1

[0049] In one exemplary embodiment, such as Figure 1 As shown, an interactive projection system based on structured light sensing and image distortion correction is provided, comprising:

[0050] The structured light sensing module 101 is used to acquire depth images of the projection surface and object images within the projection area using the principle of structured light. Specifically, the structured light sensing module includes a structured light projector and a camera. The structured light projector emits structured light coded patterns onto the projection surface and projection area, acquiring high-precision depth information through the structured light coded patterns, and simultaneously generating complete reflection images of the projection surface and object. When the structured light projector is working, the camera captures the depth image D(x,y) of the projection surface and the object image. It possesses advantages such as high integration, low cost, and high recognition accuracy, and is widely applicable to intelligent sensing terminals.

[0051] The projection modeling and object recognition module 102 is used to construct the transformation relationship from the planar coordinates of the structured light sensing module (specifically the structured light projector) to the three-dimensional world coordinates based on the projection surface depth image D(x,y), and to recognize object state data based on the object image. Specifically, it includes:

[0052] (1) The projection surface depth image D(x,y) is subjected to coordinate transformation processing to generate a 3D point cloud P(X,Y,Z) of the projection surface in real time. A point cloud reconstruction algorithm is then used to construct the transformation relationship from planar coordinates to 3D world coordinates, specifically a mesh model or texture mapping matrix. This step is used to reconstruct the 3D shape of the projection surface. The specific method is as follows:

[0053] The projection depth image D(x,y) is transformed to obtain a 3D point cloud P(X,Y,Z). Each pixel (x,y) in the depth image D(x,y) is then converted into 3D coordinates (X,Y,Z) in the camera coordinate system using the camera intrinsic parameters of the structured light sensing module. Specifically:

[0054]

[0055] Where (X,Y,Z) are the 3D point cloud coordinates; D(x,y) is the projection plane depth image; s is the scale factor of the projection plane depth image, used to transform the depth values ​​from the coordinate system of the depth image to the actual 3D spatial coordinate system; f x f is the focal length of the camera along the x-axis. y c is the focal length of the camera along the y-axis. x The x-coordinate of the principal point in the image coordinate system, c y The y-coordinate of the principal point in the image coordinate system is the intersection of the camera optical axis and the imaging plane.

[0056] Point cloud reconstruction algorithms (such as Poisson reconstruction or Delaunay triangulation) are used to convert a 3D point cloud P(X,Y,Z) into a mesh model. The mesh model represents the topological structure of the 3D point cloud, that is, the connection relationships between points.

[0057] Furthermore, the transformation relationship from planar coordinates to 3D world coordinates can also include a texture mapping matrix. This texture mapping matrix is ​​then constructed to map the texture coordinates onto the surface of the mesh model. The texture mapping matrix contains the correspondence between texture coordinates and 3D point clouds; the texture coordinates can be automatically generated based on the structured light sensing module.

[0058] Furthermore, the parameters of the projection device in the image control and projection display module can be used for calibration to ensure the accuracy of the changing relationship, but the core parameters are still the camera intrinsic parameters and projection surface depth image data in the structured light sensing module.

[0059] (2) Object state data includes the location of key points of the object and dynamic behavior; using image recognition model to identify the object image entering the projection area, the location of key points of the object (such as the head, hands, torso, feet, and animal key points) is obtained, and based on the location of key points of the object at each moment, dynamic behavior is determined (such as combining the trajectory changes of key points of human feet or hands in consecutive frames to identify the user's foot or hand dynamic behavior, including walking direction and the moment of foot landing or gesture).

[0060] The image control and projection display module 103 is used to determine the original image based on the object state data. The original image can be a static picture or an animation, such as a picture of a museum exhibit, a welcoming video of flowers blooming, or other animations. The original image and preset projection device parameters are then transmitted to the image anti-distortion processing module 104.

[0061] The image distortion correction module 104 is used to pre-transform (i.e., correct image distortion) the original image (such as a welcoming video of blooming flowers) based on the transformation relationship and projection device parameters, generating an anti-distortion image to ensure that the projected image still presents a normal visual shape on a complex projection surface. The specific method is as follows:

[0062] (1) Based on the transformation relationship and projection device parameters (including position, attitude, and viewing angle), a projection transformation matrix H is constructed to map the coordinates of the original image onto the 3D point cloud of the projection surface:

[0063] H = K·[R|t]·M;

[0064] Where K is the intrinsic parameter matrix of the projection device, which is determined by the focal length, principal point coordinates, pixel scaling factor, etc. of the projection device; [R|t] is the extrinsic parameter of the projection device, R is the rotation matrix, which describes the rotation relationship between the projection coordinate system and the world coordinate system. It is represented by three Euler angles or quaternions and reflects the attitude of the projection device. t is the translation vector, which describes the position of the origin of the projection device coordinate system in the world coordinate system. It consists of three components and determines the specific position of the projection system in the world coordinate system; M is the transformation relationship from planar coordinates to the three-dimensional world coordinate projection surface.

[0065] (2) Using the inverse matrix H of the projection transformation matrix H -1 The original image is subjected to an inverse transform to obtain an anti-distortion image, which compensates for any distortions that may occur during the projection process.

[0066]

[0067] Where I is the original image frame; I′ is the anti-distortion image frame, i.e., the image after pre-transformation, used for projection onto the projection plane; H -1 H is the inverse of the projection transformation matrix H, used to map the coordinates of the projection plane back to the coordinate system of the original image; These are mathematical operators, representing composite functions.

[0068] This expression indicates that H should be used first. -1 Transform a coordinate point, and then use the original image frame I to obtain the pixel value at that coordinate. Specifically, this means transforming the coordinates (x′, y′) of each pixel on the projection plane through H. -1 Map back to the coordinate system of the original image I to obtain the corresponding coordinates (u,v). Then sample the pixel value at coordinates (u,v) from the original image and fill the sampled pixel value into the corresponding pixel coordinates (x′,y′) in the inverted image I′. This is called reverse mapping sampling.

[0069] H -1The projection surface contains information such as deformation, unevenness, and slope. Therefore, through this reverse mapping sampling process, the pixel values ​​in the anti-distortion image frame I′ are actually pre-calculated based on the geometry of the projection surface and parameters such as the position and orientation of the projection module. When the projection module projects the anti-distortion image onto the projection surface, because the pixel values ​​have been adjusted according to the deformation, unevenness, and slope of the projection surface, the projected content can be displayed correctly and without distortion on the projection surface.

[0070] During the above image distortion correction process, geometric pre-transformation operations, including trapezoidal correction, can also be performed simultaneously to eliminate trapezoidal distortion caused by tilted projection angles or uneven projection surfaces, so that the image can visually restore normal proportions and edge shapes.

[0071] The above anti-distortion processing can be implemented in real time using GPU shaders to support video-level frame rate rendering.

[0072] The image control and projection display module 103 is also used to project the anti-distortion image onto the projection area in real time through a high-brightness micro-projection module (such as DLP, LCD, etc.) to achieve visual restoration.

[0073] The main control and rendering platform of the interactive projection system uses an embedded computing platform (such as the NVIDIA Jetson Orin series) to perform image modeling, recognition and GPU image rendering tasks, so as to achieve low latency and high frame rate projection interaction.

[0074] This application proposes an interactive projection system based on structured light perception and image distortion correction. It acquires high-precision depth information through structured light-encoded patterns and combines this with infrared diffused bottom light to generate a complete ground reflection image, thereby achieving simultaneous acquisition of structural and texture data. This system boasts advantages such as high integration, low cost, and high recognition accuracy, and can be widely applied to intelligent sensing terminals. Furthermore, by integrating structured light 3D perception and real-time distortion correction technology, this application effectively solves the image distortion problem caused by uneven surfaces. It also supports real-time interactive control based on user actions, enabling intelligent projection effects such as "gesture-activated product demonstrations" and "flowers blooming underfoot," significantly enhancing the interactive experience and visual expressiveness.

[0075] In one exemplary embodiment, such as Figure 2 As shown, an interactive projection method based on structured light perception and image distortion correction is provided, including the following steps 201 to 205. Wherein:

[0076] Step 201: The structured light sensing module acquires the projection surface depth image and object image within the projection area.

[0077] Step 202: The projection modeling and object recognition module constructs the relationship between the planar coordinates of the structured light perception module and the three-dimensional world coordinates based on the projection surface depth image, and identifies the object state data based on the object image.

[0078] Step 203: The image control and projection display module determines the original image based on the object state data, and transmits the original image and preset projection device parameters to the image anti-distortion processing module.

[0079] Step 204: The image anti-distortion processing module performs pre-transformation processing on the original image based on the change relationship and projection device parameters to generate an anti-distortion image.

[0080] Step 205: The image control and projection display module projects the anti-distortion image onto the projection area in real time.

[0081] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0082] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An interactive projection system based on structured light sensing and image distortion correction, characterized in that, include: The structured light sensing module is used to acquire depth images of the projection surface and object images within the projection area; The projection modeling and object recognition module is used to construct the transformation relationship from the planar coordinates of the structured light sensing module to the three-dimensional world coordinates based on the projection surface depth image, and to recognize the object state data based on the object image. The image control and projection display module is used to determine the original image based on the object state data, and transmit the original image and preset projection device parameters to the image anti-distortion processing module; The image anti-distortion processing module is used to perform pre-transformation processing on the original image based on the change relationship and the projection device parameters to generate an anti-distortion image; The image control and projection display module is also used to project the anti-distortion image onto the projection area in real time.

2. The interactive projection system based on structured light sensing and image distortion correction according to claim 1, characterized in that, The structured light sensing module includes a structured light projector and a camera; the structured light projector is used to emit structured light encoded patterns into the projection area; the camera is used to capture projection surface depth images and object images within the projection area when the structured light projector is working.

3. The interactive projection system based on structured light perception and image distortion correction according to claim 1, characterized in that, The transformation relationship includes a mesh model. Specifically, in constructing the transformation relationship from the planar coordinates of the structured light perception module to the three-dimensional world coordinates based on the projection surface depth image, the projection modeling and object recognition module includes: The projection plane depth image is subjected to coordinate transformation to obtain a three-dimensional point cloud; The three-dimensional point cloud is converted into a mesh model using a point cloud reconstruction algorithm.

4. The interactive projection system based on structured light perception and image distortion correction according to claim 3, characterized in that, The transformation relationship also includes a texture mapping matrix. After converting the 3D point cloud into a mesh model using the point cloud reconstruction algorithm, the projection modeling and object recognition module further includes: Construct a texture mapping matrix; the texture mapping matrix represents the correspondence between texture coordinates and 3D point clouds; The texture mapping matrix is ​​then mapped onto the surface of the mesh model.

5. The interactive projection system based on structured light perception and image distortion correction according to claim 3, characterized in that, In the process of performing coordinate transformation on the depth image of the projection surface to obtain a 3D point cloud, the projection modeling and object recognition module specifically includes: Where (X,Y,Z) are the 3D point cloud coordinates; D(x,y) is the projection plane depth image; s is the scale factor of the projection plane depth image; f x f is the focal length of the camera along the x-axis. y c is the focal length of the camera along the y-axis. x The x-coordinate of the principal point in the image coordinate system, c y The y-coordinate of the principal point in the image coordinate system.

6. The interactive projection system based on structured light sensing and image distortion correction according to claim 1, characterized in that, The object state data includes the location of key points of the object and its dynamic behavior; In terms of recognizing object state data based on the object image, the projection modeling and object recognition module specifically includes: The image of the object is processed using an image recognition model to obtain the positions of the object's key points; The dynamic behavior is determined based on the position of the object's key points at each moment.

7. The interactive projection system based on structured light sensing and image distortion correction according to claim 1, characterized in that, In the process of pre-transforming the original image based on the change relationship and the projection device parameters to generate an anti-distortion image, the image anti-distortion processing module specifically includes: Based on the aforementioned transformation relationship and the parameters of the projection device, a projection transformation matrix is ​​constructed; The original image is inversely transformed using the inverse of the projection transformation matrix to obtain an anti-distortion image.

8. The interactive projection system based on structured light sensing and image distortion correction according to claim 7, characterized in that, In terms of constructing the projection transformation matrix based on the aforementioned change relationship and the projection device parameters, the image distortion correction processing module specifically comprises: H = K·[R|t]·M; Where H is the projection transformation matrix; K is the intrinsic parameter matrix of the projection device; [R|t] is the extrinsic parameter of the projection device, R is the rotation matrix, t is the translation vector; M is the transformation relationship from the planar coordinates of the structured light sensing module to the projection plane of the three-dimensional world coordinates.

9. The interactive projection system based on structured light perception and image distortion correction according to claim 7, characterized in that, The original image is inversely transformed using the inverse of the projection transformation matrix to obtain an anti-distortion image, specifically as follows: Where I represents the original image frame; I′ represents the anti-distortion image frame; H represents the original image frame. -1 It is the inverse of the projection transformation matrix; It is a composite function.

10. An interactive projection method based on structured light perception and image distortion correction, characterized in that, include: The structured light sensing module acquires the projection surface depth image and object image within the projection area; The projection modeling and object recognition module constructs the relationship between the planar coordinates of the structured light sensing module and the three-dimensional world coordinates based on the projection surface depth image, and identifies the object state data based on the object image. The image control and projection display module determines the original image based on the object state data, and transmits the original image and preset projection device parameters to the image distortion correction module; The image distortion correction processing module performs pre-transformation processing on the original image based on the change relationship and the projection device parameters to generate an distortion correction image; The image control and projection display module projects the anti-distortion image onto the projection area in real time.

Images