A three-dimensional projection mapping method and device

Abstract

The present invention discloses a three-dimensional projection mapping method and apparatus, specifically relating to the technical field of three-dimensional projection mapping; three-dimensional data on the surface of an object is obtained in real time through multiple cameras and sensors, and the geometric model and texture information on the surface of the object are dynamically updated in combination with deep learning algorithms. By adapting to changes in real-time geometric data, lighting conditions, and viewer positions, an adaptive algorithm is used to adjust the angle, focal length, and light intensity of the projector, and perspective transformation and affine transformation are combined to perform adaptive image deformation on the virtual image, optimizing the geometric shape and texture mapping of the virtual image in real time. In addition, by feeding back changes in viewer positions and ambient lighting in real time through sensors, the system can dynamically adjust the projection content to ensure that the virtual image is always accurately aligned with the surface of the object, avoiding misalignment and distortion, and significantly improving the stability and visual quality of the interactive display effect.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional projection mapping, and particularly relates to a three-dimensional projection mapping method and device. Background Art

[0002] With the continuous development of graphics processing technology, three-dimensional projection mapping has been widely used in fields such as computer vision, augmented reality (AR), and virtual reality (VR). With the support of a GPU (graphics processing unit), three-dimensional projection mapping can project virtual images onto complex surfaces in the physical world, creating an immersive visual effect.

[0003] The prior art has the following deficiencies:

[0004] In the interactive display of museums, three-dimensional projection mapping technology is used to project virtual images onto the surfaces of complex and irregular objects, such as the display of dinosaur models. However, the current technology often faces serious accuracy problems when dealing with dynamic and irregular surfaces. Especially when there are changes in the ambient light, the position of the audience, or the adjustment of the camera angle, the projection often fails to accurately fit the surface of the object, resulting in misalignment or distortion of the virtual image. Summary of the Invention

[0005] The purpose of the present invention is to provide a three-dimensional projection mapping method and device to solve the deficiencies in the background art.

[0006] To achieve the above purpose, the present invention provides the following technical solution: A three-dimensional projection mapping method, comprising:

[0007] Using multiple cameras and sensors to obtain real-time three-dimensional data of the object surface, and dynamically updating the geometric model and texture information of the object surface in combination with a deep learning algorithm;

[0008] According to the real-time geometric data of the object surface, the lighting conditions, and the position changes of the audience, an adaptive algorithm is used to adjust the projection angle, focal length, and lighting intensity of the projector;

[0009] Performing adaptive image deformation on the virtual image, combining perspective transformation and affine transformation, projecting the virtual image onto the complex three-dimensional surface of the object, and according to the dynamic changes of the object surface, adjusting the geometric shape and texture mapping of the virtual image in real time;

[0010] Adjusting the projection content and optimizing the projection effect by real-time sensor feedback of the dynamic position changes of the audience and the lighting changes in the environment.

[0011] Preferably, through multiple high-resolution cameras, LiDAR, and structured light sensors, multi-view images and depth data of the object surface are captured in real time, including three-dimensional coordinates, normal vectors, and texture information of the object surface;

[0012] Using image stitching technology, the camera data from different perspectives are fused to form a unified three-dimensional dataset. The three-dimensional coordinate information of each pixel point on the object surface is fused through the depth map generated by the depth sensor to generate an accurate three-dimensional model;

[0013] Adopting convolutional neural networks and generative adversarial networks, geometric reconstruction and texture mapping are performed on the object surface. The CNN extracts the structural features in the image and dynamically generates a high-resolution geometric model of the object, and the GAN is used for texture synthesis.

[0014] Preferably, the geometric data of the object surface are obtained in real time, including curvature, size, and normal vector information of the object surface, and the ambient light intensity is monitored in real time through an ambient light sensor;

[0015] Using an infrared sensor or a motion capture device, the position and perspective of the audience are tracked in real time to obtain the relative distance and perspective change between the audience and the object;

[0016] Through the inverse projection algorithm, according to the geometric information of the object surface and the position of the audience, the optimal projection angle of the projector is calculated.

[0017] Preferably, the position of the audience is tracked in real time using a sensor, and the relative position of the audience is determined by calculating the three-dimensional coordinate difference between the audience and the object. The relative position of the audience is obtained through the distance calculation formula: ; where D is the viewing distance between the audience and the object, 、 、 are the three-dimensional coordinates of the audience, 、 、 are the coordinates of the corresponding points on the object surface. The viewing distance between the obtained audience and the object is compared with the preset optimal viewing distance between the audience and the object, and the absolute value of their difference is calculated and defined as the optimal viewing distance deviation.

[0018] Preferably, an ambient light sensor is used to monitor the ambient light intensity in real time to obtain the brightness information of the ambient light;

[0019] According to the intensity of the ambient light, the light adaptation parameter is calculated: ; where is the current ambient light intensity, is the standard light intensity;

[0020] Obtain the light adaptation parameters calculated within a fixed time period T , and establish a corresponding data set, calculate the standard deviation of the data set, and define it as the light adaptation fluctuation value.

[0021] Preferably, use the obtained optimal viewing distance deviation and light adaptation fluctuation value as the input items of fuzzy logic, and use the adjustment values of focal length and image sharpness as the output items of fuzzy logic;

[0022] Fuzzify the optimal viewing distance deviation and light adaptation fluctuation value;

[0023] Based on fuzzy logic reasoning, define fuzzy rules to associate input items and output items;

[0024] During the fuzzy logic reasoning process, find the corresponding fuzzy output set by combining the membership degrees of the input items;

[0025] Defuzzify the result of fuzzy reasoning to obtain specific output values;

[0026] Adjust the focal length of the projector according to the defuzzified focal length adjustment value;

[0027] Adjust the contrast and sharpness of the image according to the defuzzified sharpness adjustment value.

[0028] Preferably, use multiple cameras and depth sensors to obtain the three-dimensional data of the object surface in real time; combine perspective transformation and affine transformation to map the virtual image from a two-dimensional plane to the three-dimensional space of the object surface. Perspective transformation is used to process the area far from the projector; affine transformation is used to adjust the image of the planar area of the object surface; according to the dynamic changes of the object surface, adjust the geometric shape and texture mapping of the virtual image in real time, and adjust the shape of the image through normal vector and curvature information.

[0029] The present invention also provides a three-dimensional projection mapping device, including a data processing module, a virtual image deformation and mapping module, a projection adjustment module, and a projection adjustment module;

[0030] Data processing module: Use multiple cameras and sensors to obtain the three-dimensional data of the object surface in real time, and combine deep learning algorithms to dynamically update the geometric model and texture information of the object surface;

[0031] Projection parameter adjustment module: According to the real-time geometric data of the object surface, lighting conditions, and the position changes of the audience, use an adaptive algorithm to adjust the projection angle, focal length, and lighting intensity of the projector;

[0032] Virtual Image Deformation and Mapping Module: It performs adaptive image deformation on the virtual image, combines perspective transformation and affine transformation, projects the virtual image onto the complex three-dimensional surface of the object, and adjusts the geometric shape and texture mapping of the virtual image in real time according to the dynamic changes of the object surface;

[0033] Projection Adjustment Module: It adjusts the projection content and optimizes the projection effect by real-time sensor feedback of the dynamic position changes of the audience and the lighting changes of the environment.

[0034] In the above technical solutions, the technical effects and advantages provided by the present invention are as follows:

[0035] 1. The present invention can accurately project a virtual image onto the complex and irregular surface of an object in an interactive display environment such as a museum. By jointly using multiple cameras and sensors and combining deep learning algorithms for real-time three-dimensional reconstruction and texture mapping of the object surface, the system can dynamically update the geometric model and texture information of the object. At the same time, the system automatically adjusts the angle, focal length, and light intensity of the projector according to the changes in the position and viewing angle of the audience through the reverse projection algorithm, ensuring that the virtual image can be accurately aligned with the object surface under any ambient light change and avoiding the misalignment and distortion problems in traditional projection technologies.

[0036] 2. The present invention adopts fuzzy logic inference technology, takes the deviation of the optimal viewing distance between the audience and the object and the fluctuation value of the ambient light as input items, and realizes adaptive optimization of the projection content by dynamically adjusting the focal length and image clarity. Through perspective transformation and affine transformation, the virtual image can perform adaptive image deformation in real time according to the dynamic changes of the object surface, ensuring that the image is projected onto the three-dimensional surface of the object without distortion. This method has significant advantages in dealing with the complex geometric shape of the object surface, lighting changes, and dynamic interaction of the audience, providing a more immersive and stable visual experience. Especially in the interactive display of a museum, it improves the display effect of the virtual image and the sense of participation of the audience. Description of the Drawings

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings.

[0038] Figure 1 It is the method mind map of the present invention.

[0039] Figure 2 It is the device module mind map of the present invention. Detailed Embodiments

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0041] Example 1. Please refer to Figure 1 As shown, a three-dimensional projection mapping method in this embodiment includes:

[0042] Using multiple cameras and sensors to obtain the three-dimensional data of the object surface in real time, and dynamically updating the geometric model and texture information of the object surface in combination with deep learning algorithms;

[0043] According to the real-time geometric data of the object surface, lighting conditions, and the position changes of the audience, an adaptive algorithm is used to adjust the projection angle, focal length, and lighting intensity of the projector;

[0044] Performing adaptive image deformation on the virtual image, combining perspective transformation and affine transformation, projecting the virtual image onto the complex three-dimensional surface of the object, and adjusting the geometric shape and texture mapping of the virtual image in real time according to the dynamic changes of the object surface;

[0045] Adjusting the projection content and optimizing the projection effect by real-time sensor feedback of the dynamic position changes of the audience and the lighting changes of the environment.

[0046] In the application of three-dimensional projection mapping, obtaining the three-dimensional data of the object surface in real time is a crucial step. Through the combined use of multiple cameras and sensors, the system can achieve high-precision, real-time three-dimensional reconstruction, and dynamically update the geometric model and texture information of the object through deep learning algorithms.

[0047] To capture the three-dimensional information of the object, the system uses multiple high-resolution cameras for multi-view shooting. Usually, the cameras are set at multiple positions around the object to ensure that complete object image data is collected from different angles. These cameras are triggered synchronously for shooting to capture different surface information of the object.

[0048] In addition to cameras, the system also uses depth sensors (such as LiDAR or structured light sensors) to capture the depth data of the object surface. These sensors can emit lasers or project structured light, and calculate the distance of each pixel point by receiving the reflected signal, thereby generating a depth map of the object surface (i.e., the three-dimensional coordinates of each pixel).

[0049] Data from different cameras and sensors are transmitted to the central computing unit in real time. Using image stitching and depth map fusion techniques, this data from different perspectives and sensors is merged into a unified three-dimensional space dataset. In this way, the system can reconstruct a complete three-dimensional model of the object in real time.

[0050] After obtaining the real-time three-dimensional data, deep learning algorithms are used to dynamically update the geometric model and texture information of the object.

[0051] The system uses a convolutional neural network (CNN) to process the depth maps generated in real time. The CNN can extract the structural features of the object surface from the original depth map data and gradually generate a high-resolution geometric model of the object through the network layers. Through training, the network can learn how to process depth maps with different lighting, shapes, and surface features and accurately reconstruct the three-dimensional shape of the object surface.

[0052] In some complex applications, a three-dimensional convolutional neural network (3D CNN) is used to further process and generate three-dimensional models. These networks can process voxel data and directly convert depth information into a complete three-dimensional volume model. The 3D CNN processes each voxel through multiple layers of convolution, generates a three-dimensional model that conforms to the true shape of the object, and can accurately fit the geometric shape of each surface area.

[0053] The texture information of the object (such as surface color, pattern, etc.) is provided by the image data captured by the camera. Deep learning algorithms, especially methods based on generative adversarial networks (GANs) or autoencoders, can learn the texture features of the object surface from images taken from multiple perspectives.

[0054] GANs can be used for texture synthesis. Through the adversarial training of the generator and discriminator, the generator can generate texture images based on the input object geometric model, while the discriminator evaluates the authenticity of the texture images. In this way, GANs can effectively complete the texture information and handle texture distortions caused by lighting changes or different perspectives.

[0055] A neural network is used to match the geometric surface of the object with the image texture. By using deep learning algorithms for image segmentation and texture classification, the neural network can identify the texture features of different regions on the object surface and automatically map the appropriate texture according to the geometric features of the object surface.

[0056] Through the incremental learning method, the system can gradually optimize the geometric model and texture mapping of the object each time new perspective data is obtained. The deep learning model uses adaptive optimization algorithms, such as AdaGrad or Adam optimizer, to dynamically adjust the weights of the network, so that each new piece of data obtained can effectively improve the accuracy of the model.

[0057] To simultaneously handle geometric reconstruction and texture mapping, the system adopts multi-task learning. By training the geometric model and texture mapping tasks simultaneously within a unified neural network framework, the deep learning algorithm can better collaborate to process the three-dimensional shape and visual features of the object surface, thereby improving the update speed and accuracy.

[0058] In this application, by jointly using multiple cameras and sensors, the system can efficiently collect three-dimensional data of the object surface in real time, and combine deep learning algorithms to dynamically update the geometric model and texture information. Deep learning algorithms, such as CNN, 3D CNN, GAN, etc., play a key role in data processing and texture synthesis. The combination of incremental learning and multi-task learning enables the system to continuously optimize the three-dimensional model and texture of the object as data accumulates, providing accurate and stable three-dimensional projection mapping effects.

[0059] To achieve high-precision three-dimensional projection mapping, especially in a dynamic environment, it is necessary to adjust the parameters of the projector (such as projection angle, focal length, and light intensity) in real time.

[0060] Through a multi-camera system and depth sensors (such as LiDAR, structured light sensors, etc.), the system can obtain the three-dimensional geometric data of the object surface in real time. These data include the shape, size, texture information of the object, and any changes in the surface shape caused by dynamic changes. For example, the system can capture the details of uneven or curved areas on the object surface, which are very important for accurate three-dimensional projection.

[0061] Through the ambient light sensor, the system can monitor the lighting conditions in the surrounding environment in real time. This includes the intensity, direction, and changes of natural light, artificial lighting, and other sources. This information helps the algorithm determine how to adjust the lighting intensity of the projection to avoid distortion or excessive or insufficient brightness of the projected image.

[0062] Use sensors (such as infrared sensors, cameras, or motion capture devices) to perform real-time tracking of the audience's position and perspective. The system can obtain the specific position of the audience, determine the relative position, distance, and perspective between the audience and the object. This is crucial for dynamically adjusting the projection perspective and projection content, especially in a multi-person interaction or exhibition environment.

[0063] Adjust the projection angle according to the real-time position and perspective of the audience to ensure that the virtual image is always accurately aligned with the object surface, avoiding misalignment and distortion. The specific method is to calculate the optimal projection angle of the projector through the inverse projection algorithm.

[0064] First, a three-dimensional geometric model of the object needs to be obtained, including the surface shape, curvature, and any irregular geometric features of the object. This geometric model is usually obtained in real time by technologies such as multi-view image processing, laser scanning (LiDAR), or depth sensors (such as structured light, ToF sensors). The surface data of the object includes not only the position of each point but also the normal vector information of the surface (i.e., the orientation of each point).

[0065] The position and viewing angle of the audience are tracked in real time through sensors (such as infrared sensors, cameras, motion capture devices, etc.). The position of the audience is usually represented by a three-dimensional coordinate, and the viewing angle is determined by the relative angle between the audience and the object. This information is crucial because it determines from which angle to project the image so that the image can be accurately mapped onto the object surface.

[0066] On the premise that the geometric model of the object surface and the audience position are known, a reverse projection algorithm is used to calculate the optimal projection angle of the projector relative to the object. The goal of reverse projection is to calculate where the projector should be located and where it should point according to the audience's viewing angle and the geometric shape of the object surface, so as to ensure that the image is accurately mapped onto the object surface without distortion.

[0067] Assume that a point P(x, y, z) on the object surface and the position of the projector are known. The geometric relationship between the projector and the object surface can be represented by vector relationships in a three-dimensional coordinate system. The normal vector n(x, y, z) of the object surface represents the orientation of that point, and the viewing angle of the audience determines the incident angle of the projection.

[0068] When calculating the audience's viewing angle, first, the position relationship between the audience and each point on the object surface needs to be calculated through three-dimensional space. Assume the position of the audience is V(xv, yv, zv). The relationship between the audience's viewing angle and the object surface can be obtained by calculating the angle between the audience's line of sight and the normal vector of the object surface. This angle is the viewing angle deviation, which determines how the virtual image should be presented on the object surface.

[0069] The reverse projection algorithm calculates the optimal position and angle of the projector by reverse calculation. That is, assume how the image of a certain part of the object surface seen by the audience should be reverse-mapped from the position of the projector, and determine where the projector should be located and how its angle should be adjusted so that the light emitted from the projector is accurately projected onto the surface.

[0070] Through an optimization algorithm (such as the least squares method or gradient-based optimization), the reverse projection algorithm calculates the optimal projection angle of the projector relative to the object. This step usually involves complex matrix operations and spatial transformations. Considering the curvature of the object surface, the distance between the projector and the object surface, and the change in the audience's position, the projection angle of the projector is gradually adjusted so that the projected image accurately fits the object surface.

[0071] The projection angle calculated through the above steps ensures the precise alignment of the virtual image with the object surface. In areas where the object surface is relatively flat, the projected image has little distortion. However, when the object surface has a large curvature or irregularity, the inverse projection algorithm eliminates the misalignment and distortion problems that occur in traditional projection methods through precise adjustment of the projection angle. Especially on complex surfaces, the algorithm can dynamically adapt to the different geometric relationships of each projection point, ensuring a high degree of accuracy of the projected image.

[0072] In this application, by using the inverse projection algorithm, the system can accurately calculate the optimal projection angle of the projector relative to the object surface, ensuring that the virtual image accurately aligns with the object surface in any dynamically changing environment. This method not only effectively avoids projection distortion but also enables high-precision projection mapping under complex geometric forms, providing a more immersive visual experience.

[0073] According to the distance between the projector and the object surface and the curvature of the object surface, the focal length is dynamically adjusted to ensure that the projected image is always clear and avoid defocus or blurring. The dynamic adjustment of the focal length can ensure that the projected image remains clear and accurate in areas with different curvatures or distances on the object surface.

[0074] The system measures the distance between the projector and the object surface in real time through depth sensors (such as LiDAR, structured light sensors, ToF sensors, etc.). These sensors can provide high-precision distance data, allowing the system to obtain the distance information between each point on the object surface and the projector.

[0075] Structured light sensor: Calculates depth information by projecting structured light onto the object and analyzing the deformation of the light.

[0076] LiDAR sensor: Accurately measures the distance by emitting and receiving reflected laser signals, providing high-resolution three-dimensional surface data.

[0077] ToF sensor: Measures the time it takes for light to travel from the sensor to the object and back using the time-of-flight principle to calculate the precise distance.

[0078] The system generates a depth map based on the data from the sensors. Each pixel value in the depth map represents the distance between the projector and the corresponding point on the object surface. This depth map provides the basic data for subsequent focal length calculation.

[0079] The geometric model of the object surface (including the three-dimensional coordinates and normal vectors of each point) can provide detailed information about the curvature of the object. The normal vector of each point on the object surface can be estimated by calculating the geometric relationships of neighboring points. For example, using local surface fitting algorithms (such as principal component analysis or least squares fitting), the curvature information of each point can be obtained.

[0080] Flat area: The curvature is close to zero.

[0081] Convex area: The curvature value is positive.

[0082] Concave area: The curvature value is negative.

[0083] This curvature information can help the system judge the degree of deformation of the projected image to determine the amount of focal length adjustment.

[0084] In the convex or concave area of the object, the deformation of the projected image is more obvious. The focal length needs to be dynamically adjusted according to the change of curvature to avoid image distortion. In the flat area, the change of the focal length is relatively small.

[0085] Based on the real-time distance data between the projector and each point on the object surface and the change of the object surface curvature, the system calculates the ideal focal length of each projection point. Suppose the focal length of the projector is f, the distance of a certain point on the object surface is d, and the surface curvature of the object is κ, the ideal focal length can be adjusted by the following formula: ; where is the initial focal length, and α is an adjustment coefficient used to control the degree of change of the focal length with the change of the object surface curvature.

[0086] The focal length of the projector is controlled by its internal lens system. According to the calculated ideal focal length , the system automatically adjusts the focal length of the projector by dynamically controlling the autofocus mechanism of the projector (for example, by adjusting the movement of the lens or changing the width of the projection beam).

[0087] Flat area: In the relatively flat area of the object surface, the change of the focal length is small, and the system will maintain a relatively constant focal length.

[0088] Curved surface area: In the convex or concave area of the object, the focal length will be dynamically adjusted according to the change of the surface curvature to ensure the clarity of the image.

[0089] The system continuously monitors the change of the object surface through real-time feedback sensor data (such as depth sensors and optical sensors). When the position of the audience changes, the focal length will be adjusted immediately. For example, if the audience approaches the object and the distance decreases, the system will automatically increase the focal length; if the audience moves away, the system will decrease the focal length.

[0090] By calculating the focal length and curvature in real time, the system can effectively prevent the focal length from being too long or too short, thus avoiding image blurring or defocusing. In each object surface area, the system can maintain the best visual effect by adaptively adjusting the focal length.

[0091] In this application, by measuring the distance between the projector and the object surface in real time, analyzing the curvature changes of the object surface, and combining with the adaptive focal length adjustment mechanism, the system can dynamically adjust the focal length of the projector to avoid defocus or blurring. This process enables the virtual image to remain clear and of high quality in all regions of the object surface, whether flat, convex, or concave, ensuring the accurate presentation of the projected image.

[0092] According to the changes in ambient light and the reflection characteristics of the object surface, dynamically adjust the light intensity of the projector to ensure that the brightness of the projected image is appropriate, not affected by ambient light, avoid the projected image being too dark or too bright, and maintain the clarity and contrast of the image.

[0093] The system installs multiple light sensors (such as ambient light sensors, infrared sensors, or photoresistors) to monitor the light conditions in the surrounding environment in real time. These sensors can provide information on the intensity and direction of ambient light, helping the system understand the current changes in ambient light. For example, the sensors can detect changes in natural light (such as sunlight intensity, shadow changes) and changes in artificial light sources (such as the brightness and position of indoor lights).

[0094] After the ambient light data is collected by the sensors, it is transmitted to the computing unit in real time. The system analyzes the light intensity, direction, and distribution of light sources through algorithms to obtain the characteristics of ambient light in real time. For example, the system can judge the brightness of the current environment based on the light intensity (such as lux value) and then determine the adjustment strategy for the projection light source.

[0095] The surface texture information of the object (such as material type, smoothness, roughness, etc.) plays an important role in the intensity and manner of reflected light. Through depth sensors, cameras, or 3D scanning technologies, the system can capture the texture and reflection properties of the object surface in real time. These texture information can include characteristics such as color, gloss, and transparency. The system combines this information with the ambient light data to calculate the optimal light intensity.

[0096] The reflection characteristics of an object are usually described by a light reflection model. Commonly used models include the Phong reflection model and the Blinn-Phong reflection model. These models can simulate how the object surface reflects incident light and calculate the diffuse reflection and specular reflection parts of the object surface respectively.

[0097] Diffuse reflection: Describes the characteristic of light being evenly reflected by rough areas of the object surface.

[0098] Specular reflection: Describes the characteristic of light being reflected by smooth areas of the object surface and forming a highlighted area.

[0099] Through the reflection model, the system can combine the ambient light intensity, calculate the reflected light intensity on the object surface, and based on this, calculate the required illumination intensity of the projector. For example, if the object surface is smooth and has a high reflectivity (such as a mirror or metal surface), the system will reduce the projection illumination intensity; conversely, if the object surface is rough (such as a matte material), the system may need to increase the illumination intensity.

[0100] Calculate the optimal projection illumination intensity based on the ambient light sensing data and the reflection characteristics of the object surface : ; where is the ambient light intensity (measured in real time by the sensor), is the original illumination intensity of the projection light source, is the diffuse reflection coefficient of the object surface (obtained by calculating the object texture), is the specular reflection coefficient of the object surface, and e is an adjustment parameter that controls the influence weights of diffuse reflection and specular reflection. Through the above calculations, the system can dynamically adjust the intensity of the projection illumination, making the brightness of the projection image match the ambient light and ensuring the clarity and contrast of the image.

[0101] Based on the calculated illumination intensity, the system adjusts the projection brightness by controlling the output power of the projector's light source (such as an LED or laser source). If the ambient light is strong, the system will reduce the intensity of the projection light source; conversely, in a darker environment, the system will increase the brightness of the projection light source.

[0102] In this application, by using a light sensing sensor in real time and analyzing the reflection characteristics of the object surface, the system can dynamically calculate and adjust the illumination intensity of the projector. This adaptive illumination adjustment mechanism ensures that under different illumination conditions, the brightness of the projection image is always moderate and not affected by changes in ambient light, avoiding the phenomenon of being too bright or too dark and ensuring the best visual effect of the projection image.

[0103] Precisely project the virtual image onto the complex three-dimensional surface of the object, and in real time adjust the geometric shape and texture mapping of the virtual image according to the dynamic changes of the object surface. This process involves perspective transformation, affine transformation, and adaptive image adjustment techniques to ensure that the virtual image is precisely aligned with the object surface without distortion.

[0104] Use a multi-camera system, laser scanning (LiDAR), or depth sensor to obtain the three-dimensional data of the object surface. These devices capture the three-dimensional coordinates and texture information of each point on the object surface and generate a detailed geometric model. This model includes information such as the normal vector and curvature of the object surface, serving as the basic data for subsequent transformations.

[0105] Update the geometric information of the object surface using real-time sensors to ensure capturing the impact of dynamic changes of the object (such as minor movements or deformations of the object) on the projection effect. This data includes the shape, texture features and their changes of different regions on the object surface.

[0106] Based on the three-dimensional coordinates of the object surface, calculate the normal vector of each point. The normal vector describes the orientation of the object surface and is used to calculate the incident angle of the projection light during the projection transformation process.

[0107] Conduct curvature analysis on the object surface to determine the flat, convex or concave regions of the object. By calculating the neighborhood curvature of the surface points, the system can identify the regions with complex surface details (such as uneven regions) and adjust the deformation of the virtual image according to the geometric characteristics of these regions.

[0108] For each point P(x, y, z) on the object surface, calculate the projection position P'(x', y') of this point on the projection plane through the principle of perspective projection. The perspective transformation takes into account the distance relationship between the object surface and the projector, making the projection image appear smaller in the area where the object is far from the projector; while in the area close to the projector, the projection image appears larger, simulating the perspective effect of the real world.

[0109] Set the projection plane at Z = 0, and the projection point P'(x', y') of the object surface point P(x, y, z) can be calculated by the formula: ; where f is the focal length of the projector and z is the distance from the object surface point to the projector.

[0110] Affine transformation is used to process the planar regions on the object surface to ensure that the image maintains the characteristics of straight lines and parallel lines. It maps each point on the object surface from a two-dimensional space to another two-dimensional space through matrix operations. The affine transformation formula is: ; where a, b, c, d are the elements of the transformation matrix, controlling transformations such as rotation, scaling, and translation, and g, f are the translation vectors. The affine transformation ensures that the geometric shape of the image maintains a linear change.

[0111] Calculate the texture coordinates for each object surface point, and these coordinates correspond to a certain position in the virtual image. Through the geometric model of the object surface, map the points in the three-dimensional space to the corresponding positions on the two-dimensional texture image. This process usually uses UV mapping technology.

[0112] By comparing the calculated texture coordinates with the actual texture image, the system can correctly map the texture information of the virtual image to the object surface. In the regions with a large curvature on the object surface, the system will make adaptive adjustments to the texture according to the change of the normal vector to ensure that the texture does not stretch or distort in these regions.

[0113] In regions with large curvature changes, texture mapping adjusts according to normal vector and curvature information to adapt to the changes in the object surface and prevent texture stretching or compression.

[0114] By feeding back the perspective changes of the audience and the dynamic changes of the object surface through real-time sensors (such as cameras or depth sensors), the system will obtain new geometric data. This data is used to update the geometric model of the object surface (such as shape changes, displacements, etc.), thereby affecting the projection of the virtual image.

[0115] The dynamic changes of the object surface (such as deformation or movement) require the geometric shape of the virtual image to be updated in real time. The system uses the aforementioned perspective transformation and affine transformation, combined with the new object geometric information, to recalculate the position of each projection point and adaptively deform the virtual image.

[0116] The dynamic changes of the object surface texture (such as changes in factors like lighting and reflection) will affect the projection effect of the texture. The system adjusts the texture of the virtual image in real time by obtaining new texture data in real time (such as reflections and color changes under lighting conditions). Through adaptive texture mapping, the system can ensure that each area of the object surface always shows the most appropriate visual effect.

[0117] In this application, through technical means such as perspective transformation, affine transformation, and texture mapping, the system can accurately project the virtual image onto the complex three-dimensional surface of the object and adjust the geometric shape and texture mapping of the virtual image in real time according to the dynamic changes of the object surface. This process fully considers factors such as the geometric characteristics, normal vector, and lighting conditions of the object surface, and through adaptive image deformation technology, ensures that the virtual image is mapped onto the object surface without distortion or distortion, providing an immersive visual experience.

[0118] Use sensors (such as depth sensors, infrared sensors, motion capture devices, etc.) to track the position of the audience in real time. The system determines the relative position of the audience by calculating the three-dimensional coordinate differences between the audience and the object. The relative position of the audience can be obtained through the distance calculation formula: ; where D is the viewing distance between the audience and the object, , , are the three-dimensional coordinates of the audience, , , are the coordinates of the corresponding points on the object surface. Compare the obtained viewing distance between the audience and the object with the preset optimal viewing distance between the audience and the object, and calculate the absolute value of their difference, which is defined as the optimal viewing distance deviation. The optimal viewing distance represents the ideal distance between the audience and the object under the best viewing conditions.

[0119] If the deviation of the optimal viewing distance is greater than a predetermined deviation threshold, the system will optimize the projection effect by adjusting the position, angle, focal length, or other projection parameters of the projector. The system may automatically adjust the position or angle of the projector to make the viewing distance between the audience and the object close to the optimal viewing distance.

[0120] For example: If the audience is too far from the object, the system may prompt or automatically adjust the size and position of the projected content, or adjust the optical parameters (such as the focal length) of the projection device to achieve a better visual effect. If the audience is too close to the object, the system may reduce the size of the projected image to avoid excessive perspective distortion.

[0121] If the deviation of the optimal viewing distance is less than or equal to the predetermined deviation threshold, the system considers that the current projection effect already meets the requirements of the optimal viewing distance, so there is no need to adjust the projected content. At this time, the system can maintain the current projection settings and continue to provide a stable visual effect.

[0122] Use a light sensor to monitor the ambient light intensity in real time and obtain the brightness information of the ambient light (such as lux value). The sensor can detect changes in the surrounding light sources (natural light, indoor lights, etc.) and reflect the intensity of the light.

[0123] Based on the intensity of the ambient light, the system can calculate the light adaptation parameter : ; where is the current ambient light intensity, is the standard light intensity (usually a preset value representing the light intensity under ideal projection conditions).

[0124] Obtain the light adaptation parameters calculated within a fixed time period T , for example, if the system obtains data once per second and the time period T is 10 seconds, then 10 light adaptation parameter values will be obtained. And establish a corresponding data set, calculate the standard deviation of the data set, and define it as the light adaptation fluctuation value.

[0125] If the light adaptation fluctuation value is greater than the set standard threshold, it means that the light fluctuates greatly and the system needs to adjust the projected content. For example: Increase the projection brightness: If the ambient light is too dark, the system will increase the projection brightness. Adjust the contrast: In a strong light environment, the system will enhance the contrast of the image to avoid loss of image details. Adjust the color and brightness of the projected content: Dynamically adjust the hue and brightness of the projection to adapt to the impact of light fluctuations and ensure that the projected image is always clear.

[0126] If the light adaptation fluctuation value is less than or equal to the set standard threshold, it means that the light conditions are relatively stable and the system can maintain the current projection settings. At this time, the projection effect has adapted to the current light environment and no significant changes are required.

[0127] The standard threshold is set based on the actual environment and projection requirements, such as 0.1 or 0.2.

[0128] The obtained best viewing distance deviation and the light adaptation fluctuation value are used as the input items of fuzzy logic, and the adjustment values of the focal length and image clarity are used as the output items of fuzzy logic;

[0129] The best viewing distance deviation ΔD is fuzzified into five fuzzy sets: "very small", "small", "medium", "large", and "very large". According to the magnitude of the deviation, it is mapped to different membership functions. For example:

[0130] If ΔD is very small (close to zero), the membership function of "very small" is close to 1, and other values are close to 0.

[0131] If ΔD is large (far from the ideal viewing distance), the membership function of the "very large" set is close to 1.

[0132] The light adaptation fluctuation value σadapt is fuzzified into five fuzzy sets: "very low", "low", "medium", "high", and "very high". According to the magnitude of the fluctuation value, it is mapped to different membership functions. For example:

[0133] If the light fluctuation is small (stable light), the membership function of the "very low" set is close to 1, and the membership functions of other sets are close to 0.

[0134] If the light fluctuation is large (sharp change in ambient light), the membership function of the "very high" set is close to 1.

[0135] Based on fuzzy logic reasoning, a set of rules need to be defined to associate the input items (viewing distance deviation and light fluctuation) with the output items (adjustment of focal length and image clarity). These rules connect the input fuzzy values with the output fuzzy values to obtain the adjustment direction and amplitude of the focal length and image clarity.

[0136] Example fuzzy rules:

[0137] Rule 1: If the best viewing distance deviation is very small and the light adaptation fluctuation value is very low, the focal length adjustment is unchanged, and the image clarity remains normal.

[0138] Rule 2: If the best viewing distance deviation is large and the light adaptation fluctuation value is high, the focal length adjustment is increased, and the image clarity should be improved because the light changes greatly and the viewing distance is too large, requiring stronger focusing and brightness adjustment.

[0139] Rule 3: If the best viewing distance deviation is small and the light adaptation fluctuation value is medium, the focal length adjustment is slightly adjusted, and the image clarity adjustment is small because the ambient light is stable and the viewing distance is close to the best, with less change required.

[0140] Rule 4: If the best viewing distance deviation is very large and the light adaptation fluctuation value is very high, the focal length adjustment is increased significantly, and the image clarity should be enhanced significantly because the viewing distance is too far and the environmental light changes violently, requiring significant focus and brightness adjustments.

[0141] During the fuzzy logic reasoning process, the system combines the membership degrees of the input items by applying the above rules. For example, if the best viewing distance deviation is "medium" and the light fluctuation value is "high", the corresponding fuzzy output sets are found through Rule 2 and Rule 3.

[0142] The results of the fuzzy reasoning (i.e., the fuzzy sets of the focal length and clarity adjustments) are defuzzified to obtain specific output values. Usually, the weighted average method (also called the centroid method) is used for defuzzification.

[0143] Based on the defuzzified focal length adjustment value, the system adjusts the focal length of the projector to make it more suitable for the current viewer position and lighting conditions.

[0144] Based on the defuzzified clarity adjustment value, the system adjusts the contrast and sharpness of the image to ensure that the image is clear and bright under the current conditions.

[0145] Focal length adjustment: The system adjusts the focal length of the projector according to the output of the fuzzy reasoning to ensure that the virtual image is always clear and adapts to the distance change between the viewer and the object.

[0146] Image clarity adjustment: Based on the adjustment results of the image clarity, the system optimizes parameters such as the brightness, contrast, and sharpness of the projection to ensure that the image can provide the best visual effect in different lighting environments.

[0147] Through fuzzy logic reasoning, the system can intelligently adjust the projection focal length and image clarity based on the relative position of the viewer (best viewing distance deviation) and the change of the environmental light (light adaptation fluctuation value). Fuzzy logic can handle uncertainty and ambiguity, enabling the system to optimize the projection effect in real time and accurately in a changing environment and provide the best visual experience.

[0148] The system continuously monitors the environmental light data and automatically adjusts the projection content according to the real-time feedback. In the case of large fluctuations in the environmental light, the system will continuously perform dynamic adjustments of the brightness, contrast, and color to maintain the stability of the projection effect.

[0149] By calculating and optimizing the light adaptation fluctuation value in real time, the system can automatically adjust the projection parameters under complex and changing lighting conditions to ensure that the virtual image always presents the best effect under different environmental lighting conditions.

[0150] Example 2, please refer toFigure 2 As shown in Figure 2 , a three-dimensional projection mapping device in this embodiment includes a data processing module, a virtual image deformation and mapping module, a projection adjustment module, and a projection parameter adjustment module;

[0151] Data processing module: Use multiple cameras and sensors to obtain the three-dimensional data of the object surface in real time, and combine deep learning algorithms to dynamically update the geometric model and texture information of the object surface;

[0152] Projection parameter adjustment module: According to the real-time geometric data of the object surface, lighting conditions, and the position changes of the audience, use an adaptive algorithm to adjust the projection angle, focal length, and lighting intensity of the projector;

[0153] Virtual image deformation and mapping module: Perform adaptive image deformation on the virtual image, combine perspective transformation and affine transformation, project the virtual image onto the complex three-dimensional surface of the object, and adjust the geometric shape and texture mapping of the virtual image in real time according to the dynamic changes of the object surface;

[0154] Projection adjustment module: Adjust the projection content and optimize the projection effect by real-time sensor feedback of the dynamic position changes of the audience and the lighting changes of the environment.

[0155] The above formulas are all dimensionless and take their numerical calculations. The formulas are obtained by collecting a large amount of data for software simulation to get a formula closest to the real situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0156] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Among them, A and B can be singular or plural. In addition, the character " / " in this article generally represents an "or" relationship between the front and rear associated objects, but it may also represent an "and / or" relationship. Specifically, it can be understood by referring to the context.

[0157] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but this implementation should not be considered to exceed the scope of this application.

[0158] As described above, it is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application.

Claims

1. A three-dimensional projection mapping method, characterized in that: Including: Utilize multiple cameras and sensors to obtain three-dimensional data of the object surface in real time, and combine deep learning algorithms to dynamically update the geometric model and texture information of the object surface; According to the real-time geometric data of the object surface, lighting conditions, and the position changes of the audience, adopt an adaptive algorithm to adjust the projection angle, focal length, and lighting intensity of the projector; Perform adaptive image deformation on the virtual image, combine perspective transformation and affine transformation, project the virtual image onto the complex three-dimensional surface of the object, and adjust the geometric shape and texture mapping of the virtual image in real time according to the dynamic changes of the object surface; Adjust the projection content and optimize the projection effect by real-time sensor feedback of the dynamic position changes of the audience and the lighting changes of the environment.

2. The three-dimensional projection mapping method according to claim 1, wherein: Through multiple high-resolution cameras and LiDAR and structured light sensors, capture multi-view images and depth data of the object surface in real time, including the three-dimensional coordinates, normal vectors, and texture information of the object surface; Use image stitching technology to fuse camera data from different perspectives to form a unified three-dimensional data set, and fuse the three-dimensional coordinate information of each pixel point on the object surface through the depth map generated by the depth sensor to generate an accurate three-dimensional model; Adopt convolutional neural network and generative adversarial network to perform geometric reconstruction and texture mapping on the object surface. The CNN extracts the structural features in the image and dynamically generates a high-resolution geometric model of the object, and the GAN is used for texture synthesis.

3. A three-dimensional projection mapping method according to claim 1, characterized in that: Obtain the geometric data of the object surface in real time, including the curvature, size, and normal vector information of the object surface, and monitor the ambient light intensity in real time through the ambient light sensor; Use infrared sensors or motion capture devices to track the position and perspective of the audience in real time, and obtain the relative distance and perspective changes between the audience and the object; Through the inverse projection algorithm, calculate the optimal projection angle of the projector according to the object surface geometric information and the audience position.

4. A three-dimensional projection mapping method according to claim 3, characterized in that: Use sensors to track the position of the audience in real time, and determine the relative position of the audience by calculating the three-dimensional coordinate difference between the audience and the object. The relative position of the audience is obtained through the distance calculation formula: ; where D is the visual distance between the audience and the object, , , are the three-dimensional coordinates of the audience, , , are the coordinates of the corresponding points on the surface of the object. Compare the obtained visual distance between the audience and the object with the preset optimal visual distance between the audience and the object, and calculate the absolute value of their difference, which is defined as the optimal visual distance deviation.

5. A three-dimensional projection mapping method according to claim 4, characterized in that: Use the light sensing sensor to monitor the ambient light intensity in real time and obtain the brightness information of the ambient light; Calculate the light adaptation parameter according to the intensity of the ambient light : ; where is the current ambient light intensity, is the standard light intensity; Obtain the light adaptation parameters calculated within a fixed time period T , and establish a corresponding data set, calculate the standard deviation of the data set, which is defined as the light adaptation fluctuation value.

6. A three-dimensional projection mapping method according to claim 5, characterized in that: Take the obtained optimal viewing distance deviation and light adaptation fluctuation value as the input items of fuzzy logic, and take the adjustment values of the focal length and image clarity as the output items of fuzzy logic; Perform fuzzy processing on the optimal viewing distance deviation and light adaptation fluctuation value; Based on fuzzy logic reasoning, define fuzzy rules to associate input items and output items; During the fuzzy logic reasoning process, find the corresponding fuzzy output set by combining the membership degrees of the input items; Perform defuzzification processing on the results of fuzzy reasoning to obtain specific output values; Adjust the focal length of the projector according to the defuzzified focal length adjustment value; Adjust the contrast and sharpness of the image according to the defuzzified clarity adjustment value.

7. A three-dimensional projection mapping method according to claim 1, characterized in that: Obtain the three-dimensional data of the object surface in real time through multiple cameras and depth sensors; combine perspective transformation and affine transformation to map the virtual image from a two-dimensional plane to the three-dimensional space of the object surface. Perspective transformation is used to process areas far from the projector; affine transformation is used to adjust the image in the planar area of the object surface; according to the dynamic changes of the object surface, adjust the geometric shape and texture mapping of the virtual image in real time, and adjust the shape of the image through the normal vector and curvature information.

8. A three-dimensional projection mapping device for implementing the three-dimensional projection mapping method according to any one of claims 1-7, characterized in that: It includes a data processing module, a virtual image deformation and mapping module, a projection adjustment module, and a projection adjustment module; Data processing module: Utilize multiple cameras and sensors to obtain the three-dimensional data of the object surface in real time, and combine deep learning algorithms to dynamically update the geometric model and texture information of the object surface; Projection parameter adjustment module: According to the real-time geometric data of the object surface, lighting conditions, and the position changes of the audience, adopt an adaptive algorithm to adjust the projection angle, focal length, and lighting intensity of the projector; Virtual image deformation and mapping module: Perform adaptive image deformation on the virtual image, combine perspective transformation and affine transformation, project the virtual image onto the complex three-dimensional surface of the object, and adjust the geometric shape and texture mapping of the virtual image in real time according to the dynamic changes of the object surface; Projection adjustment module: Adjust the projection content and optimize the projection effect by real-time sensor feedback of the dynamic position changes of the audience and the lighting changes of the environment.

Images