Three-dimensional projection mapping method and device

Abstract

The invention discloses a three-dimensional projection mapping method and device, and particularly relates to the technical field of three-dimensional projection mapping. Three-dimensional data of the surface of an object are acquired in real time through multiple cameras and a sensor, a geometric model and texture information of the surface of the object are dynamically updated in combination with a deep learning algorithm, and the angle, focal length and illumination intensity of a projector are adjusted by adopting a self-adaptive algorithm according to real-time geometric data, illumination conditions and audience position changes, so that the image quality is improved. In addition, the real-time sensor feeds back the audience position and the environment illumination change, the system can dynamically adjust the projection content, it is ensured that the virtual image is accurately aligned with the surface of an object all the time, and the image quality is improved. Dislocation and distortion are avoided, and the stability and visual quality of the interactive display effect are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional projection mapping, and specifically relates to a three-dimensional projection mapping method and device. Background Art

[0002] With the continuous development of graphics processing technology, three-dimensional projection mapping has been widely used in fields such as computer vision, augmented reality (AR), and virtual reality (VR). With the support of a GPU (graphics processing unit), three-dimensional projection mapping can project virtual images onto complex surfaces in the physical world, creating an immersive visual effect.

[0003] The prior art has the following deficiencies: In the interactive display of museums, three-dimensional projection mapping technology is used to project virtual images onto the surfaces of complex and irregular objects, such as the display of dinosaur models. However, the current technology often faces serious accuracy problems when dealing with dynamic and irregular surfaces. Especially when there are changes in the ambient light, the position of the audience, or the adjustment of the camera angle, the projection often fails to accurately fit the surface of the object, resulting in misalignment or distortion of the virtual image. Summary of the Invention

[0004] The purpose of the present invention is to provide a three-dimensional projection mapping method and device to solve the deficiencies in the background art.

[0005] To achieve the above purpose, the present invention provides the following technical solution: A three-dimensional projection mapping method, comprising: Using multiple cameras and sensors to obtain the three-dimensional data of the object surface in real time, and dynamically updating the geometric model and texture information of the object surface in combination with a deep learning algorithm; According to the real-time geometric data, lighting conditions, and position changes of the audience on the object surface, an adaptive algorithm is used to adjust the projection angle, focal length, and lighting intensity of the projector; Performing adaptive image deformation on the virtual image, combining perspective transformation and affine transformation, projecting the virtual image onto the complex three-dimensional surface of the object, and adjusting the geometric shape and texture mapping of the virtual image in real time according to the dynamic changes of the object surface; Adjusting the projection content and optimizing the projection effect by real-time sensor feedback of the dynamic position changes of the audience and the lighting changes in the environment.

[0006] Preferably, multiple high-resolution cameras and LiDAR and structured light sensors are used to capture multi-view images and depth data of the object surface in real time, including the three-dimensional coordinates, normal vectors, and texture information of the object surface; Using image stitching technology to fuse camera data from different perspectives to form a unified three-dimensional dataset, and fusing the three-dimensional coordinate information of each pixel point on the object surface through the depth map generated by the depth sensor to generate an accurate three-dimensional model; Adopting a convolutional neural network and a generative adversarial network to perform geometric reconstruction and texture mapping on the object surface. The CNN extracts the structural features in the image and dynamically generates a high-resolution geometric model of the object, and the GAN is used for texture synthesis.

[0007] Preferably, geometric data of the object surface is obtained in real time, including curvature, size, and normal vector information of the object surface, and the ambient light intensity is monitored in real time through an ambient light sensor; Using an infrared sensor or a motion capture device to track the position and perspective of the audience in real time, and obtaining the relative distance and perspective change between the audience and the object; Through the inverse projection algorithm, according to the geometric information of the object surface and the position of the audience, calculate the optimal projection angle of the projector.

[0008] Preferably, use a sensor to track the position of the audience in real time, determine the relative position of the audience by calculating the three-dimensional coordinate difference between the audience and the object, and the relative position of the audience is obtained through the distance calculation formula: ; where D is the viewing distance between the audience and the object, 、 、 are the three-dimensional coordinates of the audience, 、 、 are the coordinates of the corresponding points on the object surface. Compare the obtained viewing distance between the audience and the object with the preset optimal viewing distance between the audience and the object, and calculate the absolute value of their difference, which is defined as the optimal viewing distance deviation.

[0009] Preferably, use a light sensing sensor to monitor the ambient light intensity in real time and obtain the brightness information of the ambient light; Calculate the light adaptation parameter according to the intensity of the ambient light: ; where, is the current ambient light intensity, is the standard light intensity; Obtain the light adaptation parameter calculated within a fixed time period T, and establish a corresponding data set, and calculate the standard deviation of the data set, which is defined as the light adaptation fluctuation value.

[0010] Preferably, use the obtained optimal viewing distance deviation and light adaptation fluctuation value as the input items of fuzzy logic, and use the adjustment values of focal length and image clarity as the output items of fuzzy logic; Fuzzify the optimal viewing distance deviation and the light adaptation fluctuation value; Based on fuzzy logic inference, define fuzzy rules to associate input items with output items; During the fuzzy logic inference process, find the corresponding fuzzy output set by combining the membership degrees of the input items; Defuzzify the result of the fuzzy inference to obtain a specific output value; Adjust the focal length of the projector according to the defuzzified focal length adjustment value; Adjust the contrast and sharpness of the image according to the defuzzified clarity adjustment value.

[0011] Preferably, three-dimensional data of the object surface is obtained in real time through multiple cameras and depth sensors; combining perspective transformation and affine transformation, map the virtual image from a two-dimensional plane to the three-dimensional space of the object surface, where perspective transformation is used to process areas far from the projector; affine transformation is used to adjust the image of the planar area on the object surface; according to the dynamic changes of the object surface, adjust the geometric shape and texture mapping of the virtual image in real time, and adjust the shape of the image through normal vector and curvature information.

[0012] The present invention also provides a three-dimensional projection mapping device, including a data processing module, a virtual image deformation and mapping module, a projection adjustment module, and a projection adjustment module; Data processing module: Use multiple cameras and sensors to obtain three-dimensional data of the object surface in real time, and dynamically update the geometric model and texture information of the object surface in combination with deep learning algorithms; Projection parameter adjustment module: According to the real-time geometric data of the object surface, lighting conditions, and the position changes of the audience, use an adaptive algorithm to adjust the projection angle, focal length, and lighting intensity of the projector; Virtual image deformation and mapping module: Perform adaptive image deformation on the virtual image, combine perspective transformation and affine transformation, project the virtual image onto the complex three-dimensional surface of the object, and adjust the geometric shape and texture mapping of the virtual image in real time according to the dynamic changes of the object surface; Projection adjustment module: Adjust the projection content and optimize the projection effect by real-time sensor feedback of the dynamic position changes of the audience and the lighting changes in the environment.

[0013] In the above technical solution, the technical effects and advantages provided by the present invention: 1. The present invention can accurately project virtual images onto the surfaces of complex and irregular objects in interactive display environments such as museums. By jointly using multiple cameras and sensors, and combining deep learning algorithms for real-time three-dimensional reconstruction and texture mapping of the object surface, the system can dynamically update the geometric model and texture information of the object. At the same time, the system automatically adjusts the angle, focal length, and light intensity of the projector according to the changes in the viewer's position and viewing angle through the inverse projection algorithm, ensuring that the virtual image can be accurately aligned with the object surface under any ambient light changes, avoiding the misalignment and distortion problems in traditional projection technologies.

[0014] 2. The present invention adopts fuzzy logic inference technology, taking the deviation of the optimal viewing distance between the viewer and the object and the fluctuation value of the ambient light as input items, and realizes adaptive projection content optimization by dynamically adjusting the focal length and image clarity. Through perspective transformation and affine transformation, the virtual image can perform adaptive image deformation in real time according to the dynamic changes of the object surface, ensuring that the image is projected onto the three-dimensional surface of the object without distortion. This method has significant advantages in dealing with complex geometric shapes of the object surface, light changes, and dynamic interactions of viewers, providing a more immersive and stable visual experience. Especially in the interactive display of museums, it improves the display effect of virtual images and the sense of participation of viewers. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0016] Figure 1 It is the method mind map of the present invention.

[0017] Figure 2 It is the device module mind map of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0019] Example 1, please refer to Figure 1 As shown, a three-dimensional projection mapping method in this embodiment includes: Use multiple cameras and sensors to obtain the three-dimensional data of the object surface in real time, and combine deep learning algorithms to dynamically update the geometric model and texture information of the object surface; According to the real-time geometric data of the object surface, lighting conditions, and the position changes of the audience, an adaptive algorithm is used to adjust the projection angle, focal length, and lighting intensity of the projector; Perform adaptive image deformation on the virtual image, combine perspective transformation and affine transformation, project the virtual image onto the complex three-dimensional surface of the object, and adjust the geometric shape and texture mapping of the virtual image in real time according to the dynamic changes of the object surface; Adjust the projection content and optimize the projection effect by real-time sensor feedback of the dynamic position changes of the audience and the lighting changes of the environment.

[0020] In the application of three-dimensional projection mapping, obtaining the three-dimensional data of the object surface in real time is a crucial step. Through the combined use of multiple cameras and sensors, the system can achieve high-precision, real-time three-dimensional reconstruction, and dynamically update the geometric model and texture information of the object through deep learning algorithms.

[0021] To capture the three-dimensional information of the object, the system uses multiple high-resolution cameras for multi-view shooting. Usually, the cameras are set at multiple positions around the object to ensure that complete object image data is collected from different angles. These cameras are triggered synchronously to capture different surface information of the object.

[0022] In addition to cameras, the system also uses depth sensors (such as LiDAR or structured light sensors) to capture the depth data of the object surface. These sensors can emit lasers or project structured light, and calculate the distance of each pixel point by receiving the reflected signals, thereby generating a depth map of the object surface (i.e., the three-dimensional coordinates of each pixel).

[0023] Data from different cameras and sensors are transmitted to the central computing unit in real time. Using image stitching and depth map fusion techniques, the data from different perspectives and sensors are merged into a unified three-dimensional space dataset. In this way, the system can reconstruct the complete three-dimensional model of the object in real time.

[0024] After obtaining the real-time three-dimensional data, deep learning algorithms are used to dynamically update the geometric model and texture information of the object.

[0025] The system uses a convolutional neural network (CNN) to process the depth map generated in real time. The CNN can extract the structural features of the object surface from the original depth map data, and gradually generate a high-resolution geometric model of the object through network layers. Through training, the network can learn how to process depth maps with different lighting, shapes, and surface features, and accurately reconstruct the three-dimensional shape of the object surface.

[0026] In some complex applications, a three-dimensional convolutional neural network (3D CNN) is used to further process and generate three-dimensional models. These networks can process voxel data and directly transform depth information into a complete three-dimensional volume model. The 3D CNN processes each voxel through multiple layers of convolution, generates a three-dimensional model that conforms to the true shape of the object, and can accurately fit the geometric shape of each surface area.

[0027] The texture information of the object (such as surface color, pattern, etc.) is provided by the image data captured by the camera. Deep learning algorithms, especially methods based on generative adversarial networks (GANs) or autoencoders, can learn the texture features of the object surface from images taken from multiple perspectives.

[0028] GANs can be used for texture synthesis. Through the adversarial training of the generator and the discriminator, the generator can generate texture images based on the input object geometric model, while the discriminator evaluates the authenticity of the texture images. In this way, GANs can effectively complete the texture information and handle texture distortions caused by lighting changes or different perspectives.

[0029] A neural network is used to match the geometric surface of the object with the image texture. By using deep learning algorithms for image segmentation and texture classification, the neural network can identify the texture features of different regions on the object surface and automatically map the appropriate texture according to the geometric features of the object surface.

[0030] Through an incremental learning method, the system can gradually optimize the geometric model and texture mapping of the object each time new perspective data is obtained. The deep learning model adopts an adaptive optimization algorithm, such as AdaGrad or the Adam optimizer, to dynamically adjust the weights of the network, so that each newly obtained data can effectively improve the accuracy of the model.

[0031] To simultaneously process geometric reconstruction and texture mapping, the system adopts multi-task learning. By training the geometric model and texture mapping tasks simultaneously within a unified neural network framework, deep learning algorithms can better cooperate to process the three-dimensional shape and visual features of the object surface, thereby improving the update speed and accuracy.

[0032] In this application, through the combined use of multiple cameras and sensors, the system can efficiently collect three-dimensional data of the object surface in real time, and dynamically update the geometric model and texture information by combining deep learning algorithms. Deep learning algorithms, such as CNN, 3D CNN, GAN, etc., play a key role in data processing and texture synthesis. The combination of incremental learning and multi-task learning enables the system to continuously optimize the three-dimensional model and texture of the object as the data accumulates, providing accurate and stable three-dimensional projection mapping effects.

[0033] To achieve high-precision three-dimensional projection mapping, especially in a dynamic environment, it is necessary to adjust the parameters of the projector (such as projection angle, focal length, and light intensity) in real time.

[0034] Through a multi-camera system and depth sensors (such as LiDAR, structured light sensors, etc.), the system can obtain the three-dimensional geometric data of the object surface in real time. These data include the shape, size, texture information of the object, and any changes in the surface shape caused by dynamic changes. For example, the system can capture the details of uneven or curved areas on the object surface, which are very important for accurate three-dimensional projection.

[0035] Through an ambient light sensor, the system can monitor the lighting conditions in the surrounding environment in real time. This includes the intensity, direction, and changes of natural light, artificial lighting, and other sources. This information helps the algorithm determine how to adjust the lighting intensity of the projection to avoid distortion or excessive or insufficient brightness of the projected image.

[0036] Use sensors (such as infrared sensors, cameras, or motion capture devices) to perform real-time tracking of the audience's position and perspective. The system can obtain the specific position of the audience, determine its relative position, distance, and perspective with respect to the object. This is crucial for dynamically adjusting the projection perspective and projection content, especially in a multi-person interaction or exhibition environment.

[0037] Adjust the projection angle according to the real-time position and perspective of the audience to ensure that the virtual image is always accurately aligned with the object surface, avoiding misalignment and distortion. The specific method is to calculate the optimal projection angle of the projector through inverse projection algorithms.

[0038] First, it is necessary to obtain the three-dimensional geometric model of the object, including the surface shape, curvature, and any irregular geometric features of the object. This geometric model is usually obtained in real time by techniques such as multi-view image processing, laser scanning (LiDAR), or depth sensors (such as structured light, ToF sensors). The surface data of the object includes not only the position of each point but also the normal vector (i.e., the orientation of each point) information of the surface.

[0039] The position and viewing angle of the audience are tracked in real time through sensors (such as infrared sensors, cameras, motion capture devices, etc.). The position of the audience is usually represented by a three-dimensional coordinate, and the viewing angle is determined by the relative angle between the audience and the object. This information is crucial because it determines from which angle the image should be projected so that the image can be accurately mapped onto the object surface.

[0040] Given the geometric model of the object surface and the position of the audience, the reverse projection algorithm is used to calculate the optimal projection angle of the projector relative to the object. The goal of reverse projection is to calculate where the projector should be located and where it should point according to the viewing angle of the audience and the geometry of the object surface, so as to ensure that the image is accurately mapped onto the object surface without distortion.

[0041] Suppose a point P(x, y, z) on the object surface and the position of the projector are known. The geometric relationship between the projector and the object surface can be represented by vector relationships in a three-dimensional coordinate system. The normal vector n(x, y, z) of the object surface represents the orientation of this point, and the viewing angle of the audience determines the incident angle of the projection.

[0042] When calculating the viewing angle of the audience, it is first necessary to calculate the position relationship between the audience and each point on the object surface in three-dimensional space. Suppose the position of the audience is V(xv, yv, zv). The relationship between the viewing angle of the audience and the object surface can be obtained by calculating the angle between the line of sight of the audience and the normal vector of the object surface. This angle is the viewing angle deviation, which determines how the virtual image should be presented on the object surface.

[0043] The reverse projection algorithm calculates the optimal position and angle of the projector by reverse calculation. That is, it is assumed how a certain part of the image on the object surface seen by the audience should be reverse-mapped from the position of the projector, and it is determined where the projector should be located and how its angle should be adjusted so that the light emitted from the projector is accurately projected onto the surface.

[0044] Through optimization algorithms (such as the least squares method or gradient-based optimization), the reverse projection algorithm calculates the optimal projection angle of the projector relative to the object. This step usually involves complex matrix operations and spatial transformations, taking into account the curvature of the object surface, the distance between the projector and the object surface, and the change in the position of the audience, and gradually adjusting the projection angle of the projector so that the projected image accurately fits the object surface.

[0045] The projection angle calculated through the above steps ensures the precise alignment of the virtual image with the object surface. In relatively flat areas of the object surface, the projected image has little distortion. However, when the object surface has a large curvature or irregularity, the inverse projection algorithm eliminates the misalignment and distortion problems that occur in traditional projection methods through precise adjustment of the projection angle. Especially on complex surfaces, the algorithm can dynamically adapt to the different geometric relationships of each projection point, ensuring a high degree of accuracy of the projected image.

[0046] In this application, by using the inverse projection algorithm, the system can accurately calculate the optimal projection angle of the projector relative to the object surface, ensuring that the virtual image accurately aligns with the object surface in any dynamically changing environment. This method not only effectively avoids projection distortion but also enables high-precision projection mapping under complex geometric shapes, providing a more immersive visual experience.

[0047] According to the distance between the projector and the object surface and the curvature of the object surface, the focal length is dynamically adjusted to ensure that the projected image is always clear and avoid defocus or blurring. The dynamic adjustment of the focal length can ensure that the projected image remains clear and accurate in areas of the object surface with different curvatures or distances.

[0048] The system measures the distance between the projector and the object surface in real time through depth sensors (such as LiDAR, structured light sensors, ToF sensors, etc.). These sensors can provide high-precision distance data, allowing the system to obtain the distance information between each point on the object surface and the projector.

[0049] Structured light sensor: Calculates depth information by projecting structured light onto the object and analyzing the deformation of the light.

[0050] LiDAR sensor: Accurately measures distances by emitting and receiving reflected laser signals, providing high-resolution three-dimensional surface data.

[0051] ToF sensor: Measures the time it takes for light to travel from the sensor to the object and back using the time-of-flight principle to calculate the precise distance.

[0052] The system generates a depth map based on the sensor data, and each pixel value in the depth map represents the distance between the projector and the corresponding point on the object surface. This depth map provides the basic data for subsequent focal length calculation.

[0053] The geometric model of the object surface (including the three-dimensional coordinates and normal vectors of each point) can provide detailed information about the curvature of the object. The normal vector of each point on the object surface can be estimated by calculating the geometric relationships of neighboring points. For example, using local surface fitting algorithms (such as principal component analysis or least squares fitting), the curvature information of each point can be obtained.

[0054] Flat area: The curvature is close to zero.

[0055] Convex area: The curvature value is positive.

[0056] Concave area: The curvature value is negative.

[0057] This curvature information can help the system judge the degree of deformation of the projected image to determine the amount of focal length adjustment.

[0058] In the convex or concave area of the object, the deformation of the projected image is more obvious. The focal length needs to be dynamically adjusted according to the change of curvature to avoid image distortion. In the flat area, the change of the focal length is relatively small.

[0059] Based on the real-time distance data between the projector and each point on the object surface and the change of the object surface curvature, the system calculates the ideal focal length for each projection point. Suppose the focal length of the projector is f, the distance of a certain point on the object surface is d, and the surface curvature of the object is κ, the ideal focal length can be adjusted by the following formula: ; where is the initial focal length, and α is an adjustment coefficient used to control the degree of change of the focal length with the change of the object surface curvature.

[0060] The focal length of the projector is controlled by its internal lens system. According to the calculated ideal focal length , the system automatically adjusts the focal length of the projector by dynamically controlling the autofocus mechanism of the projector (for example, by adjusting the movement of the lens or changing the width of the projection beam).

[0061] Flat area: In the relatively flat area of the object surface, the change of the focal length is small, and the system will maintain a relatively constant focal length.

[0062] Curved surface area: In the convex or concave area of the object, the focal length will be dynamically adjusted according to the change of the surface curvature to ensure the clarity of the image.

[0063] The system continuously monitors the change of the object surface through real-time feedback sensor data (such as depth sensors and optical sensors). When the position of the audience changes, the focal length will be adjusted immediately. For example, if the audience approaches the object and the distance decreases, the system will automatically increase the focal length; if the audience moves away, the system will decrease the focal length.

[0064] By calculating the focal length and curvature in real time, the system can effectively prevent the focal length from being too long or too short, thus avoiding image blurring or defocusing. In each object surface area, the system can maintain the best visual effect by adaptively adjusting the focal length.

[0065] In this application, by measuring the distance between the projector and the object surface in real time, analyzing the curvature change of the object surface, and combining with the adaptive focal length adjustment mechanism, the system can dynamically adjust the focal length of the projector to avoid defocusing or blurring. This process enables the virtual image to maintain clarity and high quality in all regions of the object surface, whether flat, convex or concave, ensuring the precise presentation of the projected image.

[0066] According to the changes in ambient light and the reflection characteristics of the object surface, dynamically adjust the light intensity of the projector to ensure that the projected image has appropriate brightness, is not affected by ambient light, avoid the projected image being too dark or too bright, and maintain the clarity and contrast of the image.

[0067] The system installs multiple light sensors (such as ambient light sensors, infrared sensors or photoresistors) to monitor the light conditions in the surrounding environment in real time. These sensors can provide information on the intensity and direction of ambient light to help the system understand the current changes in ambient light. For example, the sensors can detect changes in natural light (such as sunlight intensity, shadow changes) and changes in artificial light sources (such as the brightness and position of indoor lights).

[0068] After the ambient light data is collected by the sensors, it is transmitted to the computing unit in real time. The system analyzes the light intensity, direction and light source distribution through algorithms to obtain the characteristics of ambient light in real time. For example, the system can judge the brightness of the current environment according to the light intensity (such as lux value), and then determine the adjustment strategy of the projection light source.

[0069] The surface texture information of the object (such as material type, smoothness, roughness, etc.) plays an important role in the intensity and manner of reflected light. Through depth sensors, cameras or 3D scanning technologies, the system can capture the texture and reflection properties of the object surface in real time. These texture information can include characteristics such as color, glossiness, transparency, etc. The system combines this information with the ambient light data to calculate the optimal light intensity.

[0070] The reflection characteristics of an object are usually described by a light reflection model. Commonly used models include the Phong reflection model and the Blinn-Phong reflection model. These models can simulate how the object surface reflects incident light and calculate the diffuse reflection and specular reflection parts of the object surface respectively.

[0071] Diffuse reflection: Describes the characteristic of light being evenly reflected by the rough area of the object surface.

[0072] Specular reflection: Describes the characteristic of light being reflected by the smooth area of the object surface and forming a high-brightness area.

[0073] Through the reflection model, the system can combine the ambient light intensity, calculate the reflected light intensity of the object surface, and calculate the required illumination intensity of the projector based on this. For example, if the object surface is smooth and has a high reflectivity (such as a mirror or metal surface), the system will reduce the projection illumination intensity; conversely, if the object surface is rough (such as a matte material), the system may need to increase the illumination intensity.

[0074] Calculate the optimal projection illumination intensity based on the ambient light sensing data and the reflection characteristics of the object surface : ; where is the ambient light intensity (measured in real time by the sensor), is the original illumination intensity of the projection light source, is the diffuse reflection coefficient of the object surface (obtained by calculating the object texture), is the specular reflection coefficient of the object surface, and e is an adjustment parameter that controls the influence weights of diffuse reflection and specular reflection. Through the above calculations, the system can dynamically adjust the intensity of the projection illumination, making the brightness of the projection image match the ambient light and ensuring the clarity and contrast of the image.

[0075] Based on the calculated illumination intensity, the system adjusts the projection brightness by controlling the output power of the projector's light source (such as an LED or laser source). If the ambient light is strong, the system will reduce the intensity of the projection light source; conversely, in a darker environment, the system will increase the brightness of the projection light source.

[0076] In this application, by using the light sensing sensor in real time and analyzing the reflection characteristics of the object surface, the system can dynamically calculate and adjust the illumination intensity of the projector. This adaptive illumination adjustment mechanism ensures that under different illumination conditions, the brightness of the projection image is always moderate and not affected by changes in ambient light, avoiding over-bright or over-dark phenomena and ensuring the best visual effect of the projection image.

[0077] Precisely project the virtual image onto the complex three-dimensional surface of the object, and in real time adjust the geometric shape and texture mapping of the virtual image according to the dynamic changes of the object surface. This process involves perspective transformation, affine transformation, and adaptive image adjustment techniques to ensure that the virtual image is precisely aligned with the object surface without distortion.

[0078] Use a multi-camera system, laser scanning (LiDAR), or depth sensor to obtain the three-dimensional data of the object surface. These devices capture the three-dimensional coordinates and texture information of each point on the object surface and generate a detailed geometric model. This model includes information such as the normal vector and curvature of the object surface, serving as the basic data for subsequent transformations.

[0079] Update the geometric information of the object surface using real-time sensors to ensure that the impact of dynamic changes in the object (such as small movements or deformations of the object) on the projection effect is captured. This data includes the shape, texture features, and their changes in different regions of the object surface.

[0080] Based on the three-dimensional coordinates of the object surface, calculate the normal vector of each point. The normal vector describes the orientation of the object surface and is used to calculate the incident angle of the projection light during the projection transformation process.

[0081] Conduct curvature analysis on the object surface to determine the flat, convex, or concave regions of the object. By calculating the neighborhood curvature of the surface points, the system can identify regions with complex surface details (such as uneven regions) and adjust the deformation of the virtual image according to the geometric characteristics of these regions.

[0082] For each point P(x, y, z) on the object surface, calculate the projection position P'(x', y') of this point on the projection plane through the principle of perspective projection. The perspective transformation takes into account the distance relationship between the object surface and the projector, making the projection image appear smaller in the area where the object is far from the projector, and larger in the area close to the projector, simulating the perspective effect of the real world.

[0083] Set the projection plane at Z = 0, and the projection point P'(x', y') of the object surface point P(x, y, z) can be calculated by the formula: ; where f is the focal length of the projector, and z is the distance from the object surface point to the projector.

[0084] Affine transformation is used to process the planar regions on the object surface to ensure that the image maintains the characteristics of straight lines and parallel lines. It maps each point on the object surface from a two-dimensional space to another two-dimensional space through matrix operations. The affine transformation formula is: ; where a, b, c, d are the elements of the transformation matrix, controlling transformations such as rotation, scaling, and translation, and g, f are the translation vectors. The affine transformation ensures that the geometric shape of the image maintains a linear change.

[0085] Calculate the texture coordinates for each object surface point, and these coordinates correspond to a certain position in the virtual image. Through the geometric model of the object surface, map the points in the three-dimensional space to the corresponding positions on the two-dimensional texture image. This process usually uses UV mapping technology.

[0086] By comparing the calculated texture coordinates with the actual texture image, the system can correctly map the texture information of the virtual image to the object surface. In the regions with a large curvature of the object surface, the system will make adaptive adjustments to the texture according to the change of the normal vector to ensure that the texture does not stretch or distort in these regions.

[0087] In regions with large curvature changes, texture mapping is adjusted according to normal vector and curvature information to adapt to the changes in the object surface and prevent texture stretching or compression.

[0088] By feeding back the perspective changes of the audience and the dynamic changes of the object surface through real-time sensors (such as cameras or depth sensors), the system will obtain new geometric data. These data are used to update the geometric model of the object surface (such as shape changes, displacements, etc.), thus affecting the projection of the virtual image.

[0089] The dynamic changes of the object surface (such as deformation or movement) require the real-time update of the geometric shape of the virtual image. The system uses the aforementioned perspective transformation and affine transformation, combines the new object geometric information, recalculates the position of each projection point, and adaptively deforms the virtual image.

[0090] The dynamic changes of the object surface texture (such as changes in factors like lighting and reflection) will affect the projection effect of the texture. The system adjusts the texture of the virtual image in real time by obtaining new texture data in real time (such as reflection and color changes under lighting conditions). Through adaptive texture mapping, the system can ensure that each area of the object surface always presents the most appropriate visual effect.

[0091] In this application, through technical means such as perspective transformation, affine transformation, and texture mapping, the system can accurately project the virtual image onto the complex three-dimensional surface of the object and adjust the geometric shape and texture mapping of the virtual image in real time according to the dynamic changes of the object surface. This process fully considers factors such as the geometric characteristics, normal vector, and lighting conditions of the object surface, and through adaptive image deformation technology, ensures that the virtual image is mapped onto the object surface without distortion or distortion, providing an immersive visual experience.

[0092] Use sensors (such as depth sensors, infrared sensors, motion capture devices, etc.) to track the position of the audience in real time. The system determines the relative position of the audience by calculating the three-dimensional coordinate differences between the audience and the object. The relative position of the audience can be obtained through the distance calculation formula: ; where D is the viewing distance between the audience and the object, , , are the three-dimensional coordinates of the audience, , , are the coordinates of the corresponding points on the object surface. Compare the obtained viewing distance between the audience and the object with the preset optimal viewing distance between the audience and the object, and calculate the absolute value of their difference, which is defined as the optimal viewing distance deviation. The optimal viewing distance represents the ideal distance between the audience and the object under the best viewing conditions.

[0093] If the deviation of the optimal viewing distance is greater than the predetermined deviation threshold, the system will optimize the projection effect by adjusting the position, angle, focal length or other projection parameters of the projector. The system may automatically adjust the position or angle of the projector to make the viewing distance between the audience and the object close to the optimal viewing distance.

[0094] For example: If the audience is too far from the object, the system may prompt or automatically adjust the size and position of the projected content, or adjust the optical parameters (such as focal length) of the projection device to achieve better visual effects. If the audience is too close to the object, the system may reduce the size of the projected image to avoid excessive perspective distortion.

[0095] If the deviation of the optimal viewing distance is less than or equal to the predetermined deviation threshold, the system considers that the current projection effect already meets the requirements of the optimal viewing distance, so there is no need to adjust the projected content. At this time, the system can maintain the current projection settings and continue to provide a stable visual effect.

[0096] Use a light sensor to monitor the ambient light intensity in real time and obtain the brightness information of the ambient light (such as lux value). The sensor can detect changes in the surrounding light sources (natural light, indoor lights, etc.) and reflect the intensity of the light.

[0097] Based on the intensity of the ambient light, the system can calculate the light adaptation parameter : ; where is the current ambient light intensity, is the standard light intensity (usually a preset value representing the light intensity under ideal projection conditions).

[0098] Obtain the light adaptation parameters calculated within a fixed time period T , for example, if the system obtains data once per second and the time period T is 10 seconds, then 10 light adaptation parameter values will be obtained. And establish a corresponding data set, calculate the standard deviation of the data set, and define it as the light adaptation fluctuation value.

[0099] If the light adaptation fluctuation value is greater than the set standard threshold, it indicates that the light fluctuates greatly and the system needs to adjust the projected content. For example: Increase the projection brightness: If the ambient light is too dark, the system will increase the projection brightness. Adjust the contrast: In a strong light environment, the system will enhance the contrast of the image to avoid loss of image details. Adjust the color and brightness of the projected content: Dynamically adjust the hue and brightness of the projection to adapt to the impact of light fluctuations and ensure that the projected image is always clear.

[0100] If the light adaptation fluctuation value is less than or equal to the set standard threshold, it indicates that the light conditions are relatively stable and the system can maintain the current projection settings. At this time, the projection effect has adapted to the current light environment and no significant changes are required.

[0101] The standard threshold is set based on the actual environment and projection requirements, such as 0.1 or 0.2.

[0102] The obtained best viewing distance deviation and the light adaptation fluctuation value are used as the input items of fuzzy logic, and the adjustment values of the focal length and image sharpness are used as the output items of fuzzy logic; The best viewing distance deviation ΔD is fuzzified into five fuzzy sets: "very small", "small", "medium", "large", and "very large". According to the magnitude of the deviation, it is mapped to different membership functions. For example: If ΔD is very small (close to zero), the membership function of "very small" approaches 1, and other values approach 0.

[0103] If ΔD is large (far from the ideal viewing distance), the membership function of the "very large" set approaches 1.

[0104] The light adaptation fluctuation value σadapt is fuzzified into five fuzzy sets: "very low", "low", "medium", "high", and "very high". According to the magnitude of the fluctuation value, it is mapped to different membership functions. For example: If the light fluctuation is small (the light is relatively stable), the membership function of the "very low" set approaches 1, and the membership functions of other sets approach 0.

[0105] If the light fluctuation is large (the ambient light changes violently), the membership function of the "very high" set approaches 1.

[0106] Based on fuzzy logic reasoning, a set of rules need to be defined to associate the input items (viewing distance deviation and light fluctuation) with the output items (adjustment of focal length and image sharpness). These rules link the input fuzzy values with the output fuzzy values to obtain the adjustment direction and amplitude of the focal length and image sharpness.

[0107] Example fuzzy rules: Rule 1: If the best viewing distance deviation is very small and the light adaptation fluctuation value is very low, the focal length adjustment is unchanged, and the image sharpness remains normal.

[0108] Rule 2: If the best viewing distance deviation is large and the light adaptation fluctuation value is high, the focal length adjustment is increased, and the image sharpness should be improved because the light changes greatly and the viewing distance is too large, requiring stronger focusing and brightness adjustment.

[0109] Rule 3: If the best viewing distance deviation is small and the light adaptation fluctuation value is medium, the focal length adjustment is slightly adjusted, and the image sharpness adjustment is small because the ambient light is stable and the viewing distance is close to the best, with less change required.

[0110] Rule 4: If the best viewing distance deviation is very large and the light adaptation fluctuation value is very high, the focal length adjustment is increased significantly, and the image clarity should be enhanced significantly because the viewing distance is too far and the ambient light changes drastically, requiring significant focus and brightness adjustments.

[0111] During the fuzzy logic reasoning process, the system combines the membership degrees of the input items by applying the above rules. For example, if the best viewing distance deviation is "medium" and the light fluctuation value is "high", the corresponding fuzzy output sets are found through Rule 2 and Rule 3.

[0112] Defuzzify the result of the fuzzy reasoning (i.e., the fuzzy sets of focal length and clarity adjustments) to obtain specific output values. Usually, the weighted average method (also called the centroid method) is used for defuzzification.

[0113] Based on the defuzzified focal length adjustment value, the system adjusts the focal length of the projector to make it more suitable for the current viewer position and lighting conditions.

[0114] Based on the defuzzified clarity adjustment value, the system adjusts the contrast and sharpness of the image to ensure that the image is clear and bright under the current conditions.

[0115] Focal length adjustment: The system adjusts the focal length of the projector according to the output of the fuzzy reasoning to ensure that the virtual image is always clear and adapts to the distance change between the viewer and the object.

[0116] Image clarity adjustment: The system optimizes parameters such as the brightness, contrast, and sharpness of the projection based on the adjustment result of the image clarity to ensure that the image can provide the best visual effect in different lighting environments.

[0117] Through fuzzy logic reasoning, the system can intelligently adjust the projection focal length and image clarity based on the relative position of the viewer (best viewing distance deviation) and the change of the ambient light (light adaptation fluctuation value). Fuzzy logic can handle uncertainty and ambiguity, enabling the system to optimize the projection effect in real time and precisely in a changing environment, providing the best visual experience.

[0118] The system continuously monitors the ambient light data and automatically adjusts the projection content according to the real-time feedback. In the case of large fluctuations in the ambient light, the system will continuously perform dynamic adjustments of brightness, contrast, and color to maintain the stability of the projection effect.

[0119] By calculating and optimizing the light adaptation fluctuation value in real time, the system can automatically adjust the projection parameters under complex and changing lighting conditions to ensure that the virtual image always presents the best effect under different ambient light conditions.

[0120] Example 2, please refer to Figure 2As shown in the figure, a three-dimensional projection mapping device according to this embodiment includes a data processing module, a virtual image deformation and mapping module, a projection adjustment module, and a projection adjustment module; Data processing module: Use multiple cameras and sensors to obtain the three-dimensional data of the object surface in real time, and combine deep learning algorithms to dynamically update the geometric model and texture information of the object surface; Projection parameter adjustment module: According to the real-time geometric data of the object surface, lighting conditions, and the position changes of the audience, use an adaptive algorithm to adjust the projection angle, focal length, and lighting intensity of the projector; Virtual image deformation and mapping module: Perform adaptive image deformation on the virtual image, combine perspective transformation and affine transformation, project the virtual image onto the complex three-dimensional surface of the object, and adjust the geometric shape and texture mapping of the virtual image in real time according to the dynamic changes of the object surface; Projection adjustment module: Adjust the projection content and optimize the projection effect by real-time sensor feedback of the dynamic position changes of the audience and the lighting changes of the environment.

[0121] The above formulas are all dimensionless and take their numerical values for calculation. The formulas are obtained by collecting a large amount of data for software simulation to obtain a formula closest to the actual situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0122] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Among them, A and B can be singular or plural. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship. Specifically, it can be understood by referring to the context before and after.

[0123] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0124] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application.

Claims

1. A three-dimensional projection mapping method, characterized in that: include: Use multiple cameras and sensors to obtain three-dimensional data of the object surface in real time, and combine deep learning algorithms to dynamically update the geometric model and texture information of the object surface; Adopting adaptive algorithms to adjust the projection angle, focal length and light intensity of the projector according to the real-time geometric data of the object surface, lighting conditions and changes in the position of the audience; Adaptive image deformation of virtual images, combined with perspective transformation and affine transformation, to project virtual images onto complex three-dimensional surfaces of objects, and adjust the geometric shape and texture mapping of virtual images in real time according to the dynamic changes of the object surface; Through real-time sensor feedback of the audience's dynamic position changes and environmental lighting changes, the projection content can be adjusted and the projection effect can be optimized.

2. A three-dimensional projection mapping method according to claim 1, characterized in that: Through multiple high-resolution cameras, LiDAR, and structured light sensors, multi-view images and depth data of the object surface are captured in real time, including the 3D coordinates, normal vectors, and texture information of the object surface; Image stitching technology is used to fuse camera data from different perspectives to form a unified 3D data set. The 3D coordinate information of each pixel on the surface of the object is fused through the depth map generated by the depth sensor to generate an accurate 3D model. Convolutional neural networks and generative adversarial networks are used to perform geometric reconstruction and texture mapping on the surface of objects. CNN extracts structural features in images and dynamically generates high-resolution geometric models of objects, and GAN is used for texture synthesis.

3. A three-dimensional projection mapping method according to claim 1, characterized in that: Acquire the geometric data of the object surface in real time, including the curvature, size and normal vector information of the object surface, and monitor the ambient light intensity in real time through the ambient light sensor; Use infrared sensors or motion capture devices to track the audience's position and perspective in real time, and obtain the relative distance between the audience and the object and the perspective change; Through the reverse projection algorithm, the optimal projection angle of the projector is calculated according to the surface geometry information of the object and the position of the audience.

4. A three-dimensional projection mapping method according to claim 3, characterized in that: The sensor is used to track the audience's position in real time, and the audience's relative position is determined by calculating the three-dimensional coordinate difference between the audience and the object. The audience's relative position is obtained by the distance calculation formula: ; Where D is the viewing distance between the viewer and the object, , , is the three-dimensional coordinates of the audience, , , is the coordinate of the corresponding point on the surface of the object. The obtained viewing distance between the audience and the object is compared with the preset optimal viewing distance between the audience and the object, and the absolute value of the difference is calculated, which is defined as the optimal viewing distance deviation.

5. A three-dimensional projection mapping method according to claim 4, characterized in that: Use light sensing sensors to monitor the light intensity of the environment in real time and obtain the brightness information of the ambient light; According to the intensity of ambient light, calculate the light adaptation parameters : ;in, is the current ambient light intensity, is the standard light intensity; Get the light adaptation parameters calculated within a fixed time period of T , and establish the corresponding data set, calculate the standard deviation of the data set, and define it as the light adaptation fluctuation value.

6. A three-dimensional projection mapping method according to claim 5, characterized in that: The obtained optimal viewing distance deviation and illumination adaptation fluctuation value are used as input items of fuzzy logic, and the adjustment values ​​of focal length and image clarity are used as output items of fuzzy logic; Fuzzy the optimal viewing distance deviation and illumination adaptation fluctuation value; Based on fuzzy logic reasoning, fuzzy rules are defined to associate input items with output items; In the fuzzy logic reasoning process, the corresponding fuzzy output set is found by combining the membership of the input items; Defuzzify the result of fuzzy reasoning to obtain a specific output value; Adjusting the focus of the projector according to the deblurred focus adjustment value; Adjusts the contrast and sharpness of the image based on the deblurred clarity adjustment value.

7. A three-dimensional projection mapping method according to claim 1, characterized in that: The three-dimensional data of the object surface is acquired in real time through multiple cameras and depth sensors. The virtual image is mapped from the two-dimensional plane to the three-dimensional space of the object surface by combining perspective transformation and affine transformation. Perspective transformation is used to process the area far away from the projector; affine transformation is used to adjust the image of the plane area on the object surface. According to the dynamic changes of the object surface, the geometric shape and texture mapping of the virtual image are adjusted in real time, and the shape of the image is adjusted through the normal vector and curvature information.

8. A three-dimensional projection mapping device, used to implement a three-dimensional projection mapping method according to any one of claims 1 to 7, characterized in that: It includes a data processing module, a virtual image deformation and mapping module, a projection adjustment module and a projection adjustment module; Data processing module: uses multiple cameras and sensors to obtain three-dimensional data of the object surface in real time, and combines deep learning algorithms to dynamically update the geometric model and texture information of the object surface; Projection parameter adjustment module: Adopts adaptive algorithm to adjust the projection angle, focal length and light intensity of the projector according to the real-time geometric data of the object surface, lighting conditions and changes in the position of the audience; Virtual image deformation and mapping module: It performs adaptive image deformation on virtual images, combines perspective transformation and affine transformation, projects virtual images onto complex three-dimensional surfaces of objects, and adjusts the geometric shape and texture mapping of virtual images in real time according to the dynamic changes of the object surface; Projection adjustment module: adjusts the projection content and optimizes the projection effect by using real-time sensors to feedback the audience's dynamic position changes and environmental lighting changes.

Images