An immersive light and shadow space linkage control system and method

Abstract

The present application relates to the technical field of image data processing, and particularly relates to an immersive light and shadow space linkage control system and method, which comprises the following steps: collecting three-dimensional coordinates of a moving target at continuous time points, obtaining a system total delay time, and calculating a three-dimensional velocity vector and a three-dimensional acceleration vector; combining the system total delay time to calculate preliminary three-dimensional coordinates; calculating an acceleration rate of change vector according to the rate of change of the three-dimensional acceleration vector, and performing three-order motion correction on the preliminary three-dimensional coordinates by using the acceleration rate of change vector and the system total delay time to obtain corrected three-dimensional coordinates; mapping the corrected three-dimensional coordinates to a two-dimensional imaging plane to obtain two-dimensional rendering coordinates, and performing weighted smoothing on the two-dimensional rendering coordinates according to historical smoothed two-dimensional rendering coordinates to obtain smoothed two-dimensional rendering coordinates for visual light and shadow projection. The present application reduces the interference of transmission delay and measurement noise by high-order motion prediction and time domain smoothing, and improves the accuracy of space-time matching of light and shadow following.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to an immersive light and shadow spatial linkage control system and method. Background Technology

[0002] Immersive light and shadow spatial linkage control systems are commonly used in cultural tourism live performances, interactive light and shadow spaces, and themed exhibition halls to achieve dynamic tracking of digital light and shadow with physical entities in three-dimensional space. This system uses multi-source spatial sensors to capture the three-dimensional physical coordinates of moving targets in real time and drives projection equipment to perform dynamic projection mapping, thereby creating a highly interactive visual experience.

[0003] In existing technologies, to achieve dynamic projection mapping where light and shadow move with the target in real time, static target coordinates at the acquisition time are typically obtained using a spatial sensor as the rendering reference, or the target position is estimated using a constant-velocity extrapolation method. When the moving target is in uniform linear motion or its acceleration is close to zero and there is no sudden change in direction, existing methods can meet the basic light and shadow following requirements in conventional scenarios.

[0004] However, from the moment the sensor captures the target's physical coordinates to the moment the projection device outputs the final image, the system experiences fixed transmission and rendering delays. When the target is in a state of variable speed or change of direction, existing methods fail to account for the target's acceleration and higher-order kinematic derivatives, resulting in a spatial residual between the predicted coordinates and the target's true coordinates at the moment of actual projection. This causes a lag in the projected image's deviation from the physical target. This physical residual causes a spatial discrepancy between the digital content and the physical location, reducing the immediacy of visual interaction to some extent and making it difficult to meet the tracking requirements in complex motion scenes. Summary of the Invention

[0005] To address the technical problem of lag in light and shadow projection caused by spatial residuals between predicted and actual projection coordinates due to inherent system delays and lack of higher-order motion derivative compensation when a moving target is in a state of variable speed or change of direction, this invention provides solutions in the following aspects.

[0006] In a first aspect, the present invention provides an immersive light and shadow spatial linkage control method, the method comprising steps S1-S4:

[0007] S1. Collect the three-dimensional coordinates of the moving target at continuous moments, obtain the total system delay time, and calculate the three-dimensional velocity vector and three-dimensional acceleration vector based on the three-dimensional coordinates at multiple continuous moments. S2. Taking any moment as the current moment, calculate the preliminary three-dimensional coordinates of the moving target at the projection moment based on the three-dimensional coordinates, three-dimensional velocity vector, and three-dimensional acceleration vector at the current moment, combined with the total system delay time; S3. Calculate the acceleration rate of change vector based on the rate of change of the three-dimensional acceleration vector, and use the acceleration rate of change vector and the total system delay time to perform third-order motion correction on the initial three-dimensional coordinates to obtain the corrected three-dimensional coordinates of the moving target at the time of projection. S4. Map the corrected three-dimensional coordinates to the two-dimensional imaging plane corresponding to the projection device to obtain the two-dimensional rendering coordinates of the moving target at the projection time; perform weighted smoothing based on the two-dimensional rendering coordinates and the historical smoothed two-dimensional rendering coordinates to obtain the smoothed two-dimensional rendering coordinates of the moving target at the projection time for visual light and shadow projection.

[0008] This invention acquires the three-dimensional coordinates of a moving target and the total system delay time. Based on the three-dimensional coordinates at multiple consecutive moments, it calculates three-dimensional velocity and acceleration vectors, thereby calculating preliminary three-dimensional coordinates. Further, it uses the acceleration rate of change vector, calculated from the rate of change of the three-dimensional acceleration vector, to perform third-order motion correction to obtain corrected three-dimensional coordinates. These are then smoothed using perspective mapping and weighted smoothing based on historical smoothed two-dimensional rendering coordinates for visual lighting and shadow projection. This invention establishes a dynamic prediction mechanism that integrates velocity and acceleration to predict future orientation, overcoming the interference of the total system delay time. Simultaneously, it introduces the acceleration rate of change to represent sudden force changes, offsetting the nonlinear trajectory residuals generated during sudden stops or sharp turns. Combined with temporal smoothing filtering damping, it suppresses image jitter caused by sensor noise, enabling virtual lighting and shadow to closely match rapidly moving physical entities and improving the stability of spatiotemporal matching of lighting and shadow.

[0009] Preferably, the step of calculating the preliminary three-dimensional coordinates of the moving target at the projection moment based on the current three-dimensional coordinates, three-dimensional velocity vector, and three-dimensional acceleration vector, combined with the total system delay time, includes: using the current three-dimensional coordinates as the starting position; using the product of the current three-dimensional velocity vector and the total system delay time as the velocity displacement compensation amount; using half of the product of the current three-dimensional acceleration vector and the square of the total system delay time as the acceleration displacement compensation amount; and superimposing the velocity displacement compensation amount and the acceleration displacement compensation amount onto the starting position to obtain the preliminary three-dimensional coordinates of the moving target at the projection moment.

[0010] This invention fully reconstructs the quadratic motion evolution trajectory of an entity within a time delay by matching the time parameter and its square with the velocity and acceleration in terms of physical dimensions. This reduces the probability of the predicted trajectory deviating from the actual physical position due to neglecting acceleration, and ensures the accuracy of the basic inferred coordinates.

[0011] Preferably, the step of calculating the acceleration rate of change vector based on the rate of change of the three-dimensional acceleration vector includes: calculating the difference between the three-dimensional acceleration vector at the current moment and the three-dimensional acceleration vector at the previous moment to obtain an acceleration difference vector; and calculating the acceleration rate of change vector at the current moment based on the ratio of the acceleration difference vector to the sampling period.

[0012] Preferably, the step of using the acceleration rate of change vector and the total system delay time to perform third-order motion correction on the initial three-dimensional coordinates to obtain the corrected three-dimensional coordinates of the moving target at the time of projection includes: determining the third-order displacement correction amount based on the current acceleration rate of change vector and the cube of the total system delay time; and superimposing the third-order displacement correction amount onto the initial three-dimensional coordinates to obtain the corrected three-dimensional coordinates of the moving target at the time of projection.

[0013] This invention utilizes the correction effect of the cubic amplification of the acceleration change rate by delay time to modulate the spatial offset residual caused by sudden force changes in real time, thereby reducing the risk of image and object separation under violent motion conditions and improving the response capability of high dynamic range light and shadow tracking.

[0014] Preferably, determining the third-order displacement correction based on the current acceleration rate of change vector and the cube of the total system delay time includes: calculating the product of the cube of the total system delay time and the current acceleration rate of change vector; and determining one-sixth of the product as the third-order displacement correction.

[0015] Preferably, the step of mapping the corrected three-dimensional coordinates to the two-dimensional imaging plane corresponding to the projection device to obtain the two-dimensional rendering coordinates of the moving target at the projection time includes: converting the corrected three-dimensional coordinates into homogeneous three-dimensional coordinate vectors; using a pre-calibrated rotation and translation transformation matrix to transform the homogeneous three-dimensional coordinate vectors to the projection device coordinate system to obtain local three-dimensional coordinate vectors in the projection device coordinate system; using a pre-calibrated internal parameter matrix of the projection device to perform projection transformation on the local three-dimensional coordinate vectors to obtain homogeneous pixel coordinate vectors; and performing perspective normalization on the homogeneous pixel coordinate vectors to obtain the two-dimensional rendering coordinates of the moving target at the projection time.

[0016] This invention maps higher-order predicted coordinates to a two-dimensional imaging plane through homogeneous coordinate transformation and perspective mapping, correcting spatial perspective distortion and ensuring that the projected light and shadow maintains a geometric contour consistent with the proportion of the physical entity at different depth positions.

[0017] Preferably, the step of converting the modified three-dimensional coordinates into a homogeneous three-dimensional coordinate vector includes: obtaining the three coordinate components of the modified three-dimensional coordinates in the three-dimensional world coordinate system; and adding a homogeneous coordinate component with a non-zero constant after the three coordinate components to obtain a homogeneous three-dimensional coordinate vector.

[0018] Preferably, the step of performing weighted smoothing based on the two-dimensional rendering coordinates and the historical smoothed two-dimensional rendering coordinates to obtain the smoothed two-dimensional rendering coordinates of the moving target at the projection time includes: obtaining the two-dimensional rendering coordinates corresponding to the current sampling time and the smoothed two-dimensional rendering coordinates corresponding to the previous sampling time, and using the smoothed two-dimensional rendering coordinates corresponding to the previous sampling time as the historical smoothed two-dimensional rendering coordinates; performing weighted summation on the two-dimensional rendering coordinates corresponding to the current sampling time and the historical smoothed two-dimensional rendering coordinates to obtain the smoothed two-dimensional rendering coordinates of the moving target at the projection time.

[0019] This invention utilizes the damping attenuation characteristics of historical states to form a low-pass filtering mechanism, which suppresses the interference of discrete measurement noise on the image position, reduces the slight tremors generated at the edges of light and shadow, and improves the smoothness of visual perception.

[0020] Preferably, the calculation of the three-dimensional velocity vector and the three-dimensional acceleration vector based on the three-dimensional coordinates of multiple consecutive moments includes: calculating the three-dimensional velocity vector at the current moment according to the ratio of the difference between the three-dimensional coordinates at the current moment and the three-dimensional coordinates at the previous moment to the sampling period; and calculating the three-dimensional acceleration vector at the current moment according to the ratio of the difference between the three-dimensional velocity vector at the current moment and the three-dimensional velocity vector at the previous moment to the sampling period.

[0021] In a second aspect, the present invention provides an immersive light and shadow spatial linkage control system, which includes a memory and a processor. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, an immersive light and shadow spatial linkage control method of the first aspect of the present invention is implemented.

[0022] By adopting the above technical solution, an immersive light and shadow space linkage control method of the first aspect of the present invention is generated into a computer program and stored in a memory so that it can be loaded and executed by a processor, thereby creating a terminal device based on the memory and the processor for convenient use.

[0023] The beneficial effects of this invention are as follows: This invention obtains the total system delay time and calculates the preliminary three-dimensional coordinates by combining the three-dimensional velocity vector and the three-dimensional acceleration vector, overcoming the disconnect between physical entities and light and shadow images caused by link delay, and improving the accuracy of light and shadow tracking under regular motion. This invention calculates the rate of change of acceleration vector for performers' sudden stops or changes of direction and performs third-order motion correction on the preliminary three-dimensional coordinates, eliminating the nonlinear trajectory prediction residuals caused by sudden force changes and reducing the probability of visual mismatch under highly dynamic and complex movements. After mapping the two-dimensional rendering coordinates, this invention introduces historical smoothed two-dimensional rendering coordinates for weighted smoothing, damping and attenuating the disturbance of high-frequency noise from the sensor on the pixel coordinates, suppressing high-frequency micro-vibrations in the projected image, and improving the visual smoothness of digital light and shadow tracking. Attached Figure Description

[0024] Figure 1 A flowchart of an immersive light and shadow spatial linkage control method provided in an embodiment of the present invention; Figure 2 A schematic diagram of the composition and data path of an immersive light and shadow spatial linkage control system provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of an immersive light and shadow spatial linkage control system provided in an embodiment of the present invention. Detailed Implementation

[0025] The first aspect of this invention provides an immersive light and shadow spatial linkage control method, such as... Figure 1 As shown, the method includes steps S1-S4: Step S1: Collect basic motion data of the moving target at each moment.

[0026] It should be noted that, considering that the moving targets within the immersive light and shadow space are physical entities undergoing positional changes, such as people or interactive props, subsequent trajectory prediction and projection mapping depend on the basic position of the moving targets. Therefore, the primary task is to capture the three-dimensional coordinates of the moving targets in real time. Furthermore, system latency is a key factor causing spatial lag; obtaining accurate total system latency provides a definite time span for subsequent high-order motion state deductions.

[0027] Specifically, a spatial sensor network is used to collect the three-dimensional physical space coordinates of a moving target at various times. Simultaneously, the total system latency is obtained, which includes sensor data acquisition latency, network link transmission latency, and rendering computation latency. The sensor data acquisition latency refers to the time overhead from the occurrence of a change in position in physical space to the sensor completing signal sensing, quantization, and outputting digital coordinate data. It depends on the sensor's exposure time, sample-and-hold, and analog-to-digital conversion speed, and is a deterministic fixed value under a fixed sampling configuration. The rendering computation latency refers to the time overhead required for the rendering engine to complete prediction model calculation, perspective projection mapping, and lighting and shadow rendering output after receiving the coordinate data. It depends on the computational load of the rendering pipeline and the frame buffer refresh cycle of the graphics processor, and is also a deterministic fixed value under a fixed rendering pipeline and synchronous refresh mechanism. Both sensor acquisition latency and rendering computation latency can be precisely fixed values ​​obtained through hardware calibration or frame synchronization timing. The network link transmission latency refers to the time overhead required for digital coordinate data to be transmitted from the sensor node through the network to the central control unit or rendering server, and consists of packet serialization, link propagation, exchange forwarding, and receiving buffering. In immersive light and shadow spaces, gigabit or higher industrial Ethernet networks are typically used for networking. The network topology is closed, and the data transmission load is relatively fixed. Therefore, the fluctuation range of network link transmission delay is very small, usually on the order of microseconds, much smaller than a sampling period. Under these conditions, the instantaneous network transmission delay of each frame is approximated by the average delay parameter obtained through multiple pre-calibrated tests. The introduced prediction error is within the millimeter range, and its impact on the spatial accuracy of projection mapping is negligible.

[0028] Based on this, the statistical average of the sensor data acquisition delay, the network link transmission delay, and the rendering calculation delay are added together to obtain the total system delay time.

[0029] It should be noted that, considering that the movement of a moving target in physical space is often accompanied by changes in velocity and acceleration, static position coordinates alone cannot reflect its motion evolution trend. Therefore, it is necessary to calculate the three-dimensional velocity vector and three-dimensional acceleration vector of the moving target to characterize the first and second-order kinematic derivatives of the moving target.

[0030] Specifically, taking any given moment as the current moment, and combining the moving target at that moment... 3D coordinates The moment before 3D coordinates and the first two moments 3D coordinates The numerical difference method between adjacent sampling periods is used to calculate the moving target at the current moment. Three-dimensional velocity vector With three-dimensional acceleration vector ,in, The preset sampling period is the time interval between two consecutive data acquisitions from the space sensor network. Determining velocity and acceleration from preceding coordinates is common knowledge in this field and will not be elaborated upon here.

[0031] At this point, the basic motion data of the moving target at each moment and the system delay parameters have been obtained.

[0032] Step S2: Obtain the preliminary three-dimensional coordinates of the moving target at a future time.

[0033] It should be noted that, considering that the movement of a moving target in an immersive light and shadow space is often accompanied by changes in velocity and acceleration, projection rendering based solely on the instantaneous static coordinates at the moment of sensor acquisition cannot overcome the inherent total system latency. Therefore, it is necessary to construct a second-order motion state space prediction model that includes velocity and acceleration based on the historical motion state of the moving target, thereby deducing the future spatial position of the moving target at the moment of system response.

[0034] Furthermore, the theoretical basis of this second-order motion state-space prediction model is the position approximation equation based on Taylor series expansion in classical kinematics, and its kinematic basis is the displacement evolution formula for uniformly accelerated motion. ;in The last position, For the initial position, Let the initial velocity be , For acceleration, The time span is specified. This invention injects the first-order velocity vector and the second-order acceleration vector as motion state variables into a second-order motion state space prediction model, utilizing the coefficients of the second-order Taylor expansion. Modulating the nonlinear displacement caused by acceleration enables dynamic prediction of the future spatial position of a moving target.

[0035] Specifically, this invention utilizes Taylor series expansion from classical kinematics, based on the moving target's three-dimensional coordinates, three-dimensional velocity vector, and three-dimensional acceleration vector at the current moment, combined with the system's total delay time. Calculate the preliminary three-dimensional coordinates of the moving target at future moments. Based on the above logic, for any moving target, at time... Preliminary three-dimensional coordinates Satisfying the relation: ; in, Is the moving target at any moment Preliminary three-dimensional coordinates; Is the moving target at any moment 3D coordinates; This is the total system latency. Is the moving target at any moment The three-dimensional velocity vector; Is the moving target at any moment The three-dimensional acceleration vector.

[0036] In this relation, The base starting coordinates represent the actual physical position of the moving target at the current moment, serving as the starting reference for the predicted trajectory. The increase or decrease of the initial three-dimensional coordinates The whole structure shifts in the same direction accordingly; This is a linear time compensation quantity based on the first-order derivative of motion, reflecting the displacement caused by the moving target maintaining its current velocity and the total system delay time. Larger or three-dimensional velocity vector The larger the modulus, the greater the linear compensation displacement of that part. This is a nonlinear acceleration compensation quantity based on the second-order derivative of motion, which corrects the displacement deviation caused by the variable speed motion of the moving target, and varies with the total system delay time. As the magnitude of the acceleration vector increases, the compensation amount increases quadratically. As the magnitude of the three-dimensional acceleration vector increases, the compensation amplitude for nonlinear motion also increases in the same direction.

[0037] Thus, the preliminary three-dimensional coordinates of the moving target at a future moment have been obtained.

[0038] Step S3: Obtain the corrected three-dimensional coordinates of the moving target at future moments.

[0039] It should be noted that the high-order motion state space prediction model assumes that the total system delay time... Inner three-dimensional acceleration vector It remains constant. However, when a moving target in an immersive light and shadow space performs actions such as sudden stops, sharp changes of direction, or rapid turns, its acceleration itself undergoes rapid and drastic changes, meaning that the third-order derivative of acceleration change exists. If the rate of change of acceleration vector is ignored The dynamic impact on the trajectory is significant. When the moving target experiences a sudden change in direction or a drastic acceleration, relying solely on second-order acceleration terms can easily lead to excessive displacement estimation and spatial prediction residuals. Therefore, it is necessary to base the estimation on preliminary three-dimensional coordinates. Introducing the acceleration rate of change vector Perform high-order dynamic correction.

[0040] Furthermore, the theoretical basis of this high-order motion state space prediction model is the third-order position approximation equation based on Taylor series expansion in classical kinematics, and its physical formula is expressed as follows: ;in The last position, For the initial position, Let the initial velocity be , For acceleration, For the rate of change of acceleration, The time span is defined. By introducing the acceleration rate of change vector as a third-order motion state variable, the coefficients of the third-order Taylor expansion are used. Fine modulation of the high-order nonlinear displacement deviation caused by sudden acceleration changes can further eliminate trajectory overshoot when the motion direction changes abruptly.

[0041] Specifically, this invention constructs a dynamic correction relationship between abrupt changes in the trajectory direction and the rate of change of acceleration. A third-order correction is calculated based on the rate of change of acceleration. Combining the initial three-dimensional coordinates, the acceleration rate of change vector, and the total system delay time, the corrected three-dimensional coordinates are calculated. Based on the above logic, the corrected three-dimensional coordinates of the moving target at the future projection moment satisfy the following relationship: ; in, Is the moving target at any moment Corrected three-dimensional coordinates; Is the moving target at any moment Preliminary three-dimensional coordinates; This is the total system latency. Is the moving target at any moment The acceleration rate vector, through time... Three-dimensional acceleration vector Subtract the previous moment Three-dimensional acceleration vector The difference obtained is divided by the preset sampling period. Calculated.

[0042] In this relation, Is the moving target at any moment The initial three-dimensional coordinates are the reference position of the moving target, which serves as the physical basis for higher-order corrections. Increasing or decreasing the value corrects the three-dimensional coordinates. The whole structure shifts in the same direction accordingly; This is a nonlinear higher-order correction based on the third-order derivative of motion, representing the additional displacement caused by a drastic change in acceleration, and the total system delay time. The larger, The stronger the amplification effect of the displacement correction, the larger the magnitude of the acceleration rate of change vector, along... The magnitude of the positional correction in the direction increases in the same direction.

[0043] It should be further noted that the above-described method of calculating the third-order displacement correction using the Taylor expansion deterministic formula is a preferred embodiment of the present invention. In actual engineering implementation, the determination of the third-order displacement correction based on the current acceleration rate of change vector and the cube of the total system delay time also includes a calculation method based on physical time-domain integration: assuming that within the total system delay time, the acceleration of the moving target continuously evolves with time according to the current acceleration rate of change vector, the third-order displacement correction caused by the accumulation of acceleration abrupt changes is calculated by performing continuous quadratic time-domain integration on the acceleration rate of change vector within the total system delay time; the quadratic time-domain integration can employ existing techniques such as continuum domain analytical integration or discrete domain numerical integration, which will not be elaborated upon here. Such calculation methods based on the acceleration rate of change and the cube of the delay time to deduce higher-order displacement compensation all fall within the protection scope of the present invention.

[0044] Thus, the corrected three-dimensional coordinates of the moving target at the actual moment of future light and shadow projection were obtained.

[0045] Step S4: Calculate the smoothed two-dimensional rendering coordinates corresponding to the moving target.

[0046] It should be noted that the corrected three-dimensional coordinates of the moving target at the future projection moment are obtained. Next, it needs to be precisely mapped to the rendering position in the two-dimensional pixel coordinate system of the projection device to drive the digital lighting and shadow to fit the physical target. Since it is difficult to completely eliminate minute measurement noise during the high-frequency sampling process of the spatial sensor, directly using high-order predicted coordinates for pixel mapping may amplify the noise at high frequencies, causing slight jitter and jitter in the visual image. Therefore, it is necessary to combine the three-dimensional perspective projection transformation with the temporal first-order exponential weighted smoothing filtering mechanism to achieve the mapping from three-dimensional spatial predicted coordinates to two-dimensional pixel coordinates and suppress high-frequency jitter.

[0047] Furthermore, the theoretical basis of this mapping and smoothing model includes a pinhole camera perspective projection imaging model and a discrete first-order exponential smoothing filtering principle. The perspective projection imaging model, based on a homogeneous coordinate transformation matrix, achieves dimensionality reduction mapping from physical space coordinates to image pixel coordinates through a rigid transformation from the three-dimensional world coordinate system to the local coordinate system of the projection device and optical projection of the internal parameter matrix. The first-order exponential smoothing filtering principle utilizes the damping attenuation characteristics of historical output states, and achieves low-pass filtering of high-frequency random jitter by weighting the current observation value with the smoothed value from the previous moment.

[0048] Specifically, this invention establishes perspective transformation mapping relations and dynamic smoothing control relations. First, the corrected three-dimensional coordinates... Convert to homogeneous 3D coordinate vector Then the homogeneous three-dimensional coordinate vector Multiply by the rotation and translation transformation matrix With the internal parameter matrix of the projection device The homogeneous pixel coordinate vector is calculated; then, perspective division is performed on the homogeneous components of the homogeneous pixel coordinate vector to extract the two-dimensional components, which are then mapped to obtain the two-dimensional rendering coordinates. Finally, utilize the previous moment. Historical smooth 2D rendering coordinates Weighted smoothing control is applied to calculate the smoothed 2D rendering coordinates. .

[0049] Based on the above logic, the three-dimensional coordinates will be corrected first. Expanded into homogeneous three-dimensional coordinate vectors Next, the homogeneous pixel coordinate vector of the projection device is denoted as... It satisfies the following relation: Then, the normalized and extracted 2D rendering coordinates are... Satisfying the relation: ; in, Is the moving target at any moment The corresponding two-dimensional rendering coordinates; It is a correction of three-dimensional coordinates Expanded A homogeneous three-dimensional coordinate vector; , , These are correcting the three-dimensional coordinates. X-axis components, Y-axis components, and Z-axis components in a three-dimensional world coordinate system; It is obtained through pre-calibration. A 3D rotation and translation transformation matrix is ​​used to transform the 3D world coordinate system to the local coordinate system of the projection device; It is obtained through pre-calibration. Internal parameter matrix of a 3D projection device; It is obtained by matrix multiplication. Vizier subpixel coordinate vector; , These are the intermediate values ​​of the unnormalized pixel coordinates; It corrects the three-dimensional coordinates. Rotation and translation transformation matrix After rigid transformation to the local coordinate system of the projection device, the depth component of the resulting local three-dimensional coordinate vector, i.e., the local Z-axis coordinate value, is denoted as... .

[0050] In the above relation, The homogeneous three-dimensional coordinate vector in the three-dimensional world coordinate system is rigidly transformed to the local coordinate system of the projection device, and the local coordinate vector is output. Its translation and rotation components determine the three-dimensional spatial orientation of the moving target in the projection field of view. Using the internal parameter matrix of the projection device Optical scaling is applied to the local coordinates to output homogeneous pixel coordinate vectors. ,matrix The larger the focal length parameter, the higher the median value of the calculated unnormalized pixel coordinates. and Increases linearly proportionally; Using the depth distance component in the local coordinate system Right now The reciprocal of the unnormalized pixel coordinates is used to perform perspective normalization division and extract the first two dimension components, as well as the depth distance component. The larger the value, the farther the moving target is from the camera, and the higher the final 2D rendering coordinates. The coordinate values ​​show a non-linear shrinking trend, which conforms to the physical law of perspective where things appear larger when they are closer and smaller when they are farther away.

[0051] In one specific implementation of converting corrected 3D coordinates into homogeneous 3D coordinate vectors, the three coordinate components of the corrected 3D coordinates in the 3D world coordinate system are first obtained. Then, a preset non-zero weighting factor is obtained, and each of the three coordinate components is multiplied by this weighting factor to obtain three weighted coordinate components. Finally, these three weighted coordinate components are used as the first three components of the target vector, and the weighting factor is appended to the fourth component, concatenating them to construct a four-dimensional homogeneous 3D coordinate vector. This weighted transformation not only maintains the equivalence of projective geometry during perspective mapping but also effectively adjusts the numerical range during matrix operations, avoiding data overflow during high-precision calculations. Furthermore, the weighting factor can be automatically eliminated during subsequent perspective normalization division, accurately restoring the 2D rendering coordinates.

[0052] It should be noted that the rotation and translation transformation matrix With the internal parameter matrix of the projection device It can be obtained during the system deployment phase through a pre-defined optical calibration and spatial registration process.

[0053] Specifically, the internal parameter matrix of the projection device This is used to characterize the internal optical properties of a projection device, including parameters such as equivalent focal length and principal point coordinate pixel offset. In practical implementation, the existing Zhang Zhengyou calibration method is usually adopted. This involves projecting a checkerboard pattern of known size into physical space, acquiring images from multiple angles, extracting corner points, and then optimizing offline to obtain a fixed matrix. Rotation and translation transformation matrix The extrinsic parameter matrix, or external parameter matrix, is used to characterize the relative spatial pose relationship between the 3D world coordinate system and the local coordinate system of the projection device. In practical implementation, multiple reference target points with known 3D world coordinates can be set in the immersive light and shadow space, and their corresponding 2D pixel coordinates on the projection imaging plane can be recorded. A perspective point localization algorithm can then be used, combined with the calibrated intrinsic parameter matrix. The translation vector and rotation matrix are calculated, and then the following is constructed: 3D rotation and translation transformation matrix .

[0054] To ensure the entire perspective projection mapping process has explicit mathematical computability, in a specific embodiment, the projection device's internal parameter matrix... for A 3D matrix, the structure of which is given by the following set of exemplary calibration values: ; in, and These represent the equivalent pixel focal lengths of the projector in the X and Y axes, respectively, and determine the optical scaling ratio of the projected image. and The pixel offset representing the principal point coordinates of the projection device, corresponding to common... The resolution displays the center point of the image. Rotation and translation transformation matrix. for 3D matrix, by rotation matrix and Translation vector The assembly is as follows, and its structure and a set of exemplary calibration values ​​are as follows: ; The rotation matrix formed by the first three columns This represents the relative spatial orientation between the world coordinate system and the local coordinate system of the projection device. In this example, it indicates that the projector's pitch angle was adjusted during installation. The fourth column represents the translation vector. This indicates that the projection device is installed in front of the three-dimensional world coordinate system. Height offset The actual physical location.

[0055] Based on this, the final smooth 2D rendering coordinates are sent to the rendering engine. Satisfying the relation: ; in, Is the moving target at any moment The corresponding smooth 2D rendering coordinates; Is the moving target at any moment The corresponding two-dimensional rendering coordinates; It is a moment Output historical smooth 2D rendering coordinates; It is the preset sampling period; It is the smoothing weighting coefficient.

[0056] In this relation, The smoothing weight coefficient represents the dominant contribution of the current predicted mapping coordinates. The larger the smoothing weight coefficient, the stronger the determining effect of the currently calculated 2D rendering coordinates on the final output, and the stronger the light and shadow following response sensitivity. The damped decay retention term representing the historical state at the previous moment, and the residual weighting coefficient. The larger the value, the stronger the constraint preservation effect of the historical smooth coordinates, and the more obvious the suppression effect on the small coordinate jumps caused by high-frequency fluctuations of the sensor.

[0057] For ease of understanding, the following is an example for a highly dynamic interactive scenario in an immersive exhibition hall. In this example, the sampling frequency of the spatial sensor network is set to 50Hz, i.e., the sampling period... The setting is 0.02s. The total system latency, including data acquisition, network transmission, and rendering calculations, is determined by reading the system's preset calibration parameters. The value is set to 0.1s. Assume the time... The moving target undergoes a drastic change of direction in the X-axis direction. Through the difference calculation in step S1, the current moment of the moving target is obtained. The initial X-axis coordinate in the world coordinate system is 1m, corresponding to the X-axis velocity. The X-axis acceleration is 0.5 m / s². for According to step S2, substituting into the time-expanded second-order motion state-space prediction model, the X-axis component of the preliminary three-dimensional coordinates is calculated: Because the moving target suddenly stops at this moment, its acceleration changes abruptly, and the rate of change of acceleration along the X-axis... Calculated by difference as Using the initial coordinates directly will result in exceedance error. According to step S3, a higher-order dynamic correction model is introduced to calculate the X-axis component of the corrected 3D coordinates: ; Entering the rendering mapping stage, assuming the above-mentioned corrected 3D coordinates are normalized by rotation and translation matrices and internal parameter matrices for perspective, the resulting 2D rendering coordinates are obtained. The X-axis pixel value is 800. The system reads the historical smoothed 2D rendering coordinates output from the previous moment. The X-axis pixel value is 795. Considering the current high dynamic range scene, the smoothing weight coefficient will be adjusted. The value is set to 0.8. According to step S4, the final smooth 2D rendering coordinate X-axis pixel value sent to the rendering engine is calculated as follows: The calculation process for a single sampling period described above demonstrates the ability of this invention to correct for high-frequency abrupt motions at a microscopic timescale.

[0058] To further verify the overall system stability of this invention under macroscopic, long-term, and complex operating conditions, model verification was conducted based on a real historical dataset of an immersive exhibition hall. Sensor data from 30 consecutive days of operation were selected as training samples, and data from the following 7 days were used as test samples. Two sets of comparative models were set up: the first set was a second-order predictive ablation model that only executed up to step S2, and the second set was a model incorporating higher-order corrections and smoothing filters. Evaluation metrics included the root mean square error of trajectory tracking, the maximum spatial residual of abrupt turning, and the amplitude of high-frequency jitter in the image. The root mean square error was calculated as the square root of the squared mean of the difference between the actual rendered position of the light and shadow and the expected position of the physical target at each test moment. Statistical results from the testing phase showed that the root mean square error of the second-order predictive ablation model was 6.2 cm, the maximum spatial residual of abrupt turning was 15.8 cm, and the amplitude of high-frequency jitter in the image was 4.2 pixels; the root mean square error of the complete model decreased to 1.4 cm, the maximum spatial residual of abrupt turning decreased to 2.5 cm, and the amplitude of high-frequency jitter in the image decreased to within 0.5 pixels. Compared to the model without higher-order corrections, the overall dynamic response accuracy was improved by approximately 77.4%, effectively verifying the advantages of this invention in eliminating spatial lag and suppressing high-frequency jitter in high-dynamic scenarios.

[0059] It should be noted that the smoothing weighting coefficient The value needs to be set according to the intensity of the moving target's motion and the noise interference intensity of the spatial sensor network: for moving targets in high-dynamic movement scenarios such as high-frequency sudden stops, sharp changes of direction, or rapid turns, the value needs to be appropriately increased. Setting it to 0.85 can reduce the hysteresis effect of temporal smoothing and enhance the response sensitivity to real-time visual light and shadow tracking. For sampling scenarios where the moving target is in a stable, uniform motion or is standing still and the sensor data fluctuates significantly, the value should be appropriately reduced. If set to 0.6, the constraint preservation effect of the historical smooth coordinates of the previous moment is enhanced, improving the filtering and suppression effect on high-frequency micro-jumps and jitters in the image. In this embodiment, The preferred value is 0.8.

[0060] To further explain the smoothing weight coefficient The impact of value changes on the specific rendering calculation results and system dynamic characteristics mentioned above can be further analyzed based on the data conditions of the aforementioned embodiments: When a moving target switches from a sudden stop to a dynamic scene with equal height to a stable uniform speed movement scene, if it still maintains... While the response remains agile, its suppression of high-frequency, minute fluctuations in the sensor is somewhat conservative; if the smoothing weight coefficient is appropriately lowered... At this point, the historical smooth coordinates of the previous moment The damping weight is increased to The currently calculated smooth 2D rendering coordinate X-axis pixel value is then adjusted to... It can be seen that the smaller ones This increases the pull of historical states on the current output, causing the output coordinates to converge further towards historical stable values, thus effectively enhancing the image filtering and suppression effect in static standing or slightly shaking scenes; conversely, if the output coordinates are adjusted during a sudden stop or change of direction... Increase to The calculation result is This enhances the dominant contribution to the latest predicted mapping coordinates and reduces the lag effect caused by time-domain smoothing. Therefore, calculations based on the specific numerical values ​​above show that a reasonable setting of the smoothing weight coefficient is effective. It can adjust the dynamic response sensitivity of light and shadow tracking and the stability of visual image filtering.

[0061] like Figure 2 As shown in the figure, this is a schematic diagram of the composition and data path of the immersive light and shadow spatial linkage control system. This invention constitutes a closed-loop control architecture through front-end data acquisition, intermediate core model processing, and back-end rendering and projection. Specifically, the moving target's three-dimensional coordinates are captured by a spatial sensor network and transmitted to the motion derivative calculation module and the second-order motion state space prediction model. The motion derivative calculation module calculates the moving target's three-dimensional velocity vector, three-dimensional acceleration vector, and acceleration rate of change vector based on the current and historical three-dimensional coordinates. The system's total delay time is output using preset calibration parameters and input to the second-order motion state space prediction model and the higher-order motion state space prediction model. The second-order motion state space prediction model performs second-order motion state prediction based on the three-dimensional coordinates, three-dimensional velocity vector, three-dimensional acceleration vector, and the system's total delay time, obtaining and outputting the moving target's preliminary three-dimensional coordinates at a future time. Subsequently, the higher-order motion state space prediction model receives the preliminary three-dimensional coordinates, acceleration rate of change vector, and system's total delay time, performs higher-order dynamic correction based on the sudden change in motion trajectory direction and acceleration rate of change, obtaining and outputting the corrected three-dimensional coordinates of the moving target at the future projection moment. The spatiotemporal joint projection mapping and dynamic rendering smoothing control model performs perspective mapping and smoothing on the corrected 3D coordinates, calculates smoothed 2D rendering coordinates, and sends these smoothed 2D rendering coordinates to the rendering engine. The rendering engine drives the projection device, which projects light and shadow images onto the moving target, completing the tracking loop.

[0062] Thus, the perspective mapping and dynamic smoothing filtering from corrected 3D coordinates to 2D pixel coordinates were completed. The resulting smooth 2D rendering coordinates are used to drive the rendering engine in the immersive light and shadow space to perform visual light and shadow projection.

[0063] The second aspect of this embodiment provides an immersive light and shadow spatial linkage control system, such as Figure 3 As shown, the immersive light and shadow spatial linkage control system includes a memory and a processor. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, an immersive light and shadow spatial linkage control method according to the first aspect of the present invention is implemented.

[0064] The immersive light and shadow space linkage control system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces. Their settings and functions are known in the art and will not be described in detail here.

[0065] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. An immersive light and shadow spatial linkage control method, characterized in that, include: S1. Collect the three-dimensional coordinates of the moving target at continuous moments, obtain the total system delay time, and calculate the three-dimensional velocity vector and three-dimensional acceleration vector based on the three-dimensional coordinates at multiple continuous moments. S2. Taking any moment as the current moment, calculate the preliminary three-dimensional coordinates of the moving target at the projection moment based on the three-dimensional coordinates, three-dimensional velocity vector, and three-dimensional acceleration vector at the current moment, combined with the total system delay time; S3. Calculate the acceleration rate of change vector based on the rate of change of the three-dimensional acceleration vector, and use the acceleration rate of change vector and the total system delay time to perform third-order motion correction on the initial three-dimensional coordinates to obtain the corrected three-dimensional coordinates of the moving target at the time of projection. S4. Map the corrected three-dimensional coordinates to the two-dimensional imaging plane corresponding to the projection device to obtain the two-dimensional rendering coordinates of the moving target at the projection time; perform weighted smoothing based on the two-dimensional rendering coordinates and the historical smoothed two-dimensional rendering coordinates to obtain the smoothed two-dimensional rendering coordinates of the moving target at the projection time for visual light and shadow projection.

2. The immersive light and shadow spatial linkage control method according to claim 1, characterized in that, The calculation of the preliminary three-dimensional coordinates of the moving target at the moment of projection, based on the current three-dimensional coordinates, three-dimensional velocity vector, and three-dimensional acceleration vector, combined with the total system delay time, includes: Use the current three-dimensional coordinates as the starting position; The product of the current three-dimensional velocity vector and the total system delay time is used as the velocity displacement compensation amount. Half of the product of the current three-dimensional acceleration vector and the square of the total system delay time is used as the acceleration-displacement compensation amount. The velocity displacement compensation and acceleration displacement compensation are superimposed on the starting position to obtain the preliminary three-dimensional coordinates of the moving target at the moment of projection.

3. The immersive light and shadow spatial linkage control method according to claim 1, characterized in that, The step of calculating the acceleration rate of change vector based on the rate of change of the three-dimensional acceleration vector includes: Calculate the difference between the current three-dimensional acceleration vector and the three-dimensional acceleration vector at the previous moment to obtain the acceleration difference vector; The acceleration rate of change vector at the current moment is calculated based on the ratio of the acceleration difference vector to the sampling period.

4. The immersive light and shadow spatial linkage control method according to claim 1, characterized in that, The process of using the acceleration rate of change vector and the total system delay time to perform third-order motion correction on the initial three-dimensional coordinates to obtain the corrected three-dimensional coordinates of the moving target at the projection moment includes: The third-order displacement correction is determined by the current acceleration rate vector and the cube of the total system delay time. The third-order displacement correction is superimposed on the initial three-dimensional coordinates to obtain the corrected three-dimensional coordinates of the moving target at the moment of projection.

5. The immersive light and shadow spatial linkage control method according to claim 4, characterized in that, The determination of the third-order displacement correction based on the current acceleration rate of change vector and the cube of the total system delay time includes: Calculate the product of the cube of the total system delay time and the current acceleration rate vector; determine one-sixth of the product as the third-order displacement correction.

6. The immersive light and shadow spatial linkage control method according to claim 1, characterized in that, The step of mapping the corrected three-dimensional coordinates to the two-dimensional imaging plane corresponding to the projection device to obtain the two-dimensional rendering coordinates of the moving target at the projection time includes: The corrected three-dimensional coordinates are converted into homogeneous three-dimensional coordinate vectors; The homogeneous three-dimensional coordinate vector is transformed to the projection device coordinate system using a pre-calibrated rotation and translation transformation matrix, thus obtaining the local three-dimensional coordinate vector in the projection device coordinate system. The local three-dimensional coordinate vector is projected and transformed using the pre-calibrated internal parameter matrix of the projection device to obtain the homogeneous pixel coordinate vector; Perspective normalization is performed on the homogeneous pixel coordinate vector to obtain the two-dimensional rendering coordinates of the moving target at the projection time.

7. The immersive light and shadow spatial linkage control method according to claim 6, characterized in that, The step of converting the corrected three-dimensional coordinates into a homogeneous three-dimensional coordinate vector includes: Obtain the three coordinate components of the corrected three-dimensional coordinates in the three-dimensional world coordinate system; Adding a homogeneous coordinate component with a non-zero constant after the three coordinate components yields a homogeneous three-dimensional coordinate vector.

8. The immersive light and shadow spatial linkage control method according to claim 1, characterized in that, The step of performing weighted smoothing based on 2D rendering coordinates and historical smoothed 2D rendering coordinates to obtain the smoothed 2D rendering coordinates of the moving target at the projection time includes: Obtain the two-dimensional rendering coordinates corresponding to the current sampling time and the smoothed two-dimensional rendering coordinates corresponding to the previous sampling time, and use the smoothed two-dimensional rendering coordinates corresponding to the previous sampling time as the historical smoothed two-dimensional rendering coordinates. The weighted sum of the two-dimensional rendering coordinates corresponding to the current sampling time and the historical smoothed two-dimensional rendering coordinates is used to obtain the smoothed two-dimensional rendering coordinates of the moving target at the projection time.

9. The immersive light and shadow spatial linkage control method according to claim 1, characterized in that, The calculation of the three-dimensional velocity vector and the three-dimensional acceleration vector based on three-dimensional coordinates at multiple consecutive moments includes: The three-dimensional velocity vector at the current moment is calculated based on the ratio of the difference between the three-dimensional coordinates at the current moment and the three-dimensional coordinates at the previous moment to the sampling period. The three-dimensional acceleration vector at the current moment is calculated based on the ratio of the difference between the three-dimensional velocity vector at the current moment and the three-dimensional velocity vector at the previous moment to the sampling period.

10. An immersive light and shadow spatial linkage control system, characterized in that, The immersive light and shadow spatial linkage control system includes a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the immersive light and shadow spatial linkage control method according to any one of claims 1-9 is implemented.

Images