A digital media interaction control system and method for immersive experience

Abstract

The application provides an immersive experience-oriented digital media interaction control system and method, and relates to the technical field of interaction control.The application realizes adaptive stretching and contraction of a media element filtering range according to a user motion state by synchronously fusing limb, line of sight and voice multi-modal data, dynamically constructing a Gaussian probability density function scope by using instantaneous speed variance, solving the misselection and omission problems caused by a fixed threshold, mapping voice fundamental frequency to a spiral motion coefficient and superimposing the spiral motion coefficient to a limb motion vector equation, so that the generated enhanced trajectory has limb motion trend and voice prosody characteristics, the naturalness and continuity of the trajectory are improved, the line of sight projection method is used to determine a target interaction point, the interaction point suspension problem caused by inconsistent depth of eyes and hands is eliminated, the geometric rationality of the intention direction is ensured, a deviation checking mechanism of a theoretical focal plane depth and an actual focal plane depth is established, and picture jumping and focusing oscillation caused by depth conflicts are avoided.

Description

Technical Field

[0001] This invention relates to the field of interactive control technology, and in particular to a digital media interactive control system and method for immersive experiences. Background Technology

[0002] While immersive experience technologies are widely used, their interactive control methods still suffer from drawbacks such as low multimodal data fusion, inaccurate intent recognition, and delayed rendering response. Existing technologies typically treat body movements, eye tracking, and voice commands as independent channels, lacking deep spatiotemporal correlation modeling. This makes it difficult to accurately capture the user's true interactive intent in complex dynamic scenes. Specifically, traditional methods often use fixed spatial thresholds to filter media elements, which cannot adapt to instantaneous changes in body movement speed, easily leading to misselection or omission of targets. Simultaneously, existing trajectory prediction algorithms often ignore the fine-tuning effect of voice prosody on body movements, resulting in stiff and unnatural generated motion trajectories lacking human-like spiral enhancement features. Furthermore, in terms of camera control, existing solutions often simply overlay eye and gesture data without establishing a depth deviation verification mechanism based on geometric optics. When the user's eye focus depth is inconsistent with the hand operation depth, it can easily cause virtual camera focal plane oscillation or optical axis offset calculation errors, resulting in visual dizziness. Existing rendering strategies are mostly open-loop control, lacking dynamic gain attenuation and smooth transition mechanisms based on depth deviation. Once an interaction conflict is detected, the operation is often interrupted or an abrupt screen change occurs, which seriously damages the continuity and comfort of the immersive experience.

[0003] Therefore, it is necessary to provide a digital media interactive control system and method for immersive experiences to solve the above-mentioned technical problems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a digital media interactive control system and method for immersive experiences, achieving the beneficial effect of stable and coherent control interaction.

[0005] This invention provides a digital media interactive control method for immersive experiences, comprising:

[0006] S1: Simultaneously acquire limb key point sequences, gaze focus vector sequences, and speech audio streams; calculate instantaneous velocity vector sequences based on limb key point sequences; fit limb motion vector equations based on instantaneous velocity vector sequences; calculate variance coefficient sequences; and generate dynamic spatial domains by combining Gaussian functions.

[0007] S2: Traverse all media elements in the immersive experience scene, mark the media elements whose centroid coordinates fall within the dynamic space scope as target media elements, and encode all media elements into element enable masks based on the target media elements.

[0008] S3: Perform short-time Fourier transform on the speech audio stream to extract the fundamental frequency sequence, map the fundamental frequency sequence to a spiral motion coefficient sequence and superimpose it onto the limb motion vector equation to calculate and generate the limb enhancement trajectory sequence;

[0009] S4: Based on the limb enhancement trajectory sequence, calculate the optical axis offset and theoretical focal plane depth value of the virtual camera; based on the gaze focus vector sequence, calculate the actual focal plane depth value of the current gaze focus vector; and calculate the depth deviation based on the actual focal plane depth value and the theoretical focal plane depth value.

[0010] S5: If the absolute value of the depth deviation does not exceed the preset tolerance threshold, then a digital content rendering instruction is generated based on the optical axis offset and the theoretical depth value. If the absolute value of the depth deviation exceeds the preset tolerance threshold, then the control gain is attenuated and a smooth transition rendering instruction is generated.

[0011] Preferably, in step S1, the calculation steps of the variance coefficient sequence include:

[0012] Extract the magnitude of each instantaneous velocity vector from the instantaneous velocity vector sequence to construct a velocity amplitude sequence;

[0013] Based on the statistical dispersion characteristics of the velocity amplitude sequence, the variance coefficient sequence is calculated and normalized.

[0014] Preferably, in step S1, the step of generating the dynamic space scope includes:

[0015] The instantaneous velocity vectors in the instantaneous velocity vector sequence are normalized to obtain the unit direction vector;

[0016] Based on the variance coefficients of limb key points, the major axis standard deviation parameter is calculated through a preset nonlinear mapping function. Based on the major axis standard deviation parameter and the preset scaling factor, the minor axis standard deviation parameter is calculated.

[0017] A three-dimensional Gaussian probability density function for limb key points is constructed based on their three-dimensional spatial coordinates, unit direction vector, major axis standard deviation parameter, and minor axis standard deviation parameter at the current moment.

[0018] Extract a continuous set of spatial points from the three-dimensional Gaussian probability density function whose function value is not lower than a preset confidence threshold, and determine the closed spatial region enclosed by the continuous set of spatial points as the dynamic spatial domain of the limb key points.

[0019] Preferably, the steps for constructing the three-dimensional Gaussian probability density function of limb key points include:

[0020] The three-dimensional spatial coordinates of the limb key points at the current moment are set as the mean vector of the center point of the three-dimensional Gaussian probability density function;

[0021] The unit direction vector is set as the major axis direction of the three-dimensional Gaussian probability density function. The covariance matrix of the three-dimensional Gaussian probability density function is constructed by combining the major axis standard deviation parameter and the minor axis standard deviation parameter.

[0022] Based on the mean vector and covariance matrix of the center point, a three-dimensional Gaussian probability density function for the key points of the limb is generated.

[0023] Preferably, in step S2, the encoding step of the element enable mask includes:

[0024] Create a binary array of the same length as the total number of media elements in the scene;

[0025] The array positions corresponding to the target media elements are assigned a high level, and the array positions corresponding to non-target media elements are assigned a low level. This binary array is the element enable mask.

[0026] Preferably, in step S3, the step of generating the limb enhancement trajectory sequence includes:

[0027] Based on the fundamental frequency sequence and a preset monotonically increasing mapping function, the spiral motion coefficient sequence is calculated.

[0028] Using the normalized vector of the line-of-sight focusing vector sequence as the axis, a reference direction vector sequence for spiral motion is generated;

[0029] The spiral motion coefficients in the spiral motion coefficient sequence are used as weighting factors and multiplied by the spiral motion reference direction vector at the corresponding time in the spiral motion reference direction vector sequence to obtain the corrected vector sequence.

[0030] The modified vector sequence is added to the basic motion velocity vector at the corresponding moment in the limb motion vector equation to generate a superimposed enhanced velocity vector sequence.

[0031] By performing time integration on the superimposed enhanced velocity vector sequence in the corresponding time domain, a series of three-dimensional spatial coordinate points are obtained.

[0032] A series of three-dimensional spatial coordinate points are connected in chronological order to generate a sequence of limb enhancement trajectories.

[0033] Preferably, in step S4, the calculation of the optical axis offset includes:

[0034] Extract the real-time gaze focus vector from the gaze focus vector sequence, construct a gaze ray with the gaze origin as the origin, and calculate the vertical distance from each trajectory point in the limb enhancement trajectory sequence to the gaze ray;

[0035] The trajectory point with the smallest vertical distance is defined as the best matching point, and the vertical projection point of the best matching point on the line of sight is calculated. The three-dimensional spatial coordinates of the vertical projection point are then determined as the target interaction point.

[0036] Obtain the center coordinates of the virtual camera lens as the observation origin, and construct the intention pointing vector based on the observation origin and the target interaction point;

[0037] The angle between the intention pointing vector and the initial optical axis centerline vector of the virtual camera on the horizontal projection plane is defined as the horizontal optical axis offset.

[0038] The angle between the intention pointing vector and the initial optical axis centerline vector of the virtual camera on the vertical projection plane is defined as the vertical optical axis offset.

[0039] The horizontal and vertical optical axis offsets are combined to generate the optical axis offset of the virtual camera.

[0040] Preferably, in step S4, the theoretical focal plane depth value is obtained by calculating the magnitude of the intended pointing vector.

[0041] Preferably, in step S4, the calculation of the actual focal plane depth value includes:

[0042] Based on the geometric properties of the real-time gaze focus vector, the spatial straight-line distance from the gaze origin to the gaze convergence point is calculated;

[0043] Projecting the linear distance in space onto the optical axis of the virtual camera yields a vertical depth value relative to the lens plane, which is used as the instantaneous actual focal plane depth value at the current moment.

[0044] The instantaneous actual focal plane depth values ​​of multiple consecutive frames are then subjected to weighted average filtering to obtain the actual focal plane depth values.

[0045] This invention provides a digital media interactive control system for immersive experiences, comprising:

[0046] The multimodal perception modeling module is used to simultaneously acquire limb key point sequences, gaze focus vector sequences, and speech audio streams. It calculates instantaneous velocity vector sequences based on limb key point sequences, fits limb motion vector equations based on instantaneous velocity vector sequences, calculates variance coefficient sequences, and generates dynamic spatial domains by combining Gaussian functions.

[0047] The dynamic domain target filtering module is used to traverse all media elements in the immersive experience scene, mark media elements whose centroid coordinates fall within the dynamic space scope as target media elements, and encode all media elements into element enable masks based on the target media elements.

[0048] The speech enhancement trajectory generation module is used to extract the fundamental frequency sequence by performing short-time Fourier transform on the speech audio stream, map the fundamental frequency sequence into a spiral motion coefficient sequence and superimpose it onto the limb motion vector equation to calculate and generate the limb enhancement trajectory sequence.

[0049] The vision-motion fusion depth calculation module is used to calculate the optical axis offset and theoretical focal plane depth value of the virtual camera based on the limb enhancement trajectory sequence, calculate the actual focal plane depth value of the current gaze focus vector based on the gaze focus vector sequence, and calculate the depth deviation based on the actual focal plane depth value and the theoretical focal plane depth value.

[0050] The adaptive rendering control module is used to generate digital content rendering instructions based on the optical axis offset and the theoretical depth value if the absolute value of the depth deviation does not exceed the preset tolerance threshold, and to attenuate the control gain and generate smooth transition rendering instructions if the absolute value of the depth deviation exceeds the preset tolerance threshold.

[0051] Compared with related technologies, the digital media interactive control system and method for immersive experiences provided by this invention have the following beneficial effects:

[0052] This invention achieves adaptive scaling of the media element selection range based on user movement state by simultaneously fusing multimodal data from limbs, gaze, and speech, and dynamically constructing the domain of a Gaussian probability density function using instantaneous velocity variance. This solves the problem of misselection or omission caused by fixed thresholds. By mapping the speech fundamental frequency to spiral motion coefficients and superimposing them on the limb motion vector equation, the generated enhanced trajectory possesses both limb movement trends and speech prosody features, improving the naturalness and continuity of the trajectory. In terms of camera control, a gaze projection method is used to determine the target interaction point, that is, selecting the point on the limb trajectory closest to the gaze ray and projecting it onto the gaze, eliminating the interaction point floating problem caused by the inconsistency between eye and hand depth, and ensuring the geometric rationality of the intention direction. In addition, a deviation verification mechanism between the theoretical focal plane depth and the actual focal plane depth is established. Based on the deviation result, normal rendering instructions or smooth transition instructions with attenuated gain are dynamically selected, effectively avoiding screen jumps or focus oscillations caused by depth conflicts, and ensuring the stability and coherence of visual rendering in immersive experiences. Attached Figure Description

[0053] Figure 1 A flowchart of a digital media interactive control method for immersive experience according to the present invention;

[0054] Figure 2 This is a modular structure diagram of a digital media interactive control system for immersive experiences according to the present invention. Detailed Implementation

[0055] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the drawings, not all structures. Moreover, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0056] It should also be noted that, for ease of description, the accompanying drawings show only the parts relevant to the invention and not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. The process may correspond to a method, function, procedure, subroutine, subroutine, etc. Example

[0057] A digital media interactive control method for immersive experiences, in specific implementation, such as... Figure 1 The diagram shows a flowchart of a digital media interactive control method for immersive experiences according to the present invention, comprising:

[0058] Step S1: Simultaneously acquire the limb key point sequence, gaze focus vector sequence, and speech audio stream; calculate the instantaneous velocity vector sequence based on the limb key point sequence; fit the limb motion vector equation based on the instantaneous velocity vector sequence; calculate the variance coefficient sequence; and generate the dynamic spatial domain by combining the Gaussian function.

[0059] Specifically, in step S1, the calculation steps for the variance coefficient sequence include:

[0060] Extract the magnitude of each instantaneous velocity vector from the instantaneous velocity vector sequence to construct a velocity amplitude sequence;

[0061] Based on the statistical dispersion characteristics of the velocity amplitude sequence, the variance coefficient sequence is calculated and normalized.

[0062] Specifically, in step S1, the steps for generating the dynamic space scope include:

[0063] The instantaneous velocity vectors in the instantaneous velocity vector sequence are normalized to obtain the unit direction vector;

[0064] Based on the variance coefficients of limb key points, the major axis standard deviation parameter is calculated through a preset nonlinear mapping function. Based on the major axis standard deviation parameter and the preset scaling factor, the minor axis standard deviation parameter is calculated.

[0065] A three-dimensional Gaussian probability density function for limb key points is constructed based on their three-dimensional spatial coordinates, unit direction vector, major axis standard deviation parameter, and minor axis standard deviation parameter at the current moment.

[0066] Extract a continuous set of spatial points from the three-dimensional Gaussian probability density function whose function value is not lower than a preset confidence threshold, and determine the closed spatial region enclosed by the continuous set of spatial points as the dynamic spatial domain of the limb key points.

[0067] Specifically, the steps for constructing the three-dimensional Gaussian probability density function of limb key points include:

[0068] The three-dimensional spatial coordinates of the limb key points at the current moment are set as the mean vector of the center point of the three-dimensional Gaussian probability density function;

[0069] The unit direction vector is set as the major axis direction of the three-dimensional Gaussian probability density function. The covariance matrix of the three-dimensional Gaussian probability density function is constructed by combining the major axis standard deviation parameter and the minor axis standard deviation parameter.

[0070] Based on the mean vector and covariance matrix of the center point, a three-dimensional Gaussian probability density function for the key points of the limb is generated.

[0071] In the specific implementation process, for example, the multimodal data acquisition process is first initiated, controlling the depth camera to capture the three-dimensional coordinate sequence of the user's limb key points in real time at a frame rate of 30Hz. Simultaneously, the gaze focus unit vector sequence output in real time from the eye tracker and the audio stream recorded in real time by the microphone array are received in parallel. Based on a unified time reference, the high-sampling-frequency gaze focus unit vector sequence and audio stream are timestamped and data is extracted at each 30Hz time node, thereby generating a strictly synchronized three-dimensional coordinate sequence of limb key points, gaze focus unit vector sequence, and audio feature segments in the time dimension, which serve as input data for subsequent steps. Next, the difference between the coordinates of limb key points in two adjacent frames is calculated and divided by the time interval to obtain the instantaneous velocity vector sequence. Simultaneously, the ratio of the standard deviation to the mean of the magnitude of each instantaneous velocity vector in the instantaneous velocity vector sequence is calculated, and after normalization, a variance coefficient sequence is obtained. Subsequently... The instantaneous velocity vector sequence is normalized by dividing each vector by its magnitude to obtain a unit direction vector. A pre-defined quadratic nonlinear mapping function is used to map the variance coefficients to the major axis standard deviation parameter, which is then multiplied by a fixed scaling factor to obtain the minor axis standard deviation parameter. Next, using the current limb keypoint's three-dimensional spatial coordinates as the center point mean vector and the unit direction vector as the major axis direction, a covariance matrix is ​​constructed by combining the calculated major and minor axis standard deviation parameters, thereby generating a three-dimensional Gaussian probability density function. Finally, a continuous set of spatial points with function values ​​greater than or equal to a pre-defined confidence threshold is extracted from the three-dimensional Gaussian probability density function. The closed ellipsoidal region enclosed by this set of points is defined as the current dynamic spatial domain. The shape and size of the dynamic spatial domain are strictly determined by the user's real-time movement speed and dispersion, providing a definite geometric boundary basis for subsequent steps to accurately select target media elements falling within this specific spatial range.

[0072] Step S2: Traverse all media elements in the immersive experience scene, mark the media elements whose centroid coordinates fall within the dynamic space scope as target media elements, and encode all media elements into element enable masks based on the target media elements.

[0073] Specifically, in step S2, the encoding steps of the element enable mask include:

[0074] Create a binary array of the same length as the total number of media elements in the scene;

[0075] The array positions corresponding to the target media elements are assigned a high level, and the array positions corresponding to non-target media elements are assigned a low level. This binary array is the element enable mask.

[0076] In the specific implementation process, the boundary data of the three-dimensional closed region of the dynamic spatial domain is first read. Then, all media element objects registered in the immersive experience scene rendering engine are traversed. For each media element, the coordinates of its predefined geometric bounding box center point are extracted as centroid coordinates. Using algorithms including but not limited to ray casting and symbolic distance field determination algorithms, the centroid coordinates are accurately calculated to determine whether they are located inside the closed boundary of the dynamic spatial domain. If the centroid coordinates are inside the boundary, the media element is marked as a target media element; if the centroid coordinates are outside the boundary, it remains a non-target media element. After traversing and marking all media elements, a media element with a length equal to that in the scene is created. The system generates a one-dimensional binary array containing all media elements. The index positions of the array correspond one-to-one with the unique identifier ID of each media element. The system assigns a logic high level (value 1) to the index positions of the media elements that have been marked as target media elements, and assigns a logic low level (value 0) to the index positions of the media elements that are not target media elements. The resulting binary array is the element enable mask. This mask is directly used in the subsequent rendering pipeline to control the visibility and interactive activation state of media elements, ensuring that only media elements whose spatial location falls within the user's limb dynamic scope can be identified and responded to. This achieves deterministic filtering and digital encoding based on spatial geometric inclusion relationships.

[0077] Step S3: Perform short-time Fourier transform on the speech audio stream to extract the fundamental frequency sequence, map the fundamental frequency sequence to a spiral motion coefficient sequence and superimpose it onto the limb motion vector equation to calculate and generate the limb enhancement trajectory sequence.

[0078] Specifically, in step S3, the steps for generating the limb enhancement trajectory sequence include:

[0079] Based on the fundamental frequency sequence and a preset monotonically increasing mapping function, the spiral motion coefficient sequence is calculated.

[0080] Using the normalized vector of the line-of-sight focusing vector sequence as the axis, a reference direction vector sequence for spiral motion is generated;

[0081] The spiral motion coefficients in the spiral motion coefficient sequence are used as weighting factors and multiplied by the spiral motion reference direction vector at the corresponding time in the spiral motion reference direction vector sequence to obtain the corrected vector sequence.

[0082] The modified vector sequence is added to the basic motion velocity vector at the corresponding moment in the limb motion vector equation to generate a superimposed enhanced velocity vector sequence.

[0083] By performing time integration on the superimposed enhanced velocity vector sequence in the corresponding time domain, a series of three-dimensional spatial coordinate points are obtained.

[0084] A series of three-dimensional spatial coordinate points are connected in chronological order to generate a sequence of limb enhancement trajectories.

[0085] In the specific implementation process, firstly, a short-time Fourier transform is performed on the audio stream to extract the fundamental frequency values ​​corresponding to the signal energy peaks frame by frame and arrange them in chronological order to form a fundamental frequency sequence. Then, a preset monotonically increasing mapping function is called to calculate the fundamental frequency sequence item by item, uniquely converting each fundamental frequency value into a positive scalar to generate a spiral motion coefficient sequence. Specifically, the preset monotonically increasing mapping function is a scaling structure that binds the audio fundamental frequency value range to the spiral motion coefficient value range in a one-to-one correspondence. The determination process of this structure is based on the physiological characteristics of human vocalization to define an effective audio fundamental frequency input range, setting 80Hz as the mapping start point and 400Hz as the mapping end point, while also considering the visual comfort standards of immersive scenes. A safe spiral motion intensity output range is defined, with 0 set as the output starting point and 1.5 set as the output ending point. A fixed proportional coefficient is calculated by dividing the total span of the output range by the total span of the input range. The total span of the output range is the difference between the minimum and maximum values ​​of the spiral motion intensity output range, and the total span of the input range is the effective voice fundamental frequency input range. In specific calculations, the instantaneous fundamental frequency value of the voice signal is collected in real time. First, an 80Hz starting offset is subtracted to obtain the relative frequency value. Then, the relative frequency value is multiplied by the fixed proportional coefficient to directly calculate the unique spiral motion coefficient value. If the collected fundamental frequency value is lower than 80Hz, a static state of 0 is forced to be output; if the collected fundamental frequency value is higher than 400Hz, a 1 is forced to be output.The full-amplitude spiral state of 5 ensures that every slight increase in the user's voice pitch is precisely converted into a proportional increase in the spiral motion coefficient, eliminating any nonlinear abrupt changes and random jitter. This achieves a strictly monotonically increasing deterministic correspondence between voice control and trajectory morphology. Next, the synchronously acquired gaze focus vector sequence is read, and each gaze focus vector in the sequence is normalized to obtain a normalized vector of unit length. These normalized vectors are arranged in chronological order to form a spiral motion reference direction vector sequence, which defines the central axis of the spiral motion. Subsequently, each spiral motion coefficient in the spiral motion coefficient sequence is used as a weighting factor, and is respectively correlated with the spiral motion coefficient at the same moment in the spiral motion reference direction vector sequence. The reference direction vector of the rotational motion is multiplied by scalar to generate a corrected vector sequence with the same time length, direction along the line of sight axis, and modulus varying with the fundamental frequency of the speech. Next, the basic motion velocity vector output by the limb motion vector equation at the corresponding moment is extracted. The corrected vectors in the corrected vector sequence are added to the basic motion velocity vector to synthesize a superimposed enhanced velocity vector sequence. This superimposed enhanced velocity vector sequence integrates the original limb motion trend and the helical component driven by speech. Time integration is performed on the superimposed enhanced velocity vector sequence in a continuous time domain, accumulating the velocity vectors into a series of continuous three-dimensional spatial coordinate points. Finally, these three-dimensional spatial coordinate points are connected sequentially according to time to form a smooth and continuous three-dimensional curve, i.e., the limb enhanced trajectory sequence.

[0086] Step S4: Based on the limb enhancement trajectory sequence, calculate the optical axis offset and theoretical focal plane depth value of the virtual camera; based on the gaze focus vector sequence, calculate the actual focal plane depth value of the current gaze focus vector; and calculate the depth deviation based on the actual focal plane depth value and the theoretical focal plane depth value.

[0087] Specifically, in step S4, the calculation steps for the optical axis offset include:

[0088] Extract the real-time gaze focus vector from the gaze focus vector sequence, construct a gaze ray with the gaze origin as the origin, and calculate the vertical distance from each trajectory point in the limb enhancement trajectory sequence to the gaze ray;

[0089] The trajectory point with the smallest vertical distance is defined as the best matching point, and the vertical projection point of the best matching point on the line of sight is calculated. The three-dimensional spatial coordinates of the vertical projection point are then determined as the target interaction point.

[0090] Obtain the center coordinates of the virtual camera lens as the observation origin, and construct the intention pointing vector based on the observation origin and the target interaction point;

[0091] The angle between the intention pointing vector and the initial optical axis centerline vector of the virtual camera on the horizontal projection plane is defined as the horizontal optical axis offset.

[0092] The angle between the intention pointing vector and the initial optical axis centerline vector of the virtual camera on the vertical projection plane is defined as the vertical optical axis offset.

[0093] The horizontal and vertical optical axis offsets are combined to generate the optical axis offset of the virtual camera.

[0094] Specifically, in step S4, the theoretical focal plane depth value is obtained by calculating the magnitude of the intended pointing vector.

[0095] Specifically, in step S4, the calculation steps for the actual focal plane depth value include:

[0096] Based on the geometric properties of the real-time gaze focus vector, the spatial straight-line distance from the gaze origin to the gaze convergence point is calculated;

[0097] Projecting the linear distance in space onto the optical axis of the virtual camera yields a vertical depth value relative to the lens plane, which is used as the instantaneous actual focal plane depth value at the current moment.

[0098] The instantaneous actual focal plane depth values ​​of multiple consecutive frames are then subjected to weighted average filtering to obtain the actual focal plane depth values.

[0099] In the specific implementation process, the real-time gaze focus vector is first extracted from the gaze focus vector sequence. This gaze focus vector is generated by weighted fusion of the user's left-eye and right-eye gaze vectors. The gaze focus vector serves as intermediate data. For example, the starting endpoint of the real-time gaze focus vector is used as the gaze starting point, and an infinitely extending gaze ray is constructed along the direction of the real-time gaze focus vector. Then, all trajectory points in the limb enhancement trajectory sequence are traversed, and the vertical Euclidean distance from each trajectory point to the gaze ray is calculated. The trajectory point with the smallest calculated vertical Euclidean distance is defined as the optimal matching point. This optimal matching point represents the spatial relationship between the user's limb movement trajectory and the current gaze direction. The closest intersection point is found, and the best matching point is projected onto the line of sight along a direction perpendicular to the line of sight, resulting in a vertical projection point on the line of sight. The three-dimensional spatial coordinates of this vertical projection point are determined as the target interaction point. Then, the coordinates of the virtual camera's lens center are obtained and set as the observation origin. Based on the observation origin and the target interaction point, an intention pointing vector is constructed from the observation origin to the target interaction point. This intention pointing vector represents the direction of the user's actual interaction intention as determined by the system. This intention pointing vector and the virtual camera's preset initial optical axis centerline vector are projected onto the horizontal reference plane, and the angle between the two projection vectors is calculated and defined as the horizontal optical axis offset. Simultaneously, the... The intention pointing vector and the initial optical axis centerline vector are projected onto the vertical reference plane respectively. The angle between the two projected vectors is calculated and defined as the vertical optical axis offset. Finally, the horizontal and vertical optical axis offsets are combined to generate the final virtual camera optical axis offset, which is used to control the camera's pan-tilt rotation. Next, the geometric magnitude of the intention pointing vector is directly calculated, and this magnitude value is directly determined as the theoretical focal plane depth value. The theoretical focal plane depth value represents the ideal focusing distance derived from the limb enhancement trajectory. Simultaneously, based on the left and right eye gaze vectors and the convergence relationship of the binocular gaze, triangulation is performed to calculate the spatial intersection point of the binocular gaze as the gaze convergence point. This involves calculating the spatial straight-line distance from the starting point of the gaze to the convergence point, constructing a gaze convergence vector pointing from the starting point to the convergence point, calculating the projection component of this gaze convergence vector in the unit vector direction of the current virtual camera's optical axis, determining the magnitude of this projection component as the vertical depth value relative to the lens plane, and using it as the instantaneous actual focal plane depth value at the current moment. To eliminate jitter in the eye-tracking data, a weighted average filter is applied to the instantaneous actual focal plane depth values ​​of multiple consecutive historical frames, and the filtered output is determined as the actual focal plane depth value at the current moment. Finally, the numerical difference between the actual focal plane depth value and the theoretical focal plane depth value is calculated, and this difference is defined as the depth deviation.

[0100] Step S5: If the absolute value of the depth deviation does not exceed the preset tolerance threshold, then generate digital content rendering instructions based on the optical axis offset and the theoretical depth value. If the absolute value of the depth deviation exceeds the preset tolerance threshold, then attenuate the control gain and generate smooth transition rendering instructions.

[0101] In the specific implementation process, the absolute value of the depth deviation is first compared with the preset depth tolerance threshold. If the absolute value of the depth deviation is not greater than the preset depth tolerance threshold, it is determined that the current user's gaze focus and the limb enhancement trajectory are highly overlapping in space, indicating a clear and stable interaction intention. A high-level enable signal is immediately generated and output to the rendering control unit, triggering a digital content rendering instruction. This instruction directly calls the optical axis offset to drive the virtual camera's pan-tilt unit to rotate and directly calls the theoretical focal plane depth value to set the virtual camera's focus distance. Simultaneously, the control gain is kept at full value. Here, the control gain is defined as the output of the rendering control unit. The unit displacement command is converted into a magnification factor of the virtual camera's physical movement speed. For example, the full-scale state indicates that the magnification factor is 100%, enabling the virtual camera to follow the user's intention in real time with maximum response speed. If the absolute value of the depth deviation is greater than the depth tolerance threshold, it indicates that there is a significant spatial depth separation between the current user's gaze focus and limb movement trajectory, and there is conflict and uncertainty in the interaction intention. A low-level disable signal is immediately generated and output to the rendering control unit, triggering a smooth transition rendering command. This smooth transition rendering command first performs a control gain attenuation operation, which is defined as gradually reducing the magnification factor according to a preset time step. The system calculates the minimum magnification factor (MPF) of the current camera. It then gradually reduces the lens movement speed gain and focus response gain from full capacity to a preset safety threshold using a linear decay rate. This safety threshold is a minimum MPF greater than 0, used to suppress severe camera shake and prevent sudden image changes. Next, it initiates a focus smoothing interpolation algorithm. This algorithm reads the historical focus distance values ​​used by the virtual camera in the previous frame, using these historical focus distance values ​​as the interpolation starting point and the theoretical focal plane depth value as the interpolation ending point. Based on the current frame's time step and the decayed MPF, it calculates the intermediate transition focus distance value for the current frame and sets this intermediate transition focus distance value as the actual MPF value for the virtual camera. By adjusting the focus distance, the virtual camera's focal plane is forced to move slowly and smoothly linearly between its historical position and the theoretical target position during periods of unstable depth deviation. This avoids instantaneous blurring or visual dizziness caused by sudden changes in focus distance. The above interval judgment and magnification adjustment process is repeated continuously in each rendering cycle until the depth deviation value returns to a continuous closed value range determined by the preset tolerance threshold. At this point, the system automatically cancels the reduction in magnification and restores the control gain to its full value, completing a seamless switch from smooth transition mode to precise focus mode. Finally, a series of visually continuous and intuitively user-friendly immersive scene images are output.

[0102] The working principle of the digital media interactive control method for immersive experiences provided by this invention is as follows:

[0103] This invention achieves adaptive and precise control of a virtual camera through multimodal data fusion. First, it simultaneously collects user limb, gaze, and voice data. Limb velocity is used to fit motion equations, and a dynamic Gaussian spatial domain is constructed using variance coefficients to filter out the target media elements the user intends to point to. Next, the fundamental frequency of the voice is extracted through spectral analysis, mapped to spiral motion coefficients, and superimposed on the limb motion vector to generate an enhanced limb trajectory incorporating emotional features of the voice. Then, based on this trajectory, the theoretical optical axis offset and theoretical focal plane depth of the virtual camera are calculated, while the actual focal plane depth of the gaze focus is calculated independently. The consistency of the interaction intention is judged by comparing the deviation between the theoretical and actual depths. If the depth deviation is within a preset tolerance range, the camera is directly driven to perform precise focusing and rendering based on the theoretical parameters. If the deviation exceeds the tolerance, the gain is automatically attenuated, and a smooth transition algorithm is activated, forcing the camera to interpolate between its historical position and the theoretical target. This avoids sudden changes in the image and dizziness when the user's limb and gaze focus are inconsistent, ultimately achieving a visually continuous, responsive, and intuitively intuitive immersive interactive experience. Example

[0104] A digital media interactive control system for immersive experiences, in its specific implementation, such as... Figure 2 As shown, it illustrates a modular structure diagram of a digital media interactive control system for immersive experiences according to the present invention, comprising:

[0105] The multimodal perception modeling module 100 is used to simultaneously acquire limb key point sequences, gaze focus vector sequences, and speech audio streams. It calculates instantaneous velocity vector sequences based on limb key point sequences, fits limb motion vector equations based on instantaneous velocity vector sequences, calculates variance coefficient sequences, and generates dynamic spatial domains by combining Gaussian functions.

[0106] The dynamic domain target filtering module 200 is used to traverse each media element in the immersive experience scene, mark the media elements whose centroid coordinates fall within the dynamic space scope as target media elements, and encode all media elements into element enable masks based on the target media elements.

[0107] The speech enhancement trajectory generation module 300 is used to extract the fundamental frequency sequence by performing short-time Fourier transform on the speech audio stream, map the fundamental frequency sequence into a spiral motion coefficient sequence and superimpose it onto the limb motion vector equation to calculate and generate the limb enhancement trajectory sequence.

[0108] The vision-motion fusion depth calculation module 400 is used to calculate the optical axis offset and theoretical focal plane depth value of the virtual camera based on the limb enhancement trajectory sequence, calculate the actual focal plane depth value of the current gaze focus vector based on the gaze focus vector sequence, and calculate the depth deviation based on the actual focal plane depth value and the theoretical focal plane depth value.

[0109] The adaptive rendering control module 500 is used to generate digital content rendering instructions based on the optical axis offset and the theoretical depth value if the absolute value of the depth deviation does not exceed the preset tolerance threshold, and to attenuate the control gain and generate smooth transition rendering instructions if the absolute value of the depth deviation exceeds the preset tolerance threshold.

[0110] The working principle of the digital media interactive control system for immersive experiences provided by this invention is as follows:

[0111] This invention first utilizes a multimodal perception modeling module 100 to simultaneously collect user limb, gaze, and voice data. Based on limb velocity, it fits motion equations and constructs a dynamic Gaussian spatial domain using variance coefficients. Next, a dynamic domain target filtering module 200 traverses the scene, marking media elements falling within this domain as targets and generating enabling masks. Subsequently, a voice enhancement trajectory generation module 300 performs spectral analysis on the voice to extract the fundamental frequency, mapping it to spiral motion coefficients superimposed on the limb motion vector to calculate a limb enhancement trajectory incorporating emotional features. A vision-motion fusion depth calculation module 400 calculates the theoretical optical axis offset and theoretical focal plane depth of the virtual camera, while independently calculating the actual focal plane depth of gaze focusing and the deviation between the two. Finally, an adaptive rendering control module 500 executes decision logic based on this deviation value: if the deviation is within a preset tolerance, it directly drives the camera to precisely focus and render; if the deviation exceeds the tolerance, it automatically attenuates the control gain and initiates a smooth transition algorithm, forcing the camera to interpolate between its historical position and the theoretical target, thereby avoiding sudden changes in the scene when interactive intentions conflict, ultimately outputting a visually continuous and intuitive immersive scene.

[0112] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0114] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A digital media interactive control method for immersive experiences, characterized in that, The interactive control method includes: S1: Simultaneously acquire limb key point sequences, gaze focus vector sequences, and speech audio streams; calculate instantaneous velocity vector sequences based on limb key point sequences; fit limb motion vector equations based on instantaneous velocity vector sequences; calculate variance coefficient sequences; and generate dynamic spatial domains by combining Gaussian functions. S2: Traverse all media elements in the immersive experience scene, mark the media elements whose centroid coordinates fall within the dynamic space scope as target media elements, and encode all media elements into element enable masks based on the target media elements. S3: Perform short-time Fourier transform on the speech audio stream to extract the fundamental frequency sequence, map the fundamental frequency sequence to a spiral motion coefficient sequence and superimpose it onto the limb motion vector equation to calculate and generate the limb enhancement trajectory sequence; S4: Based on the limb enhancement trajectory sequence, calculate the optical axis offset and theoretical focal plane depth value of the virtual camera; based on the gaze focus vector sequence, calculate the actual focal plane depth value of the current gaze focus vector; and calculate the depth deviation based on the actual focal plane depth value and the theoretical focal plane depth value. S5: If the absolute value of the depth deviation does not exceed the preset tolerance threshold, then a digital content rendering instruction is generated based on the optical axis offset and the theoretical depth value. If the absolute value of the depth deviation exceeds the preset tolerance threshold, then the control gain is attenuated and a smooth transition rendering instruction is generated.

2. The digital media interactive control method for immersive experience according to claim 1, characterized in that, In step S1, the calculation steps for the variance coefficient sequence include: Extract the magnitude of each instantaneous velocity vector from the instantaneous velocity vector sequence to construct a velocity amplitude sequence; Based on the statistical dispersion characteristics of the velocity amplitude sequence, the variance coefficient sequence is calculated and normalized.

3. The digital media interactive control method for immersive experience according to claim 2, characterized in that, In step S1, the steps for generating the dynamic space scope include: The instantaneous velocity vectors in the instantaneous velocity vector sequence are normalized to obtain the unit direction vector; Based on the variance coefficients of limb key points, the major axis standard deviation parameter is calculated through a preset nonlinear mapping function. Based on the major axis standard deviation parameter and the preset scaling factor, the minor axis standard deviation parameter is calculated. A three-dimensional Gaussian probability density function for limb key points is constructed based on their three-dimensional spatial coordinates, unit direction vector, major axis standard deviation parameter, and minor axis standard deviation parameter at the current moment. Extract a continuous set of spatial points from the three-dimensional Gaussian probability density function whose function value is not lower than a preset confidence threshold, and determine the closed spatial region enclosed by the continuous set of spatial points as the dynamic spatial domain of the limb key points.

4. The digital media interactive control method for immersive experiences according to claim 3, characterized in that, The steps for constructing the three-dimensional Gaussian probability density function of limb key points include: The three-dimensional spatial coordinates of the limb key points at the current moment are set as the mean vector of the center point of the three-dimensional Gaussian probability density function; The unit direction vector is set as the major axis direction of the three-dimensional Gaussian probability density function. The covariance matrix of the three-dimensional Gaussian probability density function is constructed by combining the major axis standard deviation parameter and the minor axis standard deviation parameter. Based on the mean vector and covariance matrix of the center point, a three-dimensional Gaussian probability density function for the key points of the limb is generated.

5. The digital media interactive control method for immersive experiences according to claim 4, characterized in that, In step S2, the encoding steps of the element enable mask include: Create a binary array of the same length as the total number of media elements in the scene; The array positions corresponding to the target media elements are assigned a high level, and the array positions corresponding to non-target media elements are assigned a low level. This binary array is the element enable mask.

6. The digital media interactive control method for immersive experiences according to claim 5, characterized in that, In step S3, the steps for generating the limb enhancement trajectory sequence include: Based on the fundamental frequency sequence and a preset monotonically increasing mapping function, the spiral motion coefficient sequence is calculated. Using the normalized vector of the line-of-sight focusing vector sequence as the axis, a reference direction vector sequence for spiral motion is generated; The spiral motion coefficients in the spiral motion coefficient sequence are used as weighting factors and multiplied by the spiral motion reference direction vector at the corresponding time in the spiral motion reference direction vector sequence to obtain the corrected vector sequence. The modified vector sequence is added to the basic motion velocity vector at the corresponding moment in the limb motion vector equation to generate a superimposed enhanced velocity vector sequence. By performing time integration on the superimposed enhanced velocity vector sequence in the corresponding time domain, a series of three-dimensional spatial coordinate points are obtained. A series of three-dimensional spatial coordinate points are connected in chronological order to generate a sequence of limb enhancement trajectories.

7. The digital media interactive control method for immersive experiences according to claim 6, characterized in that, In step S4, the calculation steps for the optical axis offset include: Extract the real-time gaze focus vector from the gaze focus vector sequence, construct a gaze ray with the gaze origin as the origin, and calculate the vertical distance from each trajectory point in the limb enhancement trajectory sequence to the gaze ray; The trajectory point with the smallest vertical distance is defined as the best matching point, and the vertical projection point of the best matching point on the line of sight is calculated. The three-dimensional spatial coordinates of the vertical projection point are then determined as the target interaction point. Obtain the center coordinates of the virtual camera lens as the observation origin, and construct the intention pointing vector based on the observation origin and the target interaction point; The angle between the intention pointing vector and the initial optical axis centerline vector of the virtual camera on the horizontal projection plane is defined as the horizontal optical axis offset. The angle between the intention pointing vector and the initial optical axis centerline vector of the virtual camera on the vertical projection plane is defined as the vertical optical axis offset. The horizontal and vertical optical axis offsets are combined to generate the optical axis offset of the virtual camera.

8. A digital media interactive control method for immersive experiences according to claim 7, characterized in that, In step S4, the theoretical focal plane depth value is obtained by calculating the magnitude of the intended pointing vector.

9. A digital media interactive control method for immersive experiences according to claim 8, characterized in that, In step S4, the calculation steps for the actual focal plane depth value include: Based on the geometric properties of the real-time gaze focus vector, the spatial straight-line distance from the gaze origin to the gaze convergence point is calculated; Projecting the linear distance in space onto the optical axis of the virtual camera yields a vertical depth value relative to the lens plane, which is used as the instantaneous actual focal plane depth value at the current moment. The instantaneous actual focal plane depth values ​​of multiple consecutive frames are then subjected to weighted average filtering to obtain the actual focal plane depth values.

10. A digital media interactive control system for immersive experiences, characterized in that, An interactive control method for digital media aimed at immersive experiences, as described in any one of claims 1 to 9, wherein the interactive control system comprises: The multimodal perception modeling module is used to simultaneously acquire limb key point sequences, gaze focus vector sequences, and speech audio streams. It calculates instantaneous velocity vector sequences based on limb key point sequences, fits limb motion vector equations based on instantaneous velocity vector sequences, calculates variance coefficient sequences, and generates dynamic spatial domains by combining Gaussian functions. The dynamic domain target filtering module is used to traverse all media elements in the immersive experience scene, mark media elements whose centroid coordinates fall within the dynamic space scope as target media elements, and encode all media elements into element enable masks based on the target media elements. The speech enhancement trajectory generation module is used to extract the fundamental frequency sequence by performing short-time Fourier transform on the speech audio stream, map the fundamental frequency sequence into a spiral motion coefficient sequence and superimpose it onto the limb motion vector equation to calculate and generate the limb enhancement trajectory sequence. The vision-motion fusion depth calculation module is used to calculate the optical axis offset and theoretical focal plane depth value of the virtual camera based on the limb enhancement trajectory sequence, calculate the actual focal plane depth value of the current gaze focus vector based on the gaze focus vector sequence, and calculate the depth deviation based on the actual focal plane depth value and the theoretical focal plane depth value. The adaptive rendering control module is used to generate digital content rendering instructions based on the optical axis offset and the theoretical depth value if the absolute value of the depth deviation does not exceed the preset tolerance threshold, and to attenuate the control gain and generate smooth transition rendering instructions if the absolute value of the depth deviation exceeds the preset tolerance threshold.

Images