A synchronous interaction system of an immersive song performance special effect

Abstract

The application relates to the technical field of human-computer interaction, in particular to an immersive song performance special effect synchronous interaction system, which comprises an interaction collecting unit, which collects the interaction action of a user, determines the actual interaction time corresponding to the interaction action, and synchronously collects the environment parameter corresponding to the interaction action. A background interference evaluation unit is used for determining the background interference index of the current interaction action according to the deviation between the real-time collection value of the environment parameter and the predicted value of the environment parameter. A special effect display decision unit is used for determining the special effect display time and the special effect display type according to the comparison result of the background interference index and the preset interference threshold value and the position of the actual interaction time relative to the personalized allowable interval. A special effect display unit is used for displaying the corresponding special effect according to the special effect display time and the special effect display type. In this way, the experience interruption caused by the rush or delay of shooting is avoided, so that the continuity and reality of the immersive interaction are enhanced.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, specifically to a synchronous interactive system for immersive song performance effects. Background Technology

[0002] With the rapid development of metaverse and immersive interactive technologies, synchronous interactive systems based on song performance are becoming increasingly popular. These systems typically center around a song, combining elements such as sound, rhythm, visual effects, and user interaction to create an immersive experience where the user is drawn into the song's context. During user interaction with the system, the system needs to display corresponding dynamic feedback effects such as light effects, particles, and sound effects in real time at specific musical beats or when user actions occur. The synchronicity between these effects and user actions is a key factor determining the quality of the immersive experience.

[0003] In existing technologies, synchronous interactive systems typically collect user interaction actions and immediately trigger corresponding visual or auditory effects upon detecting the action. These methods rely on uniform, fixed synchronization evaluation criteria, such as setting a fixed time window. If the user's interaction occurs within this window near the expected beat, the synchronous interactive system considers the interaction valid and displays the effect. However, due to differences in physiological reaction speed and subjective perception of rhythm among different users, and the potential for objective interference factors such as ambient light changes and sensor noise during interaction, fixed synchronization criteria cannot adapt to the individual characteristics of all users. This leads to a deviation between the effect display time and the user's subjective expectation of synchronization, causing some users to frequently experience out-of-sync experiences, such as premature or delayed beats, thus reducing the overall immersive experience. Summary of the Invention

[0004] To address the technical problem of reduced overall immersive experience caused by frequent mismatches (rushing or lagging) when using synchronous interactive systems, this application aims to provide a synchronous interactive system for immersive song performance effects. The specific technical solution adopted is as follows: A synchronized interactive system for immersive song performance effects is provided, including: The pre-calibration unit is used to determine the user's personalized tolerance range based on the time deviation between the actual interaction time of the user's interactive action in response to the preset calibration audio signal and the preset expected special effect display time in the preset calibration audio signal.

[0005] The interactive acquisition unit is used to collect user interaction actions during immersive song performance, determine the actual interaction time corresponding to the interaction action, and simultaneously collect environmental parameters corresponding to the interaction action.

[0006] The background interference assessment unit is used to determine the background interference index of the current interactive action based on the deviation between the real-time collected values ​​of environmental parameters and the predicted values ​​of environmental parameters.

[0007] The special effects display decision unit is used to determine the timing and type of special effects display based on the comparison results between the background interference index and the preset interference threshold, as well as the position of the actual interaction time relative to the personalized allowable range.

[0008] The special effects display unit is used to display special effects corresponding to the special effects display type, based on the special effects display time and special effects display type.

[0009] In one design, the pre-calibration unit is specifically used for: It provides preset calibration audio signals containing multiple different beat intervals, including at least a low-speed beat interval and a high-speed beat interval. The total duration of the preset calibration audio signals is less than a preset duration threshold, and no feedback effects corresponding to the interactive actions are displayed during the calibration process.

[0010] Collect the actual interaction time of the user in each beat interval, and determine the time deviation between the actual interaction time and the corresponding expected special effect display time.

[0011] The interaction response stability index is determined by the absolute value of the difference between the average absolute value of the time deviation corresponding to the low-speed beat interval and the average absolute value of the time deviation corresponding to the high-speed beat interval.

[0012] The personalized allowable range is determined based on the average of all valid time deviations, the interaction response stability index, and the preset basic time tolerance.

[0013] In one design, the background interference evaluation unit is specifically used for: During the immersive song performance, the environmental parameter sequence within a preset time period before the current interactive action is obtained according to a preset sampling frequency. The preset sampling frequency is higher than the sampling frequency of the interactive action.

[0014] Based on historical data from local windows in the environmental parameter sequence, the predicted values ​​of the current environmental parameters are determined using a preset smoothing prediction algorithm.

[0015] Based on the deviation between the real-time collected values ​​and predicted values ​​of environmental parameters, and combined with the background interference weights and dimensionless removal constants corresponding to different interaction types, the background interference index is determined. The dimensionless removal constant is used to convert environmental parameters of different dimensions into dimensionless quantities.

[0016] In one design, the environmental parameters include: the ambient light signal sequence collected when the interaction action type is a visual action, the interaction coordinate sequence collected when the interaction action type is a screen touch action, and the background noise energy change sequence collected when the interaction action type is a sound action.

[0017] Different types of interactive actions correspond to different background interference weights, with visual actions having a higher background interference weight than screen touch actions and sound actions.

[0018] In one design, the special effects display decision unit is specifically used for: Determine the effective interaction range centered on the expected moment of special effects display.

[0019] Based on the positional relationship between the actual interaction time and the effective interaction interval, the number of interactions identified within the effective interaction interval, and the relationship between the background interference index and the preset interference threshold, the timing and type of special effects display are determined.

[0020] The special effects display types include positive feedback effects, negative feedback effects, transition effects, and dynamic feedback effects. Dynamic feedback effects are generated based on the mixing ratio of positive feedback effects and negative feedback effects.

[0021] In one design, when the background interference index is greater than a preset interference threshold, the special effects display decision unit is used to determine whether the actual interaction moment is within the effective interaction range.

[0022] When the actual interaction time is within the effective interaction interval and there is only one interaction within the effective interaction interval, the special effects display decision unit is also used to determine the relative position between the actual interaction time and the expected special effects display time.

[0023] When the actual interaction time is before the expected special effect display time, the special effect display time is determined as the expected special effect display time, and the special effect display type is determined as positive feedback special effect.

[0024] When the actual interaction time is after the expected effect display time, the effect display time is determined as the actual interaction time, and the effect display type is determined as positive feedback effect.

[0025] In one design, when the background interference index is less than or equal to a preset interference threshold, the special effects display decision unit is used to determine whether the actual interaction time is within the effective interaction range and whether there is only one interaction within the effective interaction range.

[0026] When the judgment result is yes, it is also used to determine the tolerance range of positive feedback special effects centered on the expected special effect display time, based on the personalized tolerance range.

[0027] When the actual interaction time is within the tolerance range of the positive feedback effect, the positive feedback effect will be displayed at the actual interaction time.

[0028] When the actual interaction occurs before the tolerance range of the positive feedback effect, a transition effect is displayed from the actual interaction to the start of the tolerance range of the positive feedback effect, and a dynamic feedback effect is displayed at the start of the transition.

[0029] When the actual interaction time falls within the tolerance range of the positive feedback effect, determine whether to display the dynamic feedback effect at the actual interaction time.

[0030] The special effects display decision unit is also used to determine the distance between the actual interaction moment and the nearest boundary in the tolerance range of positive feedback special effects. Based on the ratio of the distance to the interval length of the tolerance range of positive feedback special effects, the mixing ratio of positive feedback special effects and negative feedback special effects in dynamic feedback special effects is determined.

[0031] The special effects display unit is also used to determine the blur parameter of the dynamic feedback special effects based on the background interference index. The blur parameter is positively correlated with the background interference index and is used to adjust the sharpness intensity of the dynamic feedback special effects.

[0032] In one design, positive feedback effects are used to characterize the feedback performance when the user interaction meets the synchronization requirements.

[0033] Negative feedback effects are used to characterize the feedback behavior when the user interaction deviates from the synchronization requirements.

[0034] Transition effects are used to represent the smooth transition when there is a time difference between the interactive action and the effect display.

[0035] Positive feedback effects include crisp sound effects and bright particles, while negative feedback effects include dull sound effects and dark smoke.

[0036] In one design, the interactive acquisition unit includes a visual sensor, a screen touch sensor, and a voice acquisition device.

[0037] Interactive actions include visual actions, screen touch actions, and vocal actions.

[0038] Environmental parameters include ambient light signal sequences, interaction coordinate sequences, and background noise energy change sequences corresponding to the interaction action type.

[0039] Visual actions are determined based on image data collected by a visual sensor and a preset action recognition model; screen touch actions are determined based on touch coordinates collected by a screen touch sensor; and vocal actions are determined based on audio data collected by a voice acquisition device.

[0040] The interactive acquisition unit also includes an inertial sensor, which is used to acquire motion information of visual actions. This motion information is then combined with image data to determine the visual actions.

[0041] In one design, the pre-calibration unit is also used to monitor interactive action signals within a preset time interval before and after each expected special effects display moment of the preset calibration audio signal.

[0042] If no user interaction signal is collected within a preset time interval, the current beat is determined to be a missing interaction beat, and the data corresponding to the missing interaction beat is removed from the time deviation calculation.

[0043] The pre-calibration unit is also used to recalculate the average time deviation and the interaction response stability index based on the effective interaction times after removing missing interaction beats.

[0044] This application offers the following advantages: By establishing a user-specific tolerance range through pre-calibration, it solves the problem of fixed synchronization standards failing to adapt to individual differences; simultaneously, it utilizes environmental parameters to evaluate the background interference index in real time, effectively distinguishing between environmental noise and user subjective bias; furthermore, it dynamically determines the timing and type of special effects display based on the degree of background interference and the relative position of the actual interaction time with the personalized range. This achieves a complete closed loop from individual characteristic calibration to adaptive environmental adjustment, significantly improving the user's subjective synchronization experience, avoiding interruptions caused by premature or delayed captures, and thus enhancing the continuity and realism of immersive interaction. Attached Figure Description

[0045] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A schematic diagram of the system architecture of a synchronous interactive system for immersive song performance effects provided in one embodiment of this application; Figure 2 This is a schematic diagram of the structure of an interactive acquisition unit provided in one embodiment of this application. Detailed Implementation

[0047] To further illustrate the technical means and effects adopted by this application to achieve the intended purpose of the invention, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of an immersive song performance effect synchronous interactive system proposed in this application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0049] The following description, in conjunction with the accompanying drawings, details the specific scheme of the synchronous interactive system for immersive song performance effects provided in this application.

[0050] Please see Figure 1 It illustrates a system architecture diagram of a synchronous interactive system for immersive song performance effects provided in one embodiment of this application, such as... Figure 1 As shown, the immersive song performance effect synchronous interactive system 10 includes: a pre-calibration unit 101, an interactive acquisition unit 102, a background interference evaluation unit 103, an effect display decision unit 104, and an effect display unit 105.

[0051] The pre-calibration unit 101 is used to determine the user's personalized tolerance range based on the time deviation between the actual interaction time of the user's interactive action in response to the preset calibration audio signal and the preset expected special effect display time in the preset calibration audio signal.

[0052] The personalized tolerance range is the range of values ​​that users are allowed for the time deviation between their own interaction time and the expected time of special effect display.

[0053] Specifically, the steps for the pre-calibration unit 101 to determine the personalized tolerance range include: It provides a preset calibration audio signal, which contains multiple beat intervals with different beat speeds. The expected special effects display times are evenly distributed in each beat interval. During the calibration process, feedback special effects corresponding to interactive actions are not displayed to eliminate special effects interference.

[0054] Collect the actual interaction times of user actions performed near the expected special effects display time within each beat interval; using the clock of the preset calibration audio signal as a reference, record the time when the interaction action is detected as the actual interaction time, and determine the time deviation of each actual interaction time relative to the nearest expected special effects display time.

[0055] The collected time deviations are filtered for validity, and data corresponding to missing interaction beats that do not show any interactive action within a preset time ratio before and after the expected special effect display time are removed.

[0056] Based on the filtered effective time deviations, the difference between the average absolute value of the time deviations corresponding to the low-speed beat intervals and the average absolute value of the time deviations corresponding to the high-speed beat intervals is calculated, and the interaction response stability index is determined based on this difference; the interaction response stability index is used to measure the stability of user interaction habits under different beat speeds.

[0057] Based on the average of all valid time deviations, the center offset trend of user interaction habits is determined; and based on the center offset trend, combined with the interaction response stability index and the preset basic time tolerance, the personalized allowable range is generated. This personalized allowable range defines the range from the minimum deviation value to the maximum deviation value. When the user's interaction time is offset from the expected special effect display time within this range, the user can obtain a better synchronization experience.

[0058] The interactive acquisition unit 102 is used to collect the user's interactive actions during the immersive song performance, determine the actual interaction time corresponding to the interactive actions, and simultaneously collect the environmental parameters corresponding to the interactive actions.

[0059] In this embodiment, the interactive action refers to the action input actively made by the user in response to the rhythm of the song during immersive song performance. The actual interaction moment refers to the point in time when the synchronous interaction system 10 recognizes the occurrence of the interactive action. Environmental parameters refer to background feature data associated with the type of interactive action collected, used to evaluate the changing state of the collection environment.

[0060] Specifically, the working process of the interactive acquisition unit 102 includes: during the immersive song performance, continuously monitoring the user's behavioral input through a variety of configured sensors to acquire at least one type of interactive action; the type of interactive action may include visual actions, screen touch actions, and vocal actions.

[0061] When a user is identified to perform an interactive action that conforms to the preset interactive definition, the moment of recognition completion is determined as the actual interaction moment of the interactive action; the actual interaction moment is recorded with the clock of the song played by the synchronous interactive system 10 as the time reference.

[0062] During the acquisition of interactive actions, environmental parameters corresponding to the type of interactive action are acquired simultaneously. These environmental parameters are independent of the interactive action itself and are used to reflect the state characteristics of the background environment in which the interactive action is acquired. Different types of interactive actions correspond to different environmental parameters. For example, visual actions correspond to an ambient light signal sequence, screen touch actions correspond to an interactive coordinate sequence, and vocal actions correspond to a background noise energy change sequence.

[0063] The frequency of environmental parameter acquisition is configured to be higher than the frequency of interactive action recognition, so that there are a sufficient number of environmental parameter sampling points between two adjacent interactive actions, so that subsequent steps can evaluate the real-time changes of the acquired environment.

[0064] Background interference assessment unit 103 is used to determine the background interference index of the current interactive action based on the deviation between the real-time collected values ​​of environmental parameters and the predicted values ​​of environmental parameters.

[0065] The real-time acquired values ​​of environmental parameters in this embodiment refer to the environmental feature data synchronously acquired by the interactive acquisition unit 102 at the moment the current interactive action occurs. The predicted values ​​of environmental parameters refer to the expected feature values ​​calculated by a preset smoothing prediction algorithm based on historical environmental parameter data acquired over a period of time prior to the current interactive action. These values ​​characterize the expected level of environmental features when the acquisition environment remains stable. The background interference index is a quantitative indicator that comprehensively considers the degree of deviation between the real-time acquired values ​​and the predicted values, as well as the sensitivity of different types of interactive actions to background interference. Its value characterizes the degree of interference of the environmental background on the moment of interactive action acquisition.

[0066] Specifically, the steps for the background interference assessment unit 103 to determine the background interference index include: The environmental parameter sequence corresponding to the current interactive action is acquired from the interactive acquisition unit 102. The environmental parameter sequence includes multiple environmental parameter sampling points acquired at a sampling frequency higher than the interactive action acquisition frequency from the start of the song playing to the moment the current interactive action occurs. The type of environmental parameter matches the type of interactive action acquired. For example, visual actions correspond to ambient light signal sequences, screen touch actions correspond to interactive coordinate sequences, and vocal actions correspond to background noise energy change sequences.

[0067] In this embodiment, the background noise energy change sequence refers to the sequence of environmental noise energy changes over time, collected by a voice acquisition device after excluding the user's vocal frequency band. Specifically, the acquired audio signal can be bandpass filtered to remove the main frequency band of the user's vocalization (e.g., 80Hz to 1000Hz), and then the energy value of the remaining frequency band can be calculated to form the background noise energy change sequence. The background noise energy change sequence is used to assess the degree of interference from the environmental acoustic background and is independent of the user's vocalization action itself.

[0068] Based on local historical data in the environmental parameter sequence prior to the current interaction, a preset smoothing prediction algorithm is used to calculate the predicted value of the environmental parameters. The preset smoothing prediction algorithm can estimate the expected value under stable environmental conditions based on the changing trend of environmental parameters within a local window, so that when the actual performance of environmental parameters is stable, the predicted value is close to the actual value.

[0069] The deviation between the real-time collected values ​​and the predicted values ​​of environmental parameters is calculated, and the deviation is normalized to eliminate the differences in numerical scale between different types of environmental parameters.

[0070] Based on the sensitivity of different interaction action types to background interference, corresponding background interference weights are assigned to various environmental parameters. The normalized deviations are then weighted and synthesized to obtain the background interference index of the current interaction action. Among them, the background interference weight corresponding to visual actions that are sensitive to changes in background lighting is configured to be higher than the background interference weights corresponding to screen touch actions and sound actions.

[0071] The special effects display decision unit 104 is used to determine the special effects display time and special effects display type based on the comparison result between the background interference index and the preset interference threshold, as well as the position of the actual interaction time relative to the personalized allowable range.

[0072] In this embodiment, the preset interference threshold is a pre-set judgment boundary value used to distinguish the degree of background interference. When the background interference index is greater than the preset interference threshold, it indicates that the collection reliability of the current interaction action is low; conversely, it indicates that the collection reliability is high. The actual interaction time is provided by the interaction acquisition unit 102 and is the time point corresponding to the occurrence of the user's interaction action. The special effect display time refers to the time point when the synchronous interaction system 10 finally decides to display the feedback special effect to the user. The special effect display type refers to the type of feedback special effect selected by the synchronous interaction system 10 based on the decision result, which may include positive feedback special effects, negative feedback special effects, transition special effects, and dynamic feedback special effects.

[0073] Specifically, the process by which the special effects display decision unit 104 determines the timing and type of special effects display includes: The background interference index of the current interactive action is received from the background interference evaluation unit 103, and the background interference index is compared with a preset interference threshold (for example, it can be 0.7) to determine whether the current interactive action is in a high interference situation or a low interference situation.

[0074] The system receives the actual interaction time transmitted by the interaction acquisition unit 102 and, in conjunction with the preset expected special effects display times of the song performance system, determines whether the actual interaction time falls within the effective interaction interval centered on the expected special effects display time. The effective interaction interval refers to the time window within a preset proportion of time before and after the expected special effects display time, during which the user's interactive actions are recognized as effective interactive actions by the synchronous interaction system 10.

[0075] Based on the comparison between the background interference index and the preset interference threshold, and the relative positions of the actual interaction time with the effective interaction range and the personalized allowable range, the timing and type of special effects display are determined according to a preset decision logic. Specifically, when the background interference index is greater than the preset interference threshold, the reliability of the interaction action acquisition time is considered low, and the special effects display time is aligned with the expected special effects display time during the decision-making process. When the background interference index is less than or equal to the preset interference threshold, the reliability of the interaction action acquisition time is considered high, and the personalized allowable range is used as a benchmark. Based on the advance or lag of the actual interaction time relative to this range, the decision is made to either directly display positive feedback special effects or combine transition special effects with dynamic feedback special effects for a smooth transition. Dynamic feedback special effects are a type of special effects generated by mixing positive and negative feedback special effects in a specific ratio, which is determined based on the degree to which the actual interaction time deviates from the personalized allowable range.

[0076] Special effects display unit 105 is used to display special effects corresponding to the special effects display type according to the special effects display time and special effects display type.

[0077] Specifically, the process of displaying special effects in special effects display unit 105 includes: Receive information on the timing and type of special effects display sent by the special effects display decision unit 104.

[0078] Based on the type of special effects display, the corresponding special effects resources are retrieved from the preset special effects resource library. This library stores visual and auditory material templates for positive feedback effects, negative feedback effects, transition effects, and dynamic feedback effects. Positive feedback effects are characterized by crisp sound effects and bright particles, used to convey positive feedback that the current interaction meets synchronization requirements. Negative feedback effects are characterized by dull sound effects and dark smoke, used to convey negative feedback that the current interaction deviates from synchronization requirements. Transition effects are characterized by soft particle transitions or gradual sound changes, used to provide a smooth transition when there is a time discrepancy between the interaction action and the effect display, avoiding abrupt changes. Dynamic feedback effects are a mixture of positive and negative feedback effects in a specific ratio, used to provide progressive feedback that matches the degree of deviation when the user's interaction time is close to but not completely within the personalized tolerance range.

[0079] When the time for special effects display arrives, the retrieved special effects are shown to the user; if the special effects display type is dynamic feedback special effects, the ambiguity parameters and positive feedback special effects mixing ratio determined by the special effects display decision unit are applied simultaneously to control the visual sharpness and feedback tendency of the dynamic feedback special effects.

[0080] In one design, the pre-calibration unit 101 is specifically used for: Provides preset calibration audio signals containing multiple different beat intervals. These intervals include at least a low-speed beat interval and a high-speed beat interval. The total duration of the preset calibration audio signals is less than a preset duration threshold (e.g., 1 minute), and no feedback effects corresponding to interactive actions are displayed during calibration. The preset calibration audio signals use short, percussive percussion. The different beat intervals include a low-speed beat interval, a medium-speed beat interval, and a high-speed beat interval. The beat speed of the low-speed beat interval is less than a first preset beat value (e.g., 80 BPM). The beat speed of the medium-speed beat interval is between the first and second preset beat values ​​(e.g., 160 BPM). The beat speed of the high-speed beat interval is greater than the second preset beat value. Multiple interactive action moments are evenly distributed within each beat interval, requiring the user to complete the interactive action at the designated moments. The calibration process maintains stable acquisition conditions, including stable ambient light and noise levels.

[0081] The system collects the actual interaction times of users within each beat interval and determines the time deviation between the actual interaction time and the corresponding expected special effect display time. The time deviation is calculated as follows: using the playback clock of the preset calibration audio signal as the reference clock, the moment the interaction action is detected is recorded as the actual interaction time, and the adjacent expected special effect display time closest to the actual interaction time is recorded as the expected special effect display time. The time difference between the actual interaction time and the expected special effect display time is then calculated. The time difference is expressed by the following formula: in, Indicates time deviation, Indicates the actual moment of interaction. This indicates the expected moment for the special effects to be displayed.

[0082] The pre-calibration unit 101 is also used to monitor interactive action signals within a preset proportion of time intervals before and after each expected special effect display moment of the preset calibration audio signal. When no user interactive action signal is collected within the preset proportion (for example, 40%) of the time interval, the current beat is determined to be a missing interactive beat, and the data corresponding to the missing interactive beat is removed from the time deviation calculation.

[0083] The interaction response stability index is determined by the absolute value of the difference between the average absolute value of the time deviations corresponding to the low-speed beat intervals and the average absolute value of the time deviations corresponding to the high-speed beat intervals. The interaction response stability index is expressed by the following formula: in, Indicates the stability index of the interaction response. This represents the average of the absolute values ​​of all valid time deviations within the high-speed beat interval. This represents the average of the absolute values ​​of all effective time deviations within the low-speed beat interval. This represents a preset time normalization constant (e.g., 100ms), used to convert time differences into a dimensionless ratio. This represents absolute value operations. A higher interaction stability index indicates greater consistency in user interaction responses across different speed rhythms.

[0084] Calculate the average time deviation of all valid interaction beats (i.e., after removing missing interaction beats). .

[0085] The personalized tolerance range is determined based on the average of all valid time deviations, the interaction response stability index, and the preset baseline time tolerance. The personalized tolerance range is expressed by the following formula: in, Indicates the minimum deviation value. Indicates the maximum deviation value. This indicates the preset base time tolerance (for example, it can be 30ms). A minimum physiological tolerance constant (e.g., 15ms) is preset to ensure that even the most stable users have a basic tolerance range, avoiding excessive penalties. The minimum deviation value and the maximum deviation value together constitute the personalized allowable interval, and the length of the personalized allowable interval is the difference between the maximum deviation value and the minimum deviation value.

[0086] When the interaction response stability index d=1, it indicates that the user's interaction response is completely consistent at different beat speeds. At this time, the personalized tolerance range is determined by the preset minimum physiological tolerance constant. The interval length is determined to be 2. (For example, 30ms) is to preserve the basic physiological tolerance range for the most stable users.

[0087] The pre-calibration unit 101 is also used to recalculate the average value of the time deviation and the interaction response stability index based on the effective interaction time after removing missing interaction beats.

[0088] In a design, such as Figure 2 As shown, the interactive acquisition unit 102 includes a visual sensor 1, a screen touch sensor 2, and a voice acquisition device 3.

[0089] Interactive actions include visual actions, screen touch actions, and vocal actions.

[0090] Environmental parameters include ambient light signal sequences, interaction coordinate sequences, and background noise energy change sequences corresponding to the interaction action type.

[0091] Visual actions are determined based on image data acquired by visual sensor 1 using a preset action recognition model. The preset action recognition model is used to identify the human skeleton joints in each frame of the image, determine the movement trajectory through the skeleton key points between consecutive frames, and classify the movement trajectory with the preset trajectory model based on similarity, thereby determining the type of visual action.

[0092] The screen touch action is determined based on the touch coordinates collected by the screen touch sensor 2.

[0093] The vocalization action is determined based on the audio data collected by the voice acquisition device 3.

[0094] like Figure 2 As shown, the interactive acquisition unit 102 also includes an inertial sensor 4. The inertial sensor is used to acquire motion information of visual actions. The motion information is used to determine visual actions by combining it with image data.

[0095] During the data acquisition process, the environmental parameters are acquired at a higher frequency than the interactive actions.

[0096] In one design, the background interference evaluation unit 103 is specifically used for: During immersive song performance, the environmental parameter sequence within a preset duration prior to the current interactive action is acquired at a preset sampling frequency. The preset sampling frequency is higher than the acquisition frequency of the interactive action; the preset sampling frequency can be set to 100Hz.

[0097] Based on historical data from local windows within the environmental parameter sequence, the predicted value of the current environmental parameter is determined using a preset smoothing prediction algorithm. This preset smoothing prediction algorithm can employ a moving average method. The window size for the moving average method is empirically set to a preset number of sampling points, for example, 5 sampling points. The average value of all sampling points within the window is used as the predicted value for the next adjacent sampling point in the window.

[0098] The background interference index is determined based on the deviation between the real-time and predicted values ​​of environmental parameters, combined with the background interference weights and dimensionless removal constants corresponding to different interaction types. The dimensionless removal constant is used to convert environmental parameters of different dimensions into dimensionless quantities. The background interference index is expressed by the following formula: in, Indicates the background interference index. Indicates the first The background interference weights are assigned to different interaction action types. For example, the weight for visual actions can be set to 4, while the weights for screen touch actions and sound actions can be set to 3. This represents the number of historical time points used to calculate the average prediction error, and can be the size of the sliding window, i.e., 5 sampling points. Indicates the first Types of interactive actions in The actual value at time, Indicates the first Types of interactive actions in Predicted value at time, Represents Euclidean distance. Indicates the first The units of measurement for each type of interactive action are removed from the constant. It is a constant greater than zero. The upper limit of the background interference index is 1; when the calculated result is greater than 1, it is taken as 1. For interactive coordinate sequences, One pixel can be used; for background noise energy variation sequences, One unit of energy (e.g., 1 dB) can be used. Since each interaction action corresponds to only one type of interaction action, it is only necessary to evaluate the sequence of environmental parameters for that interaction action type, without accumulating across interaction action types.

[0099] In one design, the environmental parameters include: the ambient light signal sequence collected when the interaction action type is a visual action, the interaction coordinate sequence collected when the interaction action type is a screen touch action, and the background noise energy change sequence collected when the interaction action type is a sound action.

[0100] Different types of interactive actions correspond to different background interference weights, with visual actions having a higher background interference weight than screen touch actions and sound actions.

[0101] During an immersive song performance, the expected moment for special effects display is the moment when feedback effects should be displayed.

[0102] In one design, the special effects display decision unit 104 is specifically used for: Define the effective interaction interval centered on the expected special effects display moment. The boundaries of the effective interaction interval are determined based on the preset duration before and after the expected special effects display moment.

[0103] For example, the start time of the effective interaction interval is 40% of the beat interval before the expected special effect display time, and the end time is 40% of the beat interval after the expected special effect display time, where the beat interval is the time length between two adjacent expected special effect display times.

[0104] Based on the positional relationship between the actual interaction time and the effective interaction interval, the number of interactions identified within the effective interaction interval, and the relationship between the background interference index and the preset interference threshold, the timing and type of special effects display are determined.

[0105] Special effects display types include positive feedback effects, negative feedback effects, transition effects, and dynamic feedback effects. Positive feedback effects represent feedback when user interaction meets synchronization requirements. Negative feedback effects represent feedback when user interaction deviates from synchronization requirements. Transition effects represent a smooth transition when there is a time difference between the interaction action and the effect display. Dynamic feedback effects are generated based on a mixture of positive and negative feedback effects, the ratio of which is determined by the position of the actual interaction time relative to the tolerance range of the positive feedback effect.

[0106] In one design, the special effects display decision unit 104 is also used to execute the following general judgment logic: When the actual interaction time is not within the effective interaction range, the special effect display type is determined to be a negative feedback special effect.

[0107] If the expected effect display time is reached within the valid interaction range but no valid interaction is detected, a transition effect will be displayed starting from the expected effect display time, and the system will continue to wait for valid interaction. If no interaction is detected by the boundary of the valid interaction range, a negative feedback effect will be displayed.

[0108] When two or more interactions are detected within the effective interaction range and the background interference index is less than or equal to the preset interference threshold, it is determined that a negative feedback effect will be displayed at the moment of the second interaction.

[0109] In one design, when the background interference index is greater than the preset interference threshold, the special effects display decision unit 104 is used to determine whether the actual interaction time is within the effective interaction range.

[0110] When the actual interaction time is within the effective interaction interval and there is only one interaction within the effective interaction interval, the special effects display decision unit 104 is also used to determine the relative position between the actual interaction time and the expected special effects display time.

[0111] When the actual interaction time is before the expected special effect display time, the special effect display time is determined as the expected special effect display time, and the special effect display type is determined as positive feedback special effect.

[0112] When the actual interaction time is after the expected effect display time, the effect display time is determined as the actual interaction time, and the effect display type is determined as positive feedback effect.

[0113] In one design, when the background interference index is less than or equal to a preset interference threshold, the special effects display decision unit 104 is used to determine whether the actual interaction time is within the effective interaction interval and whether there is only one interaction within the effective interaction interval.

[0114] When the judgment result is yes, the special effects display decision unit 104 is further used to determine the positive feedback special effects tolerance range centered on the expected special effects display time, based on the personalized tolerance range. The starting time of the positive feedback special effects tolerance range is the sum of the expected special effects display time and the minimum deviation value, i.e. The end time of the tolerance range for positive feedback effects is the sum of the expected effect display time and the maximum deviation value, i.e. .in, The minimum deviation value, The maximum deviation value, This is the moment to showcase the expected special effects.

[0115] When the actual interaction moment Within the tolerance range of positive feedback effects Within the time frame, determine the actual interaction time. Display positive feedback effects.

[0116] When the actual interaction moment When the interaction occurs before the positive feedback effect is within tolerance, determine the actual interaction time. The transition effect is displayed at the beginning, and at the moment when the positive feedback effect tolerance range is reached. It displays dynamic feedback effects in real time.

[0117] When the actual interaction moment When the positive feedback effect is beyond the tolerance range, determine when to display the dynamic feedback effect at the actual interaction moment.

[0118] The special effects display decision unit 104 is also used to determine the distance between the actual interaction moment and the nearest boundary within the tolerance range of the positive feedback effect, and to determine the mixing ratio of positive and negative feedback effects in the dynamic feedback effect based on the ratio between the distance and the interval length of the tolerance range of the positive feedback effect. The distance is negatively correlated with the mixing ratio of positive feedback effects in the dynamic feedback effect. The mixing ratio is expressed by the following formula: in, This indicates the proportion of positive feedback effects in the dynamic feedback effects. This represents the distance between the actual interaction moment and the nearest boundary within the tolerance range of the positive feedback effect. This indicates the length of the personalized allowable range. The maximum deviation value, The minimum deviation value is represented by max(·), which indicates the function that takes the maximum value.

[0119] The special effects display unit 105 is also used to determine the blur parameter of the dynamic feedback special effects based on the background interference index. The blur parameter is positively correlated with the background interference index, i.e., the blur parameter... ,in, Positive coefficients (e.g.) ), For example, constants (e.g.) The blur parameter is used to adjust the sharpness of the dynamic feedback effect. The higher the background interference index, the larger the blur parameter, and the weaker the sharpness of the dynamic feedback effect.

[0120] In one design, positive feedback effects include crisp sound effects and bright particles. Negative feedback effects include dull sound effects and dark smoke. Transition effects include softened particles with adjustable softening parameters and duration. The softening parameter is used to adjust the smoothing effect of the particle transition.

[0121] The synchronous interaction system 10 is also used to continuously execute the above steps for subsequent interaction processes after completing the synchronous display of an interaction process. Based on real-time interaction parameters, it identifies the actual interaction time and the expected special effect display time. By combining the real-time background interference index with the user interaction characteristics during the calibration phase, it determines the special effect display logic, continuously provides a synchronous interactive experience, and forms dynamic time-domain compensation to offset the problem of asynchronous special effects caused by the user's subjective perception deviation.

[0122] The immersive song performance effect synchronous interactive system provided in this application has the following beneficial effects: By establishing a user-personalized tolerance range through pre-calibration, it solves the problem of fixed synchronization standards not adapting to individual differences; simultaneously, it utilizes environmental parameters to evaluate the background interference index in real time, effectively distinguishing between environmental noise and user subjective bias; furthermore, it dynamically decides the timing and type of effect display based on the degree of background interference and the relative position of the actual interaction time with the personalized range. This achieves a complete closed loop from individual characteristic calibration to environmental adaptive adjustment, significantly improving the user's subjective synchronization experience, avoiding interruptions caused by premature or delayed timing, and thus enhancing the continuity and realism of immersive interaction.

[0123] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0124] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A synchronous interactive system for immersive song performance effects, characterized in that, include: The pre-calibration unit is used to determine the user's personalized tolerance range based on the time deviation between the actual interaction time of the user's interactive action in response to the preset calibration audio signal and the preset expected special effect display time in the preset calibration audio signal. The interactive acquisition unit is used to acquire the user's interactive actions during the immersive song performance, determine the actual interaction time corresponding to the interactive actions, and simultaneously acquire the environmental parameters corresponding to the interactive actions. The background interference assessment unit is used to determine the background interference index of the current interactive action based on the deviation between the real-time collected value of the environmental parameter and the predicted value of the environmental parameter. The special effects display decision unit is used to determine the special effects display time and special effects display type based on the comparison result between the background interference index and the preset interference threshold, and the position of the actual interaction time relative to the personalized allowable range. The special effects display unit is used to display special effects corresponding to the special effects display type according to the special effects display time and the special effects display type.

2. The synchronous interactive system for immersive song performance effects according to claim 1, characterized in that, The pre-calibration unit is specifically used for: Provide a preset calibration audio signal containing multiple different beat intervals, wherein the multiple different beat intervals include at least a low-speed beat interval and a high-speed beat interval; the total duration of the preset calibration audio signal is less than a preset duration threshold, and no feedback effects corresponding to the interactive action are displayed during the calibration process; Collect the actual interaction time of the user in each of the beat intervals, and determine the time deviation between the actual interaction time and the corresponding expected special effect display time; The interaction response stability index is determined by the absolute value of the difference between the average absolute value of the time deviation corresponding to the low-speed beat interval and the average absolute value of the time deviation corresponding to the high-speed beat interval. The personalized tolerance range is determined based on the average of all valid time deviations, the interaction response stability index, and the preset basic time tolerance.

3. The synchronous interactive system for immersive song performance effects according to claim 1, characterized in that, The background interference assessment unit is specifically used for: During the immersive song performance, an environmental parameter sequence within a preset time period prior to the current interactive action is acquired according to a preset sampling frequency, wherein the preset sampling frequency is higher than the acquisition frequency of the interactive action; Based on the historical data of local windows in the environmental parameter sequence, the predicted value of the current environmental parameter is determined by a preset smoothing prediction algorithm; Based on the deviation between the real-time collected values ​​of the environmental parameters and the predicted values, and combined with the background interference weights and dimensionless removal constants corresponding to different interaction action types, the background interference index is determined; the dimensionless removal constant is used to convert environmental parameters of different dimensions into dimensionless quantities.

4. The synchronous interactive system for immersive song performance effects according to claim 3, characterized in that, The environmental parameters include: the ambient light signal sequence collected when the interaction action type is a visual action, the interaction coordinate sequence collected when the interaction action type is a screen touch action, and the background noise energy change sequence collected when the interaction action type is a sound action. The different types of interactive actions correspond to different background interference weights, with the background interference weight for visual actions being higher than that for screen touch actions and sound actions.

5. The synchronous interactive system for immersive song performance effects according to claim 1, characterized in that, The special effects display decision unit is specifically used for: Determine the effective interaction range centered on the expected moment of special effects display; Based on the positional relationship between the actual interaction time and the effective interaction interval, the number of interactions identified within the effective interaction interval, and the relationship between the background interference index and the preset interference threshold, the special effect display time and the special effect display type are determined. The special effects display types include positive feedback effects, negative feedback effects, transition effects, and dynamic feedback effects. The dynamic feedback effects are special effects generated based on the mixing ratio of the positive feedback effects and the negative feedback effects.

6. The synchronous interactive system for immersive song performance effects according to claim 5, characterized in that, When the background interference index is greater than the preset interference threshold, the special effects display decision unit is used to: determine whether the actual interaction time is within the effective interaction range; When the actual interaction time is within the effective interaction interval and there is only one interaction within the effective interaction interval, the special effects display decision unit is also used to determine the relative position between the actual interaction time and the expected special effects display time. When the actual interaction time is before the expected special effect display time, the special effect display time is determined as the expected special effect display time, and the special effect display type is determined to be a positive feedback special effect. When the actual interaction time is after the expected special effect display time, the special effect display time is determined as the actual interaction time, and the special effect display type is determined to be a positive feedback special effect.

7. The synchronous interactive system for immersive song performance effects according to claim 5, characterized in that, When the background interference index is less than or equal to the preset interference threshold, the special effects display decision unit is used to determine whether the actual interaction time is within the effective interaction interval and whether there is only one interaction within the effective interaction interval. When the judgment result is yes, it is also used to determine the tolerance range of positive feedback special effects centered on the expected special effect display time based on the personalized tolerance range; When the actual interaction time is within the tolerance range of the positive feedback effect, it is determined to display the positive feedback effect at the actual interaction time. When the actual interaction time is before the tolerance range of the positive feedback effect, a transition effect is displayed from the actual interaction time to the start time of the tolerance range of the positive feedback effect, and a dynamic feedback effect is displayed at the start time. When the actual interaction time is after the tolerance range of the positive feedback effect, it is determined to display the dynamic feedback effect at the actual interaction time. The special effects display decision unit is also used to determine the distance between the actual interaction time and the nearest boundary in the tolerance range of the positive feedback special effects, and to determine the mixing ratio of positive feedback special effects and negative feedback special effects in the dynamic feedback special effects based on the ratio of the distance to the interval length of the tolerance range of the positive feedback special effects. The special effects display unit is also used to determine the blur parameter of the dynamic feedback special effect based on the background interference index. The blur parameter is positively correlated with the background interference index and is used to adjust the sharpness intensity of the dynamic feedback special effect.

8. The synchronous interactive system for immersive song performance effects according to claim 5, characterized in that, The positive feedback effect is used to characterize the feedback performance when the user interaction performance meets the synchronization requirements; The negative feedback effect is used to characterize the feedback performance when the user interaction performance deviates from the synchronization requirement; The transition effect is used to represent the smooth connection when there is a time difference between the interactive action and the effect display. The positive feedback effects include crisp sound effects and bright particles, while the negative feedback effects include dull sound effects and dark smoke.

9. The synchronous interactive system for immersive song performance effects according to claim 1, characterized in that, The interactive acquisition unit includes a visual sensor, a screen touch sensor, and a voice acquisition device. The interactive actions include visual actions, screen touch actions, and sound actions. The environmental parameters include an ambient light signal sequence, an interaction coordinate sequence, and a background noise energy change sequence corresponding to the interaction action type. The visual action is determined based on the image data collected by the visual sensor through a preset action recognition model; the screen touch action is determined based on the touch coordinates collected by the screen touch sensor; and the vocal action is determined based on the audio data collected by the voice acquisition device. The interactive acquisition unit also includes an inertial sensor, which is used to acquire motion information of the visual action, and the motion information is used to determine the visual action in combination with the image data.

10. The synchronous interactive system for immersive song performance effects according to claim 2, characterized in that, The pre-calibration unit is also used to monitor interactive action signals within a preset time interval before and after each expected special effect display moment of the preset calibration audio signal. When no user interaction signal is collected within the time interval of the preset ratio, the current beat is determined to be a missing interaction beat, and the data corresponding to the missing interaction beat is removed from the time deviation calculation. The pre-calibration unit is also used to recalculate the average value of the time deviation and the interaction response stability index based on the effective interaction time after removing missing interaction beats.

Images