Multi-sensory virtual reality equipment control method and multi-sensory virtual reality mask

Abstract

The invention discloses a multi-sensory virtual reality equipment control method and a multi-sensory virtual reality mask, and belongs to the technical field of virtual reality, a synchronization time window of a current period is constructed according to multi-sensory instructions which are received in the current period and have the same timestamp, and the synchronization time window of the current period is constructed based on the receiving time of the multi-sensory instructions at a multi-sensory virtual reality equipment end. Calculating the expected sensing time of each sensory feedback module in the current period according to the current sensing time of each sensory feedback module and the internal predicted delay time and subjective compensation time of each sensory feedback module in the current period, and fusing the expected sensing time of each sensory feedback module in the current period to obtain a target common sensing time of the current period; therefore, when the target common sensing moment of the current period is in the synchronization time window, the execution starting moment of each sensory feedback module is calculated based on the target common sensing moment of the current period, and the corresponding sensory feedback module is triggered at the execution starting moment of each sensory feedback module, so that multi-sensory synchronous feedback is realized.

Description

Technical Field

[0001] This invention belongs to the field of virtual reality technology, and more specifically, relates to a control method for a multi-sensory virtual reality device and a multi-sensory virtual reality mask. Background Technology

[0002] In recent years, the rapid development of virtual reality technology has greatly promoted the research and application of immersive interactive experiences. Through head-mounted display devices, users can obtain highly realistic visual and auditory experiences, leading to their widespread application in fields such as education and training, medical rehabilitation, gaming and entertainment, and industrial simulation. However, existing virtual reality systems primarily rely on visual and auditory feedback, lacking effective stimulation of other senses, resulting in significant deficiencies in the user's sense of immersion and realism.

[0003] With the rise of multi-sensory interaction technology, researchers have begun to integrate physical stimuli such as temperature, vibration, and smell into virtual reality experiences to achieve more realistic environmental perception. For example, temperature changes can simulate fire or snow environments, and odor feedback can recreate scenes such as gardens and kitchens. However, the transmission time of instructions from the server to the virtual reality device for each sensory feedback module in a multi-sensory interaction device may be inconsistent, and there is a certain delay between the triggering of each sensory feedback module and the actual attainment of the corresponding intensity of sensory stimulation, with these delays varying among different sensory feedback modules. Existing multi-sensory interaction devices typically use independent control modes for each sensory feedback module, lacking a unified multimodal coordination mechanism. This leads to the problem of asynchronous timing of different sensory stimuli, and the feedback control algorithm is simple, failing to consider the response delays of different sensory modules, making it difficult to achieve a stable real-time experience. Therefore, how to design a virtual reality device control scheme that can achieve synchronous multimodal sensory feedback has become a key technical problem for improving immersive interactive experiences. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a multi-sensory virtual reality device control method and a multi-sensory virtual reality mask, thereby solving the technical problem of how to achieve multi-modal sensory synchronous feedback.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a multi-sensory virtual reality device control method is provided, comprising: For multi-sensory instructions with the same timestamp received in the current cycle, a synchronization time window for the current cycle is constructed; the multi-sensory instructions include instructions corresponding to any two or more of the senses, namely temperature, touch, and smell. Determine the reception time of the multi-sensory command at the multi-sensory virtual reality device, calculate the expected delay time of each sensory feedback module in the multi-sensory virtual reality device within the current cycle, and determine the subjective compensation time of each sensory feedback module. Based on the reception time of the multi-sensory instructions at the multi-sensory virtual reality device, and the expected delay time and subjective compensation time of each sensory feedback module in the current cycle, the expected perception time of each sensory feedback module in the current cycle is calculated. The expected perception moment of each sensory feedback module in the current cycle is fused to obtain the common perception moment of the target in the current cycle. When the common perception time of the target in the current cycle is within the synchronization time window, the start execution time of each sensory feedback module is calculated based on the common perception time of the target in the current cycle, and the corresponding sensory feedback module is triggered at the start execution time of each sensory feedback module, so that the wearer of the multi-sensory virtual reality device can synchronize the perception time of each sense corresponding to the multi-sensory command.

[0006] According to the above-described multi-sensory virtual reality device control method, based on the reception time of the multi-sensory command at the multi-sensory virtual reality device, and the internal estimated delay time and subjective compensation time of each sensory feedback module in the current cycle, the expected perception time of each sensory feedback module in the current cycle is calculated, specifically including: For any sensory feedback module, the reception time of the instruction corresponding to the sensory feedback module in the multi-sensory instructions on the multi-sensory virtual reality device is determined, and the reception time of the instruction corresponding to the sensory feedback module on the multi-sensory virtual reality device, the internal expected delay time and subjective compensation time of the sensory feedback module in the current cycle are accumulated to obtain the expected perception time of the sensory feedback module in the current cycle.

[0007] Based on the above-mentioned multi-sensory virtual reality device control method, the desired perception moment of each sensory feedback module in the current cycle is fused to obtain the common perception moment of the target in the current cycle, specifically including: The initial target co-sensing time is calculated based on the following formula. :

[0008] in, For sensory feedback module i, the expected perception moment in the current cycle, The weight of sensory feedback module i in the current cycle; The initial target common perception time is smoothed based on the target common perception time of the previous cycle to obtain the target common perception time of the current cycle.

[0009] According to the above-mentioned multi-sensory virtual reality device control method, the initial target co-perception time is smoothed based on the target co-perception time of the previous cycle to obtain the target co-perception time of the current cycle, specifically including: The initial target common sensing time is smoothed based on the following formula:

[0010] in, As a smoothing factor, To jointly perceive the goals of the previous cycle. To jointly perceive the moment for the goals of the current cycle.

[0011] According to the above-described multi-sensory virtual reality device control method, the weight of any sensory feedback module in the current cycle is determined as follows: Based on the weight of any of the sensory feedback modules i in the previous cycle And the transformation relationship between the weight and delay variance of any sensory feedback module i in any period, and calculate the delay variance of the sensory feedback module i in the previous period. ; The delay variance of any sensory feedback module i in the current period is calculated based on the following formula. :

[0012] in, For smoothing coefficients, and These are the actual internal delay times of any sensory feedback module i in the first two cycles; Based on the transformation relationship between the weight and delay variance of any sensory feedback module i in any period, and the delay variance of any sensory feedback module i in the current period, calculate the weight of any sensory feedback module i in the current period. Wherein, the weight of any sensory feedback module i in any period t With delay variance The transformation relationship between them is: .

[0013] Based on the above multi-sensory virtual reality device control method, the expected delay time of any sensory feedback module in any cycle is calculated using the following formula:

[0014] in, For any sensory feedback module i, the estimated delay time within period t is... For any sensory feedback module i, the estimated delay time within period t-1 is given. Let be the actual delay time of any sensory feedback module i within the period t-1.

[0015] According to the above-mentioned multi-sensory virtual reality device control method, the start execution time of each sensory feedback module is calculated based on the target common perception time of the current cycle, specifically including: For any sensory feedback module, the start execution time of the sensory feedback module is obtained by subtracting the target co-perception time of the current cycle from the expected delay time of the sensory feedback module within the current cycle.

[0016] Based on the above-mentioned multi-sensory virtual reality device control method, a synchronization time window is constructed for multi-sensory instructions with the same timestamp received in the current cycle, including: The execution delay time of each sensory feedback module in the current cycle and the previous cycle is determined; the execution delay time of any sensory feedback module in any cycle is the sum of the expected delay time of the sensory feedback module within the cycle, the transmission time of the instruction corresponding to the sensory feedback module received in the cycle, and the subjective compensation time of the sensory feedback module. If the execution delay time of each sensory feedback module is different in the current cycle and the previous cycle, the synchronization time window size of the current cycle is determined based on the difference between the default window size and the execution delay time of each sensory feedback module in the current cycle and the previous cycle. Based on the timestamp of the multi-sensory instruction and the size of the synchronization time window for the current period, a synchronization time window for the current period is constructed.

[0017] According to the above-described multi-sensory virtual reality device control method, the instruction for any sensory element in the multi-sensory instructions includes a frame header field, a preamble field, a timestamp field, a working bit field, a duration field, an intensity bit field, and a checksum field. The frame header field is a fixed identifier field; the preamble field indicates the sensory feedback module corresponding to the instruction; the timestamp field is a timestamp indicating the time the instruction was generated; the working bit field indicates the specific working element within the corresponding sensory feedback module; the duration field indicates the duration of the corresponding sensory stimulus; the intensity bit field indicates the intensity of the corresponding sensory stimulus; and the checksum field is used to prevent transmission errors of the corresponding instruction.

[0018] According to a second aspect of the present invention, a multi-sensory virtual reality mask is provided, comprising: Temperature feedback module, used to generate temperature changes through semiconductor cooling heating element; A tactile feedback module is used to provide vibratory tactile stimulation based on a miniature vibration motor. An olfactory feedback module is used to release odors through a piezoelectric atomizing plate, a micro fan, and an airflow control structure; The main control module is used to execute the multi-sensory virtual reality device control method as described in the first aspect.

[0019] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: By constructing a synchronization time window for the current period based on multi-sensory commands with the same timestamp received in the current period, the reception time of the multi-sensory commands on the multi-sensory virtual reality device is determined. The internal expected delay time of each sensory feedback module in the multi-sensory virtual reality device in the current period is calculated, and the subjective compensation time of each sensory feedback module is determined. Then, based on the reception time of the multi-sensory commands on the multi-sensory virtual reality device, as well as the internal expected delay time and subjective compensation time of each sensory feedback module in the current period, the expected perception time of each sensory feedback module in the current period is calculated. The expected perception times of each sensory feedback module in the current period are then fused to obtain the target common perception time of the current period. Thus, when the target common perception time of the current period is within the synchronization time window, the start execution time of each sensory feedback module is calculated based on the target common perception time of the current period, and the corresponding sensory feedback module is triggered at the start execution time of each sensory feedback module. This allows the wearer of the multi-sensory virtual reality device to synchronize the perception time of each sense corresponding to the multi-sensory commands, achieving synchronous feedback of multiple senses. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the control method for a multi-sensory virtual reality device provided in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of instruction encoding provided for an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of a double buffer provided for an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of the structure of a multi-sensory virtual reality mask provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] This invention provides a control method for a multi-sensory virtual reality device, such as... Figure 1 As shown, it includes: Step 110: For multi-sensory instructions with the same timestamp received in the current cycle, construct a synchronization time window for the current cycle; the multi-sensory instructions include instructions corresponding to any two or more of the senses, namely temperature, touch and smell. Step 120: Determine the reception time of the multi-sensory command at the multi-sensory virtual reality device, calculate the expected delay time of each sensory feedback module in the multi-sensory virtual reality device within the current cycle, and determine the subjective compensation time of each sensory feedback module. Step 130: Based on the reception time of the multi-sensory instruction at the multi-sensory virtual reality device, and the expected delay time and subjective compensation time of each sensory feedback module in the current cycle, calculate the expected perception time of each sensory feedback module in the current cycle. Step 140: The expected perception moments of each sensory feedback module in the current cycle are fused to obtain the common perception moment of the target in the current cycle. Step 150: When the common perception time of the target in the current cycle is within the synchronization time window, calculate the start execution time of each sensory feedback module based on the common perception time of the target in the current cycle, and trigger the corresponding sensory feedback module at the start execution time of each sensory feedback module, so that the wearer of the multi-sensory virtual reality device can synchronize the perception time of each sense corresponding to the multi-sensory command.

[0026] Here, the rendering program of the virtual reality host outputs the current scene's frame and event parameters in each rendering frame cycle, including but not limited to: frame images, ambient temperature labels, object interaction intensity, explosion / impact events, odor trigger points, and odor concentration levels. This data is read in real-time via a plugin or SDK interface and encapsulated into a structured event object before being sent to the edge server. Upon receiving the structured event object, the edge server unpacks and parses it, and based on the parsed content, constructs instructions for each sensory organ using a unified encoding format before sending them to the virtual reality device (e.g., a virtual reality mask). In some embodiments, the unified encoding format defines the following fields: frame header field (F), preamble field (S), timestamp field (T), working bit field (W), duration field (D), intensity bit field (I), and checksum field (C). The frame header field is a fixed identifier field; the preamble field indicates the sensory feedback module corresponding to the given instruction; the timestamp field is the timestamp of the instruction generation time; the working bit field indicates the specific working element in the corresponding sensory feedback module; the duration field indicates the duration of the corresponding sensory stimulus; the intensity bit field indicates the intensity of the corresponding sensory stimulus; and the checksum field is used to prevent transmission errors of the corresponding instruction. An example instruction encoding is shown below. Figure 2 As shown.

[0027] In some embodiments, to reduce the size of the instruction code and thus alleviate the instruction transmission pressure, continuous similar codes can be compressed. For continuous sensory instructions with identical preamble fields, the preamble fields can be merged, retaining only one preamble field along with a continuous frame count field; for the timestamp field, only the absolute timestamp of the first frame is recorded, and the time increment ΔTi is written to subsequent frames to avoid repeatedly transmitting the complete timestamp field. The checksum field, originally present in each frame, can be updated to a unified checksum for the continuous sensory instruction codes to reduce instruction packet redundancy overhead.

[0028] For example: For the same sensory module, three frames of haptic commands are output consecutively: F, S=01, T=100, W1, D1, I1, C1 F, S=01, T=600, W2, D2, I2, C2 F, S=01, T=1100, W3, D3, I3, C3 The compressed code obtained after using the above encoding method is: F, S=01, COUNT=03, T=100, ΔT=500, W1, D1, I1, W2, D2, I2, W3, D3,I3, C block C block To standardize the verification code.

[0029] When most fields of consecutive instructions are consistent, a field merging encoding method can be adopted. This method replaces the frame-by-frame encoding of multiple repeated instructions by adding a repeat count field REPEAT. The value of REPEAT represents the number of repeated fields on the left. The original checksum for each frame is replaced with a unified checksum for consecutive instructions to reduce packet size and communication bandwidth usage.

[0030] For example: For different sensory modules, multiple sensory commands are triggered at the same time, and the W, D, I fields of multiple sensory commands are the same: F, S=01, T1, W1, D1, I1, C1 F, S=10, T2, W2, D2, I2, C2 F, S=11, T3, W3, D3, I3, C3 The compressed code obtained after using the above encoding method is: F, S=01,10,11, T1, W1, D1, I1, REPEAT=04, C block .

[0031] Furthermore, the edge server periodically (e.g., every 500ms) aggregates the multi-sensory events that need to be triggered, forming a command packet containing several coded frames. This ensures that all frames within the command packet have a uniform generation time range, thereby maintaining strict synchronization on the virtual reality device. The command batch packetization mechanism includes the following steps: 1. The edge server collects all sensory instruction frames in chronological order, using a synchronization time window of size W; 2. Establish a unified packet header, containing the following information: 1) Packaging header identifier, fixed as 0xFF 2) Package timestamp 3) Size of the synchronization time window 4) Current batch instruction count.

[0032] After receiving the instruction sent by the edge server in the current cycle, the multi-sensory virtual reality device parses and schedules the received instruction using the aforementioned unified encoding format. In some embodiments, after receiving consecutive instruction frames, the multi-sensory virtual reality device can employ a double-buffering mechanism to store and schedule the instructions. This double-buffering mechanism provides two buffers: one buffer stores the instruction to be executed in the current cycle, and the other buffer stores the instruction for the next cycle; these two buffers are periodically exchanged. Here, the duration of any cycle varies with the size of the synchronization time window for that cycle. Figure 3 As shown, assuming two buffers, buffer A and buffer B, in cycle T, instructions from buffer A are read, parsed, synchronized, and distributed to trigger the corresponding sensory feedback module. In cycle T+1, instructions from buffer B are read, parsed, synchronized, and distributed. In cycle T+2, the process of parsing, synchronizing, and distributing instructions from buffer A continues, and so on. Even if jitter occurs during communication, the dual-buffered queue mechanism decouples instruction reception and execution time within the execution cycle, providing additional buffer time for instruction transmission. When the communication jitter amplitude is within the buffer cycle tolerance range, this mechanism can effectively ensure the complete reception and orderly execution of instruction data in each buffer, thereby improving the stability and continuity of instruction execution on the virtual reality device side.

[0033] Specifically, if multi-sensory instructions with the same timestamp are received, they will be processed synchronously to ensure that the virtual reality device wearer perceives multiple sensory stimuli at consistent times, achieving consistency in multi-sensory feedback. Here, multi-sensory instructions include instructions corresponding to any two or more of the following senses: temperature, touch, and smell.

[0034] Specifically, this embodiment of the invention introduces a synchronization time window mechanism to restrict the execution and feedback of multi-sensory instructions within a synchronization time window. Specifically, for multi-sensory instructions with the same timestamp received in the current period, a synchronization time window for the current period is first constructed. For example, the synchronization time window for the current period can be constructed based on the timestamp of the multi-sensory instruction, combined with the window length set for the current period, thus obtaining a window that can be represented as... The synchronization time window, in which The time it takes for the virtual reality device to receive the instruction. W Indicates the window length of the synchronization time window.

[0035] To improve the stability of multi-sensory feedback synchronization, some embodiments provide a window adaptive adjustment mechanism. Specifically, the execution delay time of each sensory feedback module in the virtual reality device in the current cycle and the previous cycle can be determined. The execution delay time of any sensory feedback module in any cycle is the sum of the expected delay time of the sensory feedback module within that cycle, the transmission time of the instruction corresponding to the sensory feedback module received in that cycle, and the subjective compensation time of the sensory feedback module. Here, the expected delay time of any sensory feedback module within any cycle is the system-predicted delay from triggering the sensory feedback module to actually reaching the corresponding intensity of sensory stimulation, and the transmission time of the instruction corresponding to the sensory feedback module is the time from the generation of the instruction at the edge server to its transmission to the virtual reality device. Subsequently, the execution delay times of each sensory feedback module in the current cycle and the previous cycle are compared to increase or decrease the window size based on a default window size (e.g., 500ms). Specifically, the average execution delay time of each sensory feedback module in the current cycle and the average execution delay time of each sensory feedback module in the previous cycle can be determined, and then the difference between these two averages is calculated. If the difference is not zero, the default window size is adjusted based on this difference to obtain the synchronization time window size for the current period. For example, the synchronization time window for the current period can be determined as follows: If δ best (k) is greater than or less than δ best (k-1), then W(k)=500±|δ best (k)-δ best (k-1)|.

[0036] Where, δ best (k) and δ best (k-1) represent the average execution delay time of the current period and the average execution delay time of the previous period, respectively, and W(k) is the size of the synchronization time window of the current period.

[0037] Alternatively, it can be based on the size of the synchronization time window from the previous cycle. The synchronization time window size for the current period obtained above. Smoothing is performed, and the smoothing process can be represented by the following formula:

[0038] in, This is the smoothing coefficient. For example, .

[0039] Based on the timestamp of the multi-sensory instruction and the size of the synchronization time window for the current period, a synchronization time window for the current period can be constructed.

[0040] The synchronization time window is specifically applied in the following aspects: 1) Edge servers collect all sensory event frames according to the size of the synchronization time window and arrange them in chronological order; 2) The synchronization time window is also the unit time for the virtual reality device hardware to update its state once. The synchronization time window determines the time range within which the expected perception moment of each sensory feedback module is calculated, i.e., the minimum multi-sensory resolution. Theoretically, the smaller the synchronization time window, the higher the frequency of calculating the expected perception moment of each sensory feedback module, the more timely the signal processing, the higher the realism and fidelity, and the higher the hardware requirements of the device. Therefore, the role of the synchronization time window is to provide the time range for calculating the expected perception moment of each sensory feedback module, thereby mapping physically delayed temperature, tactile, and olfactory signals to a unified sensory integration time domain. This allows multimodal stimuli to form a single event attribution at the cognitive level of the brain, thus achieving cross-modal immersive experience synchronization, rather than relying solely on physical delay alignment.

[0041] After establishing the synchronization time window for the current cycle, the reception time of multi-sensory commands at the multi-sensory virtual reality device is determined. The expected delay time within the current cycle for each sensory feedback module in the multi-sensory virtual reality device is calculated, and the subjective compensation time for each sensory feedback module is determined. As mentioned above, the expected delay time within the current cycle for each sensory feedback module is the system-predicted delay from triggering the module to actually reaching the corresponding intensity of sensory stimulation. The subjective compensation time for each sensory feedback module is a compensation time set based on the subjective perception delay of various sensory stimuli by the human body, used to compensate for the subjective perception delay of various sensory stimuli by triggering the modules in advance. In some embodiments, the objective law of "slow for cold and heat, fast for vibration, and slowest for odor" can be followed to set corresponding subjective compensation times for each sensory feedback module. For example, the subjective compensation time for the temperature feedback module can be set to -200ms, the subjective compensation time for the tactile feedback module to -100ms, and the subjective compensation time for the olfactory feedback module to -300ms.

[0042] Based on the reception time of multi-sensory commands at the multi-sensory virtual reality device, and the internal estimated delay time and subjective compensation time of each sensory feedback module in the current cycle, the expected perception time of each sensory feedback module in the current cycle is calculated. Here, the expected perception time of each sensory feedback module in the current cycle represents the ideal time perceived by the virtual reality device wearer within the current cycle for that corresponding sensory feedback module. In some embodiments, for any sensory feedback module, the reception time of the command corresponding to that sensory feedback module in the multi-sensory commands at the multi-sensory virtual reality device is determined, and the reception time of the command corresponding to that sensory feedback module at the multi-sensory virtual reality device, the internal estimated delay time and subjective compensation time of that sensory feedback module in the current cycle are accumulated to obtain the expected perception time of that sensory feedback module in the current cycle.

[0043] In some embodiments, the estimated delay time of any sensory feedback module within any cycle can be calculated based on the following formula:

[0044] in, For any sensory feedback module i, the estimated delay time within period t is... For any sensory feedback module i, the estimated delay time within period t-1 is given. Let be the actual delay time of any sensory feedback module i within the period t-1.

[0045] Subsequently, the expected perception moments of each sensory feedback module in the current cycle are fused to obtain the common perception moment of the target in the current cycle. This common perception moment is the ideal moment when the sensory stimuli given by each sensory feedback module are simultaneously perceived, calculated based on the expected perception moment of each sensory feedback module in the current cycle.

[0046] In some embodiments, the initial target co-sensing moment can be calculated based on the following formula. :

[0047] in, For sensory feedback module i, the expected perception moment in the current cycle, The weight of sensory feedback module i in the current cycle; Then, based on the target co-perception time of the previous cycle, the above initial target co-perception time is smoothed to obtain the target co-perception time of the current cycle.

[0048] In other embodiments, the initial common perception moment of the target can be smoothed based on the following formula:

[0049] in, As a smoothing factor, To jointly perceive the goals of the previous cycle. To jointly perceive the moment for the goals of the current cycle.

[0050] Considering the inherent delay between the triggering of a sensory feedback module and the actual arrival of the corresponding intensity of sensory stimulus, and the varying degrees of delay across different sensory feedback modules, the system's prediction accuracy for these delays also differs. Therefore, to improve the stability and accuracy of multi-sensory feedback synchronization, when fusing the expected perception times of each sensory feedback module in the current cycle to determine the common perception time of the target in the current cycle, the expected perception time of the sensory feedback module that provides the most accurate prediction of the aforementioned delay can be used as a reference, and other sensory feedback modules can align themselves with this reference sensory feedback module. Specifically, in some embodiments, the weight of any sensory feedback module in the current cycle can be determined based on the following method: Based on the weight of sensory feedback module i in the previous cycle And the transformation relationship between the weight and delay variance of any sensory feedback module i in any period, and calculate the delay variance of the sensory feedback module i in the previous period. ; The delay variance of sensory feedback module i in the current period is calculated based on the following formula. :

[0051] in, For smoothing coefficients, and These represent the actual delay times within the first two cycles for sensory feedback module i. Here, the actual delay time within any cycle for any sensory feedback module is the real delay from triggering the sensory feedback module to actually reaching the corresponding intensity of sensory stimulation.

[0052] Subsequently, based on the transformation relationship between the weight and delay variance of any sensory feedback module i in any period, and the delay variance of the sensory feedback module i in the current period, the weight of the sensory feedback module i in the current period is calculated.

[0053] Here, the weight of any sensory feedback module i in any period t With delay variance The transformation relationship between them is: .

[0054] It is evident that the expected perception moment of the sensory feedback module that makes the more accurate prediction of the delay is also given a greater weight, and when the expected perception moments of each sensory feedback module are integrated in the current cycle, the contribution of the expected perception moment of that sensory feedback module is also greater.

[0055] Here, by introducing the shared moment of target perception, the multi-sensory synchronization of virtual reality can be elevated from "consistent signal arrival time" to "consistent user perception fusion time." This embodiment of the invention calculates the objective delay of different sensory feedback modules and sets independent subjective optimal advance delays. However, if control is implemented solely based on the optimal trigger point of each sensory feedback module, multimodal event attribution will be fragmented, and the wearer's brain will be unable to uniformly bind multi-sensory stimuli such as temperature, touch, and smell to the same virtual event. Therefore, this embodiment of the invention uses the shared moment of target perception as the multimodal synchronization target, enabling the unification of multi-sensory signals at the cognitive level rather than the physical level, fundamentally improving the degree of virtual immersion.

[0056] After obtaining the common perception moment of the target in the current period, it is determined whether this common perception moment falls within the aforementioned constructed synchronization time window. If the common perception moment is not within the constructed synchronization time window, it is adjusted to fall within the synchronization time window. Specifically, if the common perception moment is less than the lower limit of the synchronization time window for the current period, the lower limit of the synchronization time window for the current period is determined as the common perception moment; if the common perception moment is greater than the upper limit of the synchronization time window for the current period, the upper limit of the synchronization time window for the current period is determined as the common perception moment.

[0057] Subsequently, based on the target shared perception time of the current cycle, the start execution time of each sensory feedback module is calculated, and the corresponding sensory feedback module is triggered at its start execution time, so that the wearer of the multi-sensory virtual reality device can synchronize the perception time of each sense corresponding to the multi-sensory command. It should be noted that if the calculated start execution time of any sensory feedback module has passed (i.e., is less than the current time), that sensory feedback module is triggered immediately.

[0058] In some embodiments, for any sensory feedback module, the start execution time of the sensory feedback module can be obtained by subtracting the target co-perception time of the current cycle from the expected delay time of the sensory feedback module within the current cycle.

[0059] In summary, the method provided by this invention constructs a synchronization time window for the current period based on multi-sensory instructions with the same timestamp received in the current period, determines the reception time of the multi-sensory instructions on the multi-sensory virtual reality device, calculates the expected delay time of each sensory feedback module in the current period, and determines the subjective compensation time of each sensory feedback module. Then, based on the reception time of the multi-sensory instructions on the multi-sensory virtual reality device, the expected delay time and subjective compensation time of each sensory feedback module in the current period, the expected perception time of each sensory feedback module in the current period is calculated, and the expected perception times of each sensory feedback module in the current period are fused to obtain the target common perception time of the current period. Thus, when the target common perception time of the current period is within the synchronization time window, the start execution time of each sensory feedback module is calculated based on the target common perception time of the current period, and the corresponding sensory feedback module is triggered at the start execution time of each sensory feedback module, so that the wearer of the multi-sensory virtual reality device can synchronize the perception time of each sense corresponding to the multi-sensory instructions, thereby achieving synchronous feedback of multiple senses.

[0060] This invention provides a multi-sensory virtual reality mask, comprising: Temperature feedback module, used to generate temperature changes through semiconductor cooling heating element; A tactile feedback module is used to provide vibratory tactile stimulation based on a miniature vibration motor. An olfactory feedback module is used to release odors through a piezoelectric atomizing plate, a micro fan, and an airflow control structure; The main control module is used to execute the multi-sensory virtual reality device control method provided in any of the above embodiments.

[0061] Specifically, such as Figure 4 As shown, the multi-sensory virtual reality mask includes a main control module 10, a temperature feedback module 20, a tactile feedback module 30, an olfactory feedback module 40, and a power management and safety protection module 50. The main control module 10 is used to synchronously control each sensory feedback module; the temperature feedback module 20 includes a TEC element and drive circuit, and a temperature sensor, used to generate temperature changes through a semiconductor cooling heating element; the tactile feedback module 30 includes a micro vibration motor and its drive circuit, used to provide vibrational tactile stimulation based on the micro vibration motor; the olfactory feedback module 40 includes an atomizing plate, an odor canister, and a micro fan, used to release odors through the atomizing plate, micro fan, and airflow control structure; the power management and safety protection module 50 includes a lithium battery pack, a power management chip, current and voltage sensors, etc., used to power each module and protect against abnormalities such as overcurrent, overheating, and short circuits.

[0062] This invention provides an electronic device, including: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0063] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0064] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0065] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-sensory virtual reality device control method, characterized by, The method comprises the following steps: constructing a synchronization time window of the current period for multi-sensory instructions with the same timestamp received in the current period; the multi-sensory instructions comprise instructions of any two or more of temperature, touch and olfactory; determining the receiving time of the multi-sensory instructions at the multi-sensory virtual reality device, calculating the internal expected delay time of each sensory feedback module in the multi-sensory virtual reality device in the current period, and determining the subjective compensation time of each sensory feedback module; based on the receiving time of the multi-sensory instructions at the multi-sensory virtual reality device, and the internal expected delay time and subjective compensation time of each sensory feedback module in the current period, calculating the expected perception time of each sensory feedback module in the current period; fusing the expected perception time of each sensory feedback module in the current period to obtain the target common perception time of the current period; when the target common perception time of the current period is within the synchronization time window, calculating the start execution time of each sensory feedback module based on the target common perception time of the current period, and triggering the corresponding sensory feedback module at the start execution time of each sensory feedback module, so that the wearer of the multi-sensory virtual reality device synchronizes the perception time of each sensory corresponding to the multi-sensory instructions.

2. The multi-sensory virtual reality device control method of claim 1, wherein, based on the receiving time of the multi-sensory instructions at the multi-sensory virtual reality device, and the internal expected delay time and subjective compensation time of each sensory feedback module in the current period, calculating the expected perception time of each sensory feedback module in the current period, specifically comprising: for any sensory feedback module, determining the receiving time of the instruction corresponding to the any sensory feedback module in the multi-sensory instructions at the multi-sensory virtual reality device, and adding the receiving time of the instruction corresponding to the any sensory feedback module at the multi-sensory virtual reality device, the internal expected delay time and subjective compensation time of the any sensory feedback module in the current period to obtain the expected perception time of the any sensory feedback module in the current period.

3. The multi-sensory virtual reality device control method of claim 1, wherein, fusing the expected perception time of each sensory feedback module in the current period to obtain the target common perception time of the current period, specifically comprising: The initial target common perception time is calculated based on the following formula : wherein, is the desired perception time of the sensory feedback module i in the current cycle, is the weight of the sensory feedback module i in the current cycle; based on the target common perception time of the last period, smoothing the initial target common perception time to obtain the target common perception time of the current period.

4. The multi-sensory virtual reality device control method of claim 3, wherein, based on the target common perception time of the last period, smoothing the initial target common perception time to obtain the target common perception time of the current period, specifically comprising: the initial target common perception time is smoothed based on the following formula: wherein, is a smoothing factor, is a target common perception time of the previous cycle, is a target common perception time of the current cycle.

5. The multi-sensory virtual reality device control method of claim 3, wherein, the weight of any sensory feedback module in the current period is determined based on the following method: a weight of any sensory feedback module i in a previous period based on the transformation relationship between the weight of any sensory feedback module i in any period and the delay variance of any sensory feedback module i in any period and the transformation relationship between the weight of any sensory feedback module i in any period and the delay variance of any sensory feedback module i in any period, a delay variance of any sensory feedback module i in a previous period ; The delay variance of any of the sensory feedback modules i in the current cycle is calculated based on the following equation : wherein, is a smoothing coefficient, and are the internal actual delay times of the preceding two periods for the respective sensory feedback mold i. based on the transformation relationship between the weight and delay variance of any sensory feedback module i in any period, and the delay variance of the any sensory feedback module i in the current period, the weight of the any sensory feedback module i in the current period is calculated; wherein the weight of any sensory feedback module i at any cycle t and the delay variance is transformed between the two as: 。 6. The multi-sensory virtual reality device control method of claim 5, wherein, the internal expected delay time of any sensory feedback module in any period is calculated based on the following formula: wherein, is the internal predicted delay time for any sensory feedback module i at cycle t, is the internal predicted delay time for any sensory feedback module i at cycle t-1, is the internal actual delay time for any sensory feedback module i at cycle t-1.

7. The multi-sensory virtual reality device control method of claim 1, wherein, calculating the start execution time of each sensory feedback module based on the target common perception time of the current period, specifically comprising: For any sensory feedback module, the target common perception moment of the current period is subtracted from the internal expected delay time of the any sensory feedback module in the current period to obtain the start execution moment of the any sensory feedback module.

8. The multi-sensory virtual reality device control method of claim 1, wherein, For the multi-sensory instructions with the same timestamp received in the current period, a synchronization time window is constructed, including: The execution delay time of each sensory feedback module in the current period and the last period is determined; the execution delay time of any sensory feedback module in any period is the sum of the internal expected delay time of the any sensory feedback module in the period, the transmission time of the instruction corresponding to the any sensory feedback module received in the period, and the subjective compensation time of the any sensory feedback module; If the execution delay time of each sensory feedback module in the current period and the last period is different, the synchronization time window size of the current period is determined based on the difference between the default window size and the execution delay time of each sensory feedback module in the current period and the last period; Based on the timestamp of the multi-sensory instructions and the synchronization time window size of the current period, the synchronization time window of the current period is constructed.

9. The multi-sensory virtual reality device control method of any one of claims 1 to 8, wherein, Any sensory instruction in the multi-sensory instructions includes a frame header field, a preamble field, a timestamp code field, a working bit code field, a time length code field, an intensity bit code field, and a check code field; wherein the frame header field is a fixed identification field, the preamble field indicates the corresponding sensory feedback module of the instruction, the timestamp code field is the timestamp of the instruction generation time, the working bit code field indicates the specific working element in the corresponding sensory feedback module, the time length code field indicates the duration of the corresponding sensory stimulation, the intensity bit code field indicates the intensity of the corresponding sensory stimulation, and the check code field is used to avoid transmission errors of the corresponding instruction.

10. A multi-sensory virtual reality visor, characterized in that, It includes: A temperature feedback module for generating cold and hot changes through a semiconductor refrigeration and heating sheet; A tactile feedback module for providing vibration tactile stimulation based on a micro-vibration motor; An olfactory feedback module for releasing odor through a piezoelectric atomizing sheet, a micro-fan, and an air flow control structure; A master control module for executing the multi-sensory virtual reality device control method according to any one of claims 1 to 9.

Images