Multi-sensory virtual reality device control method and multi-sensory virtual reality mask

Abstract

The application discloses a multi-sensory virtual reality device control method and a multi-sensory virtual reality mask, and belongs to the technical field of virtual reality. The method comprises the following steps: constructing a synchronous time window of a current period according to multi-sensory instructions with the same time stamp received in the current period; calculating expected perception time of each sensory feedback module in the current period based on the receiving time of the multi-sensory instructions at the multi-sensory virtual reality device end, the internal expected delay time and the subjective compensation time of each sensory feedback module in the current period; fusing the expected perception time of each sensory feedback module in the current period to obtain a target common perception time of the current period; calculating the starting execution time of each sensory feedback module based on the target common perception time of the current period when the target common perception time of the current period is in the synchronous time window; and triggering the corresponding sensory feedback module at the starting execution time of each sensory feedback module, so as to realize the synchronous feedback of multi-sensory.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of virtual reality, and more particularly relates to a multi-sensory virtual reality device control method and a multi-sensory virtual reality mask. BACKGROUND

[0002] In recent years, the rapid development of virtual reality technology has greatly promoted the research and application of immersive interactive experience. Through a head-mounted display device, users can obtain a highly realistic visual and auditory experience, which is widely used in education and training, medical rehabilitation, game entertainment and industrial simulation. However, the existing virtual reality system mainly focuses on visual and auditory feedback, and lacks effective stimulation of other senses, resulting in obvious deficiencies in user immersion and realism.

[0003] With the rise of multi-sensory interaction technology, researchers have begun to try to integrate temperature, vibration, smell and other physical stimuli into virtual reality experiences to achieve more realistic environmental perception. For example, simulate fire or ice and snow environment through temperature change, restore garden, kitchen and other scenes through smell feedback. However, the transmission time of instructions corresponding to each sensory feedback module in multi-sensory interactive devices from the server to the virtual reality device may not be consistent, and there is a certain delay from triggering to actually reaching the corresponding intensity of sensory stimulation for each sensory feedback module, and the above delay of different sensory feedback modules is also inconsistent. However, the existing multi-sensory interactive devices usually adopt independent control mode for each sensory feedback module, lack a unified multi-modal coordination mechanism, leading to the problem that different sensory stimuli cannot be synchronized in time sequence, and the feedback control algorithm is simple and does not consider the response delay of different sensory modules, making it difficult to achieve stable real-time experience. Therefore, how to design a virtual reality device control scheme that can realize multi-modal sensory synchronous feedback has become a key technical problem to improve immersive interactive experience. SUMMARY

[0004] In view of the above defects or improvement needs of the prior art, the present application provides a multi-sensory virtual reality device control method and a multi-sensory virtual reality mask, thereby solving the technical problem of how to realize multi-modal sensory synchronous feedback.

[0005] To achieve the above purpose, according to the first aspect of the present application, a multi-sensory virtual reality device control method is provided, comprising:

[0006] For the multi-sensory instructions with the same timestamp received in the current period, a synchronous time window of the current period is constructed; the multi-sensory instructions include instructions for any two or more of temperature, touch and smell;

[0007] determining a receiving time of the multi-sensory instruction at the multi-sensory virtual reality device end, calculating an internal expected delay time of each sensory feedback module in the multi-sensory virtual reality device in a current period, and determining a subjective compensation time of the each sensory feedback module;

[0008] based on the receiving time of the multi-sensory instruction at the multi-sensory virtual reality device end, and the internal expected delay time and the subjective compensation time of each sensory feedback module in the current period, calculating an expected perception time of each sensory feedback module in the current period;

[0009] fusing the expected perception time of each sensory feedback module in the current period to obtain a target common perception time in the current period;

[0010] when the target common perception time in the current period is within the synchronization time window, calculating a starting execution time of each sensory feedback module based on the target common perception time in the current period, and triggering the corresponding sensory feedback module at the starting execution time of each sensory feedback module, so that the wearer of the multi-sensory virtual reality device perceives the multi-sensory instruction corresponding to each sense at the same time.

[0011] According to the multi-sensory virtual reality device control method described above, based on the receiving time of the multi-sensory instruction at the multi-sensory virtual reality device end, and the internal expected delay time and the subjective compensation time of each sensory feedback module in the current period, the expected perception time of each sensory feedback module in the current period is calculated, which specifically includes:

[0012] for any sensory feedback module, determining the receiving time of the instruction corresponding to the any sensory feedback module in the multi-sensory instruction at the multi-sensory virtual reality device end, and adding the receiving time of the instruction corresponding to the any sensory feedback module at the multi-sensory virtual reality device end, the internal expected delay time and the subjective compensation time of the any sensory feedback module in the current period to obtain the expected perception time of the any sensory feedback module in the current period.

[0013] According to the multi-sensory virtual reality device control method described above, the expected perception time of each sensory feedback module in the current period is fused to obtain the target common perception time in the current period, which specifically includes:

[0014] the initial target common perception time is calculated based on the following formula :

[0015]

[0016] wherein, is the expected perception time of sensory feedback module i in the current period, is the weight of sensory feedback module i in the current period.

[0017] smooth the initial target common perception time based on the target common perception time of the last period to obtain the target common perception time of the current period.

[0018] According to the multi-sensory virtual reality device control method, the initial target common perception time is smoothed based on the target common perception time of the last period to obtain the target common perception time of the current period, and specifically includes:

[0019] The initial target common perception time is smoothed based on the following formula:

[0020]

[0021] wherein, is a smoothing factor, is the target common perception time of the last period, is the target common perception time of the current period.

[0022] According to the multi-sensory virtual reality device control method, the weight of any sensory feedback module in the current period is determined based on the following manner:

[0023] The weight of any sensory feedback module i in the last period is based on the weight of any sensory feedback module i in the last period , and the transformation relationship between the weight and the delay variance of any sensory feedback module i in any period, to calculate the delay variance of any sensory feedback module i in the last period .

[0024] The delay variance of any sensory feedback module i in the current period is calculated based on the following formula :

[0025]

[0026] wherein, is a smoothing factor, and are the internal actual delay times of the sensory feedback module i in the last two periods;

[0027] The weight of any sensory feedback module i in the current period is calculated based on the transformation relationship between the weight and the delay variance of any sensory feedback module i in any period, and the delay variance of any sensory feedback module i in the current period.

[0028] wherein, the transformation relationship between the weight and the delay variance of any sensory feedback module i in any period t is:

[0029] .

[0030] According to the multi-sensory virtual reality device control method, the internal expected delay time of any sensory feedback module in any period is calculated based on the following formula:

[0031]

[0032] wherein, is the internal expected delay time of any sensory feedback module i in period t, is the internal expected delay time of any sensory feedback module i in period t-1, is the internal actual delay time of any sensory feedback module i in period t-1.

[0033] According to the multi-sensory virtual reality device control method, the start execution time of each sensory feedback module is calculated based on the target common perception time of the current period, specifically including:

[0034] For any sensory feedback module, the target common perception time of the current period is subtracted from the internal expected delay time of the any sensory feedback module in the current period to obtain the start execution time of the any sensory feedback module.

[0035] According to the multi-sensory virtual reality device control method, for the multi-sensory instructions with the same timestamp received in the current period, a synchronization time window is constructed, including:

[0036] The execution delay time of each sensory feedback module in the current period and the previous period is determined; the execution delay time of any sensory feedback module in any period is the sum of the internal expected delay time of the any sensory feedback module in the period, the transmission time of the instruction corresponding to the any sensory feedback module received in the period, and the subjective compensation time of the any sensory feedback module;

[0037] If the execution delay time of each sensory feedback module in the current period and the previous period is different, the synchronization time window size of the current period is determined based on the difference between the default window size and the execution delay time of each sensory feedback module in the current period and the previous period;

[0038] Based on the timestamp of the multi-sensory instructions and the synchronization time window size of the current period, the synchronization time window of the current period is constructed.

[0039] According to the multi-sensory virtual reality device control method, the instruction of any sense in the multi-sensory instruction includes a frame header field, a preamble field, a timestamp code field, a working bit code field, a time length code field, an intensity bit code field and a check code field; the frame header field is a fixed identification field, the preamble field indicates the corresponding sense feedback module of the instruction, the timestamp code field is a timestamp of the instruction generation time, the working bit code field indicates the specific working element in the corresponding sense feedback module, the time length code field indicates the duration of the corresponding sense stimulation, the intensity bit code field indicates the intensity of the corresponding sense stimulation, and the check code field is used to avoid transmission errors of the corresponding instruction.

[0040] According to a second aspect of the present application, a multi-sensory virtual reality mask is provided, comprising:

[0041] A temperature feedback module is configured to generate cold and hot changes through a semiconductor refrigeration and heating sheet.

[0042] A tactile feedback module is configured to provide vibration tactile stimulation based on a micro-vibration motor.

[0043] An olfactory feedback module is configured to release an odor through a piezoelectric atomizing sheet, a micro-fan and an air flow control structure.

[0044] A master control module is configured to execute the multi-sensory virtual reality device control method according to the first aspect.

[0045] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0046] By receiving the multi-sensory instructions with the same timestamp for the current period, a synchronous time window for the current period is constructed, the receiving time of the multi-sensory instructions at the multi-sensory virtual reality device end is determined, the internal expected delay time of each sense feedback module in the multi-sensory virtual reality device for the current period is calculated, and the subjective compensation time of each sense feedback module is determined. Then, based on the receiving time of the multi-sensory instructions at the multi-sensory virtual reality device end, the internal expected delay time and the subjective compensation time of each sense feedback module for the current period, the expected perception time of each sense feedback module for the current period is calculated, and the expected perception time of each sense feedback module for the current period is fused to obtain the target common perception time for the current period. Thus, when the target common perception time for the current period is within the synchronous time window, the start execution time of each sense feedback module is calculated based on the target common perception time for the current period, and the corresponding sense feedback module is triggered at the start execution time of each sense feedback module, so that the wearer of the multi-sensory virtual reality device synchronizes the perception time of each sense corresponding to the multi-sensory instruction, and realizes the synchronous feedback of multi-senses. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 A flowchart of a multi-sensory virtual reality device control method provided by an embodiment of the present application is shown.

[0048] Figure 2 An instruction encoding diagram provided by an embodiment of the present application is shown.

[0049] Figure 3 A double buffer diagram provided by an embodiment of the present application is shown.

[0050] Figure 4 A structure diagram of a multi-sensory virtual reality mask provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0052] An embodiment of the present application provides a multi-sensory virtual reality device control method, as shown in Figure 1 , comprising:

[0053] Step 110, for the multi-sensory instructions with the same timestamp received in the current period, a synchronization time window of the current period is constructed; the multi-sensory instructions include instructions corresponding to any two or more of temperature, touch and smell;

[0054] Step 120, the receiving time of the multi-sensory instructions at the multi-sensory virtual reality device end is determined, the internal expected delay time of each sensory feedback module in the multi-sensory virtual reality device in the current period is calculated, and the subjective compensation time of the each sensory feedback module is determined;

[0055] Step 130, based on the receiving time of the multi-sensory instructions at the multi-sensory virtual reality device end, and the internal expected delay time and subjective compensation time of each sensory feedback module in the current period, the expected perception time of each sensory feedback module in the current period is calculated;

[0056] Step 140, the expected perception time of each sensory feedback module in the current period is fused to obtain the target common perception time of the current period;

[0057] Step 150, when the target common perception time of the current period is within the synchronization time window, the starting execution time of each sensory feedback module is calculated based on the target common perception time of the current period, and the corresponding sensory feedback module is triggered at the starting execution time of each sensory feedback module, so that the wearer of the multi-sensory virtual reality device synchronizes the perception time of each sense corresponding to the multi-sensory instruction.

[0058] Here, the rendering program of the virtual reality host outputs the frame picture and event parameters of the current scene in each rendering frame period, including but not limited to: frame picture, environmental temperature label, object interaction intensity, explosion / impact event, smell trigger point, smell concentration level, etc. The above data is read in real time through a plug-in or SDK interface, and is packaged as a structured event object and sent to the edge server. After receiving the structured event object, the edge server unpacks and analyzes the structured event object, and according to the analysis content, constructs the instructions of each sense using a unified encoding format, and then sends them to the virtual reality device end (such as a virtual reality mask). In some embodiments, the unified encoding format defines the following fields: frame header field (F), preamble field (S), timestamp code field (T), working bit code field (W), time length code field (D), intensity bit code field (I), and check code field (C). The frame header field is a fixed identification field, the preamble field indicates the corresponding sensory feedback module of the corresponding instruction, the timestamp code field is the timestamp of the instruction generation time, the working bit code field indicates the specific working element in the corresponding sensory feedback module, the time length code field indicates the duration of the corresponding sensory stimulation, the intensity bit code field indicates the intensity of the corresponding sensory stimulation, and the check code field is used to avoid transmission errors of the corresponding instruction. An exemplary instruction encoding is shown in Figure 2 .

[0059] In some embodiments, in order to reduce the size of the instruction encoding to reduce the instruction transmission pressure, continuous similar encoding can be compressed. For continuous sensory instructions with the same preamble field, the preamble field can be merged, and only one preamble field is retained with a continuous frame number field; for the timestamp code field, only the absolute timestamp of the first frame is recorded, and the time increment ΔTi is written for the subsequent frames to avoid repeated transmission of the complete timestamp field. For the original check code field of each frame, a unified check code for continuous sensory instruction encoding can be updated to reduce the instruction packet redundancy overhead.

[0060] For example: for the same sensory module, three frames of tactile instructions are continuously output:

[0061] F, S=01, T=100, W1, D1, I1, C1

[0062] F, S=01, T=600, W2, D2, I2, C2

[0063] F, S=01, T=1100, W3, D3, I3, C3

[0064] The compressed code obtained after using the above encoding method is:

[0065] F, S=01, COUNT=03, T=100, ΔT=500, W1, D1, I1, W2, D2, I2, W3, D3,I3, C block C block To standardize the verification code.

[0066] When most fields of consecutive instructions are consistent, a field merging encoding method can be adopted. This method replaces the frame-by-frame encoding of multiple repeated instructions by adding a repeat count field REPEAT. The value of REPEAT represents the number of repeated fields on the left. The original checksum for each frame is replaced with a unified checksum for consecutive instructions to reduce packet size and communication bandwidth usage.

[0067] For example: For different sensory modules, multiple sensory commands are triggered at the same time, and the W, D, I fields of multiple sensory commands are the same:

[0068] F, S=01, T1, W1, D1, I1, C1

[0069] F, S=10, T2, W2, D2, I2, C2

[0070] F, S=11, T3, W3, D3, I3, C3

[0071] The compressed code obtained after using the above encoding method is:

[0072] F, S=01,10,11, T1, W1, D1, I1, REPEAT=04, C block .

[0073] Furthermore, the edge server periodically (e.g., every 500ms) aggregates the multi-sensory events that need to be triggered, forming a command packet containing several coded frames. This ensures that all frames within the command packet have a uniform generation time range, thereby maintaining strict synchronization on the virtual reality device. The command batch packetization mechanism includes the following steps:

[0074] 1. The edge server collects all sensory instruction frames in chronological order, using a synchronization time window of size W;

[0075] 2. Establish a unified packet header, containing the following information:

[0076] 1) Packaging header identifier, fixed as 0xFF

[0077] 2) Package timestamp

[0078] 3) Size of the synchronization time window

[0079] 4) Current batch instruction count.

[0080] After receiving the instruction sent by the edge server in the current cycle, the multi-sensory virtual reality device parses and schedules the received instruction using the aforementioned unified encoding format. In some embodiments, after receiving consecutive instruction frames, the multi-sensory virtual reality device can employ a double-buffering mechanism to store and schedule the instructions. This double-buffering mechanism provides two buffers: one buffer stores the instruction to be executed in the current cycle, and the other buffer stores the instruction for the next cycle; these two buffers are periodically exchanged. Here, the duration of any cycle varies with the size of the synchronization time window for that cycle. Figure 3 As shown, assuming two buffers, buffer A and buffer B, in cycle T, instructions from buffer A are read, parsed, synchronized, and distributed to trigger the corresponding sensory feedback module. In cycle T+1, instructions from buffer B are read, parsed, synchronized, and distributed. In cycle T+2, the process of parsing, synchronizing, and distributing instructions from buffer A continues, and so on. Even if jitter occurs during communication, the dual-buffered queue mechanism decouples instruction reception and execution time within the execution cycle, providing additional buffer time for instruction transmission. When the communication jitter amplitude is within the buffer cycle tolerance range, this mechanism can effectively ensure the complete reception and orderly execution of instruction data in each buffer, thereby improving the stability and continuity of instruction execution on the virtual reality device side.

[0081] Specifically, if multi-sensory instructions with the same timestamp are received, they will be processed synchronously to ensure that the virtual reality device wearer perceives multiple sensory stimuli at consistent times, achieving consistency in multi-sensory feedback. Here, multi-sensory instructions include instructions corresponding to any two or more of the following senses: temperature, touch, and smell.

[0082] Specifically, this embodiment of the invention introduces a synchronization time window mechanism to restrict the execution and feedback of multi-sensory instructions within a synchronization time window. Specifically, for multi-sensory instructions with the same timestamp received in the current period, a synchronization time window for the current period is first constructed. For example, the synchronization time window for the current period can be constructed based on the timestamp of the multi-sensory instruction, combined with the window length set for the current period, thus obtaining a window that can be represented as... The synchronization time window, in which The time it takes for the virtual reality device to receive the instruction. W Indicates the window length of the synchronization time window.

[0083] To improve the stability of multi-sensory feedback synchronization, some embodiments provide a window adaptive adjustment mechanism. Specifically, the execution delay time of each sensory feedback module in the virtual reality device in the current cycle and the previous cycle can be determined. The execution delay time of any sensory feedback module in any cycle is the sum of the expected delay time of the sensory feedback module within that cycle, the transmission time of the instruction corresponding to the sensory feedback module received in that cycle, and the subjective compensation time of the sensory feedback module. Here, the expected delay time of any sensory feedback module within any cycle is the system-predicted delay from triggering the sensory feedback module to actually reaching the corresponding intensity of sensory stimulation, and the transmission time of the instruction corresponding to the sensory feedback module is the time from the generation of the instruction at the edge server to its transmission to the virtual reality device. Subsequently, the execution delay times of each sensory feedback module in the current cycle and the previous cycle are compared to increase or decrease the window size based on a default window size (e.g., 500ms). Specifically, the average execution delay time of each sensory feedback module in the current cycle and the average execution delay time of each sensory feedback module in the previous cycle can be determined, and then the difference between these two averages is calculated. If the difference is not zero, the default window size is adjusted based on this difference to obtain the synchronization time window size for the current period. For example, the synchronization time window for the current period can be determined as follows:

[0084] If δ best (k) is greater than or less than δ best (k-1), then W(k)=500±|δ best (k)-δ best (k-1)|.

[0085] Where, δ best (k) and δ best (k-1) represent the average execution delay time of the current period and the average execution delay time of the previous period, respectively, and W(k) is the size of the synchronization time window of the current period.

[0086] Alternatively, it can be based on the size of the synchronization time window from the previous cycle. The synchronization time window size for the current period obtained above. Smoothing is performed, and the smoothing process can be represented by the following formula:

[0087]

[0088] in, This is the smoothing coefficient. For example, .

[0089] Based on the timestamp of the multi-sensory instruction and the size of the synchronization time window for the current period, a synchronization time window for the current period can be constructed.

[0090] The synchronization time window is specifically applied in the following aspects: 1) Edge servers collect all sensory event frames according to the size of the synchronization time window and arrange them in chronological order; 2) The synchronization time window is also the unit time for the virtual reality device hardware to update its state once. The synchronization time window determines the time range within which the expected perception moment of each sensory feedback module is calculated, i.e., the minimum multi-sensory resolution. Theoretically, the smaller the synchronization time window, the higher the frequency of calculating the expected perception moment of each sensory feedback module, the more timely the signal processing, the higher the realism and fidelity, and the higher the hardware requirements of the device. Therefore, the role of the synchronization time window is to provide the time range for calculating the expected perception moment of each sensory feedback module, thereby mapping physically delayed temperature, tactile, and olfactory signals to a unified sensory integration time domain. This allows multimodal stimuli to form a single event attribution at the cognitive level of the brain, thus achieving cross-modal immersive experience synchronization, rather than relying solely on physical delay alignment.

[0091] After establishing the synchronization time window for the current cycle, the reception time of multi-sensory commands at the multi-sensory virtual reality device is determined. The expected delay time within the current cycle for each sensory feedback module in the multi-sensory virtual reality device is calculated, and the subjective compensation time for each sensory feedback module is determined. As mentioned above, the expected delay time within the current cycle for each sensory feedback module is the system-predicted delay from triggering the module to actually reaching the corresponding intensity of sensory stimulation. The subjective compensation time for each sensory feedback module is a compensation time set based on the subjective perception delay of various sensory stimuli by the human body, used to compensate for the subjective perception delay of various sensory stimuli by triggering the modules in advance. In some embodiments, the objective law of "slow for cold and heat, fast for vibration, and slowest for odor" can be followed to set corresponding subjective compensation times for each sensory feedback module. For example, the subjective compensation time for the temperature feedback module can be set to -200ms, the subjective compensation time for the tactile feedback module to -100ms, and the subjective compensation time for the olfactory feedback module to -300ms.

[0092] Based on the reception time of multi-sensory commands at the multi-sensory virtual reality device, and the internal estimated delay time and subjective compensation time of each sensory feedback module in the current cycle, the expected perception time of each sensory feedback module in the current cycle is calculated. Here, the expected perception time of each sensory feedback module in the current cycle represents the ideal time perceived by the virtual reality device wearer within the current cycle for that corresponding sensory feedback module. In some embodiments, for any sensory feedback module, the reception time of the command corresponding to that sensory feedback module in the multi-sensory commands at the multi-sensory virtual reality device is determined, and the reception time of the command corresponding to that sensory feedback module at the multi-sensory virtual reality device, the internal estimated delay time and subjective compensation time of that sensory feedback module in the current cycle are accumulated to obtain the expected perception time of that sensory feedback module in the current cycle.

[0093] In some embodiments, the estimated delay time of any sensory feedback module within any cycle can be calculated based on the following formula:

[0094]

[0095] in, For any sensory feedback module i, the estimated delay time within period t is... For any sensory feedback module i, the estimated delay time within period t-1 is given. Let be the actual delay time of any sensory feedback module i within the period t-1.

[0096] Subsequently, the expected perception moments of each sensory feedback module in the current cycle are fused to obtain the common perception moment of the target in the current cycle. This common perception moment is the ideal moment when the sensory stimuli given by each sensory feedback module are simultaneously perceived, calculated based on the expected perception moment of each sensory feedback module in the current cycle.

[0097] In some embodiments, the initial target co-sensing moment can be calculated based on the following formula. :

[0098]

[0099] in, For sensory feedback module i, the expected perception moment in the current cycle, The weight of sensory feedback module i in the current cycle;

[0100] Then, based on the target co-perception time of the previous cycle, the above initial target co-perception time is smoothed to obtain the target co-perception time of the current cycle.

[0101] In other embodiments, the initial common perception moment of the target can be smoothed based on the following formula:

[0102]

[0103] in, As a smoothing factor, To jointly perceive the goals of the previous cycle. To jointly perceive the moment for the goals of the current cycle.

[0104] Considering the inherent delay between the triggering of a sensory feedback module and the actual arrival of the corresponding intensity of sensory stimulus, and the varying degrees of delay across different sensory feedback modules, the system's prediction accuracy for these delays also differs. Therefore, to improve the stability and accuracy of multi-sensory feedback synchronization, when fusing the expected perception times of each sensory feedback module in the current cycle to determine the common perception time of the target in the current cycle, the expected perception time of the sensory feedback module that provides the most accurate prediction of the aforementioned delay can be used as a reference, and other sensory feedback modules can align themselves with this reference sensory feedback module. Specifically, in some embodiments, the weight of any sensory feedback module in the current cycle can be determined based on the following method:

[0105] Based on the weight of sensory feedback module i in the previous cycle And the transformation relationship between the weight and delay variance of any sensory feedback module i in any period, and calculate the delay variance of the sensory feedback module i in the previous period. ;

[0106] The delay variance of sensory feedback module i in the current period is calculated based on the following formula. :

[0107]

[0108] in, For smoothing coefficients, and These represent the actual delay times within the first two cycles for sensory feedback module i. Here, the actual delay time within any cycle for any sensory feedback module is the real delay from triggering the sensory feedback module to actually reaching the corresponding intensity of sensory stimulation.

[0109] Subsequently, based on the transformation relationship between the weight and delay variance of any sensory feedback module i in any period, and the delay variance of the sensory feedback module i in the current period, the weight of the sensory feedback module i in the current period is calculated.

[0110] Here, the weight of any sensory feedback module i in any period t With delay variance The transformation relationship between them is:

[0111] .

[0112] It is evident that the expected perception moment of the sensory feedback module that makes the more accurate prediction of the delay is also given a greater weight, and when the expected perception moments of each sensory feedback module are integrated in the current cycle, the contribution of the expected perception moment of that sensory feedback module is also greater.

[0113] Here, by introducing the shared moment of target perception, the multi-sensory synchronization of virtual reality can be elevated from "consistent signal arrival time" to "consistent user perception fusion time." This embodiment of the invention calculates the objective delay of different sensory feedback modules and sets independent subjective optimal advance delays. However, if control is implemented solely based on the optimal trigger point of each sensory feedback module, multimodal event attribution will be fragmented, and the wearer's brain will be unable to uniformly bind multi-sensory stimuli such as temperature, touch, and smell to the same virtual event. Therefore, this embodiment of the invention uses the shared moment of target perception as the multimodal synchronization target, enabling the unification of multi-sensory signals at the cognitive level rather than the physical level, fundamentally improving the degree of virtual immersion.

[0114] After obtaining the common perception moment of the target in the current period, it is determined whether this common perception moment falls within the aforementioned constructed synchronization time window. If the common perception moment is not within the constructed synchronization time window, it is adjusted to fall within the synchronization time window. Specifically, if the common perception moment is less than the lower limit of the synchronization time window for the current period, the lower limit of the synchronization time window for the current period is determined as the common perception moment; if the common perception moment is greater than the upper limit of the synchronization time window for the current period, the upper limit of the synchronization time window for the current period is determined as the common perception moment.

[0115] Subsequently, based on the target shared perception time of the current cycle, the start execution time of each sensory feedback module is calculated, and the corresponding sensory feedback module is triggered at its start execution time, so that the wearer of the multi-sensory virtual reality device can synchronize the perception time of each sense corresponding to the multi-sensory command. It should be noted that if the calculated start execution time of any sensory feedback module has passed (i.e., is less than the current time), that sensory feedback module is triggered immediately.

[0116] In some embodiments, for any sensory feedback module, the start execution time of the sensory feedback module can be obtained by subtracting the target co-perception time of the current cycle from the expected delay time of the sensory feedback module within the current cycle.

[0117] In summary, the method provided by this invention constructs a synchronization time window for the current period based on multi-sensory instructions with the same timestamp received in the current period, determines the reception time of the multi-sensory instructions on the multi-sensory virtual reality device, calculates the expected delay time of each sensory feedback module in the current period, and determines the subjective compensation time of each sensory feedback module. Then, based on the reception time of the multi-sensory instructions on the multi-sensory virtual reality device, the expected delay time and subjective compensation time of each sensory feedback module in the current period, the expected perception time of each sensory feedback module in the current period is calculated, and the expected perception times of each sensory feedback module in the current period are fused to obtain the target common perception time of the current period. Thus, when the target common perception time of the current period is within the synchronization time window, the start execution time of each sensory feedback module is calculated based on the target common perception time of the current period, and the corresponding sensory feedback module is triggered at the start execution time of each sensory feedback module, so that the wearer of the multi-sensory virtual reality device can synchronize the perception time of each sense corresponding to the multi-sensory instructions, thereby achieving synchronous feedback of multiple senses.

[0118] This invention provides a multi-sensory virtual reality mask, comprising:

[0119] Temperature feedback module, used to generate temperature changes through semiconductor cooling heating element;

[0120] A tactile feedback module is used to provide vibratory tactile stimulation based on a miniature vibration motor.

[0121] An olfactory feedback module is used to release odors through a piezoelectric atomizing plate, a micro fan, and an airflow control structure;

[0122] The main control module is used to execute the multi-sensory virtual reality device control method provided in any of the above embodiments.

[0123] Specifically, such as Figure 4 As shown, the multi-sensory virtual reality mask includes a main control module 10, a temperature feedback module 20, a tactile feedback module 30, an olfactory feedback module 40, and a power management and safety protection module 50. The main control module 10 is used to synchronously control each sensory feedback module; the temperature feedback module 20 includes a TEC element and drive circuit, and a temperature sensor, used to generate temperature changes through a semiconductor cooling heating element; the tactile feedback module 30 includes a micro vibration motor and its drive circuit, used to provide vibrational tactile stimulation based on the micro vibration motor; the olfactory feedback module 40 includes an atomizing plate, an odor canister, and a micro fan, used to release odors through the atomizing plate, micro fan, and airflow control structure; the power management and safety protection module 50 includes a lithium battery pack, a power management chip, current and voltage sensors, etc., used to power each module and protect against abnormalities such as overcurrent, overheating, and short circuits.

[0124] This invention provides an electronic device, including: a computer-readable storage medium and a processor;

[0125] The computer-readable storage medium is used to store executable instructions;

[0126] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0127] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0128] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0129] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A control method for a multi-sensory virtual reality device, characterized in that, include: For multi-sensory instructions with the same timestamp received in the current cycle, construct a synchronization time window for the current cycle; The multi-sensory instructions include instructions corresponding to any two or more of the following senses: temperature, touch, and smell. Determine the reception time of the multi-sensory command at the multi-sensory virtual reality device, calculate the expected delay time of each sensory feedback module in the multi-sensory virtual reality device within the current cycle, and determine the subjective compensation time of each sensory feedback module. Based on the reception time of the multi-sensory instructions at the multi-sensory virtual reality device, and the expected delay time and subjective compensation time of each sensory feedback module in the current cycle, the expected perception time of each sensory feedback module in the current cycle is calculated. The expected perception moment of each sensory feedback module in the current cycle is fused to obtain the common perception moment of the target in the current cycle. When the common perception time of the target in the current cycle is within the synchronization time window, the start execution time of each sensory feedback module is calculated based on the common perception time of the target in the current cycle, and the corresponding sensory feedback module is triggered at the start execution time of each sensory feedback module, so that the wearer of the multi-sensory virtual reality device can synchronize the perception time of each sense corresponding to the multi-sensory command.

2. The multi-sensory virtual reality device control method as described in claim 1, characterized in that, Based on the reception time of the multi-sensory commands at the multi-sensory virtual reality device, and the expected delay time and subjective compensation time of each sensory feedback module within the current cycle, the expected perception time of each sensory feedback module in the current cycle is calculated, specifically including: For any sensory feedback module, the reception time of the instruction corresponding to the sensory feedback module in the multi-sensory instructions on the multi-sensory virtual reality device is determined, and the reception time of the instruction corresponding to the sensory feedback module on the multi-sensory virtual reality device, the internal expected delay time and subjective compensation time of the sensory feedback module in the current cycle are accumulated to obtain the expected perception time of the sensory feedback module in the current cycle.

3. The multi-sensory virtual reality device control method as described in claim 1, characterized in that, The expected perception moment of each sensory feedback module in the current cycle is fused to obtain the common perception moment of the target in the current cycle, specifically including: The initial target co-sensing time is calculated based on the following formula. : in, For sensory feedback module i, the expected perception moment in the current cycle, The weight of sensory feedback module i in the current cycle; The initial target common perception time is smoothed based on the target common perception time of the previous cycle to obtain the target common perception time of the current cycle.

4. The multi-sensory virtual reality device control method as described in claim 3, characterized in that, The initial target common perception time is smoothed based on the target common perception time of the previous cycle to obtain the target common perception time of the current cycle, specifically including: The initial target common sensing time is smoothed based on the following formula: in, As a smoothing factor, To jointly perceive the goals of the previous cycle. To jointly perceive the moment for the goals of the current cycle.

5. The multi-sensory virtual reality device control method as described in claim 3, characterized in that, The weight of any sensory feedback module in the current cycle is determined based on the following method: Based on the weight of any of the sensory feedback modules i in the previous cycle And the transformation relationship between the weight and delay variance of any sensory feedback module i in any period, and calculate the delay variance of the sensory feedback module i in the previous period. ; The delay variance of any sensory feedback module i in the current period is calculated based on the following formula. : in, For smoothing coefficients, and These are the actual internal delay times of any sensory feedback module i in the first two cycles; Based on the transformation relationship between the weight and delay variance of any sensory feedback module i in any period, and the delay variance of any sensory feedback module i in the current period, calculate the weight of any sensory feedback module i in the current period. Wherein, the weight of any sensory feedback module i in any period t With delay variance The transformation relationship between them is: 。 6. The multi-sensory virtual reality device control method as described in claim 5, characterized in that, The expected delay time of any sensory feedback module within any cycle is calculated using the following formula: in, For any sensory feedback module i, the estimated delay time within period t is... For any sensory feedback module i, the estimated delay time within period t-1 is given. Let be the actual delay time of any sensory feedback module i within the period t-1.

7. The multi-sensory virtual reality device control method as described in claim 1, characterized in that, The start execution time of each sensory feedback module is calculated based on the target co-perception time of the current cycle, specifically including: For any sensory feedback module, the start execution time of the sensory feedback module is obtained by subtracting the target co-perception time of the current cycle from the expected delay time of the sensory feedback module within the current cycle.

8. The multi-sensory virtual reality device control method as described in claim 1, characterized in that, For multi-sensory instructions with the same timestamp received in the current cycle, a synchronization time window is constructed, including: The execution delay time of each sensory feedback module in the current cycle and the previous cycle is determined; the execution delay time of any sensory feedback module in any cycle is the sum of the expected delay time of the sensory feedback module within the cycle, the transmission time of the instruction corresponding to the sensory feedback module received in the cycle, and the subjective compensation time of the sensory feedback module. If the execution delay time of each sensory feedback module is different in the current cycle and the previous cycle, the synchronization time window size of the current cycle is determined based on the difference between the default window size and the execution delay time of each sensory feedback module in the current cycle and the previous cycle. Based on the timestamp of the multi-sensory instruction and the size of the synchronization time window for the current period, a synchronization time window for the current period is constructed.

9. The multi-sensory virtual reality device control method according to any one of claims 1 to 8, characterized in that, The multi-sensory instructions for any of the senses include a frame header field, a preamble field, a timestamp field, a working bit field, a duration field, an intensity bit field, and a checksum field. The frame header field is a fixed identifier field; the preamble field indicates the sensory feedback module corresponding to the instruction; the timestamp field is a timestamp indicating the time the instruction was generated; the working bit field indicates the specific working element within the corresponding sensory feedback module; the duration field indicates the duration of the corresponding sensory stimulus; the intensity bit field indicates the intensity of the corresponding sensory stimulus; and the checksum field is used to prevent transmission errors of the corresponding instruction.

10. A multi-sensory virtual reality mask, characterized in that, include: Temperature feedback module, used to generate temperature changes through semiconductor cooling heating element; A tactile feedback module is used to provide vibratory tactile stimulation based on a miniature vibration motor. An olfactory feedback module is used to release odors through a piezoelectric atomizing plate, a micro fan, and an airflow control structure; The main control module is used to execute the multi-sensory virtual reality device control method as described in any one of claims 1 to 9.

Images