Airflow generating method, device and electronic equipment
By integrating audio and airflow generation devices in virtual reality/augmented reality devices to simulate airflow in virtual scenes, the problem of users not being able to experience facial airflow is solved, and the immersion and experience quality is improved.
Patent Information
- Application Number
- CN202311055849.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-22
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-08-22
AI Technical Summary
In virtual reality/augmented reality technology, the head-mounted display device isolates external airflow when worn, resulting in users being unable to experience the facial airflow in the virtual scene, making it difficult to achieve an immersive experience.
By integrating an audio playback device and an airflow generation device in an electronic device, the size and direction of the airflow sound are calculated based on the audio signal, and the airflow generation device outputs simulated airflow, enhancing the user's virtual experience.
It realizes simulating airflow in virtual scenes, improving user immersion and experience quality, and enhancing the interaction with virtual scenes.
Smart Images

Figure CN116774836B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of virtual reality / augmented reality technology, and in particular to an airflow generation method, device and electronic equipment. Background Art
[0002] With the development of computer graphics technology, VR technology and AR technology are gradually applied to people's lives. VR technology and AR technology use electronic equipment 100 to simulate a three-dimensional virtual scene, providing a simulated experience in vision, hearing, touch or other senses, so that users feel as if they are in the scene. In addition, users can also interact with the simulated virtual scene.
[0003] In some virtual scenes, users can experience some virtual scenes through head mounted display (HMD). For example, through HMD, users can experience virtual stimulating scenes such as riding a convertible, roller coaster, and mountain climbing. However, in these virtual stimulating scenes, the HMD is worn on the user's eyes and face, isolating the external airflow, making the airflow inside the HMD basically still. Users cannot experience the feeling of airflow blowing over their faces in virtual stimulating scenes, and it is difficult to achieve an immersive effect. Therefore, how to improve the user's immersive experience needs further research. Summary of the invention
[0004] The present application provides an airflow generation method, device and electronic device, which can generate simulated airflow based on audio signals in a virtual scene, achieve an immersive effect, and enhance the user's virtual experience.
[0005] In a first aspect, the present application provides an airflow generating method, wherein the electronic device comprises a display device, an audio playback device and an airflow generating device, the audio playback device comprises a first audio playback device and a second audio playback device, the airflow generating device comprises a first airflow generating device and a second airflow generating device, and the method comprises: displaying a virtual object through a display device; playing a first audio through a first audio playback device, and playing a second audio through a second audio playback device; calculating a first airflow size of an airflow sound in the first audio and a second airflow size of an airflow sound in the second audio; outputting an airflow of the first airflow size through the first airflow generating device, and outputting an airflow of the second airflow size through the second airflow generating device.
[0006] Among them, the electronic device can be a VR device or an AR device, for example, the electronic device can be a head-mounted display device, etc.
[0007] The present invention is not limited to two audio playback devices. The electronic device may include only one, or the electronic device may include more than two. This application does not make any limitation on this.
[0008] The present invention is not limited to two airflow generating devices. The electronic device may include only one airflow generating device, or the electronic device may include more than two airflow generating devices. This application does not make any limitation on this.
[0009] The size of the airflow refers to the size of the thrust in the forward direction. The greater the airflow, the greater the thrust. The smaller the airflow, the smaller the thrust.
[0010] The electronic device is used to be worn on the user's head so that the user's eyes are located inside the electronic device, and the user's eyes can view the virtual object displayed by the display device.
[0011] The first and second may be located on the left and right sides of the electronic device respectively.
[0012] When the electronic device is worn on the user's head, relative to the position of the user's eyes, the first airflow generating device and the second airflow generating device can be respectively located at the upper left front, upper right front, or lower left front, lower right front, or upper left front, lower right front, or lower left front, upper right front, etc. of the user's eyes.
[0013] In some embodiments, the first airflow generating device and the second airflow generating device may be located outside the display area of the display device to avoid blocking the user from observing the virtual object displayed on the display device. Through this method, a simulated airflow can be generated based on an audio signal in a virtual scene to achieve an immersive effect and enhance the user's virtual experience.
[0014] In combination with the first aspect, in a possible implementation method, calculating a first airflow size of an airflow sound in a first audio and a second airflow size of an airflow sound in a second audio specifically includes: processing the first audio through a low-pass filter to obtain a first low-frequency signal; processing the second audio through a low-pass filter to obtain a second low-frequency signal; calculating the airflow size of the first low-frequency signal to obtain the first airflow size; calculating the airflow size of the second low-frequency signal to obtain the second airflow size.
[0015] Generally speaking, the frequency of airflow sounds is relatively low. The high-frequency signals in the audio can be filtered out by a low-pass filter, leaving only the low-frequency signals. The low-frequency signals are used as airflow sounds, and the airflow size of the airflow sounds is calculated based on the low-frequency signals. This can improve the accuracy of the airflow size obtained by electronic devices.
[0016] In combination with the first aspect, in a possible implementation method, the airflow size of the first low-frequency signal is calculated to obtain the first airflow size, specifically including: based on the first low-frequency signal and the second low-frequency signal, a first wind noise probability is obtained, and the first wind noise probability is used to indicate the proportion of airflow sound contained in the first low-frequency signal and the second low-frequency signal; based on the first low-frequency signal and the first wind noise probability, a first airflow signal is obtained; the airflow size of the first airflow signal is calculated to obtain the first airflow size; the airflow size of the second low-frequency signal is calculated to obtain the second airflow size, specifically including: based on the second low-frequency signal and the first wind noise probability, a second airflow signal is obtained; and the airflow size of the second airflow signal is calculated to obtain the second airflow size.
[0017] Generally speaking, the frequency of airflow sound is relatively low, and the high-frequency signal in the audio can be filtered out by a low-pass filter, leaving only the low-frequency signal. However, the low-frequency signal may contain other noises, such as the sound of musical instruments or the voice of a boy talking. Therefore, it is also necessary to filter out the noise in the low-frequency signal to obtain the real airflow sound. Specifically, a first wind noise probability can be obtained based on the first low-frequency signal and the second low-frequency signal, and the first wind noise probability is used to indicate the proportion of airflow sound contained in the first low-frequency signal and the second low-frequency signal. Then, based on the first low-frequency signal and the first wind noise probability, a first airflow signal is obtained, and then based on the second low-frequency signal and the first wind noise probability, a second airflow signal is obtained. Then, based on the first airflow signal and the second airflow signal, the first airflow size and the second airflow size are obtained respectively. In this way, the accuracy of the airflow size obtained by the electronic device can be further improved, and the accuracy of the airflow emitted by the electronic device can also be improved.
[0018] In combination with the first aspect, in a possible implementation method, the airflow size of the first low-frequency signal is calculated to obtain the first airflow size, specifically including: filtering out non-airflow signals in the first low-frequency signal through a first neural network to obtain the first airflow signal; calculating the airflow size of the first airflow signal to obtain the first airflow size; calculating the airflow size of the second low-frequency signal to obtain the second airflow size, specifically including: filtering out non-airflow signals in the second low-frequency signal through the first neural network to obtain the second airflow signal; calculating the airflow size of the second airflow signal to obtain the first airflow size.
[0019] In some embodiments, the first neural network may be a neural network such as a NN network, which is not limited in the present application.
[0020] In other implementations, a neural network may be trained, and the first airflow signal and the second airflow signal may be obtained by filtering based on the trained neural network. In this way, the amount of calculation of the electronic device may be reduced.
[0021] In combination with the first aspect, in a possible implementation, before outputting an airflow of a first airflow size through a first airflow generating device and outputting an airflow of a second airflow size through a second airflow generating device, the method also includes: calculating a first angle based on the first airflow size and the second airflow size; outputting an airflow of the first airflow size through the first airflow generating device and outputting an airflow of the second airflow size through the second airflow generating device, specifically including: outputting an airflow of the first airflow size along a first angle through the first airflow generating device and outputting an airflow of the second airflow size along the first angle through the second airflow generating device.
[0022] In some embodiments, the airflow generating device can not only emit simulated airflow of corresponding size based on the size of the airflow sound, but also emit airflow in corresponding direction based on the direction of the airflow sound in the audio, further enhancing the user experience.
[0023] Specifically, the electronic device can calculate a first angle of the emitted airflow based on the first airflow size and the second airflow size, and then generate a simulated airflow of corresponding size based on the first angle.
[0024] In combination with the first aspect, in a possible implementation, an airflow of a first airflow size is output along a first angle by a first airflow generating device, and an airflow of a second airflow size is output along the first angle by a second airflow generating device, specifically including: when the first angle is greater than a first preset angle and less than a second preset angle, an airflow of the first airflow size is output along the first angle by the first airflow generating device, and an airflow of the second airflow size is output along the first angle by the second airflow generating device; wherein the first preset angle is the minimum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception, and the second preset angle is the maximum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception.
[0025] In combination with the first aspect, in a possible implementation, the method also includes: when the first angle is less than the first preset angle, an airflow of a first airflow size is output along the first preset angle by the first airflow generating device, and an airflow of a second airflow size is output along the first preset angle by the second airflow generating device; or, when the first angle is greater than the second preset angle, an airflow of the first airflow size is output along the second preset angle by the first airflow generating device, and an airflow of the second airflow size is output along the second preset angle by the second airflow generating device.
[0026] In this way, the electronic device can determine the calculated airflow angle to prevent the airflow angle from exceeding the maximum angle or minimum angle of airflow that can be perceived by the face. After the calculated airflow angle exceeds the maximum angle or minimum angle of airflow that can be perceived by the face, the airflow can be output only according to the first preset angle or the second preset angle.
[0027] In combination with the first aspect, in a possible implementation, before outputting an airflow of a first airflow size through a first airflow generating device and outputting an airflow of a second airflow size through a second airflow generating device, the method also includes: calculating a second angle based on the first airflow size, the second airflow size and a second preset angle, wherein the second preset angle is the maximum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception; outputting an airflow of a first airflow size through the first airflow generating device and outputting an airflow of a second airflow size through the second airflow generating device specifically includes: outputting an airflow of the first airflow size along the second angle through the first airflow generating device, and outputting an airflow of the second airflow size along the second angle through the second airflow generating device.
[0028] The airflow angle calculated in this manner is between the first preset angle and the second preset angle, and will not exceed the maximum angle and the minimum angle of the airflow that can be perceived by the human face.
[0029] In combination with the first aspect, in a possible implementation, calculating the first angle based on the first airflow size and the second airflow size specifically includes: according to the formula Calculate the first angle; where, is the first angle, is the first airflow size, is the second airflow size.
[0030] In some embodiments, a positive minimum value ε may be added to the denominator in the above formula. For example, the value of ε may be 0.0001, 0.00003, etc.
[0031] In combination with the first aspect, in a possible implementation, calculating the first angle based on the first airflow size and the second airflow size specifically includes: according to the formula Calculate the first angle; where, is the first angle, is the first airflow size, is the second airflow size, B is the second preset angle, and -B is the first preset angle.
[0032] In some embodiments, a positive minimum value ε may be added to the denominator in the above formula. For example, the value of ε may be 0.0001, 0.00003, etc.
[0033] In combination with the first aspect, in a possible implementation, the second angle is calculated based on the first airflow size, the second airflow size, and the second preset angle, specifically including: according to the formula Calculate the second angle; where, is the second angle, is the first airflow size, is the second airflow size, and B is the second preset angle.
[0034] In some embodiments, a positive minimum value ε may be added to the denominator in the above formula. For example, the value of ε may be 0.0001, 0.00003, etc.
[0035] In combination with the first aspect, in a possible implementation, the second angle is calculated based on the first airflow size, the second airflow size, and the second preset angle, specifically including: according to the formula Calculate the second angle; where, is the second angle, is the first airflow size, is the second airflow size, and B is the second preset angle.
[0036] In some embodiments, a positive minimum value ε may be added to the denominator in the above formula. For example, the value of ε may be 0.0001, 0.00003, etc.
[0037] In a second aspect, the present application provides an airflow generating device, which includes a display device, an audio playback device, an airflow generating device and a processing device, the audio playback device includes a first audio playback device and a second audio playback device, and the airflow generating device includes a first airflow generating device and a second airflow generating device, wherein the display device is used to display a virtual object; the first audio playback device is used to play a first audio; the second audio playback device is used to play a second audio; the processing device is used to calculate a first airflow size of an airflow sound in the first audio and a second airflow size of an airflow sound in the second audio; the first airflow generating device is used to output an airflow of a first airflow size; and the second airflow generating device is used to output an airflow of a second airflow size.
[0038] In combination with the second aspect, in a possible implementation method, the processing device is specifically used to: process the first audio through a low-pass filter to obtain a first low-frequency signal; process the second audio through a low-pass filter to obtain a second low-frequency signal; calculate the airflow size of the first low-frequency signal to obtain the first airflow size; calculate the airflow size of the second low-frequency signal to obtain the second airflow size.
[0039] In combination with the second aspect, in a possible implementation method, the processing device is specifically used to: obtain a first wind noise probability based on the first low-frequency signal and the second low-frequency signal, the first wind noise probability is used to indicate the proportion of airflow sound contained in the first low-frequency signal and the second low-frequency signal; obtain a first airflow signal based on the first low-frequency signal and the first wind noise probability; calculate the airflow size of the first airflow signal to obtain the first airflow size; obtain a second airflow signal based on the second low-frequency signal and the first wind noise probability; calculate the airflow size of the second airflow signal to obtain the second airflow size.
[0040] In combination with the second aspect, in a possible implementation method, the processing device is specifically used to: filter out non-airflow signals in the first low-frequency signal through a first neural network to obtain a first airflow signal; calculate the airflow size of the first airflow signal to obtain the first airflow size; filter out non-airflow signals in the second low-frequency signal through the first neural network to obtain a second airflow signal; calculate the airflow size of the second airflow signal to obtain the first airflow size.
[0041] In combination with the second aspect, in a possible implementation, the processing device is also used to calculate the first angle based on the first airflow size and the second airflow size; the first airflow generating device is specifically used to output an airflow of the first airflow size along the first angle; the second airflow generating device is specifically used to output an airflow of the second airflow size along the first angle.
[0042] In combination with the second aspect, in a possible implementation, the first airflow generating device is specifically used to output an airflow of a first airflow size along a first angle when the first angle is greater than a first preset angle and less than a second preset angle; the second airflow generating device is specifically used to output an airflow of a second airflow size along a first angle when the first angle is greater than the first preset angle and less than the second preset angle; wherein the first preset angle is the minimum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception, and the second preset angle is the maximum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception.
[0043] In combination with the second aspect, in a possible implementation, the first airflow generating device is also used to output an airflow of a first airflow size along a first preset angle when the first angle is less than the first preset angle; the second airflow generating device is also used to output an airflow of a second airflow size along the first preset angle when the first angle is less than the first preset angle; or, the first airflow generating device is also used to output an airflow of a first airflow size along a second preset angle when the first angle is greater than the second preset angle; the second airflow generating device is also used to output an airflow of a second airflow size along the second preset angle when the first angle is greater than the second preset angle.
[0044] In combination with the second aspect, in a possible implementation method, the processing device is also used to calculate the second angle based on the first airflow size, the second airflow size and the second preset angle, wherein the second preset angle is the maximum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception; the first airflow generating device is specifically used to output an airflow of a first airflow size along the second angle; the second airflow generating device is specifically used to output an airflow of a second airflow size along the second angle.
[0045] In conjunction with the second aspect, in a possible implementation, the processing device is specifically configured to: Calculate the first angle; where, is the first angle, is the first airflow size, is the second airflow size.
[0046] In conjunction with the second aspect, in a possible implementation, the processing device is specifically configured to: Calculate the first angle; where, is the first angle, is the first airflow size, is the second airflow size, B is the second preset angle, and -B is the first preset angle.
[0047] In conjunction with the second aspect, in a possible implementation, the processing device is specifically configured to: Calculate the second angle; where, is the second angle, is the first airflow size, is the second airflow size, and B is the second preset angle.
[0048] In conjunction with the second aspect, in a possible implementation, the processing device is specifically configured to: Calculate the second angle; where, is the second angle, is the first airflow size, is the second airflow size, and B is the second preset angle.
[0049] In a third aspect, the present application provides an electronic device comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes an interaction method in a virtual reality / augmented reality scenario provided in any possible implementation of any of the above aspects.
[0050] In a fourth aspect, the present application provides a chip system, which is applied to an electronic device. The chip system includes a processing circuit and an interface circuit. The interface circuit is used to receive instructions and transmit them to the processing circuit. The processing circuit is used to run instructions to execute an interaction method in a virtual reality / augmented reality scenario provided in any possible implementation of any of the above aspects.
[0051] In a fifth aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes an interaction method in a virtual reality / augmented reality scene provided in any possible implementation of any of the above aspects.
[0052] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes an interaction method in a virtual reality / augmented reality scene in any possible implementation of any of the above aspects.
[0053] For the description of the beneficial effects of the second to sixth aspects, reference may be made to the description of the beneficial effects in the first aspect, and this application will not repeat them here. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 A schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application;
[0055] Figure 2 A schematic diagram of the device structure of the electronic device 100 is shown;
[0056] Figure 3 A schematic diagram showing a user wearing an electronic device 100;
[0057] Figure 4-Figure 6 A schematic diagram showing the positions of a left-side airflow generating device and a right-side airflow generating device is shown;
[0058] Figure 7 Another schematic diagram showing the positions of the left-side airflow generating device and the right-side airflow generating device;
[0059] Figure 8 shows a front view projected backward from the user's face;
[0060] Fig. 9 Another front view projected backward from the user's face is shown;
[0061] Figures 10A-10G A schematic diagram showing the angles of a group of airflow outputs from a left airflow generating device and a right airflow generating device is shown;
[0062] Figure 11A-11B A schematic diagram of obtaining the size and / or direction of the airflow on the left and right sides based on audio in the present application is shown;
[0063] Figure 12A-12B Another schematic diagram of obtaining the size and / or direction of the left and right airflows based on audio in the present application is shown;
[0064] Fig.13A schematic diagram showing the electronic device 100 separating the left airflow signal and the right airflow signal from the audio in the first mode;
[0065] Fig.14A A schematic diagram showing another electronic device 100 obtaining the left airflow size and the right airflow size based on the airflow size estimation module;
[0066] Fig. 14B A schematic diagram of training a NN network is shown;
[0067] Fig. 14C A schematic diagram showing the electronic device 100 separating an airflow signal from audio data based on a NN network;
[0068] Fig.15 A schematic diagram of a flow chart of a method for generating airflow provided in the present application;
[0069] Fig.16 A schematic diagram of an airflow generating device provided in the present application. DETAILED DESCRIPTION
[0070] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0071] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.
[0072] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. A common form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icon, button, menu, tab, text box, dialog box, status bar, navigation bar, widget, etc. displayed on the display screen of the electronic device 100.
[0073] In the embodiment of the present application, the electronic device 100 uses VR technology to provide a virtual environment, which means that the electronic device 100 uses VR display technology to render and display one or more virtual objects. The virtual object can be generated by the electronic device 100 itself using computer graphics technology, computer simulation technology, etc., or it can be generated by other electronic devices 100 using computer graphics technology, computer simulation technology, etc. and sent to the electronic device 100. Other electronic devices 100 can be servers, or mobile phones, computers, etc. connected or paired with the electronic device 100. Virtual objects can also be referred to as virtual images or virtual elements. Virtual objects can be two-dimensional or three-dimensional. Virtual objects are fake objects that do not really exist in the physical world. Virtual objects can be virtual objects that imitate objects that exist in the real physical world, thereby bringing an immersive experience to users. Virtual objects may include virtual animals, virtual characters, virtual trees, virtual buildings, virtual labels, icons, pictures or videos, etc.
[0074] In the following embodiments of the present application, the electronic device 100 providing a virtual environment using VR technology may also be referred to as the electronic device 100 providing a virtual reality scene.
[0075] The following first introduces the electronic device 100 provided in an embodiment of the present application.
[0076] The virtual objects displayed by the electronic device 100 can interact with the user. In some embodiments, the user can directly interact with the virtual objects displayed by the electronic device 100 through somatosensory interactions such as hand / arm movements, head movements, eye movements, etc. In other embodiments, the electronic device 100 can be used in conjunction with a handheld device, and the user can interact with the virtual objects displayed by the electronic device 100 by manipulating the handheld device. The handheld device can be, for example, a controller, a gyro mouse, a stylus, or other handheld computing device. The handheld device can be configured with a variety of sensors such as acceleration sensors, gyro sensors, magnetic sensors, etc., which can be used to detect its own movement. The handheld device can communicate with the electronic device 100 through short-range transmission technologies such as Bluetooth, near field communication (NFC), and ZigBee.
[0077] The electronic device 100 can be mounted on the user's head. For example, the electronic device 100 can be a virtual reality display device such as VR glasses, a VR head-mounted display (HMD), a VR all-in-one machine, etc. In some other embodiments of the present application, the electronic device 100 can also be a non-portable electronic device 100 such as a desktop computer supporting VR technology, a smart TV, a vehicle with a display screen, etc.
[0078] The following embodiments of the present application are described by taking the electronic device 100 as an HMD as an example.
[0079] refer to Figure 1 , Figure 1 A schematic diagram of the structure of an exemplary electronic device 100 provided in the present application.
[0080] like Figure 1 As shown, the electronic device 100 may include a processor 110, a memory 120, a communication module 130, a sensor module 140, a button 150, an input / output interface 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a display device 180, a camera 190, a battery 1100, and the like.
[0081] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figure. For example, it may also include an infrared transceiver, an ultrasonic transceiver, a motor, and a flash lamp, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0082] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.
[0083] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0084] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0085] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.
[0086] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor, charger, camera 190, etc. through different I2C bus interfaces.
[0087] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module via the I2S interface to achieve the function of answering a call through a Bluetooth headset.
[0088] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module in the communication module 130 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
[0089] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the communication module 130. For example, the processor 110 communicates with the Bluetooth module in the communication module 130 through the UART interface to implement the Bluetooth function.
[0090] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display device 180 and the camera 190. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 190 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display device 180 communicate via the DSI interface to implement the display function of the electronic device 100.
[0091] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 190, the display device 180, the communication module 130, the sensor module 140, the microphone 170C, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.
[0092] The USB interface is an interface that complies with the USB standard specification, and can be specifically a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices 100, such as mobile phones. The USB interface can be USB3.0, which is used to be compatible with high-speed display port (DP) signal transmission, and can transmit high-speed video and audio data.
[0093] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0094] The electronic device 100 can implement a wireless communication function through the communication module 130. The communication module 130 may include an antenna, a wireless communication module, a mobile communication module, a modem processor, a baseband processor, and the like.
[0095] The antenna is used to transmit and receive electromagnetic wave signals. The electronic device 100 may include multiple antennas, each of which may be used to cover a single or multiple communication frequency bands. Different antennas may also be reused to improve the utilization of the antennas. For example, an antenna may be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna may be used in combination with a tuning switch.
[0096] The mobile communication module can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module may receive electromagnetic waves through an antenna, filter, amplify, and process the received electromagnetic waves, and transmit them to a modulation and demodulation processor for demodulation. The mobile communication module may also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna. In some embodiments, at least some of the functional modules of the mobile communication module may be arranged in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module may be arranged in the same device as at least some of the modules of the processor 110.
[0097] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, etc.), or displays an image or video through a display device 180. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and may be set in the same device as the mobile communication module or other functional modules.
[0098] The wireless communication module can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module can be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module can also receive the signal to be sent from the processor 110, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna.
[0099] In some embodiments, the antenna of the electronic device 100 is coupled to the mobile communication module so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).
[0100] The electronic device 100 implements the display function through a GPU, a display device 180, and an application processor. The GPU is a microprocessor for image processing, which connects the display device 180 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.
[0101] In the embodiment of the present application, the display device 180 may be used to present one or more virtual objects, so that the electronic device 100 provides a virtual reality scene for the user. The display device 180 may present virtual objects in one or more of the following ways:
[0102] 1. In some embodiments, the display device 180 is a display screen, and the display screen may include a display panel. The display panel of the display device 180 can be used to display virtual objects, thereby presenting a three-dimensional virtual environment to the user. The user can see the virtual object from the display panel and experience the virtual reality scene. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc.
[0103] 2. In some embodiments, the display device 180 may include an optical device for projecting an optical signal (e.g., a light beam) directly onto the user's retina. The user can directly see the virtual object through the optical signal projected by the optical device and experience the three-dimensional virtual environment. The optical device may be a micro projector, etc.
[0104] The number of display devices 180 in the electronic device 100 may be two, corresponding to the two eyeballs of the user respectively. The contents displayed on the two display devices may be displayed independently. Different images may be displayed on the two display devices to enhance the stereoscopic effect of the image. In some possible embodiments, the number of display devices 180 in the electronic device 100 may also be one, corresponding to both eyeballs of the user.
[0105] The electronic device 100 can implement a shooting function through an ISP, a camera 190, a video codec, a GPU, a display device 180, and an application processor.
[0106] The ISP is used to process the data fed back by the camera 190. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 190.
[0107] The camera 190 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 190, where N is a positive integer greater than 1. The camera 190 may include, but is not limited to, a traditional color camera (RGB camera), a depth camera (RGB depth camera), a dynamic vision sensor (DVS) camera, and the like.
[0108] In some embodiments, the camera 190 may be a depth camera. The depth camera may collect spatial information of the real environment. The spatial information of the real environment may refer to the above description.
[0109] In some embodiments, the camera 190 may capture hand images or body images of the user, and the processor 110 may be used to analyze the images captured by the camera 190 to identify hand movements or body movements input by the user.
[0110] In some embodiments, the camera 190 can be used in conjunction with an infrared device (such as an infrared transmitter) to detect the user's eye movements, such as eye gaze direction, blinking operations, gaze operations, etc., to achieve eye tracking.
[0111] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0112] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0113] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of electronic device 100, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0114] The memory 120 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application program (such as VR / AR / MR application) required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area may store data (such as audio data, a phone book, etc.) created during the use of the electronic device 100. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0115] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the microphone 170C, the headphone interface 170D, and the application processor.
[0116] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module can also be used to encode and decode audio signals. In some embodiments, the audio module can be arranged in the processor 110, or some functional modules of the audio module can be arranged in the processor 110.
[0117] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker.
[0118] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. The electronic device 100 may be provided with at least one microphone. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 may also be provided with three, four or more microphones 170C to realize collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.
[0119] In some embodiments, microphone 170C may detect voice signals used to control portable electronic device 100. Processor 110 may then process the voice signals to recognize voice commands.
[0120] The headphone jack is used to connect wired headphones. The headphone jack can be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0121] In some embodiments, the electronic device 100 may include one or more keys 150, which can control the electronic device 100 and provide the user with access to functions on the electronic device 100. The key 150 can be in the form of a button, a switch, a dial, and a touch or near-touch sensing device (such as a touch sensor). Specifically, for example, the user can turn on the display device 180 of the electronic device 100 by pressing a button. The key 150 includes a power button, a volume button, etc. The key 150 can be a mechanical key. It can also be a touch key. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100.
[0122] In some embodiments, the electronic device 100 may include an input / output interface 160, which may connect other devices to the electronic device 100 through appropriate components. The components may include, for example, an audio / video jack, a data connector, and the like.
[0123] The sensor module 140 may include various sensors, for example, a proximity light sensor, a distance sensor, a gyro sensor, an ambient light sensor, an acceleration sensor, a temperature sensor, a magnetic sensor, a bone conduction sensor, a fingerprint sensor, and the like.
[0124] The proximity light sensor may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor to detect gesture operations at a specific position of the electronic device 100 to achieve the purpose of associating gesture operations with operation commands.
[0125] The distance sensor can be used to measure the distance. The electronic device 100 can measure the distance by infrared or laser.
[0126] The gyroscope sensor can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) can be determined by the gyroscope sensor. The gyroscope sensor can also be used for navigation and somatosensory game scenes.
[0127] The ambient light sensor is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display device 180 according to the sensed brightness of the ambient light. The ambient light sensor can also be used to automatically adjust the white balance when taking pictures.
[0128] The acceleration sensor can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device 100 and applied to applications such as pedometers.
[0129] In some embodiments of the present application, the electronic device 100 may track the movement of the user's head based on an acceleration sensor, a gyroscope sensor, a magnetic sensor, and the like.
[0130] The temperature sensor is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 1100 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 1100 to avoid abnormal shutdown due to low temperature.
[0131] In the embodiment of the present application, the memory 120 may be used to store the spatial information of the virtual object corresponding to one or more applications. The processor 110 may be used to determine whether the virtual object is superimposed on the real object, that is, to determine whether there is a superimposed area, based on the spatial information of the real object and the spatial information of the virtual object. The detailed operation of the processor 110 in determining whether there is a superimposed area may refer to the relevant description of the subsequent embodiments.
[0132] In the embodiment of the present application, one or more of the display device 180, the speaker 170A, the flashlight or the motor can be used to output prompt information for indicating that there is currently an overlay area. Here, the specific operation of the display device 180, the speaker 170A, the flashlight or the motor outputting the prompt information can refer to the relevant description of the subsequent embodiments.
[0133] In the embodiment of the present application, one or more of the display device 180, the speaker 170A, the flashlight or the motor can be used to output prompt information for guiding the user to perform effective interaction. Here, the specific operation of the display device 180, the flashlight or the motor outputting the prompt information can refer to the relevant description of the subsequent embodiments.
[0134] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc., and this application does not limit this. For example, the electronic device 100 in the embodiment of this application may be equipped with iOS, Android, Microsoft or other operating systems.
[0135] First, the device structure of the electronic device 100 in the embodiment of the present application is introduced.
[0136] Figure 2 A schematic diagram of the device structure of the electronic device 100 is shown.
[0137] like Figure 2 As shown, the electronic device 100 may include a controller, a power circuit, a display, a battery, a right speaker, a left speaker, a left airflow generating device, a right airflow generating device, and the like.
[0138] The controller may also be referred to as a processor 110, and the controller is used to control the display to display virtual objects.
[0139] In some embodiments, the display may also be referred to as display device 180 .
[0140] When the display outputs virtual objects, the controller can also control the left and right speakers to emit audio, thereby enhancing the user's virtual experience through images and audio.
[0141] Not limited to the left speaker and the right speaker, the electronic device 100 may also include more other speakers. For example, when the electronic device 100 supports multiple channels, such as when the electronic device 100 supports three channels, the electronic device 100 may also include a mid-side speaker, the left speaker is used to output the left channel audio, the right speaker is used to output the right channel audio, and the mid-side speaker is used to output the center channel audio.
[0142] Optionally, the left speaker and the right speaker and other more speakers may be referred to as speaker 170A.
[0143] The present application is merely described by taking the electronic device 100 including a left speaker and a right speaker as an example, which does not constitute a limitation.
[0144] When the electronic device 100 emits audio through the speaker, the controller can detect wind noise based on the audio output by the speaker to obtain the size and direction of the wind noise in the left channel and the size and direction of the wind noise in the right channel. The controller can control the left airflow generating device to emit an airflow of corresponding size and direction based on the size and direction of the wind noise in the left channel, and control the right airflow generating device to emit an airflow of corresponding size and direction based on the size and direction of the wind noise in the right channel, so that the user can experience an immersive feeling based on the simulated airflow, thereby improving the user experience.
[0145] It should be noted that Figure 2 Only some devices of the electronic device 100 are shown. The electronic device 100 may also include other devices. For details, please refer to Figure 1 The description in the embodiments will not be repeated in this application.
[0146] Description of the location of the airflow generating device in the electronic device 100.
[0147] In the present application, the electronic device 100 can generate a simulated airflow based on the audio output by the speaker, and generate and emit a simulated airflow of corresponding size and direction through the left airflow generating device and the right airflow generating device.
[0148] In one possible implementation, the left-side airflow generating device and the right-side airflow generating device may include a fan, an air duct or other device that can output airflow, and the position and direction of the device can be changed to enable the left-side airflow generating device and the right-side airflow generating device to emit airflow in corresponding directions through the device.
[0149] In other possible implementations, a fan, an air duct or other device that can output airflow may be included, and the position and direction of the device are fixed, but the position and direction of the left airflow generating device and the right airflow generating device can be changed to enable the left airflow generating device and the right airflow generating device to emit airflow in corresponding directions through the device.
[0150] In some embodiments, a device in an airflow generating apparatus that can output airflow can be referred to as an airflow generating device.
[0151] Not limited to the above implementation methods, the left airflow generating device and the right airflow generating device can also emit simulated airflows of corresponding sizes and directions in other ways, and this application does not limit this.
[0152] First, the positions of the left airflow generating device and the right airflow generating device in the electronic device 100 are described.
[0153] Figure 3 A schematic diagram showing a user wearing the electronic device 100 is shown.
[0154] like Figure 3 As shown, the user wears an electronic device 100 (such as a head-mounted display device) on his head, and the display of the electronic device 100 is located directly in front of the user's eyes. The user can view the virtual object output by the display of the electronic device 100 through both eyes.
[0155] Figure 4-Figure 6 A schematic diagram showing the positions of a left-side airflow generating device and a right-side airflow generating device is shown.
[0156] Figure 4 1 is a front view of the electronic device 100. The electronic device 100 may include a plurality of head fixing straps, which are used to fix the electronic device 100 on the user's head when the user wears the electronic device 100.
[0157] Figure 5 FIG. 1 is a rear view of the electronic device 100. The electronic device 100 may also include a plurality of head fixing straps, Figure 5 The internal structure of the electronic device 100 can be seen. The electronic device 100 may include a display screen inside, and the display screen is used to display virtual objects. The user can watch the virtual objects displayed by the electronic device 100 through both eyes to achieve the purpose of virtual experience.
[0158] like Figure 5 As shown, it can be seen that the electronic device 100 may include a left-side airflow generating device and a right-side airflow generating device.
[0159] The left airflow generating device is located at the upper left front of the electronic device 100. The right airflow generating device is located at the upper right front of the electronic device 100.
[0160] It should be noted that Figure 5 The position of the left airflow generating device at the upper left front of the electronic device 100 and the position of the right airflow generating device at the upper right front of the electronic device 100 are only shown as examples. The left airflow generating device can also be located at other positions at the upper left front of the electronic device 100, and the right airflow generating device can also be located at other positions at the upper right front of the electronic device 100, and the present application does not limit this.
[0161] Figure 6 FIG. 1 is another rear view of the electronic device 100. The electronic device 100 may also include a plurality of head fixing straps, Figure 6The internal structure of the electronic device 100 can be seen. The electronic device 100 may include a display screen inside, and the display screen is used to display virtual objects. The user can watch the virtual objects displayed by the electronic device 100 through both eyes to achieve the purpose of virtual experience.
[0162] like Figure 6 As shown, it can be seen that the electronic device 100 may include a left-side airflow generating device and a right-side airflow generating device.
[0163] The left airflow generating device is located at the lower left front of the electronic device 100. The right airflow generating device is located at the lower right front of the electronic device 100.
[0164] It should be noted that Figure 6 The position of the left airflow generating device at the lower left front of the electronic device 100 and the position of the right airflow generating device at the lower right front of the electronic device 100 are only shown as examples. The left airflow generating device can also be located at other positions at the lower left front of the electronic device 100, and the right airflow generating device can also be located at other positions at the lower right front of the electronic device 100. This application does not limit this.
[0165] Figure 7-Figure 9 Another schematic diagram of the positions of the left-side airflow generating device and the right-side airflow generating device is shown.
[0166] in, Figure 7 A top view projected downward from the top of the user's head is shown.
[0167] like Figure 7 As shown, when the electronic device 100 is worn, the electronic device 100 can be worn on the user's face so that the user's eyes are located inside the electronic device 100, and the user's eyes can view the virtual objects displayed by the electronic device 100 through the display screen inside the electronic device 100 to achieve the purpose of virtual experience.
[0168] Among them, the electronic device 100 is pre-installed with a left airflow generating device and a right airflow generating device. The left airflow generating device is located at the upper left front of the electronic device 100, and the right airflow generating device is located at the upper right front of the electronic device 100, or the left airflow generating device is located at the lower left front of the electronic device 100, and the right airflow generating device is located at the lower right front of the electronic device 100. Or the left airflow generating device is located at the upper left front of the electronic device 100, and the right airflow generating device is located at the upper right front of the electronic device 100. Or the left airflow generating device is located at the lower left front of the electronic device 100, and the right airflow generating device is located at the lower right front of the electronic device 100.
[0169] Figure 8 A front view projected backwards from the user's face is shown.
[0170] like Figure 8 As shown, the right airflow generating device may be located at the upper right front of the electronic device 100 , and the left airflow generating device may be located at the upper left front of the electronic device 100 .
[0171] It should be noted that the right airflow generating device and the left airflow generating device are both located inside the electronic device 100, so that the right airflow generating device and the left airflow generating device can generate airflow and blow the airflow on the user's face, so that the user can feel the airflow.
[0172] Furthermore, the right airflow generating device and the left airflow generating device are not located within the display area of the electronic device 100, so as to avoid the right airflow generating device and the left airflow generating device blocking the user's field of vision.
[0173] Fig. 9 Another front view is shown projected backwards from the user's face.
[0174] like Fig. 9 As shown, the right airflow generating device may be located at the lower right front of the electronic device 100 , and the left airflow generating device may be located at the lower left front of the electronic device 100 .
[0175] It should be noted that the right airflow generating device and the left airflow generating device are both located inside the electronic device 100, so that the right airflow generating device and the left airflow generating device can generate airflow and blow the airflow on the user's face, so that the user can feel the airflow.
[0176] Furthermore, the right airflow generating device and the left airflow generating device are not located within the display area of the electronic device 100, so as to avoid the right airflow generating device and the left airflow generating device blocking the user's field of vision.
[0177] It should be noted that Figure 3-Figure 9 The positions of the right airflow generating device and the left airflow generating device in the electronic device 100 are only exemplarily shown. In some embodiments, the right airflow generating device and the left airflow generating device may not be located in parallel positions, for example, the right airflow generating device is located at the upper right front of the electronic device 100, and the left airflow generating device is located at the lower left front of the electronic device 100. For another example, the right airflow generating device is located at the lower right front of the electronic device 100, and the left airflow generating device is located at the upper left front of the electronic device 100. This application does not limit this.
[0178] The present application needs to calculate the size and direction of the left airflow sound and the size and direction of the right airflow sound in the audio based on the size and direction of the audio output by the electronic device 100. After obtaining the size and direction of the left airflow sound and the size and direction of the right airflow sound, the electronic device 100 can control the right airflow generating device to emit an airflow of corresponding size and direction according to the size and direction of the right airflow sound, and control the left airflow generating device to emit an airflow of corresponding size and direction according to the size and direction of the left airflow sound.
[0179] It should be noted that the present application is only described using two-channel audio and two airflow generating devices as examples.
[0180] In other embodiments, more audio channels and more airflow generating devices may be included. For example, for three-channel audio, the electronic device 100 may include three airflow generating devices. For example, for five-channel audio, the electronic device 100 may include five airflow generating devices, which is not limited in the present application.
[0181] The angle of the airflow generating device in the electronic device 100 can emit the airflow direction of the airflow.
[0182] In the present application, the electronic device 100 can calculate the airflow size and airflow direction of the airflow based on the audio output by the electronic device 100, and the airflow generating device in the electronic device 100 can emit an airflow of corresponding size and corresponding direction based on the calculated airflow size and airflow direction. Next, the airflow direction is explained.
[0183] In some embodiments, the airflow direction can be determined by the airflow angle emitted by the airflow generating device. The airflow direction is related to the airflow angle.
[0184] Fig. 10A A schematic diagram showing the airflow directions of the airflows that can be emitted by the right-side airflow generating device and the left-side airflow generating device is shown.
[0185] like Fig. 10A As shown, when the right airflow generating device is located at the upper right front of the electronic device 100, and the left airflow generating device is located at the upper left front of the electronic device 100, the right airflow generating device and the left airflow generating device are located on the same horizontal plane.
[0186] like Fig. 10A As shown, the line between the left airflow generating device and the right airflow generating device can be called a first baseline.
[0187] Projecting downward from directly above the user's head, we get Fig. 10A The first baseline is shown.
[0188] In a possible implementation, the reference positions of the left and right airflow generating devices on the first baseline are fixed, but the left and right airflow generating devices can rotate from left to right around the reference position to emit airflow in a preset direction.
[0189] In other possible implementations, the reference positions of the left-side airflow generating device and the right-side airflow generating device on the first baseline are fixed, and the left-side airflow generating device and the right-side airflow generating device cannot rotate. An airflow in a preset direction can be emitted by controlling the airflow generating device inside the airflow generating device to rotate from left to right around the reference position.
[0190] Optionally, the reference positions of the left and right airflow generating devices may not be fixed, and the positions of the left and right airflow generating devices may also be moved along the first baseline, thereby changing the reference positions of the left and right airflow generating devices on the first baseline.
[0191] The left-side airflow generating device and the right-side airflow generating device can also emit airflow in a preset direction in other ways, which is not limited in this application.
[0192] like Fig. 10B As shown, the horizontal plane where the left airflow generating device and the right airflow generating device sit can intersect with the vertical plane where the center line of the head from front to back is located, and a second baseline is obtained. The angle between the airflow emitted by the left airflow generating device and the second baseline can be called the airflow direction θ. The airflow direction emitted by the right airflow generating device is the same as the airflow direction emitted by the left airflow generating device.
[0193] Optionally, no matter where the left airflow generating device and the right airflow generating device are located on the electronic device 100, projection from the top of the user's head to the horizontal plane can obtain Figure 10A-10B The first baseline and the second baseline are shown, but this application does not limit them.
[0194] The value range of the airflow direction θ may be between [-A, A], and the value range of the airflow direction θ may also be between [-B, B].
[0195] For example, -A may be -90 degrees, and A may be 90 degrees.
[0196] For example, -B may be -45 degrees and B may be 45 degrees.
[0197] The airflow direction θ may also be other preset values, which is not limited in this application.
[0198] The following embodiments are exemplified by taking the airflow direction θ of the airflow emitted by the left airflow generating device.
[0199] like Fig. 10C As shown, when the left airflow generating device and the right airflow generating device emit airflow horizontally to the left, the airflow direction θ is 90 degrees, that is, the angle between the airflow emitted by the left airflow generating device and the second baseline is 90 degrees.
[0200] like Fig. 10D As shown, when the left airflow generating device and the right airflow generating device emit airflow to the left front, the airflow direction θ is 45 degrees, that is, the angle between the airflow emitted by the left airflow generating device and the second baseline is 45 degrees.
[0201] like Fig. 10E As shown, when the left airflow generating device and the right airflow generating device emit airflow upward perpendicular to the first baseline, the airflow direction θ is 0 degrees, that is, the angle between the airflow emitted by the left airflow generating device and the second baseline is 0 degrees.
[0202] like Fig.10F As shown, when the left airflow generating device and the right airflow generating device emit airflow to the right front, the airflow direction θ is -45 degrees, that is, the angle between the airflow emitted by the left airflow generating device and the second baseline is -45 degrees.
[0203] like Figure 10G As shown, when the left airflow generating device and the right airflow generating device emit airflow horizontally to the right, the airflow direction θ is -90 degrees, that is, the angle between the airflow emitted by the left airflow generating device and the second baseline is 0 degrees.
[0204] Optionally, in other embodiments, the direction of the airflow emitted by the left airflow generating device and the right airflow generating device along the left front may be a negative value, and the direction of the airflow emitted by the left airflow generating device and the right airflow generating device along the right front may be a positive value, and the present application does not limit this. Figures 10A-10G It is only used for exemplary explanation of this application.
[0205] Figure 11A-11B A schematic diagram of obtaining the size and / or direction of the left and right airflows based on audio in the present application is shown.
[0206] In some embodiments, the electronic device 100 can obtain the size and direction of the left and right airflows based on the audio processed by the audio processing algorithm.
[0207] like Fig.11A As shown, the electronic device 100 includes an audio processing algorithm module, an airflow size and / or airflow direction estimation module, a left speaker, a right speaker, a left airflow generating device, and a right airflow generating device.
[0208] The electronic device 100 may output left-channel audio and right-channel audio.
[0209] The electronic device 100 can input the left channel audio and the right channel audio into the audio processing algorithm module, process the left channel audio and the right channel audio, and obtain the left channel processed audio and the right channel processed audio. The electronic device 100 then inputs the left channel processed audio into the left speaker, and then inputs the right channel processed audio into the right speaker. The electronic device 10 can play the left channel processed audio through the left speaker and play the right channel processed audio through the right speaker. This allows the user to hear the audio output by the electronic device 100.
[0210] The audio processing algorithm includes but is not limited to one or more of the following: audio equalization algorithm, sound effect adjustment algorithm, low frequency enhancement algorithm, dynamic range compression algorithm, etc. In this way, the audio effect of the electronic device 100 can be better.
[0211] The electronic device 100 can obtain the left airflow size and / or direction and the right airflow size and / or direction respectively based on the left channel processed audio and the right channel processed audio.
[0212] Specifically, the electronic device 100 can input the left channel processed audio and the right channel processed audio into the airflow size and / or airflow direction estimation module respectively, and the airflow size and / or airflow direction estimation module can obtain the left airflow size and / or direction based on the left channel processed audio, and output the left airflow size and / or direction. Similarly, the airflow size and / or airflow direction estimation module can obtain the right airflow size and / or direction based on the right channel processed audio, and output the right airflow size and / or direction.
[0213] After obtaining the left airflow size and / or direction and the right airflow size and / or direction, the airflow size and / or airflow direction estimation module can send the obtained left airflow size and / or direction and the right airflow size and / or direction to the left airflow generating device and the right airflow generating device respectively.
[0214] The left-side airflow generating device can emit an airflow of a preset size and / or preset direction based on the size and / or direction of the left-side airflow.
[0215] The right-side airflow generating device can emit an airflow of a preset size and / or preset direction based on the size and / or direction of the left-side airflow.
[0216] like Fig. 11B As shown, the electronic device 100 includes an audio processing algorithm module, an airflow size and / or airflow direction estimation module, a left speaker, a right speaker, a left airflow generating device, and a right airflow generating device.
[0217] The electronic device 100 may output left-channel audio and right-channel audio.
[0218] The electronic device 100 can input the left channel audio and the right channel audio into the audio processing algorithm module, process the left channel audio and the right channel audio, and obtain the left channel processed audio and the right channel processed audio. The electronic device 100 then inputs the left channel processed audio into the left speaker, and then inputs the right channel processed audio into the right speaker. The electronic device 100 can play the left channel processed audio through the left speaker and play the right channel processed audio through the right speaker. This allows the user to hear the audio output by the electronic device 100.
[0219] The audio processing algorithm includes but is not limited to one or more of the following: audio equalization algorithm, sound effect adjustment algorithm, low frequency enhancement algorithm, dynamic range compression algorithm, etc. In this way, the audio effect of the electronic device 100 can be better.
[0220] exist Fig. 11B In the embodiment, the electronic device 100 can obtain the size and / or direction of the left airflow and the size and / or direction of the right airflow based on the left channel audio and the right channel audio, respectively.
[0221] Specifically, the electronic device 100 can input the left channel audio and the right channel audio into the airflow size and / or airflow direction estimation module respectively, and the airflow size and / or airflow direction estimation module can obtain the left airflow size and / or direction based on the left channel audio, and output the left airflow size and / or direction. Similarly, the airflow size and / or airflow direction estimation module can obtain the right airflow size and / or direction based on the right channel audio, and output the right airflow size and / or direction.
[0222] After obtaining the left airflow size and / or direction and the right airflow size and / or direction, the airflow size and / or airflow direction estimation module can send the obtained left airflow size and / or direction and the right airflow size and / or direction to the left airflow generating device and the right airflow generating device respectively.
[0223] The left-side airflow generating device can emit an airflow of a preset size and / or preset direction based on the size and / or direction of the left-side airflow.
[0224] The right-side airflow generating device can emit an airflow of a preset size and / or preset direction based on the size and / or direction of the left-side airflow.
[0225] It should be noted that the present application may not include an audio processing algorithm module, that is, no audio is processed, and the audio is directly input into the left speaker and the right speaker.
[0226] The left airflow size and / or direction and the right airflow size and / or direction are obtained based on the audio.
[0227] Figure 12A-12BAnother schematic diagram of obtaining the size and / or direction of the left and right airflows based on audio in the present application is shown.
[0228] In some embodiments, the electronic device 100 can obtain only the airflow size based on the audio without calculating the airflow direction. In this way, the left airflow generating device and the right airflow generating device only need to emit airflows of a preset size.
[0229] In this case, in some embodiments, the reference positions of the left airflow generating device and the right airflow generating device are fixed, and the positions of the airflow generating components inside the airflow generating device are also fixed, and the left airflow generating device and the right airflow generating device can output airflows of different sizes in a fixed direction.
[0230] In other embodiments, the positions of the left airflow generating device and the right airflow generating device can also be changed, or the positions of the airflow generating devices inside the airflow generating devices can be changed, so that the left airflow generating device and the right airflow generating device can output airflows of different sizes in different directions.
[0231] like Fig. 12A As shown, the electronic device 100 can input the left channel audio and the right channel audio into the airflow feature estimation module respectively, and the airflow feature estimation module can obtain the left airflow size based on the left channel audio, and the airflow feature estimation module then sends the left airflow size to the left airflow generating device. The left airflow generating device can emit an airflow of a preset size based on the left airflow size.
[0232] The airflow characteristic estimation module can also obtain the left airflow size based on the right channel audio, and then send the right airflow size to the right airflow generating device. The right airflow generating device can emit a preset airflow size based on the right airflow size.
[0233] In other embodiments, the electronic device 100 can obtain only the airflow size based on the audio, and then estimate the airflow direction based on the airflow size. In this way, the left airflow generating device and the right airflow generating device need to emit airflows of a preset size and a preset direction.
[0234] In this case, in one possible implementation, the reference positions of the left airflow generating device and the right airflow generating device are fixed, and the left airflow generating device and the right airflow generating device can rotate from left to right around the reference position to emit airflow in a preset direction. In other possible implementations, the reference positions of the left airflow generating device and the right airflow generating device are fixed or not fixed, and the left airflow generating device and the right airflow generating device cannot rotate, but the left airflow generating device and the right airflow generating device can rotate from left to right around the reference position to emit airflow in a preset direction by controlling the airflow generating device inside the airflow generating device.
[0235] The left-side airflow generating device and the right-side airflow generating device can also emit airflow in a preset direction in other ways, which is not limited in this application.
[0236] like Fig. 12B As shown, the airflow characteristic estimation module includes an airflow size estimation module and an airflow direction estimation module.
[0237] Among them, the electronic device 100 can input the left channel audio and the right channel audio into the airflow size estimation module respectively, and the airflow size estimation module can obtain the left airflow size based on the left channel audio, and obtain the right airflow size based on the right channel audio.
[0238] Afterwards, on the one hand, the airflow size estimation module needs to send the left airflow size to the left airflow generating device, and send the right airflow size to the right airflow generating device. On the other hand, the airflow size estimation module also needs to send the left airflow size and the right airflow size to the airflow direction estimation module.
[0239] After receiving the left airflow size and the right airflow size, the airflow direction estimation module can obtain the left airflow direction based on the left airflow size, and obtain the right airflow direction based on the right airflow size.
[0240] The airflow direction estimation module then sends the left airflow direction to the left airflow generating device, and sends the right airflow direction to the right airflow generating device.
[0241] After receiving the left airflow size and the left airflow direction, the left airflow generating device can emit an airflow of a preset size and a preset direction based on the left airflow size and the left airflow direction.
[0242] Similarly, after receiving the right airflow size and the right airflow direction, the right airflow generating device can emit an airflow of a preset size and a preset direction based on the right airflow size and the right airflow direction.
[0243] Next, it is introduced how the electronic device 100 calculates the size of the left airflow and the size of the right airflow.
[0244] The present application provides but is not limited to two methods for measuring the size of the left airflow and the size of the right airflow.
[0245] Method 1: The electronic device 100 can calculate the left low-frequency signal based on the left channel audio, and calculate the right low-frequency signal based on the right channel audio. The electronic device 100 then calculates the probability of wind noise based on the left low-frequency signal and the right low-frequency signal. The greater the probability of wind noise, the greater the probability of wind noise. Then, the electronic device 100 calculates the left airflow signal based on the wind noise probability and the left low-frequency signal, and calculates the right airflow signal based on the wind noise probability and the right low-frequency signal.
[0246] Among them, the reason why the left low-frequency signal is calculated from the left channel audio and the right low-frequency signal is calculated based on the right channel audio is that, generally speaking, the frequency of wind noise signals is relatively low, and the left low-frequency signal and the right low-frequency signal can be filtered from the left channel audio and the right channel audio. The left low-frequency signal can be considered to be partly or entirely of the left airflow signal, and the right low-frequency signal can also be considered to be partly or entirely of the right airflow signal.
[0247] The reason why the electronic device 100 calculates the left airflow signal based on the wind noise probability and the left low-frequency signal, and calculates the right airflow signal based on the wind noise probability and the right low-frequency signal is that, generally speaking, the low-frequency signal may include low-frequency noise such as musical instrument sound or male voice in addition to the airflow signal. In order to improve the accuracy of the left airflow signal and the right airflow signal finally obtained, the electronic device 100 can further determine the left airflow signal based on the wind noise probability and the left low-frequency signal, and further determine the right airflow signal based on the wind noise probability and the right low-frequency signal. In this way, the accuracy of the electronic device 100 separating the left airflow signal and the right airflow signal from the audio signal can be improved.
[0248] Optionally, in some embodiments, after obtaining the left low-frequency signal and the right low-frequency signal, the electronic device 100 may not consider the influence of low-frequency noise such as musical instrument sounds or male voices, and directly use the left low-frequency signal as the left airflow signal and the right low-frequency signal as the right airflow signal. This application does not limit this.
[0249] Fig.13 A schematic diagram showing the electronic device 100 separating the left airflow signal and the right airflow signal from the audio in the first mode is shown.
[0250] like Fig.13 As shown, the airflow size estimation module includes a low-pass filter, an airflow probability estimation module and an airflow size estimation module.
[0251] The low-pass filter may receive the left channel audio and the right channel audio, and perform low-pass filtering on the left channel audio and the right channel audio to obtain a left low-frequency signal and a right low-frequency signal.
[0252] Afterwards, on the one hand, the low-pass filter needs to input the left low-frequency signal and the right low-frequency signal into the airflow probability estimation module. On the other hand, the low-pass filter also needs to input the left low-frequency signal and the right low-frequency signal into the left airflow signal synthesis module and the right airflow signal synthesis module.
[0253] Among them, the airflow probability estimation module can calculate the airflow probability based on the left low-frequency signal and the right low-frequency signal. The airflow probability is used to indicate the probability that the left low-frequency signal and the right low-frequency signal are airflow signals. In other words, the airflow probability is used to indicate the proportion of the airflow signal in the left low-frequency signal and the right low-frequency signal to the total signal amount. The greater the airflow probability, the more airflow signals are contained in the left low-frequency signal and the right low-frequency signal.
[0254] After determining the airflow probability, the airflow probability estimation module may send the airflow probability to the left airflow signal synthesis module and the right airflow signal synthesis module respectively.
[0255] The left airflow signal synthesis module can obtain the left airflow signal based on the left low-frequency signal and the airflow probability. The left airflow signal synthesis module also needs to send the left airflow signal to the airflow size estimation module.
[0256] The right airflow signal synthesis module can obtain the right airflow signal based on the right low-frequency signal and the airflow probability. The right airflow signal synthesis module also needs to send the right airflow signal to the airflow size estimation module.
[0257] The airflow magnitude estimation module may output a left airflow magnitude based on the left airflow signal.
[0258] Similarly, the airflow size estimation module can also output the right airflow size based on the right airflow signal.
[0259] In some embodiments, after obtaining the left low-frequency signal and the right low-frequency signal, the low-pass filter can directly input the left low-frequency signal and the right low-frequency signal into the airflow size estimation module. The airflow size estimation module can output the left airflow size based on the left low-frequency signal. Similarly, the airflow size estimation module can also output the right airflow size based on the right low-frequency signal.
[0260] Next, it is explained in detail how the above modules obtain the left airflow size and the right airflow size.
[0261] 1. The electronic device 100 obtains a left low-frequency signal based on the left channel audio through a low-pass filter, and obtains a right low-frequency signal based on the right channel audio through a low-pass filter.
[0262] In some embodiments, the electronic device 100 can obtain the left low-frequency signal and the right low-frequency signal by using formula (1) and formula (2) respectively.
[0263] Formula (1)
[0264] Formula (2)
[0265] Among them, the formula (1) shows Represents the low-frequency signal on the left. Indicates the left audio signal. Represents a low-pass filter. For example, It can be {a, b, a}, a=0.18, b=0.64, or a=0.64, b=0.58. N represents the number of sampling points of the left low-frequency signal.
[0266] The formula (2) shows Represents the low-frequency signal on the left. Indicates the left audio signal. represents the low-pass filter coefficients {a, b, a}. For example, a=0.18, b=0.64, or a=0.64, b=0.58. N represents the number of sampling points of the left low-frequency signal.
[0267] Optionally, since the audio signal needs to be processed in real time, the audio signal is usually split into some continuous segments, which can also be called frames. The length of each audio frame can be a fixed number of milliseconds or tens of milliseconds. Each frame of audio can contain N sampling points (N is a positive integer). Exemplarily, the sampling rate of the audio signal is 48000Hz, and the length of each audio frame is 10ms, then a frame of audio can include 480 sampling points, and the value of each sampling point can be identified by a floating point number, for example, the value of each sampling point can be between [-1.0, 1.0].
[0268] The above formulas (1) and (2) are only used to exemplify how the electronic device 100 obtains the left low-frequency signal and the right low-frequency signal based on the left channel audio and the right channel audio. The electronic device 100 can also obtain the left low-frequency signal and the right low-frequency signal based on the left channel audio and the right channel audio in other ways, and the present application does not limit this.
[0269] Second, the electronic device 100 can calculate the wind noise probability based on the left low-frequency signal and the right low-frequency signal.
[0270] Since the frequency of the airflow is relatively low, the electronic device 100 can filter out the high-frequency signal through a low-pass filter and only retain the low-frequency signal, which may include the airflow signal in the audio.
[0271] In some embodiments, in addition to the airflow signal, the low-frequency signal may also include low-frequency noise such as musical instrument sounds or male voices. In order to improve the accuracy of the final left airflow signal and the right airflow signal, the electronic device 100 also needs to remove the non-airflow signal in the low-frequency signal.
[0272] Optionally, the electronic device 100 calculates the wind noise probability based on the left low-frequency signal and the right low-frequency signal. The greater the wind noise probability, the more airflow sound is contained in the low-frequency signal and the less non-airflow sound is contained in the low-frequency signal.
[0273] In some embodiments, the electronic device 100 can obtain the wind noise probability by using formula (3) and formula (4) respectively.
[0274]
[0275] p=1-c Formula (4)
[0276] Among them, the formula (3) shows Represents the low-frequency signal on the left. represents the right low-frequency signal, and c in formula (3) represents the correlation between the left low-frequency signal and the right low-frequency signal. The value of c is between [0, 1]. The higher the value of c, the higher the correlation between the left low-frequency signal and the right low-frequency signal, and the fewer airflow signals are contained in the left low-frequency signal and the right low-frequency signal.
[0277] is a very small positive real number (such as 0.0001, 0.00003, etc.), which is used to avoid the situation where the denominator is 0 in the calculation of formula (3). In some embodiments, formula (3) may not include positive real numbers , this application does not limit this.
[0278] p in formula (4) represents the probability of wind noise. The value of p is between [0, 1]. The larger the value of p is, the more airflow signals are contained in the left low-frequency signal and the right low-frequency signal.
[0279] The above formula (3) and formula (4) are only used to illustrate how the electronic device 100 obtains the wind noise probability based on the left low-frequency signal and the right low-frequency signal. The electronic device 100 can also obtain the wind noise probability based on the left low-frequency signal and the right low-frequency signal in other ways, and the present application does not limit this.
[0280] 3. The electronic device 100 obtains a left airflow signal and a right airflow signal based on the left low-frequency signal, the right low-frequency signal and the wind noise probability.
[0281] After the electronic device 100 determines the wind noise probability, the electronic device 100 can obtain the left airflow signal based on the left low-frequency signal and the wind noise probability. Then, the right airflow signal is obtained based on the right airflow signal and the wind noise probability. In this way, the electronic device 100 can exclude non-airflow signals in the low-frequency signal, and the accuracy of the electronic device 100 obtaining the left airflow signal and the right airflow signal can be improved.
[0282] In some embodiments, the electronic device 100 can obtain the left airflow signal through formula (5) and the right airflow signal through formula (6).
[0283] Formula (5)
[0284] Formula (6)
[0285] Among them, in formula (5) represents the low-frequency signal on the left, p represents the probability of wind noise, Indicates the left airflow signal.
[0286] In formula (6) represents the low-frequency signal on the right, p represents the probability of wind noise, Indicates right side airflow signal.
[0287] The above formula (5) and formula (6) are only used to exemplify how the electronic device 100 obtains the left airflow signal based on the left low-frequency signal and the wind noise probability, and how to obtain the right airflow signal based on the right low-frequency signal and the wind noise probability. The electronic device 100 can also obtain the left airflow signal based on the left low-frequency signal and the wind noise probability by other methods, and obtain the right airflow signal based on the right low-frequency signal and the wind noise probability by other methods. This application does not limit this.
[0288] 4. The electronic device 100 obtains the size of the left airflow based on the left airflow signal, and obtains the size of the right airflow based on the right airflow signal.
[0289] In some embodiments, the electronic device 100 can obtain the size of the airflow on the left side by using formula (7) and the size of the airflow on the right side by using formula (8).
[0290]
[0291] Among them, in formula (7) Indicates the left airflow signal, Indicates the airflow size on the left.
[0292] In formula (8) Indicates the right airflow signal, Indicates the airflow size on the right.
[0293] The above formulas (7) and (8) are only used to illustrate how the electronic device 100 obtains the left airflow size based on the left airflow signal and how it obtains the right airflow size based on the right airflow signal. The electronic device 100 can also obtain the left airflow size based on the left airflow signal by other methods and obtain the right airflow size based on the right airflow signal by other methods. This application does not limit this.
[0294] In some embodiments, formula (7) It can also represent the low-frequency signal on the left. Formula (8) It can also represent the right low-frequency signal, that is, the electronic device 100 can directly obtain the left airflow size and the right airflow size based on the left low-frequency signal and the right low-frequency signal, and this application does not limit this.
[0295] Method 2: In some embodiments, the electronic device 100 may also separate the left airflow signal from the left channel audio and the right airflow signal from the right channel audio based on a deep learning network.
[0296] Exemplarily, the deep learning network can be a NN network.
[0297] Fig.14A A schematic diagram showing another electronic device 100 obtaining the left airflow size and the right airflow size based on the airflow size estimation module.
[0298] like Fig.14A As shown, the airflow size estimation module includes a NN network and an airflow size estimation module.
[0299] Among them, the electronic device 100 can input the left channel audio and the right channel audio into the NN network, and the NN network can output the left airflow signal based on the left channel audio, and output the right airflow signal based on the right channel audio. Afterwards, the NN network inputs the left airflow signal into the airflow size estimation module, and the airflow size estimation module can output the left airflow size. The NN network then inputs the right airflow signal into the airflow size estimation module, and the airflow size estimation module can output the right airflow size.
[0300] First, it is introduced how the electronic device 100 is trained to obtain a deep learning network.
[0301] Fig. 14B A schematic diagram of training a NN network is shown.
[0302] like Fig. 14BAs shown, the training interface may include airflow sound and non-airflow sound data. The non-airflow sound data may include but is not limited to human voice, musical instrument sound, special effect noise, etc.
[0303] Then, the airflow sound data in the time domain can be converted into airflow sound data in the frequency domain through time-frequency transformation.
[0304] The airflow sound data and the non-airflow sound data can be mixed to obtain a mixed signal. The mixed signal can be converted from the time domain mixed signal to the frequency domain mixed signal by time-frequency transformation.
[0305] Afterwards, the real airflow sound mask can be obtained based on the airflow sound data in the frequency domain and the mixed signal in the frequency domain. The real airflow sound mask can be understood as the proportion of the airflow sound data in the mixed signal (0.3), and the real airflow sound mask is accurate data. The larger the real airflow sound mask, the more airflow sound data is contained in the mixed signal, and the smaller the real airflow sound mask, the less airflow sound data is contained in the mixed signal.
[0306] At the same time, the mixed signal in the frequency domain can be input into the NN network, and the NN network can output a predicted airflow sound mask. The predicted airflow sound mask can be the proportion of the airflow sound data predicted by the NN network from the mixed signal in the frequency domain in the mixed signal. The predicted airflow sound mask is not accurate data, but predicted data. Therefore, it is necessary to compare the difference between the predicted airflow sound mask and the real airflow sound mask, and then adjust the network parameters of the NN network so that the difference between the predicted airflow sound mask output by the NN network and the real airflow sound mask is within the preset range. The training is completed, and the predicted airflow sound mask output by the NN network is close to the real airflow sound mask.
[0307] like Fig. 14B As shown, after the NN network obtains the predicted airflow sound mask, the predicted airflow sound mask can be input into the loss function, and the real airflow sound mask can also be input into the loss function, and the difference between the predicted airflow sound mask and the real airflow sound mask can be calculated. If the difference is within the preset range, the predicted airflow sound mask output by the NN network is close to the real airflow sound mask, and the NN network training is completed. If the difference is greater than the preset range, it is necessary to adjust the NN network parameters, train the NN network again, and then compare the difference between the predicted airflow sound mask output by the NN network after the network parameters are adjusted and the real airflow sound mask. If the difference is within the preset range, the training is completed, and the predicted airflow sound mask output by the NN network is close to the real airflow sound mask. If the difference is still greater than the range, the NN network can continue to be trained in the above manner until the difference between the predicted airflow sound mask output by the NN network and the real airflow sound mask is within the preset range, and the training is completed.
[0308] Fig. 14CA schematic diagram showing the electronic device 100 separating an airflow signal from audio data based on a NN network is shown.
[0309] like Fig. 14C As shown, the left channel audio can be transformed in time-frequency, and the left channel audio in the time domain can be converted into the left channel audio in the frequency domain. The left channel audio in the frequency domain is then input into the trained NN network, and the NN network can output the left predicted airflow sound mask. Afterwards, the left airflow signal in the frequency domain can be obtained based on the left predicted airflow sound mask and the left channel audio in the frequency domain. Finally, the left airflow signal in the frequency domain is transformed in time-frequency, and the left airflow signal in the frequency domain is converted into the left airflow signal in the time domain. The left airflow signal in the time domain is the final left airflow signal.
[0310] Similarly, if Fig. 14C As shown, the right channel audio can be transformed in time-frequency, and the right channel audio in the time domain can be converted into the right channel audio in the frequency domain. Then the right channel audio in the frequency domain is input into the trained NN network, and the NN network can output the right predicted airflow sound mask. After that, the right airflow signal in the frequency domain can be obtained based on the right predicted airflow sound mask and the right channel audio in the frequency domain. Finally, the right airflow signal in the frequency domain is transformed in time-frequency, and the right airflow signal in the frequency domain is converted into the right airflow signal in the time domain. The right airflow signal in the time domain is the right airflow signal finally obtained.
[0311] exist Fig. 14C In the embodiment, after obtaining the left airflow signal and the right airflow signal based on the NN network, the electronic device 100 can obtain the left airflow size and the right airflow size based on the left airflow signal and the right airflow signal according to formula (7) and formula (8), which will not be described in detail in this application.
[0312] The airflow signal can be obtained by separating the airflow signal from the audio in addition to the two methods described above. The airflow signal can also be obtained by separating the airflow signal from the audio in other ways, which is not limited in the present application.
[0313] Next, it is described how the electronic device 100 calculates the air-spinning direction based on the magnitude of the left air-flow signal and the magnitude of the right air-flow.
[0314] Optionally, in the present application, the directions of the airflows emitted by the left-side airflow generating device and the right-side airflow generating device may be consistent.
[0315] In other embodiments, the directions of the airflows emitted by the left-side airflow generating device and the right-side airflow generating device may also be inconsistent, and the present application does not limit this.
[0316] In some embodiments, when the airflow blows from the right side to the left side toward the user's face, the wind sound heard by the user's right ear is greater than the wind sound heard by the user's left ear. When the airflow blows from the left side to the right side toward the user's face, the wind sound heard by the user's left ear is greater than the wind sound heard by the user's right ear. When the airflow blows perpendicularly to the user's face, the wind sound heard by the user's left ear is roughly equal to the wind sound heard by the user's right ear. Based on this principle, the electronic device 100 can calculate the airflow spinning direction based on the size of the left airflow signal and the size of the right airflow.
[0317] Based on the above description, the value range of the airflow direction θ is between [-A, A].
[0318] Exemplarily, A may be π / 2, and the value range of the airflow direction θ is between [-π / 2, π / 2].
[0319] Among them, when the airflow direction is -π / 2, the left airflow generating device and the right airflow generating device can emit airflow parallel to the user's face and horizontally to the left.
[0320] When the airflow direction is 0, the left airflow generating device and the right airflow generating device can emit airflow perpendicular to the user's face.
[0321] Among them, when the airflow direction is π / 2, the left airflow generating device and the right airflow generating device can emit airflow parallel to the user's face and horizontally to the right.
[0322] This application provides four methods for calculating the airflow direction.
[0323] Method 1: The electronic device 100 can calculate the airflow direction θ based on formula (9).
[0324] Formula (9)
[0325] As shown in formula (9), Indicates the left airflow signal, represents the right airflow signal, and θ represents the airflow direction. Wherein, ε is a very small positive real number such as 0.0001, 0.00003, etc., which is used to avoid the situation where the denominator is 0 in formula (9) when calculating θ. Optionally, in other embodiments, ε may not be included in formula (9), and this application does not limit this.
[0326] The airflow direction θ calculated in formula (9) is the actual airflow direction calculated based on the airflow size on the left and the airflow size on the right. The value range of the airflow direction θ is between [-A, A].
[0327] Formula (9) does not limit the maximum angle and minimum angle of the airflow direction θ. In some embodiments, when the airflow direction θ is greater than the maximum angle or minimum angle of the airflow that can be sensed by the face, the airflow generated by the airflow generating device cannot reach the face, and the user may not feel the airflow. To avoid this situation, the maximum angle and minimum angle of the airflow that can be sensed by the user's face can be limited.
[0328] The electronic device 100 can obtain the airflow direction θ based on the left airflow size, the right airflow size and the preset maximum angle B. The value of the airflow direction θ is limited between the maximum angle B and the minimum angle -B at which the human face can perceive the airflow.
[0329] For details, please refer to the descriptions in Method 2 to Method 4.
[0330] Method 2: The electronic device 100 can calculate the airflow direction θ based on formula (10).
[0331] Formula (10)
[0332] As shown in formula (10), Indicates the left airflow signal, Represents the right airflow signal. Wherein, ε is a very small positive real number such as 0.0001, 0.00003, etc., which is used to avoid the situation where the denominator is 0 in formula (10) when calculating θ. Optionally, in other embodiments, ε may not be included in formula (10), and this application does not limit this. The value range of the airflow direction θ calculated in formula (10) is between [-B, B], where B is less than A.
[0333] In formula (10), the electronic device 100 can calculate the airflow direction θ based on the size of the airflow on the left and the size of the airflow on the right. In method 2, the maximum angle and the minimum angle of the airflow direction θ are limited. Because if the airflow direction θ calculated based on the size of the airflow on the left and the size of the airflow on the right is greater than the maximum angle or less than the minimum angle, the airflow emitted by the airflow generating device cannot reach the face of the user, and the user may not feel the airflow. In order to avoid this situation, the maximum angle and the minimum angle at which the user's face can feel the airflow can be limited.
[0334] When the airflow direction θ is greater than the maximum angle or less than the minimum angle, the electronic device 100 can control the airflow generating device to only emit an airflow of a preset size toward the maximum angle or less than the minimum angle, thereby avoiding the situation where the user cannot feel the airflow, reducing the power consumption of the electronic device 100, and improving the user experience.
[0335] When the airflow direction θ is greater than the minimum angle and less than the maximum angle, that is, between the minimum angle and the maximum angle, the electronic device 100 can emit an airflow of a preset magnitude according to the calculated actual airflow direction θ.
[0336] Exemplarily, B may be π / 4, and the value range of the airflow direction θ is between [-π / 4, π / 4].
[0337] For example, when the electronic device 100 calculates the airflow direction θ to be 3π / 4 based on the left airflow size and the right airflow size, 3π / 4 is greater than π / 4. The electronic device 100 can control the left airflow generating device and the right airflow generating device to only emit an airflow of a preset size in the direction of π / 4.
[0338] When the electronic device 100 calculates the airflow direction θ to be -3π / 4 based on the airflow size on the left and the airflow size on the right, -3π / 4 is less than -π / 4. The electronic device 100 can control the airflow generating device on the left and the airflow generating device on the right to emit an airflow of a preset size only in the direction of -π / 4.
[0339] When the electronic device 100 calculates the airflow direction θ to be 0 degrees based on the left airflow size and the right airflow size, and 0 is greater than -π / 4 and less than π / 4, the electronic device 100 can control the left airflow generating device and the right airflow generating device to emit an airflow of a preset size in the direction of 0 degrees.
[0340] That is to say, in method two, when the airflow direction θ calculated by the electronic device 100 based on the left airflow size and the right airflow size is between [-B, B], the electronic device 100 can control the left airflow generating device and the right airflow generating device to emit an airflow of a preset size toward the calculated actual airflow direction.
[0341] When the airflow direction θ calculated by the electronic device 100 based on the left airflow size and the right airflow size is less than -B, the electronic device 100 can control the left airflow generating device and the right airflow generating device to only emit an airflow of a preset size toward the -B direction. When the airflow direction θ calculated by the electronic device 100 based on the left airflow size and the right airflow size is greater than B, the electronic device 100 can control the left airflow generating device and the right airflow generating device to only emit an airflow of a preset size toward the B direction.
[0342] Method 3: The electronic device 100 can calculate the airflow direction θ based on formula (11).
[0343] Formula (11)
[0344] As shown in formula (11), Indicates the left airflow signal, Represents the right airflow signal. B represents the maximum airflow direction that the left airflow generating device and the right airflow generating device can emit, and the maximum direction is the maximum direction in which the user can feel the airflow. Among them, ε is a very small positive real number such as 0.0001, 0.00003, etc., which is used to avoid the situation where the denominator is 0 in formula (11) when calculating θ. Optionally, in other embodiments, ε may not be included in formula (11), and this application does not limit this. The value range of the airflow direction θ calculated in formula (11) is between [-B, B], where B is less than A.
[0345] According to formula (11), the airflow direction θ calculated by the electronic device 100 based on the left airflow size and the right airflow size can be limited to between [-B, B], that is, the airflow direction θ calculated by the electronic device 100 based on the left airflow size and the right airflow size can be limited to between the minimum angle -B and the maximum angle B. To avoid the situation where the airflow direction θ calculated by the electronic device 100 is greater than the maximum angle B or less than the minimum angle -B, the airflow generated by the airflow generating device cannot blow on the face of the user, and the user may not feel the airflow.
[0346] It should be noted that the airflow direction θ calculated according to formula (11) is different from the actual airflow direction calculated by the electronic device 100 based on the left airflow size and the right airflow size. The airflow direction θ calculated according to formula (11) will be smaller than the actual airflow direction calculated by the electronic device 100 based on the left airflow size and the right airflow size, thereby avoiding the situation where the actual airflow direction calculated by the electronic device 100 based on the left airflow size and the right airflow size is greater than the maximum angle B or less than the minimum angle -B.
[0347] Method 4: The electronic device 100 can calculate the airflow direction θ based on formula (12).
[0348] Formula (12)
[0349] As shown in formula (12), Indicates the left airflow signal, Represents the right airflow signal. B represents the maximum airflow direction that the left airflow generating device and the right airflow generating device can emit, and the maximum direction is the maximum direction in which the user can feel the airflow. Among them, ε is a very small positive real number such as 0.0001, 0.00003, etc., which is used to avoid the situation where the denominator is 0 in formula (12) when calculating θ. Optionally, in other embodiments, ε may not be included in formula (12), and this application does not limit this. The value range of the airflow direction θ calculated in formula (12) is between [-B, B], where B is less than A.
[0350] According to formula (12), the airflow direction θ calculated by the electronic device 100 based on the left airflow size and the right airflow size can be limited to between [-B, B], that is, the airflow direction θ calculated by the electronic device 100 based on the left airflow size and the right airflow size can be limited to between the minimum angle -B and the maximum angle B. To avoid the situation where the airflow direction θ calculated by the electronic device 100 is greater than the maximum angle B or less than the minimum angle -B, the airflow generated by the airflow generating device cannot blow on the face of the user, and the user may not feel the airflow.
[0351] The methods of calculating the airflow direction θ are not limited to the above four methods. The electronic device 100 can also calculate the airflow direction based on other methods, which is not limited in the present application.
[0352] Fig.15 A schematic flow chart of an airflow generating method provided in the present application.
[0353] The electronic device includes a display device, an audio playback device and an airflow generating device. The audio playback device includes a first audio playback device and a second audio playback device. The audio playback device airflow generating device includes a first airflow generating device and a second airflow generating device.
[0354] Among them, the electronic device can be a VR device or an AR device, for example, the electronic device can be a head-mounted display device, etc.
[0355] For example, an electronic device may be Figure 3-Figure 6 An electronic device 100 is shown.
[0356] The display device may be the display in the above-mentioned embodiment.
[0357] The first audio playback device may be the left speaker in the above embodiment.
[0358] The second audio playback device may be the right speaker in the above embodiment.
[0359] The first airflow generating device may be the left airflow generating device in the above embodiment.
[0360] The second airflow generating device may be the right airflow generating device in the above embodiment.
[0361] The present invention is not limited to two airflow generating devices. The electronic device may include only one airflow generating device, or the electronic device may include more than two airflow generating devices. This application does not make any limitation on this.
[0362] The present invention is not limited to two audio playback devices. The electronic device may include only one, or the electronic device may include more than two. This application does not make any limitation on this.
[0363] S1501. Display a virtual object via a display device.
[0364] S1502: Play the first audio through the first audio playback device, and play the second audio through the second audio playback device.
[0365] The first audio may be the left audio in the above embodiment.
[0366] The second audio may be the right audio in the above embodiment.
[0367] S1503: Calculate a first airflow size of the airflow sound in the first audio and a second airflow size of the airflow sound in the second audio.
[0368] S1504. Output an airflow of a first airflow size through the first airflow generating device, and output an airflow of a second airflow size through the second airflow generating device.
[0369] The electronic device is used to be worn on the user's head so that the user's eyes are located inside the electronic device, and the user's eyes can view the virtual object displayed by the display device.
[0370] When the electronic device is worn on the user's head, relative to the position of the user's eyes, the first airflow generating device and the second airflow generating device can be respectively located at the upper left front, upper right front, or lower left front, lower right front, or upper left front, lower right front, or lower left front, upper right front, etc. of the user's eyes.
[0371] In some embodiments, the first airflow generating device and the second airflow generating device may be located outside the display area of the display device to avoid blocking the user from observing the virtual object displayed on the display device. Through this method, a simulated airflow can be generated based on an audio signal in a virtual scene to achieve an immersive effect and enhance the user's virtual experience.
[0372] In one possible implementation, calculating a first airflow size of an airflow sound in a first audio and a second airflow size of an airflow sound in a second audio specifically includes: processing the first audio through a low-pass filter to obtain a first low-frequency signal; processing the second audio through a low-pass filter to obtain a second low-frequency signal; calculating the airflow size of the first low-frequency signal to obtain a first airflow size; calculating the airflow size of the second low-frequency signal to obtain a second airflow size.
[0373] Generally speaking, the frequency of airflow sounds is relatively low. The high-frequency signals in the audio can be filtered out by a low-pass filter, leaving only the low-frequency signals. The low-frequency signals are used as airflow sounds, and the airflow size of the airflow sounds is calculated based on the low-frequency signals. This can improve the accuracy of the airflow size obtained by electronic devices.
[0374] For details, please refer to Fig.13 Description in the examples.
[0375] In a possible implementation, the airflow size of the first low-frequency signal is calculated to obtain the first airflow size, specifically including: obtaining a first wind noise probability based on the first low-frequency signal and the second low-frequency signal, the first wind noise probability being used to indicate the proportion of airflow sound contained in the first low-frequency signal and the second low-frequency signal; obtaining a first airflow signal based on the first low-frequency signal and the first wind noise probability; calculating the airflow size of the first airflow signal to obtain the first airflow size; calculating the airflow size of the second low-frequency signal to obtain the second airflow size, specifically including: obtaining a second airflow signal based on the second low-frequency signal and the first wind noise probability; calculating the airflow size of the second airflow signal to obtain the second airflow size.
[0376] Generally speaking, the frequency of airflow sound is relatively low, and the high-frequency signal in the audio can be filtered out by a low-pass filter, leaving only the low-frequency signal. However, the low-frequency signal may contain other noises, such as the sound of musical instruments or the voice of a boy talking. Therefore, it is also necessary to filter out the noise in the low-frequency signal to obtain the real airflow sound. Specifically, a first wind noise probability can be obtained based on the first low-frequency signal and the second low-frequency signal, and the first wind noise probability is used to indicate the proportion of airflow sound contained in the first low-frequency signal and the second low-frequency signal. Then, based on the first low-frequency signal and the first wind noise probability, a first airflow signal is obtained, and then based on the second low-frequency signal and the first wind noise probability, a second airflow signal is obtained. Then, based on the first airflow signal and the second airflow signal, the first airflow size and the second airflow size are obtained respectively. In this way, the accuracy of the airflow size obtained by the electronic device can be further improved, and the accuracy of the airflow emitted by the electronic device can also be improved.
[0377] In some embodiments, the electronic device 100 can obtain the first low-frequency signal and the second low-frequency signal by using the above formula (1) and formula (2) respectively.
[0378] In some embodiments, the electronic device 100 may obtain the first wind noise probability by using the above formula (3) and formula (4) respectively.
[0379] In some embodiments, the electronic device 100 can obtain the first airflow signal through the above formula (5) and obtain the second airflow signal through the above formula (6).
[0380] For details, please refer to Fig.13 Description in the examples.
[0381] In one possible implementation, the calculation of the airflow size of the first low-frequency signal to obtain the first airflow size specifically includes: filtering out non-airflow signals in the first low-frequency signal through a first neural network to obtain the first airflow signal; calculating the airflow size of the first airflow signal to obtain the first airflow size; calculating the airflow size of the second low-frequency signal to obtain the second airflow size specifically includes: filtering out non-airflow signals in the second low-frequency signal through the first neural network to obtain the second airflow signal; calculating the airflow size of the second airflow signal to obtain the first airflow size.
[0382] In some embodiments, the first neural network may be a neural network such as a NN network, which is not limited in the present application.
[0383] In other implementations, a neural network may be trained, and the first airflow signal and the second airflow signal may be obtained by filtering based on the trained neural network. In this way, the amount of calculation of the electronic device may be reduced.
[0384] For details, please refer to Fig.14A , Fig. 14B , Fig. 14C Description in the examples.
[0385] In one possible implementation, before outputting an airflow of a first airflow size through a first airflow generating device and outputting an airflow of a second airflow size through a second airflow generating device, the method also includes: calculating a first angle based on the first airflow size and the second airflow size; outputting an airflow of the first airflow size through the first airflow generating device and outputting an airflow of the second airflow size through the second airflow generating device, specifically including: outputting an airflow of the first airflow size along a first angle through the first airflow generating device and outputting an airflow of the second airflow size along the first angle through the second airflow generating device.
[0386] In some embodiments, the airflow generating device can not only emit simulated airflow of corresponding size based on the size of the airflow sound, but also emit airflow in corresponding direction based on the direction of the airflow sound in the audio, further enhancing the user experience.
[0387] Specifically, the electronic device can calculate a first angle of the emitted airflow based on the first airflow size and the second airflow size, and then generate a simulated airflow of corresponding size based on the first angle.
[0388] In a possible implementation, an airflow of a first airflow size is output along a first angle by a first airflow generating device, and an airflow of a second airflow size is output along the first angle by a second airflow generating device, specifically including: when the first angle is greater than a first preset angle and less than a second preset angle, an airflow of the first airflow size is output along the first angle by the first airflow generating device, and an airflow of the second airflow size is output along the first angle by the second airflow generating device; wherein the first preset angle is the minimum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception, and the second preset angle is the maximum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception.
[0389] In one possible implementation, the method also includes: when the first angle is less than the first preset angle, outputting an airflow of a first airflow size along the first preset angle through the first airflow generating device, and outputting an airflow of a second airflow size along the first preset angle through the second airflow generating device; or, when the first angle is greater than the second preset angle, outputting an airflow of the first airflow size along the second preset angle through the first airflow generating device, and outputting an airflow of the second airflow size along the second preset angle through the second airflow generating device.
[0390] In this way, the electronic device can determine the calculated airflow angle to prevent the airflow angle from exceeding the maximum angle or minimum angle of airflow that can be perceived by the face. After the calculated airflow angle exceeds the maximum angle or minimum angle of airflow that can be perceived by the face, the airflow can be output only according to the first preset angle or the second preset angle.
[0391] In some embodiments, the first preset angle may be -45 degrees.
[0392] In some embodiments, the second preset angle may be 50 degrees.
[0393] For details, please refer to Figures 10A-10G Description in the examples.
[0394] In a possible implementation, before outputting an airflow of a first airflow size through a first airflow generating device and outputting an airflow of a second airflow size through a second airflow generating device, the method also includes: calculating a second angle based on the first airflow size, the second airflow size and a second preset angle, wherein the second preset angle is the maximum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception; outputting an airflow of a first airflow size through the first airflow generating device and outputting an airflow of a second airflow size through the second airflow generating device specifically includes: outputting an airflow of the first airflow size along the second angle through the first airflow generating device and outputting an airflow of the second airflow size along the second angle through the second airflow generating device.
[0395] In some embodiments, the second angle is smaller than the first angle.
[0396] The airflow angle calculated in this manner is between the first preset angle and the second preset angle, and will not exceed the maximum angle and the minimum angle of the airflow that can be perceived by the human face.
[0397] In a possible implementation, calculating the first angle based on the first airflow size and the second airflow size specifically includes: according to the formula Calculate the first angle; where, is the first angle, is the first airflow size, is the second airflow size.
[0398] In some embodiments, a positive minimum value ε may be added to the denominator in the above formula. For example, the value of ε may be 0.0001, 0.00003, etc.
[0399] For details, please refer to the description in the above formula (9).
[0400] In a possible implementation, calculating the first angle based on the first airflow size and the second airflow size specifically includes: according to the formula Calculate the first angle; where, is the first angle, is the first airflow size, is the second airflow size, B is the second preset angle, and -B is the first preset angle.
[0401] In some embodiments, a positive minimum value ε may be added to the denominator in the above formula. For example, the value of ε may be 0.0001, 0.00003, etc.
[0402] For details, please refer to the description in the above formula (10).
[0403] In a possible implementation, the second angle is calculated based on the first airflow size, the second airflow size, and the second preset angle, specifically including: according to the formula Calculate the second angle; where, is the second angle, is the first airflow size, is the second airflow size, and B is the second preset angle.
[0404] In some embodiments, a positive minimum value ε may be added to the denominator in the above formula. For example, the value of ε may be 0.0001, 0.00003, etc.
[0405] For details, please refer to the description in the above formula (11).
[0406] In a possible implementation, the second angle is calculated based on the first airflow size, the second airflow size, and the second preset angle, specifically including: according to the formula Calculate the second angle; where, is the second angle, is the first airflow size, is the second airflow size, and B is the second preset angle.
[0407] In some embodiments, a positive minimum value ε may be added to the denominator in the above formula. For example, the value of ε may be 0.0001, 0.00003, etc.
[0408] For details, please refer to the description in the above formula (12).
[0409] Fig.16 A schematic diagram of an airflow generating device provided in the present application.
[0410] like Fig.16 As shown, the device 1600 includes a display device 180, an audio playback device 1602, an airflow generating device 1603 and a processing device 1604. The audio playback device 1602 includes a first audio playback device 1605 and a second audio playback device 1606, and the airflow generating device 1603 includes a first airflow generating device 1607 and a second airflow generating device 1608, wherein:
[0411] The display device is used to display a virtual object.
[0412] The first audio playing device 1605 is used to play the first audio.
[0413] The second audio playing device 1606 is used to play the second audio.
[0414] The processing device 1604 is used to calculate a first airflow size of the airflow sound in the first audio and a second airflow size of the airflow sound in the second audio.
[0415] The first airflow generating device 1607 is used to output an airflow of a first airflow size.
[0416] The second airflow generating device 1608 is used to output an airflow of a second airflow size.
[0417] In one possible implementation, the processing device 1604 is specifically used to: process the first audio through a low-pass filter to obtain a first low-frequency signal; process the second audio through a low-pass filter to obtain a second low-frequency signal; calculate the airflow size of the first low-frequency signal to obtain the first airflow size; calculate the airflow size of the second low-frequency signal to obtain the second airflow size.
[0418] In one possible implementation, the processing device 1604 is specifically used to: obtain a first wind noise probability based on the first low-frequency signal and the second low-frequency signal, the first wind noise probability being used to indicate the proportion of airflow sound contained in the first low-frequency signal and the second low-frequency signal; obtain a first airflow signal based on the first low-frequency signal and the first wind noise probability; calculate the airflow size of the first airflow signal to obtain the first airflow size; obtain a second airflow signal based on the second low-frequency signal and the first wind noise probability; calculate the airflow size of the second airflow signal to obtain the second airflow size.
[0419] In one possible implementation, the processing device 1604 is specifically used to: filter out non-airflow signals in the first low-frequency signal through a first neural network to obtain a first airflow signal; calculate the airflow size of the first airflow signal to obtain the first airflow size; filter out non-airflow signals in the second low-frequency signal through the first neural network to obtain a second airflow signal; calculate the airflow size of the second airflow signal to obtain the first airflow size.
[0420] In one possible implementation, the processing device 1604 is also used to calculate the first angle based on the first airflow size and the second airflow size; the first airflow generating device 1607 is specifically used to output an airflow of the first airflow size along the first angle; the second airflow generating device 1608 is specifically used to output an airflow of the second airflow size along the first angle.
[0421] In one possible implementation, the first airflow generating device 1607 is specifically used to output an airflow of a first airflow size along a first angle when the first angle is greater than a first preset angle and less than a second preset angle; the second airflow generating device 1608 is specifically used to output an airflow of a second airflow size along a first angle when the first angle is greater than the first preset angle and less than the second preset angle; wherein the first preset angle is the minimum angle of the airflow output by the first airflow generating device 1607 or the second airflow generating device 1608 for face perception, and the second preset angle is the maximum angle of the airflow output by the first airflow generating device 1607 or the second airflow generating device 1608 for face perception.
[0422] In one possible implementation, the first airflow generating device 1607 is further used to output an airflow of a first airflow size along a first preset angle when the first angle is less than the first preset angle; the second airflow generating device 1608 is further used to output an airflow of a second airflow size along the first preset angle when the first angle is less than the first preset angle; or, the first airflow generating device 1607 is further used to output an airflow of a first airflow size along a second preset angle when the first angle is greater than the second preset angle; the second airflow generating device 1608 is further used to output an airflow of a second airflow size along the second preset angle when the first angle is greater than the second preset angle.
[0423] In one possible implementation, the processing device 1604 is further used to calculate the second angle based on the first airflow size, the second airflow size and the second preset angle, wherein the second preset angle is the maximum angle of the airflow output by the first airflow generating device 1607 or the second airflow generating device 1608 for face perception; the first airflow generating device 1607 is specifically used to output an airflow of a first airflow size along the second angle; and the second airflow generating device 1608 is specifically used to output an airflow of a second airflow size along the second angle.
[0424] In a possible implementation, the processing device 1604 is specifically configured to: Calculate the first angle; where, is the first angle, is the first airflow size, is the second airflow size.
[0425] In a possible implementation, the processing device 1604 is specifically configured to: Calculate the first angle; where, is the first angle, is the first airflow size, is the second airflow size, B is the second preset angle, and -B is the first preset angle.
[0426] In a possible implementation, the processing device 1604 is specifically configured to: Calculate the second angle; where, is the second angle, is the first airflow size, is the second airflow size, and B is the second preset angle.
[0427] In a possible implementation, the processing device 1604 is specifically configured to: Calculate the second angle; where, is the second angle, is the first airflow size, is the second airflow size, and B is the second preset angle.
[0428] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.
[0429] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.
[0430] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.
[0431] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for generating airflow, characterized in that: The method is applied to an electronic device, the electronic device comprising a display device, an audio playback device and an airflow generating device, the audio playback device comprising a first audio playback device and a second audio playback device, the airflow generating device comprising a first airflow generating device and a second airflow generating device, the method comprising: displaying a virtual object through the display device; Play the first audio through the first audio playback device, and play the second audio through the second audio playback device; Calculating a first airflow magnitude of the airflow sound in the first audio and a second airflow magnitude of the airflow sound in the second audio; calculating a first angle based on the first airflow magnitude and the second airflow magnitude; When the first angle is greater than the first preset angle and less than the second preset angle, the first airflow generating device outputs an airflow of the first airflow size along the first angle, and the second airflow generating device outputs an airflow of the second airflow size along the first angle; When the first angle is smaller than the first preset angle, the first airflow generating device outputs an airflow of the first airflow size along the first preset angle, and the second airflow generating device outputs an airflow of the second airflow size along the first preset angle; When the first angle is greater than the second preset angle, the first airflow generating device outputs an airflow of the first airflow size along the second preset angle, and the second airflow generating device outputs an airflow of the second airflow size along the second preset angle; Among them, the first preset angle is the minimum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception, and the second preset angle is the maximum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception.
2. A method for generating airflow, characterized in that: The method is applied to an electronic device, the electronic device comprising a display device, an audio playback device and an airflow generating device, the audio playback device comprising a first audio playback device and a second audio playback device, the airflow generating device comprising a first airflow generating device and a second airflow generating device, the method comprising: displaying a virtual object through the display device; Play the first audio through the first audio playback device, and play the second audio through the second audio playback device; Calculating a first airflow magnitude of the airflow sound in the first audio and a second airflow magnitude of the airflow sound in the second audio; Calculating a second angle based on the first airflow size, the second airflow size and a second preset angle, wherein the second preset angle is a maximum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception, and the second angle is less than the second preset angle; The first airflow generating device outputs an airflow of the first airflow size along the second angle, and the second airflow generating device outputs an airflow of the second airflow size along the second angle.
3. The method according to claim 1 or 2, characterized in that: The calculating a first airflow size of the airflow sound in the first audio and a second airflow size of the airflow sound in the second audio specifically includes: Processing the first audio signal through a low-pass filter to obtain a first low-frequency signal; Processing the second audio through the low-pass filter to obtain a second low-frequency signal; Calculating the airflow size of the first low-frequency signal to obtain the first airflow size; The airflow size of the second low-frequency signal is calculated to obtain the second airflow size.
4. The method according to claim 3, characterized in that The calculating the airflow size of the first low-frequency signal to obtain the first airflow size specifically includes: obtaining a first wind noise probability based on the first low-frequency signal and the second low-frequency signal, wherein the first wind noise probability is used to indicate a ratio of airflow sound contained in the first low-frequency signal and the second low-frequency signal; obtaining a first airflow signal based on the first low-frequency signal and the first wind noise probability; Calculating the airflow size of the first airflow signal to obtain the first airflow size; The calculating the airflow size of the second low-frequency signal to obtain the second airflow size specifically includes: obtaining a second airflow signal based on the second low-frequency signal and the first wind noise probability; The airflow magnitude of the second airflow signal is calculated to obtain the second airflow magnitude.
5. The method according to claim 3, characterized in that: The calculating the airflow size of the first low-frequency signal to obtain the first airflow size specifically includes: Filtering out non-airflow signals in the first low-frequency signal through a first neural network to obtain a first airflow signal; Calculating the airflow size of the first airflow signal to obtain the first airflow size; The calculating the airflow size of the second low-frequency signal to obtain the second airflow size specifically includes: Filtering out non-airflow signals in the second low-frequency signal through the first neural network to obtain a second airflow signal; The airflow magnitude of the second airflow signal is calculated to obtain the first airflow magnitude.
6. The method according to claim 1, characterized in that The calculating the first angle based on the first airflow size and the second airflow size specifically includes: According to the formula calculating the first angle; Wherein, θ is the first angle, and F L is the first airflow size, the F R is the second air flow size.
7. The method according to claim 1, characterized in that The calculating the first angle based on the first airflow size and the second airflow size specifically includes: According to the formula calculating the first angle; Wherein, θ is the first angle, and F L is the first airflow size, the F R is the second airflow size, B is the second preset angle, and -B is the first preset angle.
8. The method according to claim 2, characterized in that The calculating the second angle based on the first airflow size, the second airflow size and the second preset angle specifically includes: According to the formula calculating the second angle; Wherein, θ is the second angle, and F L is the first airflow size, the F R is the second airflow size, and B is the second preset angle.
9. The method according to claim 2, characterized in that: The calculating the second angle based on the first airflow size, the second airflow size and the second preset angle specifically includes: According to the formula calculating the second angle; Wherein, θ is the second angle, and F L is the first airflow size, the F R is the second airflow size, and B is the second preset angle.
10. An airflow generating device, characterized in that: The device comprises a display device, an audio playback device, an airflow generating device and a processing device, wherein the audio playback device comprises a first audio playback device and a second audio playback device, and the airflow generating device comprises a first airflow generating device and a second airflow generating device, wherein: The display device is used to display the virtual object; The first audio playing device is used to play the first audio; The second audio playing device is used to play the second audio; The processing device is used to calculate a first airflow size of the airflow sound in the first audio and a second airflow size of the airflow sound in the second audio; The processing device is further used to calculate a first angle based on the first airflow size and the second airflow size; When the first angle is greater than the first preset angle and less than the second preset angle, the first airflow generating device is used to output an airflow of the first airflow size along the first angle, and the second airflow generating device is used to output an airflow of the second airflow size along the first angle; In the case where the first angle is smaller than the first preset angle, the first airflow generating device is used to output an airflow of the first airflow size along the first preset angle, and the second airflow generating device is used to output an airflow of the second airflow size along the first preset angle; When the first angle is greater than the second preset angle, the first airflow generating device is used to output an airflow of the first airflow size along the second preset angle, and the second airflow generating device is used to output an airflow of the second airflow size along the second preset angle; Among them, the first preset angle is the minimum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception, and the second preset angle is the maximum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception.
11. An airflow generating device, characterized in that: The device comprises a display device, an audio playback device, an airflow generating device and a processing device, wherein the audio playback device comprises a first audio playback device and a second audio playback device, and the airflow generating device comprises a first airflow generating device and a second airflow generating device, wherein: The display device is used to display the virtual object; The first audio playing device is used to play the first audio; The second audio playing device is used to play the second audio; The processing device is used to calculate a first airflow size of the airflow sound in the first audio and a second airflow size of the airflow sound in the second audio; The processing device is further used to calculate a second angle based on the first airflow size, the second airflow size and the second preset angle, wherein the second preset angle is a maximum angle of the airflow output by the first airflow generating device or the second airflow generating device for face perception, and the second angle is smaller than the second preset angle; The first airflow generating device is used to output an airflow of the first airflow size along the second angle; The second airflow generating device is used to output the airflow of the second airflow size along the second angle.
12. The device according to claim 10 or 11, characterized in that The processing device is specifically used for: Processing the first audio signal through a low-pass filter to obtain a first low-frequency signal; Processing the second audio through the low-pass filter to obtain a second low-frequency signal; Calculating the airflow size of the first low-frequency signal to obtain the first airflow size; The airflow size of the second low-frequency signal is calculated to obtain the second airflow size.
13. The device according to claim 12, characterized in that The processing device is specifically used for: obtaining a first wind noise probability based on the first low-frequency signal and the second low-frequency signal, wherein the first wind noise probability is used to indicate a ratio of airflow sound contained in the first low-frequency signal and the second low-frequency signal; obtaining a first airflow signal based on the first low-frequency signal and the first wind noise probability; Calculating the airflow size of the first airflow signal to obtain the first airflow size; obtaining a second airflow signal based on the second low-frequency signal and the first wind noise probability; The airflow magnitude of the second airflow signal is calculated to obtain the second airflow magnitude.
14. The device according to claim 12, characterized in that The processing device is specifically used for: Filtering out non-airflow signals in the first low-frequency signal through a first neural network to obtain a first airflow signal; Calculating the airflow size of the first airflow signal to obtain the first airflow size; Filtering out non-airflow signals in the second low-frequency signal through the first neural network to obtain a second airflow signal; The airflow magnitude of the second airflow signal is calculated to obtain the first airflow magnitude.
15. The device according to claim 10, characterized in that The processing device is specifically used to calculating the first angle; Wherein, θ is the first angle, and F L is the first airflow size, the F R is the second air flow size.
16. The device according to claim 10, characterized in that The processing device is specifically used to calculating the first angle; Wherein, θ is the first angle, and F L is the first airflow size, the F R is the second airflow size, B is the second preset angle, and -B is the first preset angle.
17. The device according to claim 11, characterized in that The processing device is specifically used to calculating the second angle; Wherein, θ is the second angle, and F L is the first airflow size, the F R is the second airflow size, and B is the second preset angle.
18. The device according to claim 11, characterized in that The processing device is specifically used to calculating the second angle; Wherein, θ is the second angle, and F L is the first airflow size, the F R is the second airflow size, and B is the second preset angle.
19. An electronic device, characterized in that: It comprises one or more processors and one or more memories; wherein, the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method as described in any one of claims 1-9.
20. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes a processing circuit and an interface circuit, the interface circuit is used to receive instructions and transmit them to the processing circuit, and the processing circuit is used to run the instructions to execute the method as described in any one of claims 1-9.
21. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 9.
