A prompt method, extended reality device and medium
By obtaining and rendering audio streams in the real environment in extended real-life devices, the problem of users ignoring important sounds in the virtual space is solved, achieving a safer and immersive user experience.
Patent Information
- Application Number
- CN202210668433.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-14
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-06-14
AI Technical Summary
When users use extended real-life devices, the virtual space is isolated from the real environment, causing users to ignore important sounds in the real environment, such as mobile phone ringtones, door ringtones, etc., which affects the security and interactive experience.
By extending the real-life device to obtain the initial position in the virtual space and the user's head pose changes, update the spatial position of the target object, and render the audio stream in the real-life space based on this position, output it to the binaural playback device to provide immersion and sound Tip to avoid users missing important voices.
When users are immersed in virtual space, they can effectively remind users of the sound in the real environment, improve security levels and interactive experience, and ensure that users do not miss important events.
Smart Images

Figure CN116048241B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a prompting method, an extended reality device, and a computer-readable storage medium. Background Art
[0002] With the continuous development of technology, a new interactive mode, extended reality (XR), has emerged. Extended reality refers to a virtual space that combines reality and virtuality and can be interacted with by humans and machines, which is generated by computer technology and wearable devices. Generally, extended reality includes augmented reality (AR), virtual reality (VR), mixed reality (MR) and other forms.
[0003] When users are experiencing the above XR devices, they usually wear wearable devices. The interaction of video signals is usually achieved through glasses, and the interaction of sound signals is usually achieved through headphones, which can be wired headphones or true wireless stereo (TWS) headphones. When users wear wearable devices, they will be immersed in a virtual space. At this time, the virtual space and the real environment of the user are isolated. The sounds generated by the sound sources in the real environment, such as mobile phone call reminders, smart locks reminding guests of visits, and alarm sounds of alarms, are easily ignored by users. Therefore, it is currently necessary to prompt users in a timely manner to prevent users from ignoring the sounds in the real environment. Summary of the invention
[0004] The purpose of the present application is to provide a prompting method, an extended reality device, a computer-readable storage medium, and a computer program product, which can prompt a user with sounds in a real environment when the user is immersed in a virtual space.
[0005] In order to achieve the above objectives, this application adopts the following technical solutions:
[0006] In the first aspect, the present application provides a prompting method, which can be performed by an extended reality device. Specifically, the method includes: the extended reality device obtains the initial spatial position of the target object in the virtual space, and then determines the updated spatial position of the target object based on the initial spatial position and the change in the user's head posture, and based on the updated spatial position, renders the audio stream generated by the target object in the real space, and then uses the rendered spatial audio stream for prompting. It can be seen that in this method, when the user is immersed in the virtual space, the extended reality device can obtain the audio stream of the target object in the real space, and based on the initial spatial position of the target object in the real space, render the audio stream of the target object, and output it to the binaural playback device to prompt the user, so as to prevent the user from missing some sounds in the real space. In this way, the extended reality device can not only give the user an immersive experience, but also prevent the user from being completely out of the real space, thereby preventing the user from missing some important prompts in the real space, such as mobile phone ringtones, doorbells, baby crying, etc.
[0007] In some possible implementations, the method further includes: the extended reality device obtains the category of the audio stream of the target object in the real space, and when the category of the audio stream in the real space is a preset category, the audio stream of the target object in the real space is rendered based on the updated spatial position. After classifying the audio stream in the real space, the extended reality device can select different display methods based on audio streams of different categories. This can improve the safety level and virtual interaction capabilities of users immersed in the virtual space. For example, for ordinary events in the real space, the extended reality device prompts by presenting prompt information, which can not only ensure the user's sense of immersion, but also prevent the user from missing some events in the real space;
[0008] In some possible implementations, the method further includes: the extended reality device determines the prompt information corresponding to the category of the audio stream of the target object according to the correspondence between the category of the audio stream of the target object and the prompt information, and displays the prompt information corresponding to the category of the audio stream of the target object to the user. The prompt information includes text and / or images.
[0009] In some possible implementations, when the category of the target object's audio stream is the highest level among the preset categories, the screen flashes to display prompt information corresponding to the category of the target object's audio stream. For example, for important events, the extended reality device displays prompt information while controlling the display screen to flash red, which can more clearly inform the user that there are important events in the real space that need to be handled.
[0010] In some possible implementations, the audio stream of the target object in the real space is obtained by collecting the audio stream of the target object or by receiving the audio stream transmitted by the target object.
[0011] In some possible implementations, the method further includes: the extended reality device acquires a virtual space audio stream; and then the extended reality device provides a prompt through the real space audio stream and the virtual space audio stream.
[0012] In a second aspect, the present application provides an extended reality device, including:
[0013] An acquisition unit, used for acquiring an initial spatial position of a target object in real space;
[0014] An updating unit, configured to determine an updated spatial position of the target object according to the initial spatial position and a change in the spatial position of the extended reality device;
[0015] A rendering unit, configured to render the audio stream of the target object in the real space based on the updated spatial position to obtain a real space audio stream;
[0016] A prompting unit is used to provide prompts through the real space audio stream.
[0017] In some possible implementations, the acquisition unit is further configured to acquire a category of an audio stream of the target object in the real space;
[0018] The rendering unit is specifically configured to render the audio stream of the target object in the real space based on the updated spatial position when the category of the audio stream of the target object in the real space is a preset category.
[0019] In some possible implementations, the prompt unit is further used to determine the prompt information corresponding to the category of the audio stream of the target object based on the correspondence between the category of the audio stream of the target object and the prompt information, and display the prompt information corresponding to the category of the audio stream of the target object to the user.
[0020] In some possible implementations, the prompt information includes text and / or images.
[0021] In some possible implementations, when the category of the audio stream of the target object is the highest level among the preset categories, the prompt unit is specifically configured to display prompt information corresponding to the category of the audio stream of the target object by flashing the screen.
[0022] In some possible implementations, the audio stream of the target object in the real space is obtained by collecting the audio stream of the target object or by receiving the audio stream transmitted by the target object.
[0023] In some possible implementations, the acquisition unit is further used to acquire the virtual space audio stream;
[0024] The prompting unit is specifically used to provide prompting through the real space audio stream and the virtual space audio stream.
[0025] In a third aspect, the present application provides an extended reality device, including: a display screen, a processor, and a memory;
[0026] The display screen is used to display prompt information;
[0027] The memory is used to store computer programs;
[0028] The processor is used to execute the computer program to implement the method according to any one of the first aspects.
[0029] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a computer, the method described in any one of the first aspects is implemented.
[0030] In a fifth aspect, the present application provides a computer program product, which includes instructions. When the computer program product is run in an extended reality device, the extended reality device executes any one of the methods in the first aspect.
[0031] The technical solution of this application has the following beneficial effects:
[0032] The present application provides a prompting method, which can be performed by an XR device. XR can first obtain the initial spatial position of the target object in the real space, and obtain the updated spatial position of the target object based on the change parameters of the user's head posture collected by the XR device, and then based on the updated spatial position of the target object, perform spatialization and immersive rendering on the audio stream of the target object, and output it to the binaural playback device. In this way, even if the user is immersed in the virtual space, the user can feel the sound generated by the target object in the real space, and can also perceive the direction of the sound of the target object.
[0033] Furthermore, the XR device can also classify the audio stream of the target object. For example, the audio stream can be classified based on the acoustic signal characteristics of the audio stream to obtain the category of the audio stream of the target object. When the category of the audio stream of the target object is a preset category, the XR device prompts the user. For example, it can play the audio stream in the real space to the user, or it can display prompt information to the user, so that the user can not only immerse himself in the virtual space, but also avoid missing some important events in the real space.
[0034] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;
[0036] Figure 2 A structural schematic diagram of a VR device provided in an embodiment of the present application;
[0037] Figure 3 Another structural schematic diagram provided for an embodiment of the present application;
[0038] Figure 4 A flowchart of a prompting method provided in an embodiment of the present application;
[0039] Figure 5 A schematic diagram of a living room scene provided in an embodiment of the present application;
[0040] Fig. 6A A schematic diagram of an initial spatial position provided in an embodiment of the present application;
[0041] Figure 6B A schematic diagram of a spatial position change provided in an embodiment of the present application;
[0042] Figure 7 A schematic diagram of an audio stream processing provided in an embodiment of the present application;
[0043] Fig. 8A A schematic diagram of a virtual world screen provided in an embodiment of the present application;
[0044] Figure 8B A schematic diagram of a prompt information provided in an embodiment of the present application;
[0045] Figure 8C A schematic diagram of another prompt information provided in an embodiment of the present application;
[0046] Fig. 9 A flowchart of another prompting method provided in an embodiment of the present application;
[0047] Fig.10 A schematic diagram of an extended reality device provided in an embodiment of the present application;
[0048] Fig.11 A schematic diagram of another extended reality device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] It should be noted that the terms "first", "second" and "third" in the specification, claims and drawings of this application are used to distinguish different objects, rather than to limit a specific order. In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.
[0050] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the application scenario of the technical solution of the present application is first described below.
[0051] Extended reality (XR) can provide a virtual space for human-computer interaction, which can bring users (such as experiencers) an "immersive feeling" of seamless transition between the virtual world and the real world.
[0052] Typically, XR includes various forms such as augmented reality (AR) and virtual reality (VR).
[0053] VR refers to the use of computers to simulate three-dimensional space, simulating a sense of immersion and presence for users through their senses, allowing users to experience a virtual simulated world. VR devices block the user's line of sight through head-mounted displays, creating a completely virtual visual environment; and provide sound signals through headphones, using spatial audio binaural rendering technology to complete the playback of virtual scene sound and image, so that users can experience a three-dimensional virtual immersion.
[0054] AR refers to adding virtual content to the real space, mapping virtual information (such as objects, pictures, videos, sounds, etc.) in the real environment. Similarly, AR devices can also observe the real world through head-mounted displays, and add the above objects, pictures, videos, sounds, etc. by calculating the specified positions in the real world, so that the real world and the virtual world are seamlessly connected, such as adding a picture of a clock on the wall in the real world.
[0055] XR can be seen as a combination of the above-mentioned AR and VR, which can merge the real and virtual worlds to create a new visual virtual space.
[0056] The following is an example of a user wearing a VR device. The VR device and the terminal device exchange information through a wired connection or a wireless connection. The terminal device can be a mobile device, a laptop, a computer or a tablet computer, etc., which is not specifically limited in the embodiments of the present application. Users can watch movies, listen to music, play games or study by wearing VR devices.
[0057] See also Figure 1 , which is a schematic diagram of an application scenario provided by this application.
[0058] like Figure 1 As shown, in this scenario, after the user 110 wears the VR device 120, the VR device 120 can render a virtual space 130 for the user 110, and the user 110 will be immersed in the virtual space 130. The virtual space 130 refers to the three-dimensional space rendered by the VR device, which can provide the user with a sense of immersion and presence, allowing the user to experience the simulated world. The virtual space 130 is different from the real space 140. The real space 140 refers to the three-dimensional space truly perceived by the user, that is, the real world truly perceived by the user.
[0059] The VR device 120 uses the technology of spatial audio binaural rendering to play the sound of the virtual scene, that is, to construct the sound source characteristics such as the direction, distance, and movement of the sound in the virtual space, and provide the user with the audio of the virtual space. However, since the virtual space and the real environment where the user is located are isolated, this will cause the user to ignore the sound from the target object 141 in the real space 140, thereby causing the user to miss the information in the real space.
[0060] The target object 141 in the real space 140 may be an electronic device, such as a smart speaker, a mobile phone, a smart door lock, or an alarm, etc. At this time, the sound may be the sound of a smart speaker broadcasting weather conditions, the sound of a smart door lock reminding a guest to visit, the sound of a mobile phone incoming call, or the alarm sound of an alarm, etc.
[0061] Figure 1The target objects 141 displayed include three, namely a smart speaker, a mobile phone, and a smart door lock.
[0062] It should be noted that Figure 1 The target object is an electronic device as an example for explanation, but the target object is not limited to electronic devices. In some embodiments, the target object may also be a non-electronic device, such as a baby, a pet, etc. In this case, the sound may be the sound of a baby crying or the sound of a pet barking.
[0063] In order to solve the above technical problems, the present application provides a prompting method, which includes: obtaining the initial spatial position of the target object 141 in the virtual space, and then determining the updated spatial position of the target object 141 based on the initial spatial position and the change in the user's head posture, and based on the updated spatial position, rendering the audio stream generated by the target object in the real space, and then using the rendered spatial audio stream for prompting.
[0064] It can be seen that in this method, when the user is immersed in the virtual space, the VR device can obtain the audio stream of the target object in the real space, and based on the initial spatial position of the target object in the real space, render the audio stream of the target object, and output it to the binaural playback device to prompt the user, thereby preventing the user from missing some sounds in the real space. In this way, the VR device can not only give the user an immersive experience, but also prevent the user from being completely out of the real space, thereby preventing the user from missing some important prompts in the real space, such as mobile phone ringing, doorbell, baby crying, etc.
[0065] The prompt method provided in the embodiment of the present application can be applied to XR devices. Figure 2 As shown, Figure 2 The schematic diagram of the structure of an XR device is shown. The XR device 200 includes a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, an antenna 1, a wireless communication module 260, an audio module 270, a speaker 270A, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, an indicator 292, a display screen 294, etc. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a touch sensor 280K, etc.
[0066] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the XR device. In other embodiments, the XR device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0067] The processor 210 may include one or more processing units, for example: the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. For example, in the present application, the processor can perform the following steps: obtain the initial spatial position of the target object 141 in the virtual space, and then determine the updated spatial position of the target object 141 based on the initial spatial position and the change in the user's head posture, and based on the updated spatial position, render the audio stream generated by the target object in the real space, and then use the rendered spatial audio stream for prompting.
[0068] The controller can be the nerve center and command center of the XR device. The controller can generate operation control signals according to the instruction opcode and timing signal to complete the control of fetching and executing instructions.
[0069] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.
[0070] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and / or a universal serial bus (USB) interface, etc.
[0071] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 210 may include multiple sets of I2C buses. The processor 210 may be coupled to the touch sensor 280K, the charger, the flash, etc. through different I2C bus interfaces. For example, the processor 210 may be coupled to the touch sensor 280K through the I2C interface, so that the processor 210 communicates with the touch sensor 280K through the I2C bus interface to realize the touch function of the XR device.
[0072] The I2S interface can be used for audio communication. In some embodiments, the processor 210 can include multiple groups of I2S buses. The processor 210 can be coupled to the audio module 270 via the I2S bus to achieve communication between the processor 210 and the audio module 270. In some embodiments, the audio module 270 can transmit an audio signal to the wireless communication module 260 via the I2S interface to achieve the function of listening to sound through a Bluetooth headset.
[0073] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 270 and the wireless communication module 260 can be coupled via a PCM bus interface. In some embodiments, the audio module 270 can also transmit audio signals to the wireless communication module 260 via the PCM interface to realize the function of listening to sound through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
[0074] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 210 and the wireless communication module 260. For example, the processor 210 communicates with the Bluetooth module in the wireless communication module 260 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 270 can transmit an audio signal to the wireless communication module 260 through the UART interface to implement the function of playing sound through a Bluetooth headset.
[0075] The MIPI interface can be used to connect the processor 210 to peripheral devices such as the display screen 294. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 210 and the display screen 294 communicate through the DSI interface to realize the display function of the XR device.
[0076] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 210 with the display screen 294, the wireless communication module 260, the audio module 270, the sensor module 280, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.
[0077] The USB interface 230 is an interface that complies with the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 230 can be used to connect a charger to charge the XR device, and can also be used to transfer data between the XR device and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as mobile phones.
[0078] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation on the XR device. In other embodiments of the present application, the XR device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0079] The wireless communication function of the XR device can be implemented through the antenna 1 wireless communication module 260, the modem processor and the baseband processor.
[0080] The antenna 1 is used to transmit and receive electromagnetic wave signals. The antenna in the XR device can be used to cover a single or multiple communication frequency bands.
[0081] The wireless communication module 260 can provide wireless communication solutions for applications on XR devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210. The wireless communication module 260 can also receive the signal to be sent from the processor 210, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through antenna 2.
[0082] In some embodiments, the antenna 1 of the XR device is coupled to the wireless communication module 260 so that the XR device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).
[0083] The XR device implements display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.
[0084] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the XR device may include 1 or N display screens 294, where N is a positive integer greater than 1.
[0085] A series of graphical user interfaces (GUIs) may be displayed on the display screen 294 of the XR device, and these GUIs are the main screens of the XR device.
[0086] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the XR device. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, files such as music and videos can be saved in the external memory card.
[0087] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the XR device by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the XR device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 210 executes various functional applications and data processing of the XR device by running the instructions stored in the internal memory 221, and / or the instructions stored in the memory provided in the processor.
[0088] The XR device can implement audio functions such as music playback and recording through the audio module 270, the speaker 270A, the microphone 270C, the headphone interface 270D, and the application processor.
[0089] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.
[0090] Speaker 270A, also called a "speaker", is used to convert audio electrical signals into sound signals. The XR device can listen to music through speaker 270A; in other examples, when the XR device is connected to headphones (Bluetooth headphones or wired headphones), audio is played through the headphones.
[0091] Microphone 270C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. The user can make a sound by approaching the microphone 270C with his mouth to input the sound signal into the microphone 270C. The XR device can be provided with at least one microphone 270C. In other embodiments, the XR device can be provided with two microphones 270C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the XR device can also be provided with three, four or more microphones 270C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording functions, etc. In an embodiment of the present application, the microphone 270C can be used to collect sound signals of the target object in the real space, and then the XR device can obtain the audio stream of the target object.
[0092] The earphone interface 270D is used to connect a wired earphone. The earphone interface 270D may be a USB interface 230, or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0093] The gyro sensor 280B can be used to determine the motion posture of the XR device. In some embodiments, the angular velocity of the XR device around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 280B. The gyro sensor 280B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyro sensor 280B detects the angle of the XR device shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the XR device through reverse motion to achieve anti-shake. The gyro sensor 280B can also be used for navigation and somatosensory game scenes. In some examples, the gyro sensor 280B can be used to detect the position parameter transformation of the XR device. For example, when the user wears the XR device, the XR device can detect the user's posture change (such as the change in spatial position) through the gyro sensor 280B.
[0094] The magnetic sensor 280D includes a Hall sensor. The XR device can use the magnetic sensor 280D to detect the opening and closing of the flip leather case. In some embodiments, when the XR device is a flip machine, the XR device can detect the opening and closing of the flip cover according to the magnetic sensor 280D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the features such as automatic unlocking of the flip cover are set.
[0095] The accelerometer 280E can detect the magnitude of the acceleration of the XR device in all directions (generally three axes). When the XR device is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the XR device and is applied to applications such as horizontal and vertical screen switching and pedometers.
[0096] Distance sensor 280F is used to measure distance. The XR device can measure distance through infrared or laser. In some embodiments, when shooting a scene, the XR device can use distance sensor 280F to measure distance to achieve fast focus.
[0097] The touch sensor 280K is also called a "touch device". The touch sensor 280K can be set on the display screen 294. The touch sensor 280K and the display screen 294 form a touch screen, also called a "touch screen". The touch sensor 280K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 294. In other embodiments, the touch sensor 280K can also be set on the surface of the XR device, which is different from the position of the display screen 294.
[0098] The button 290 includes a power button, a volume button, etc. The button 290 may be a mechanical button or a touch button. The XR device may receive the button input and generate a key signal input related to the user settings and function control of the XR device.
[0099] The indicator 292 may be an indicator light, which may be used to indicate the charging status, power change, etc.
[0100] like Figure 3 As shown, this figure is a structural schematic diagram of another XR device provided in an embodiment of the present application.
[0101] When the XR device executes the prompt method provided in the embodiment of the present application, its processor 210 includes: Figure 3 The initial spatial position acquisition module 310 and the sound signal spatialization processing module 320 shown in FIG. The sound signal spatialization processing module 320 includes a spatial position update module 321 , a sound signal receiving and managing module 322 , and a spatial audio rendering module 323 .
[0102] Among them, the initial spatial position acquisition module 310 is used to obtain the initial spatial position of the target object; the spatial position update module 321 is used to obtain the change in the spatial position (such as head posture) of the XR device worn by the user, and based on the change in the user's head posture and the initial spatial position of the target object, the spatial position of the target object is updated to obtain the updated spatial position of the target object. The sound signal receiving and management module 322 is used to receive the audio stream in the virtual space and the audio stream in the real space. The spatial audio rendering module 323 is used to render the audio stream in the virtual space and the audio stream in the real space to obtain a binaural audio stream and output it to the binaural playback device.
[0103] The following combination Figure 4 , the contents of the prompt method provided by the embodiment of the present application are introduced by cooperating with the initial spatial position acquisition module 310, the spatial position update module 321, the sound signal receiving and management module 322, and the spatial audio rendering module 323.
[0104] like Figure 4 As shown in FIG. 1 , this figure is a flowchart of a prompting method provided in an embodiment of the present application. The method can be executed by a VR device worn by a user. Of course, in other embodiments, the method can also be executed by an AR device worn by a user. For ease of understanding, the prompting method provided in an embodiment of the present application is introduced in combination with the above scenario. The method includes:
[0105] S401: The initial spatial position acquisition module 320 acquires the initial spatial position of the target object.
[0106] The initial spatial position of the target object refers to the spatial position of the target object in the real space 140, which can be represented by spherical coordinates. In some embodiments, after the VR device is started, the initial spatial position acquisition module 320 starts to collect the initial spatial position of the target object in the real space. The initial spatial position acquisition module 320 can estimate the initial spatial position of the target object by one or more methods such as ultrasonic technology, ultra-wideband (UWB) technology, wifi network, inertial measurement unit (IMU), etc., and then obtain the initial spatial position of the target object.
[0107] Taking the initial spatial position acquisition module 320 locating the initial spatial position of the target object through ultra-wideband technology as an example, the VR device can perform multiple unidirectional or bidirectional information exchanges with the target object based on the UWB communication protocol, and perform ranging and direction finding through the wireless signals used to exchange information to obtain the relative distance and relative azimuth between the VR device and the target object, that is, the initial spatial position of the target object.
[0108] For ease of understanding, the following Figure 5 The following scenario is used as an example. Figure 5 The scene shown is a living room scene. Of course, in other embodiments, the prompt method provided in the embodiment of the present application can also be applied to bedrooms, conference rooms, etc. Figure 5 In the scene shown, the user wears a VR device to watch a movie in the living room. After the user wears the VR device, he or she is in a virtual space, and the target objects in the real space (such as smart speakers 520, smart phones 510 and smart door locks 530) are connected through the network. Through this connection, information transmission and interaction can be achieved between the target object and the VR device, and the above target object is measured by the exchanged information and wireless information. The initial spatial position of the target object is obtained. Of course, based on the above connection, the target object can also transmit the sound prompts, forecasts, and warning signals emitted by the target object, as well as the initial spatial position of the pre-set target object.
[0109] Of course, when the target object is a non-electronic device, the initial spatial position of the non-electronic device can be recorded in the VR device in advance. For example, the target object can be a baby, and the position of the baby can be recorded in the VR device in advance, and then the initial spatial position acquisition module 320 can acquire the initial spatial position of the non-electronic device.
[0110] S402 , the initial spatial position acquisition module 310 transmits the initial spatial position of the target object to the spatial position update module 321 .
[0111] After the initial spatial position acquisition module 310 acquires the initial spatial position of the target object, it may transmit the initial spatial position of the target object to the spatial position update module 321 so that the spatial position update module 321 can update the spatial position of the target object in real time.
[0112] S403: The spatial position updating module 321 obtains the change in the spatial position of the VR device.
[0113] In some examples, the user can wear the VR device on the head, and when the user's head turns, the spatial position of the VR device changes accordingly. The VR device is equipped with a gyroscope sensor, an acceleration sensor, etc. When the user's head posture changes, that is, when the spatial position of the VR device changes, the spatial position update module 321 can obtain the change in the spatial position of the VR device, that is, the change parameter of the head posture, through the above gyroscope sensor, acceleration sensor, etc.
[0114] S404: The spatial position updating module 321 determines the updated spatial position of the target object according to the initial spatial position of the target object and the change in the spatial position of the VR device.
[0115] After the spatial position update module 321 determines the change in the spatial position of the VR device, it can update the spatial position of the target object based on the change in the spatial position of the VR device and the initial spatial position of the target object to obtain an updated spatial position of the target object.
[0116] like Fig. 6A As shown in FIG. 1 , this figure is a schematic diagram of an initial spatial position of a target object provided in an embodiment of the present application. In some examples, the initial spatial position of the target object can be represented by spherical coordinates, such as Fig. 6A As shown, the initial spatial position of the smart phone 510 is The initial spatial position of the smart speaker 520 is The initial spatial position of the smart door lock 530 is
[0117] like Figure 6B As shown in the figure, this figure is a schematic diagram of a VR device rotation provided by an embodiment of the present application. When the user's head rotates, the spatial position update module 321 device can detect the change parameters of the user's head posture using a gyroscope sensor, for example, the change parameters of the user's head posture are The spatial position updating module 321 can be based on the change parameters of the head posture And the initial spatial position of the target object, the updated spatial position of the target object is obtained. For example, the updated spatial position of the target object is calculated by the following formula:
[0118]
[0119]
[0120]
[0121] Among them, in formula (1) is the updated spatial position of the smartphone 510, in formula (2): is the updated spatial position of the smart speaker 520, in formula (3): It is the updated spatial position of the smart door lock 530.
[0122] It should be noted that the above representation of spatial parameters by spherical coordinates is only an example. In other implementations, the spatial position may also be represented by other forms of data, such as the direction and distance of the target object.
[0123] S405 . The spatial position updating module 321 transmits the updated spatial position of the target object to the spatial audio rendering module 323 .
[0124] After the spatial position update module 321 determines the real-time spatial position of the target object (i.e., the updated spatial position), the real-time spatial position is transmitted to the spatial audio rendering module 323, so that the spatial audio rendering module 323 renders the audio stream of the target object based on the real-time spatial position and outputs it to the binaural playback device.
[0125] S406: The sound signal receiving and managing module 322 obtains the audio stream in the virtual space and the audio stream of the target object in the real space.
[0126] The audio stream in the virtual space may be a superposition of the sounds generated by virtual sound source 1, virtual sound source 2, ... virtual sound source N in the virtual space constructed by the VR device. The audio stream in the real space may be a combination of real sound source 1, real sound source 2, ... real sound source M in the real space. The real sound source may be formed by the sound generated by the target object in the real space. In some examples, the audio stream in the real space may be used to remind the user of some important events in the real space, such as important meetings, crying babies, etc.
[0127] In some embodiments, the sound signal receiving and management module 322 may collect audio streams in the real space through a microphone. For example, the sound signal receiving and management module 322 may collect sound signals of a target object in the real space through a microphone to obtain the audio stream of the target object.
[0128] The target object can be a non-electronic device or an electronic device. When the target object is a non-electronic device, for example, the target object can be a baby, and the sound signal receiving and management module 322 can collect the audio stream generated by the baby in the real space through a microphone, such as the baby's crying and shouting; when the target object is an electronic device, for example, the target object can be a smart phone, and the sound signal receiving and management module 322 can collect the audio stream played by the smart phone in the real space through a microphone, such as an incoming call ringtone, an alarm ringtone, etc.
[0129] In other embodiments, when the target object is an electronic device, the electronic device can establish a VR device. Figure 5 The connection shown (for example, it can be in the same local area network, or it can be connected via Bluetooth or other means). The sound signal receiving and management module 322 can receive the audio stream transmitted by the electronic device and played in the real space. Taking the electronic device as a smart phone as an example, the audio stream in the real space can be the incoming ringtone or alarm ringtone of the smart phone, etc. The sound signal receiving and management module 322 can receive the incoming ringtone or alarm ringtone transmitted by the smart phone through the above connection, and then obtain the audio stream played by the smart phone in the real space.
[0130] In other embodiments, the sound signal receiving and management module 322 can also receive an audio prompt signal transmitted by an electronic device. Taking the electronic device as a smart phone and the audio stream in the real space as an incoming call ringtone as an example, when the mobile phone receives an incoming call, the sound signal receiving and management module 322 can receive the audio prompt signal transmitted by the smart phone. The audio prompt signal is used to prompt the VR device that there is an incoming call on the smart phone in the real space. At this time, the VR device can obtain the audio stream corresponding to the audio prompt signal from the correspondence between the audio prompt signal and the audio stream. For example, audio prompt signal 1 corresponds to the audio stream of the incoming call ringtone, and audio prompt signal 2 corresponds to the audio stream of the alarm ringtone. After the sound signal receiving and management module 322 receives the audio prompt signal 1, it can obtain the audio stream of the incoming call ringtone corresponding to the audio prompt signal 1 from the correspondence between the audio prompt signal and the audio stream.
[0131] The above is merely an introduction to an example of a VR device obtaining an audio stream in a real space in an embodiment of the present application. Those skilled in the art may select an appropriate method based on actual needs to enable a VR device to obtain an audio stream in a real space.
[0132] After the sound signal receiving and management module 322 obtains the audio stream of the target object in the real space, it can fill the audio stream of the target object in the real space into the container corresponding to the updated spatial position of the target object. Among them, the updated spatial position of the target object is different, and the container corresponding to the updated spatial position of the target object is different. In some examples, the container can be a buffer, and different caches correspond to different addresses. For example, there are target objects 1, 2, 3, and 4 in the real space. Among them, target object 1 can be a smart phone, and the corresponding container is C1; target object 2 can be a smart door lock, and the corresponding container is C2; target object 3 can be a smart speaker, and the corresponding container is C3; target object 4 can be a baby, and the corresponding container is C4. In this way, the spatial audio rendering module 323 processes the audio stream in the real space to obtain an audio stream based on channel coding.
[0133] S407 : The sound signal receiving and managing module 322 determines the category of the audio stream of the target object.
[0134] The sound signal receiving and management module 322 can also classify audio streams in the real space. In some embodiments, the categories of audio streams can be predefined. For example, the audio streams can be divided into three categories: primary category, secondary category and tertiary category. Different levels of audio streams can reflect sounds of different importance, such as the audio streams in the primary category are the most important, the audio streams in the secondary category are important, and the audio streams in the tertiary category are not important. For example, the primary category may include alarm sounds, important meeting forecast sounds, baby crying sounds, etc.; the secondary category may include mobile phone prompt sounds (such as incoming call ringtones, alarm clock ringtones, etc.), doorbell sounds, smart speaker weather broadcast sounds, etc.; the tertiary category may include TV program sounds, smart speaker playing music sounds, etc.
[0135] Based on this, after the acoustic signal receiving and management module 322 obtains the audio stream of the real space, it can classify the audio stream to obtain the category of the audio stream. In some examples, the acoustic signal receiving and management module 322 can classify the audio stream in the real space based on the acoustic signal characteristics of the audio stream in the real space, and then obtain the category of the audio stream in the real space. Among them, the acoustic signal characteristics can be Mel-Frequency Cepstral Coefficients (MFCC) or Mel-energy. After the acoustic signal receiving and management module 322 obtains the acoustic signal characteristics of the above audio stream, it can process the acoustic signal characteristics of the above audio stream based on machine learning or deep learning methods such as Support Vector Machines (SVM), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Deep Neural Networks (DNN) to obtain the category of the audio stream.
[0136] S408 : The sound signal receiving and managing module 322 transmits the audio stream of the preset category and the audio stream in the virtual space to the spatial audio rendering module 323 .
[0137] See also Figure 7 , which is a schematic diagram of a VR device processing an audio stream provided by an embodiment of the present application. The VR device further includes a display module 340, and the display module 330 is used to display prompt information corresponding to the category of the audio stream to the user when the category of the audio stream is a preset category. The display module 330 will be introduced later.
[0138] In some examples, the preset category may be a category set by the user. Before the user experiences the immersive feeling of the virtual world through the VR device, the VR device may receive the user's setting of the preset category. For example, the user may set the above-mentioned primary category and secondary category as preset categories. In other examples, the preset category may also be a default category, such as the default primary category is a preset category.
[0139] When the sound signal receiving and managing module 322 determines that the category of the audio stream in the real space is a preset category, the audio stream in the virtual space and the audio stream of the preset category in the real space can be transmitted to the spatial audio rendering module 323. For example, the preset categories are a primary category and a secondary category, and the sound signal receiving and managing module 322 can transmit the audio stream of the primary category and the audio stream of the secondary category to the spatial audio rendering module 323, so that the spatial audio rendering module 323 renders the audio stream of the primary category and the audio stream of the secondary category in the real space.
[0140] It should be noted that the embodiment of the present application does not specifically limit the execution order of the above S401 to S405 and S406 to S408. S401 to S405 and S406 to S408 can be executed simultaneously or sequentially, such as executing S406 to S408 first and then executing S401 to S405.
[0141] S409: The spatial audio rendering module 323 renders the audio stream in the virtual space and renders the audio stream of the preset category based on the updated spatial position to obtain a binaural audio stream.
[0142] The spatial audio rendering module 323 is used to render the audio stream in the virtual space and the audio stream in the real space to obtain a binaural audio stream and output it to the binaural playback device. There are two types of audio streams in the virtual space: the first type is an audio stream based on sound object coding; the second type is an audio stream based on channel coding.
[0143] In the case where the audio stream in the virtual space is the first type, the spatial audio rendering module 323 needs to convert the audio stream based on sound object coding into an audio stream based on channel coding. Among them, the channel can be each speaker of the speaker system, and the spatial audio rendering module 323 projects the audio stream based on sound object coding on the speaker system to obtain an audio stream based on channel coding. In the case where the audio stream in the virtual space is the second type, the spatial audio rendering module 323 can directly map the audio stream based on channel coding to the corresponding channel.
[0144] It should be noted that the above-mentioned channel-coded audio streams include but are not limited to stereo, 5.1, 7.1, 5.1.4, 7.1.4 and other multi-channel audio streams.
[0145] Generally, the audio stream in the real space received by the sound signal receiving and managing module 322 is an audio stream based on the sound object. Similarly, the spatial audio rendering module 323 needs to convert the audio stream in the real space into an audio stream based on channel coding, and the processing method is similar to the processing method of the audio stream in the virtual space, which will not be repeated here.
[0146] After completing the conversion of the audio streams in the virtual space and the real space, the spatial audio rendering module 323 can directly use the audio stream based on channel coding to complete the spatial audio rendering, and obtain a binaural audio stream including each sound source in the virtual space and the real space. In this way, when the user is immersed in the virtual space, the VR device can provide the user with audio streams of some important events in the real space.
[0147] Next, during the rendering process of the audio stream in the real space, the spatial audio rendering module 323 renders the audio stream of the target object in the real space based on the updated space of the target object, thereby obtaining a binaural audio stream. The binaural audio stream can not only prompt the user of important events, but also enable the user to feel the direction of the audio stream, thereby improving the user experience.
[0148] S410. The spatial audio rendering module 323 outputs the binaural audio stream to a binaural playback device.
[0149] A binaural playback device refers to a device used for performing sound interaction with a user, for example, the binaural playback device may be a headset.
[0150] After the spatial audio rendering module 323 outputs the binaural audio stream to the binaural playback device, the binaural playback device can play the audio stream in the real space while playing the audio stream in the virtual space. In this way, even when the user is immersed in the virtual space rendered by the VR device, he can still feel the prompts of important information in the real space.
[0151] In some embodiments, the acoustic signal receiving and managing module 322 may also transmit the category of the target object's audio stream to the display module 330. After receiving the category of the target object's audio stream, the display module 330 may display prompt information to the user in different display modes based on different categories. For example, the preset categories include primary categories and secondary categories, and the display modes corresponding to the primary categories and secondary categories may be preset.
[0152] For ease of understanding, the following is a schematic diagram of the virtual world screen when no prompt information is displayed. Fig. 8A As shown in FIG. 3 , this figure is a schematic diagram of a virtual world screen provided by an embodiment of the present application. During the user's experience of the VR device, the VR device blocks the user's line of sight through a head-mounted display, creating a complete virtual visual environment. The display module 330 is also used to present the user with a virtual world screen through the head-mounted display (such as a display screen), thereby providing the user with an immersive experience. Fig. 8A The virtual world shown does not include any prompt information. Fig. 8A The screen shots shown are for illustrative purposes only.
[0153] In some examples, the display method of the first-level category can be to present prompt information on the screen, which includes prompt text (such as an important event that needs to be handled) and prompt image (such as a danger sign), and the screen flashes during the display of the prompt text and prompt image, such as a red screen flashing. Figure 8B As shown, this figure is a schematic diagram of a prompt information provided by an embodiment of the present application. It can be seen that the display module 330 controls the display screen to flash red while presenting the prompt information, which can more clearly inform the user that there are important events in the real space that need to be handled.
[0154] In some examples, the secondary category can be displayed by presenting prompt information in the screen, which includes prompt text (such as there is a common event that needs to be handled) and prompt image (such as the image corresponding to the common event). For example, when the audio stream in the real space is a doorbell, the image of the prompt information can be an image of a "door". Figure 8C As shown, this figure is a schematic diagram of another prompt information provided by an embodiment of the present application. The display module 330 prompts common events in the real space by presenting prompt information, which can not only ensure the user's immersion, but also prevent the user from missing some events in the real space.
[0155] It can be seen that the display module 330 can select different display methods to display prompt information to the user based on the category of the audio stream, so as to remind the user that there are some events in the real space that need to be handled, thereby improving the user's safety level and virtual interaction ability when immersed in the virtual space.
[0156] It should be noted that the above Figure 8B and Figure 8C This is just an example of the prompt information. The embodiment of the present application is introduced by assuming that the VR device only prompts the audio streams of the first and second categories, so as to better ensure the user's immersion while preventing the user from missing some important events in the real space. Of course, in other embodiments, the VR device can also prompt the audio streams of the third category, which will not be repeated here.
[0157] Based on the above description, an embodiment of the present application provides a prompting method, which can be executed by an XR device. XR can first obtain the initial spatial position of the target object in the real space, and obtain the updated spatial position of the target object based on the change parameters of the user's head posture collected by the XR device, and then based on the updated spatial position of the target object, perform spatialization and immersive rendering on the audio stream of the target object, and output it to the binaural playback device. In this way, even if the user is immersed in the virtual space, the user can feel the sound generated by the target object in the real space, and can also perceive the direction of the sound of the target object.
[0158] Furthermore, the XR device can also classify the audio stream of the target object. For example, the audio stream can be classified based on the acoustic signal characteristics of the audio stream to obtain the category of the audio stream of the target object. When the category of the audio stream of the target object is a preset category, the XR device prompts the user. For example, it can play the audio stream in the real space to the user, or it can display prompt information to the user, so that the user can not only immerse himself in the virtual space, but also avoid missing some important events in the real space.
[0159] For ease of understanding, the following Fig. 9 , from the perspective of XR devices, the prompt method provided in the embodiment of the present application is briefly introduced. Fig. 9 As shown in FIG. 1 , the figure is a flowchart of a prompt method provided in an embodiment of the present application. The method can be executed by an XR device, and the XR device can include the above-mentioned VR device and AR device. Specifically, the method includes:
[0160] S901: The XR device obtains the initial spatial position of the target object in the real space.
[0161] The XR device can obtain the initial spatial position of the target object through the above-mentioned initial spatial position acquisition module 310, which will not be repeated here.
[0162] S902: The XR device determines an updated spatial position of the target object based on the initial spatial position and the change in the spatial position of the extended reality device.
[0163] The process of the XR device determining the updated spatial position of the target object can be referred to the above-mentioned spatial position updating module 321, which will not be repeated here.
[0164] S903: The XR device renders the audio stream of the target object in the real space based on the updated spatial position to obtain a real space audio stream.
[0165] The process of the XR device rendering the audio stream of the target object can be referred to the above-mentioned spatial audio rendering module 323, which will not be repeated here.
[0166] S904: The XR device provides a prompt through the real space audio stream.
[0167] The XR device plays the above-mentioned real space audio stream (i.e., binaural audio stream) through a binaural playback device such as headphones to prompt the user.
[0168] In some embodiments, the XR device may also obtain the category of the audio stream of the target object in the real space; when the category of the audio stream of the target object in the real space is a preset category, the audio stream of the target object in the real space is rendered based on the updated spatial position.
[0169] In some embodiments, the XR device determines the prompt information corresponding to the category of the audio stream of the target object according to the correspondence between the category of the audio stream of the target object and the prompt information, and displays the prompt information corresponding to the category of the audio stream of the target object to the user. The prompt information may include text, images, or a combination of text and images.
[0170] In some embodiments, when the category of the audio stream of the target object is the highest level among the preset categories, the XR device may display prompt information corresponding to the category of the audio stream of the target object by flashing the screen.
[0171] In some embodiments, the audio stream of the target object in the real space is obtained by collecting the audio stream of the target object or by receiving the audio stream transmitted by the target object.
[0172] The present application embodiment provides an extended reality device, such as Fig.10 As shown, the extended reality device includes:
[0173] An acquisition unit 1001 is used to acquire an initial spatial position of a target object in a real space;
[0174] An updating unit 1002, configured to determine an updated spatial position of the target object according to the initial spatial position and a change in the spatial position of the extended reality device;
[0175] A rendering unit 1003, configured to render the audio stream of the target object in the real space based on the updated spatial position to obtain a real space audio stream;
[0176] The prompting unit 1004 is used to provide prompts through the real space audio stream.
[0177] In some possible implementations, the acquisition unit 1001 is further configured to acquire a category of an audio stream of a target object in the real space;
[0178] The rendering unit 1003 is specifically configured to render the audio stream of the target object in the real space based on the updated spatial position when the category of the audio stream of the target object in the real space is a preset category.
[0179] In some possible implementations, the prompt unit 1004 is further used to determine the prompt information corresponding to the category of the audio stream of the target object based on the correspondence between the category of the audio stream of the target object and the prompt information, and display the prompt information corresponding to the category of the audio stream of the target object to the user.
[0180] In some possible implementations, the prompt information includes text and / or images.
[0181] In some possible implementations, when the category of the audio stream of the target object is the highest level among the preset categories, the prompt unit 1004 is specifically configured to display prompt information corresponding to the category of the audio stream of the target object by flashing the screen.
[0182] In some possible implementations, the audio stream of the target object in the real space is obtained by collecting the audio stream of the target object or by receiving the audio stream transmitted by the target object.
[0183] In some possible implementations, the acquisition unit 1001 is further configured to acquire a virtual space audio stream;
[0184] The prompting unit 1004 is specifically configured to provide prompting through the real space audio stream and the virtual space audio stream.
[0185] The present application also provides an extended reality device, such as Fig.11 As shown, the extended reality device may include: a display screen 1110, one or more processors 1120, a memory 1130, and one or more computer programs 1140. The above components may be connected via one or more communication buses 1150. The one or more computer programs 1140 are stored in the above memory 1130 and are configured to be executed by the one or more processors 1120. The one or more computer programs 1140 include instructions, which may be used to execute the following steps: Figure 4 or Fig. 9 The display screen 1110 may be used to display prompt information, etc.
[0186] This embodiment also provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device executes Figure 4 , Fig. 9 The relevant method steps in the embodiment are used to implement the method in the above embodiment.
[0187] This embodiment also provides a computer program product including instructions. When the computer program product is run on an electronic device, the electronic device executes the following Figure 4 , Fig. 9 The relevant method steps in the embodiment are used to implement the method in the above embodiment.
[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the extended reality device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the extended reality device and unit described above can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0189] In the several embodiments provided in this embodiment, it should be understood that the disclosed extended reality device and method can be implemented in other ways. For example, the extended reality device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0190] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0191] In addition, each functional unit in each embodiment of the present embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0192] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk and other media that can store program codes.
[0193] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A prompting method, characterized in that: Applied to an extended reality device, the method comprises: Acquire an initial spatial position of a target object in real space and acquire a category of an audio stream of the target object, wherein the initial spatial position is a position of the target object in real space relative to the extended reality device; Determine, according to the initial spatial position and the amount of change in the spatial position of the extended reality device, an updated spatial position of the target object, wherein the updated spatial position is the position of the target object after the position of the target object in the real space is updated relative to the extended reality device; Based on the updated spatial position, rendering the audio stream of the target object in the real space to obtain a real space audio stream; While playing the audio stream in the virtual space, playing the audio stream in the real space; When the category of the audio stream belongs to a preset category, prompt information corresponding to the category of the audio stream is determined, and the prompt information is presented in a screen of the virtual space, wherein the prompt information includes prompt text and a prompt image.
2. The method according to claim 1, characterized in that The rendering of the audio stream of the target object in the real space based on the updated spatial position includes: When the category of the audio stream of the target object in the real space is a preset category, the audio stream of the target object in the real space is rendered based on the updated spatial position.
3. The method according to claim 1, characterized in that The preset categories include a first category and a second category. The first category includes: alarm sounds, meeting forecast sounds, and baby crying sounds; the second category includes: mobile phone reminder sounds, doorbell sounds, and smart speaker weather broadcast sounds.
4. The method according to claim 3, characterized in that The prompt information corresponding to the first category includes first category prompt text and first category prompt image, and the prompt information corresponding to the second category includes second category prompt text and second category prompt image; the importance of the audio stream indicated by the first category prompt text is higher than the importance of the audio stream indicated by the second category prompt text, and the importance of the audio stream indicated by the first category prompt image is higher than the importance of the audio stream indicated by the second category prompt image.
5. The method according to claim 3 or 4, characterized in that: When the category of the audio stream of the target object is the first category among the preset categories, presenting the prompt information in the screen of the virtual space includes: When displaying the prompt information corresponding to the category of the audio stream of the target object, the display screen of the extended reality device is controlled to flash.
6. The method according to claim 1, characterized in that The audio stream of the target object in the real space is obtained by collecting the audio stream of the target object or by receiving the audio stream transmitted by the target object.
7. An extended reality device, characterized in that: include: an acquisition unit, configured to acquire an initial spatial position of a target object in a real space and to acquire a category of an audio stream of the target object, wherein the initial spatial position is a position of the target object in the real space relative to the extended reality device; An updating unit, configured to determine an updated spatial position of the target object according to the initial spatial position and the amount of change in the spatial position of the extended reality device, wherein the updated spatial position is the position of the target object after the position of the target object in the real space is updated relative to the extended reality device; A rendering unit, configured to render the audio stream of the target object in the real space based on the updated spatial position to obtain a real space audio stream; A prompting unit, configured to play the audio stream of the real space while playing the audio stream of the virtual space; When the category of the audio stream belongs to a preset category, prompt information corresponding to the category of the audio stream is determined, and the prompt information is presented in a screen of the virtual space, wherein the prompt information includes at least one of a prompt text and a prompt image.
8. An extended reality device, characterized in that: include: Display, processor and memory; The display screen is used to display prompt information; The memory is used to store computer programs; The processor is used to execute the computer program to implement the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Voice monitoring method and device for virtual reality equipment and virtual reality equipment
CN106325518A
Incoming call reminding method and virtual reality device
CN108243275A
Content discovery
CN109791436A
Application of geometric acoustics for immersive virtual reality (VR)
CN111158459A
Information prompting method and device, terminal equipment and computer readable storage medium
CN111198608A
