Extended reality information notification method, device, electronic device and storage medium

Abstract

The embodiment of the present application discloses an extended reality information notification method, device, electronic device and storage medium, the method comprising: in response to receiving notification information, the extended reality device determines the spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device; generates notification audio for the spatial display position; displays the notification information at the spatial display position in the three-dimensional display space and outputs the notification audio. By displaying the notification information in the three-dimensional display space of the extended reality device and using the notification audio to provide information notification reminders, an audio notification based on the three-dimensional display space is implemented, helping users to quickly perceive positioning information from the three-dimensional display space of the extended reality device, thereby improving user experience.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of extended reality technology, and specifically to an extended reality information notification method, device, electronic device, and storage medium. Background Art

[0002] Extended Reality (XR) is a comprehensive term that covers virtual reality (VR), augmented reality (AR) and mixed reality (MR). Extended reality technology expands human perception and interaction methods, interacts with the real world through digital technology, and creates virtual, augmented, and mixed reality experiences. Extended reality provides a more intuitive and convenient content presentation experience through virtual reality scenes or extended reality scenes. It has developed rapidly in the fields of entertainment, games, education, medical care, corporate production, and social communication, showing great potential and influence. Extended reality devices represented by wearable smart glasses are considered to be the best carriers of "XR+AI" technology. They have been widely used in industrial inspections, education and training, marketing displays and other fields, and are gradually moving towards personal consumer electronics products, and are expected to replace mobile phones as the next generation of mobile communication terminals for humans.

[0003] Similar to mobile phones, today's extended reality devices integrate multiple functions such as communication, office, consumption, and entertainment. Unlike mobile phones, extended reality devices can simultaneously display various information on a huge virtual display screen that is integrated with the real field of view, and more information can be displayed based on the size of the virtual display screen. The virtual display screen can also be three-dimensional, for example, displaying information in a three-dimensional display space in all directions of the user, increasing the amount of information that can be displayed and the diversity of information display styles.

[0004] However, when there is a variety of information in the field of vision, it is more time-consuming and laborious for users wearing extended reality devices to discover and process the information. It is not easy for users to quickly locate the information they want to process in the three-dimensional display space, especially for newly received or updated information. It is difficult for users to perceive the information in the first time, which is also a cognitive burden for users, thereby affecting the user experience of the extended reality device. Summary of the invention

[0005] The embodiments of the present application provide an extended reality information notification method, device, electronic device and storage medium, which can help users quickly perceive positioning information from the three-dimensional display space of the extended reality device and improve the user experience.

[0006] The present application provides an extended reality information notification method, which is applied to an extended reality device. The method includes:

[0007] In response to receiving the notification information, determining a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device;

[0008] generating a notification audio for the spatial display location;

[0009] The notification information is displayed at the spatial display position in the three-dimensional display space and the notification audio is output.

[0010] Accordingly, an embodiment of the present application further provides an extended reality information notification device, which is applied to an extended reality device, and the device includes:

[0011] A response module, configured to determine, in response to receiving the notification information, a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device;

[0012] A generating module, used to generate a notification audio for the spatial display position;

[0013] A notification module is used to display the notification information at the spatial display position of the three-dimensional display space and output the notification audio.

[0014] Optionally, in some embodiments of the present application, the virtual display screen presented in the three-dimensional display space is in a hovering state, and the notification information is presented in the virtual display screen;

[0015] After displaying the notification information at the spatial display position of the three-dimensional display space and outputting the notification audio, the method further includes:

[0016] In response to detecting the behavior information of the target user, identifying a control instruction corresponding to the behavior information;

[0017] Processing the notification information in the virtual display screen in a hovering state according to the control instruction to obtain an information processing result;

[0018] The behavior information includes at least one of body rotation, gesture movement, ear deflection, ear closeness, voice or eye movement.

[0019] Optionally, in some embodiments of the present application, in response to receiving the notification information, determining the spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device includes:

[0020] In response to receiving the notification information, determining attribute information corresponding to the notification information;

[0021] Determine, according to the attribute information, a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device;

[0022] The attribute information includes at least one of the type information, receiving time information, title feature information, number of items, or sender identity information of the notification information.

[0023] Optionally, in some embodiments of the present application, determining, according to the attribute information, a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device includes:

[0024] If the attribute information includes the sender identity information corresponding to the notification information, determining the interpersonal relationship level information corresponding to the sender identity information;

[0025] Determining a spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device according to the interpersonal relationship level information;

[0026] The generating of the notification audio for the spatial display position includes:

[0027] A notification audio for the spatial display position is generated according to the interpersonal relationship level information.

[0028] Optionally, in some embodiments of the present application, generating a notification audio for the spatial display position includes:

[0029] Determining the position information of the spatial display position relative to the extended reality device;

[0030] Determine the sound type information according to the attribute information of the notification information;

[0031] A spatial audio is generated according to the position information and the sound type information, and the spatial audio is used as the notification audio.

[0032] Optionally, in some embodiments of the present application, displaying the notification information at the spatial display position in the three-dimensional display space and outputting the notification audio includes:

[0033] Determine the background card type and card color tone information according to the attribute information of the notification information;

[0034] Generating a background card for the background card type and the card color tone information at the spatial display position of the three-dimensional display space;

[0035] The notification information is displayed in the background card, and the notification audio is output.

[0036] Optionally, in some embodiments of the present application, in response to receiving the notification information, determining the spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device includes:

[0037] In response to receiving the notification information, determining an initial display position of the notification information in a three-dimensional display space corresponding to the extended reality device;

[0038] If it is detected that there is a real scene object in the gaze direction corresponding to the initial display position, a spatial display position other than the initial display position is determined from the three-dimensional display space.

[0039] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned method for notifying extended reality information when executing the computer program.

[0040] In a fourth aspect, an embodiment of the present application further provides a storage medium, which includes a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned extended reality information notification method are implemented.

[0041] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0042] In response to receiving notification information, the extended reality device of an embodiment of the present application determines the spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device, generates notification audio for the spatial display position, displays the notification information at the spatial display position in the three-dimensional display space, and outputs the notification audio.

[0043] Among them, by outputting notification audio, the notification audio is used to notify the user of information, and the notification audio is determined based on the spatial display position of the notification information, so that the user can quickly perceive the location of the notification information through the notification audio.

[0044] Among them, by displaying the notification information in the three-dimensional display space of the extended reality device and using the notification audio for information notification reminder, an audio notification based on the three-dimensional display space is realized, helping users to quickly perceive positioning information from the three-dimensional display space of the extended reality device and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 It is a schematic diagram of a scenario in which an extended reality device provided in an embodiment of the present application executes the extended reality information notification method;

[0047] Figure 2 It is a flowchart of the extended reality information notification method provided in an embodiment of the present application;

[0048] Figure 3 is a schematic diagram of a three-dimensional display space provided in an embodiment of the present application;

[0049] Figure 4 It is a schematic diagram of the relationship between notification information of different applications and spatial display positions provided in an embodiment of the present application;

[0050] Figure 5 is a schematic diagram of the relationship between different interpersonal relationship levels and spatial display positions provided in an embodiment of the present application;

[0051] Figure 6 This is a first effect diagram of the notification information display provided by the embodiment of the present application;

[0052] Figure 7 This is a second effect diagram of the notification information display provided by the embodiment of the present application;

[0053] Figure 8 This is a third effect diagram of the notification information display provided in the embodiment of the present application;

[0054] Fig. 9 is a schematic diagram of the structure of an extended reality information notification device provided in an embodiment of the present application;

[0055] Fig.10 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0056] Description of Figure Numbers:

[0057] 10-extended reality device; 201-response module; 202-generation module; 203-notification module; 301-processor; 302-memory; 303-power supply; 304-input unit. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0059] The embodiments of the present application provide an extended reality information notification method, device, electronic device and computer-readable storage medium. Specifically, the embodiments of the present application provide an extended reality information notification device suitable for an electronic device, the electronic device including an extended reality device, the extended reality device including but not limited to a head-mounted display, wearable glasses, and the extended reality device can also be an integrated extended reality device built into a computing processing unit, or a split extended reality device external to the computing processing unit. Among them, the extended reality device includes but is not limited to an airborne optical display system, i.e., a head-up display system, applied to vehicles such as aircraft, automobiles, and ships, for example, an AR-HUD (Augmented reality hud-up display) mounted on an intelligent networked car, an extended reality application applied to handheld mobile devices such as mobile phones, laptops, and tablets (such as extended reality games, virtual tourism, telemedicine, or virtual experiments, etc.), a near-eye display system applied to wearable devices such as head-mounted displays and smart glasses, etc.

[0060] See also Figure 1 , Figure 1 is a schematic diagram of a scenario in which an extended reality device according to an embodiment of the present application performs the extended reality information notification method, such as Figure 1 Taking the extended reality device as optical transmission wearable smart AR glasses as an example, the specific execution process of the extended reality device executing the extended reality information notification method is as follows:

[0061] In response to receiving the notification information, the extended reality device 10 determines the spatial display position of the notification information from the three-dimensional display space, generates notification audio for the spatial display position, displays the notification information at the spatial display position in the three-dimensional display space, and outputs the notification audio.

[0062] Among them, in the embodiment of the present application, by outputting notification audio, information notification reminders are provided to the user by using the notification audio, and the notification audio is determined based on the spatial display position of the notification information, so that the user can quickly perceive the location of the notification information through the notification audio.

[0063] Among them, by displaying the notification information in the three-dimensional display space of the extended reality device and using the notification audio for information notification reminder, an audio notification based on the three-dimensional display space is realized, helping users to quickly perceive positioning information from the three-dimensional display space of the extended reality device and improve user experience.

[0064] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0065] See also Figure 2 , Figure 2 1 is a flowchart of an extended reality information notification method provided by an embodiment of the present application. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings. Specifically, the extended reality information notification method is applied to an extended reality device, and the specific process of the extended reality device executing the extended reality information notification method is as follows:

[0066] 101. In response to receiving notification information, determine a spatial display position of the notification information in a three-dimensional display space corresponding to the extended reality device.

[0067] Among them, the notification information can be information or messages received through an application on the extended reality device, for example, an incoming call, SMS message, WeChat message, shopping message or game upgrade message is received through a call application, SMS application, WeChat application, consumer application or entertainment application on the extended reality device.

[0068] The three-dimensional display space is the space in which the extended reality device can project output information, for example, see Figure 3 As shown, the display space is 180 degrees, 360 degrees or 720 degrees around the extended reality device. For example, after the user wears the extended reality device, the information is displayed in the space in front of the user, or the information is displayed on the side and back of the user, or the information is displayed in the space above the user. Among them, the spatial display position refers to the display position of the notification information in the three-dimensional display space.

[0069] 102. Generate a notification audio for the spatial display location.

[0070] Among them, the notification audio is targeted at the spatial display position, that is, the notification audio is output by simulating the sound source of the spatial display position, so that the user can perceive the notification information by locating the location source of the notification audio.

[0071] For example, multiple small speakers are set up in the extended reality device to achieve multi-channel audio output. The sounds of these channels will have different delays, volumes and frequency characteristics, thereby simulating the effect of sound coming from different directions. Alternatively, the notification audio can be generated through the head-related transfer function (HRTF). HRTF can be used to simulate the changes in sound effects when sound reaches the human ear from different directions. Since the human body structure such as the head, auricle, and ear canal have an impact on the propagation and reception of sound, HRTF can calculate the changes in sound from different directions when it reaches the human ear based on these factors, including sound delay, attenuation, filtering, etc. By simulating these parameters, the direction and distance of the sound can be accurately located and tracked, providing users with a more immersive experience. Alternatively, spatial audio technology can be used to enable users to perceive sounds from specific sound directions, which are not limited here.

[0072] 103. Display the notification information at the spatial display position in the three-dimensional display space and output the notification audio.

[0073] Among them, by displaying the notification information at the spatial display position and outputting the notification audio for the spatial display position, it is convenient for the user to perceive the display position of the notification information through the notification audio, and it is convenient for the user to focus his observation attention on the notification information at the first time.

[0074] In summary, in the embodiment of the present application, by outputting a notification audio, the user is notified of information using the notification audio, and the notification audio is determined based on the spatial display position of the notification information, so that the user can quickly perceive the location of the notification information through the notification audio. Among them, by displaying the notification information in the three-dimensional display space of the extended reality device and using the notification audio to notify the user of information, an audio notification based on the three-dimensional display space is realized, which helps the user quickly perceive the positioning information from the three-dimensional display space of the extended reality device and improves the user experience.

[0075] Among them, since the notification information can be displayed in the three-dimensional display space, the position where the notification information is displayed is not always maintained in front of the user. In order to remind the user through the notification audio and facilitate the user to turn the body to adjust the attention line of sight and focus on the notification information, in the embodiment of the present application, the virtual display screen presented in the three-dimensional display space of the extended reality device can be set to a hovering state, wherein the hovering state refers to the content in the virtual display screen "floating" or "hovering" at a certain position in the real world space, that is, the position of the virtual display screen in the real field of view is fixed and will not change with the change of the user's head posture, as if they are real objects. This technology provides users with a more immersive experience. For example, after projecting the virtual display screen, by detecting the deflection angle of the extended reality device, the virtual display screen is kept "floating" in the original position through projection compensation. Correspondingly, in the embodiment of the present application, the notification information is displayed in the hovering virtual display screen presented in the three-dimensional display space.

[0076] Further, after the notification information is displayed in the three-dimensional display space and the notification audio is output, the user's behavior action can be detected, and the processing of the notification information can be responded to based on the behavior action. That is, optionally, in some embodiments of the present application, after the step of "displaying the notification information at the spatial display position of the three-dimensional display space and outputting the notification audio", the method further includes:

[0077] In response to detecting the behavior information of the target user, identifying a control instruction corresponding to the behavior information;

[0078] Processing the notification information in the virtual display screen in a hovering state according to the control instruction to obtain an information processing result;

[0079] The behavior information includes at least one of body rotation, gesture movement, ear deflection, ear closeness, voice or eye movement.

[0080] For example, if it is detected that the user's body is turning or the ear is tilted, the notification information will be highlighted, for example, a border animation will be added to the notification information so that the notification information is in a focused state at this time; if it is detected that the target user's finger is pointing at the notification information, or the ear is close to the direction of the notification information, the broadcast of the notification information can be triggered; if voice is detected, the voice is recognized and corresponding control instructions are generated to control the notification information; if it is detected that the user's eyes are in a gaze state, or there is an action of enlarging the eyes, stretching the head and eyes forward, the font of the notification information will be enlarged so that the user can easily browse the content of the notification information.

[0081] Optionally, in an embodiment of the present application, in order to standardize the display position of information or make full use of the characteristics of display priority of different positions, in an embodiment of the present application, after receiving the notification information, the attribute information of the notification information may also be analyzed, and the spatial display position of the notification information in the three-dimensional display space may be determined based on the attribute information, that is, optionally, in some embodiments of the present application, the step of "in response to receiving the notification information, determining the spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device" includes:

[0082] In response to receiving the notification information, determining attribute information corresponding to the notification information;

[0083] Determine, according to the attribute information, a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device;

[0084] The attribute information includes at least one of the type information, receiving time information, title feature information, number of items, or sender identity information of the notification information.

[0085] Among them, attribute information is the attribute corresponding to the notification information, reflecting the characteristics of the notification information. Type information refers to the type of notification information, for example, type information includes office information, entertainment information, social information, current affairs information or application upgrade information, etc. For different types of information, different spatial display positions are determined. Among them, different types of notification information can be distinguished by the different applications that receive the information. For example, please refer to Figure 4 As shown, Figure 4 is a schematic diagram of the relationship between notification information of different applications and spatial display positions provided by an embodiment of the present application, wherein: Figure 4 It shows that the spatial display position of social applications is close to the left, where the left direction refers to the display direction and close refers to the distance to the extended reality device or the user; the spatial display position of the browser application is far to the right, etc.

[0086] Among them, the receiving time information refers to the time when the notification information is received; the title feature information refers to the feature information of the title of the notification information, such as the length of the title text, the size of the title font, the color of the title font or the keywords contained in the title, etc. The importance of the notification information can be analyzed by identifying the text of the title, and then the spatial display position can be determined; the number of information refers to the number of messages received by the same application within a period of time. Accordingly, the more the number, the higher the priority, and the notification information of the application can be displayed in front of the user so that the user can quickly perceive the notification information.

[0087] The sender identity information refers to the identity information of the user who sent the notification information. For example, the sender identity information may be a relative, friend, colleague, stranger, etc., and the spatial display position may be determined based on the identity relationship with the sender user. That is, optionally, in some embodiments of the present application, the step of "determining the spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device according to the attribute information" includes:

[0088] If the attribute information includes the sender identity information corresponding to the notification information, determining the interpersonal relationship level information corresponding to the sender identity information;

[0089] Determining a spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device according to the interpersonal relationship level information;

[0090] The generating of the notification audio for the spatial display position includes:

[0091] A notification audio for the spatial display position is generated according to the interpersonal relationship level information.

[0092] Among them, the interpersonal relationship level information refers to the interpersonal relationship level, which can be determined based on the identity information of the current target user and the sender user. For example, if the sender user is a relative of the current target user, the interpersonal relationship level is the highest; if the sender user is a friend or colleague of the current target user, the interpersonal relationship level is medium.

[0093] Accordingly, different types of notification audio may be generated for different levels of interpersonal relationship information. Figure 5 , Figure 5 It is a schematic diagram of the relationship between different interpersonal relationship levels and spatial display positions provided in an embodiment of the present application. For example, when the interpersonal relationship level is high, a pleasant and brisk notification audio can be generated, and when the interpersonal relationship level is medium, a regular notification audio can be generated.

[0094] Among them, in the embodiment of the present application, the attribute information can also be obtained by analyzing the facial features, voice features, and location features of the sender (such as the caller), for example, by activating sensors such as cameras, microphones, and accelerometers to collect the caller's facial expressions, voice features, location, and other information. Combined with the current user's personalized settings and habits, the system performs real-time data fusion and analysis to identify the caller's emotions, intentions, and nature of the call, and then determine the attribute information of the notification information.

[0095] Optionally, in an embodiment of the present application, for a spatial display position, the position information of the spatial display position with respect to an extended reality device or a user may be determined, and a notification audio may be generated according to the position information. That is, optionally, in some embodiments of the present application, the step of “generating a notification audio for the spatial display position” includes:

[0096] Determining the position information of the spatial display position relative to the extended reality device;

[0097] Determine the sound type information according to the attribute information of the notification information;

[0098] A spatial audio is generated according to the position information and the sound type information, and the spatial audio is used as the notification audio.

[0099] For example, if the spatial display position is directly in front and the notification information is social information, a pleasant low-frequency spatial audio is generated and output; if the spatial display position is directly behind and the notification information is entertainment, a regular low-frequency spatial audio is generated and output.

[0100] Optionally, in an embodiment of the present application, a background card for displaying notification information may be configured, and notification information displaying different attribute information may be distinguished by background card type and card color information. That is, optionally, in some embodiments of the present application, the step of "displaying the notification information at the spatial display position in the three-dimensional display space and outputting the notification audio" includes:

[0101] Determine the background card type and card color tone information according to the attribute information of the notification information;

[0102] Generating a background card for the background card type and the card color tone information at the spatial display position of the three-dimensional display space;

[0103] The notification information is displayed in the background card, and the notification audio is output.

[0104] The background card type is the type of the background card, for example, the background card type may include a circle, a square, an ellipse or a triangle, etc., and the background card may also be a plane or a three-dimensional structure. Figure 6 As shown, the notification information for "Wife Calls" uses an elliptical three-dimensional background card and uses warm colors for display. Please refer to Figure 7 As shown, the notification information for "Peter's call" uses a square three-dimensional background card and is displayed in cool colors.

[0105] Depend on Figure 4 and Figure 5It can be seen that for notification information with different attribute information, different vibration frequencies, materials and colors can be used to distinguish each notification information. Among them, the vibration frequency refers to the vibration frequency of the extended reality device body, and the material refers to the material of the background card, including transparent material, solid material, translucent material, semi-solid material, etc. For example, please refer to Figure 6 For the notification information of "Wife Calls", the background card with "transparent" material is used. Please refer to Figure 7 , a solid material background card is used for the notification information of "Peter is calling". The color corresponds to the card tone information. Among them, in the embodiment of the present application, the user's eye gaze information can also be identified, and the overall tone of the current three-dimensional display space can be adjusted according to the notification information of the user's eye gaze. For example, when it is determined through the eye gaze information that the user has browsed the notification information of "Wife is calling", the three-dimensional display space is adjusted to a warm tone as a whole based on the warm tone of the current background card. When it is determined that the user has browsed the notification information of "Peter is calling", the three-dimensional display space is adjusted to a cool tone as a whole based on the cool tone of the current turtle essence card.

[0106] Optionally, in an embodiment of the present application, since there may be some real scene objects in the three-dimensional display space where the notification information is displayed, in order to improve the display effect of the notification information and avoid blocking the real scene objects or causing the display effect of the notification information to be poor due to the color tone of the real scene objects, the notification information can be displayed at other locations other than the display scene objects. That is, optionally, in some embodiments of the present application, the step of "in response to receiving the notification information, determining the spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device" includes:

[0107] In response to receiving the notification information, determining an initial display position of the notification information in a three-dimensional display space corresponding to the extended reality device;

[0108] If it is detected that there is a preset real scene object in the gaze direction corresponding to the initial display position, a spatial display position other than the initial display position is determined from the three-dimensional display space.

[0109] Among them, the preset real-scene objects are pre-set or screened real-scene objects, for example, real-scene objects that are of concern to users or affect the display effect and are screened out based on the importance and color tone of the real-scene objects, such as the face of the person being talked to, an area with particularly high ambient brightness, an office computer display screen that is being used, etc. That is, in an embodiment of the present application, notification information does not need to avoid all real-scene objects, thereby improving the flexibility of notification information display.

[0110] In summary, the embodiment of the present application outputs notification audio to realize information notification reminder to the user by using notification audio, and determines the notification audio based on the spatial display position of the notification information, so that the user can quickly perceive the location of the notification information through the notification audio.

[0111] Among them, by displaying the notification information in the three-dimensional display space of the extended reality device and using the notification audio for information notification reminder, an audio notification based on the three-dimensional display space is realized, helping users to quickly perceive positioning information from the three-dimensional display space of the extended reality device and improve user experience.

[0112] In summary, the effects achieved by the embodiments of the present application at least include: Figure 8 As shown, when the user is currently browsing the virtual display screen in the front, if a notification message is received and displayed on the right side of the user, the sound from the right side will remind the user that there is a new message displayed on the right side, and the user can turn around and quickly locate and browse the notification message displayed on the right side, avoiding the user from screening one by one in the three-dimensional display space to find and locate the newly received notification message. When the user's eye gaze point stays in the area in the center of the front, a notification message is virtually displayed in an area above the user, and this area is not in the user's attention area. At this time, the extended reality device guides the user to adjust his line of sight to view the notification message through spatial audio (the user does not need to make body movements or change head posture at this time, and this situation is not shown in the figure).

[0113] In order to better implement the extended reality information notification method of the present application, the present application also provides an extended reality information notification device based on the above extended reality information notification method. The meanings of the terms are the same as those in the above extended reality information notification method, and the specific implementation details can refer to the description in the method embodiment.

[0114] See also Fig. 9 , Fig. 9 : is a schematic diagram of the structure of an extended reality information notification device provided in an embodiment of the present application, wherein the extended reality information notification device is applied to an extended reality device, and the device can be specifically as follows:

[0115] A response module 201, configured to determine, in response to receiving the notification information, a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device;

[0116] A generating module 202, configured to generate a notification audio for the spatial display position;

[0117] The notification module 203 is used to display the notification information at the spatial display position of the three-dimensional display space and output the notification audio.

[0118] Optionally, in some embodiments of the present application, the virtual display screen presented in the three-dimensional display space is in a hovering state, and the notification information is presented in the virtual display screen;

[0119] After displaying the notification information at the spatial display position of the three-dimensional display space and outputting the notification audio, the method further includes:

[0120] In response to detecting the behavior information of the target user, identifying a control instruction corresponding to the behavior information;

[0121] Processing the notification information in the virtual display screen in a hovering state according to the control instruction to obtain an information processing result;

[0122] The behavior information includes at least one of body rotation, gesture movement, ear deflection, ear closeness, voice or eye movement.

[0123] Optionally, in some embodiments of the present application, in response to receiving the notification information, determining the spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device includes:

[0124] In response to receiving the notification information, determining attribute information corresponding to the notification information;

[0125] Determine, according to the attribute information, a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device;

[0126] The attribute information includes at least one of the type information, receiving time information, title feature information, number of items, or sender identity information of the notification information.

[0127] Optionally, in some embodiments of the present application, determining, according to the attribute information, a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device includes:

[0128] If the attribute information includes the sender identity information corresponding to the notification information, determining the interpersonal relationship level information corresponding to the sender identity information;

[0129] Determining a spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device according to the interpersonal relationship level information;

[0130] The generating of the notification audio for the spatial display position includes:

[0131] A notification audio for the spatial display position is generated according to the interpersonal relationship level information.

[0132] Optionally, in some embodiments of the present application, generating a notification audio for the spatial display position includes:

[0133] Determining the position information of the spatial display position relative to the extended reality device;

[0134] Determine the sound type information according to the attribute information of the notification information;

[0135] A spatial audio is generated according to the position information and the sound type information, and the spatial audio is used as the notification audio.

[0136] Optionally, in some embodiments of the present application, displaying the notification information at the spatial display position in the three-dimensional display space and outputting the notification audio includes:

[0137] Determine the background card type and card color tone information according to the attribute information of the notification information;

[0138] Generating a background card for the background card type and the card color tone information at the spatial display position of the three-dimensional display space;

[0139] The notification information is displayed in the background card, and the notification audio is output.

[0140] Optionally, in some embodiments of the present application, in response to receiving the notification information, determining the spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device includes:

[0141] In response to receiving the notification information, determining an initial display position of the notification information in a three-dimensional display space corresponding to the extended reality device;

[0142] If it is detected that there is a real scene object in the gaze direction corresponding to the initial display position, a spatial display position other than the initial display position is determined from the three-dimensional display space.

[0143] In the embodiment of the present application, in response to receiving notification information, the response module 201 determines the spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device, the generation module 202 generates a notification audio for the spatial display position, and the notification module 203 displays the notification information at the spatial display position in the three-dimensional display space and outputs the notification audio.

[0144] Among them, the embodiment of the present application outputs notification audio to realize information notification reminder to the user by using notification audio, and determines the notification audio based on the spatial display position of the notification information, so that the user can quickly perceive the location of the notification information through the notification audio.

[0145] Among them, by displaying the notification information in the three-dimensional display space of the extended reality device and using the notification audio for information notification reminder, an audio notification based on the three-dimensional display space is realized, helping users to quickly perceive positioning information from the three-dimensional display space of the extended reality device and improve user experience.

[0146] In addition, the present application also provides an electronic device, such as Fig.10 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0147] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0148] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0149] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0150] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0151] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0152] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application program stored in the memory 302, thereby implementing the steps in any one of the augmented reality information notification methods provided in the embodiments of the present application.

[0153] Among them, the electronic device includes an extended reality device, which presents a virtual scene image in a three-dimensional display space, and notifies the user of information by displaying the position control and outputting the notification audio. The extended reality device can be wearable smart glasses, which include a frame, temples, a processor, a sensor, an optical display component, a microphone, a speaker, etc. The frame and temples are the supporting structure of the entire smart glasses, and a cavity is formed inside, and the cavity is used to place the circuit and the above-mentioned electronic components. The processor is used to calculate and process various data and receive and send instructions. The sensor includes a camera, an eye tracker, an IMU, a gyroscope, etc. The camera can be used to obtain images of the user's hands, face, and body; the eye tracker can be used to directly obtain the user's current line of sight and the position of the gaze point; the IMU and the gyroscope can be used to obtain the position of the smart glasses relative to the real environment and the user's head posture. The optical display component includes a micro-projection optical machine and an optical coupler. The micro-projection optical machine can be based on Micro-Oled, Micro-Led, LCOS, or LBS. The optical coupler can be an optical lens or an optical waveguide sheet, which is used to display virtual information in the real field of view. The microphone is used to capture the user's voice, and the speaker is used to generate and play spatial audio.

[0154] Specifically, in response to receiving notification information, the extended reality device of an embodiment of the present application determines the spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device, generates notification audio for the spatial display position, displays the notification information at the spatial display position in the three-dimensional display space, and outputs the notification audio.

[0155] Among them, by outputting notification audio, the notification audio is used to notify the user of information, and the notification audio is determined based on the spatial display position of the notification information, so that the user can quickly perceive the location of the notification information through the notification audio.

[0156] Among them, by displaying the notification information in the three-dimensional display space of the extended reality device and using the notification audio for information notification reminder, an audio notification based on the three-dimensional display space is realized, helping users to quickly perceive positioning information from the three-dimensional display space of the extended reality device and improve user experience.

[0157] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0158] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0159] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any of the extended reality information notification methods provided in the present application.

[0160] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0161] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0162] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the extended reality information notification methods provided in the present application, the beneficial effects that can be achieved by any of the extended reality information notification methods provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0163] The above is a detailed introduction to an extended reality information notification method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for notifying extended reality information, characterized in that: Applied to an extended reality device, the method comprises: In response to receiving the notification information, determining a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device; generating a notification audio for the spatial display location, wherein the notification audio includes audio based on the orientation of the spatial display location; Displaying the notification information at the spatial display position in the three-dimensional display space and outputting the notification audio; In response to receiving the notification information, determining a spatial display position of the notification information from a three-dimensional display space corresponding to the extended reality device includes: In response to receiving the notification information, determining an initial display position of the notification information in a three-dimensional display space corresponding to the extended reality device; If it is detected that there is a preset real scene object in the gaze direction corresponding to the initial display position, a spatial display position other than the initial display position is determined from the three-dimensional display space.

2. The method for notifying extended reality information according to claim 1, characterized in that: The virtual display screen presented in the three-dimensional display space is in a hovering state, and the notification information is presented in the virtual display screen; After displaying the notification information at the spatial display position of the three-dimensional display space and outputting the notification audio, the method further includes: In response to detecting the behavior information of the target user, identifying a control instruction corresponding to the behavior information; Processing the notification information in the virtual display screen in a hovering state according to the control instruction to obtain an information processing result; The behavior information includes at least one of body rotation, gesture movement, ear deflection, ear closeness, voice or eye movement.

3. The method for notifying extended reality information according to claim 1, characterized in that: In response to receiving the notification information, determining a spatial display position of the notification information from a three-dimensional display space corresponding to the extended reality device includes: In response to receiving the notification information, determining attribute information corresponding to the notification information; Determine, according to the attribute information, a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device; The attribute information includes at least one of the type information, receiving time information, title feature information, number of items, or sender identity information of the notification information.

4. The method for notifying extended reality information according to claim 3, characterized in that: The determining, according to the attribute information, a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device includes: If the attribute information includes the sender identity information corresponding to the notification information, determining the interpersonal relationship level information corresponding to the sender identity information; Determining a spatial display position of the notification information from the three-dimensional display space corresponding to the extended reality device according to the interpersonal relationship level information; The generating of the notification audio for the spatial display position includes: A notification audio for the spatial display position is generated according to the interpersonal relationship level information.

5. The method for notifying extended reality information according to claim 1, characterized in that: The generating of the notification audio for the spatial display position includes: Determining the position information of the spatial display position relative to the extended reality device; Determine the sound type information according to the attribute information of the notification information; A spatial audio is generated according to the position information and the sound type information, and the spatial audio is used as the notification audio.

6. The method for notifying extended reality information according to claim 1, characterized in that: The step of displaying the notification information at the spatial display position in the three-dimensional display space and outputting the notification audio includes: Determine the background card type and card color tone information according to the attribute information of the notification information; Generating a background card for the background card type and the card color tone information at the spatial display position of the three-dimensional display space; The notification information is displayed in the background card, and the notification audio is output.

7. An extended reality information notification device, characterized in that: Applied to an extended reality device, the device comprises: A response module, configured to determine, in response to receiving the notification information, a spatial display position of the notification information in the three-dimensional display space corresponding to the extended reality device; a generating module, configured to generate a notification audio for the spatial display position, wherein the notification audio includes an audio based on the orientation of the spatial display position; a notification module, configured to display the notification information at the spatial display position in the three-dimensional display space and output the notification audio; In response to receiving the notification information, determining a spatial display position of the notification information from a three-dimensional display space corresponding to the extended reality device includes: In response to receiving the notification information, determining an initial display position of the notification information in a three-dimensional display space corresponding to the extended reality device; If it is detected that there is a preset real scene object in the gaze direction corresponding to the initial display position, a spatial display position other than the initial display position is determined from the three-dimensional display space.

8. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the method for notifying extended reality information as described in any one of claims 1 to 6 when executing the computer program.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the extended reality information notification method according to any one of claims 1 to 6 are implemented.

Images