Information prompting method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an information prompting method and device, electronic equipment and a computer readable storage medium. The method is applied to augmented reality equipment. The method comprises the steps of obtaining at least one piece of prompt information in an environment; acquiring a user scene; determining target prompt information from the at least one piece of prompt information according to the user scene; displaying the target prompt information by using a virtual screen of the augmented reality device; and when it is detected that the user does not watch the target prompt information, playing a spatial audio to prompt the user to watch the target prompt information. Therefore, according to the scheme, the target prompt information which is not missed by the user in the current user scene can be determined, and the user is prompted to watch the target prompt information, so that the user is ensured to watch the target prompt information.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and specifically to an information prompting method, device, electronic device, and computer-readable storage medium. Background Art

[0002] One application of augmented reality (AR) glasses is to display navigation information. In the related art, AR glasses superimpose arrow icons on the real environment to instruct users to go straight, turn, or make a U-turn. However, if the user does not see the arrow icon, the user may go the wrong way, which may even pose a safety hazard to the user.

[0003] For example, after the user initially walks straight for a distance according to the straight arrow icon superimposed on the sidewalk, he turns his head to look at the scenery beside the road, but does not notice that the arrow icon superimposed on the sidewalk has been transformed into a turn arrow. The user may continue to walk straight and thus take the wrong road. Summary of the invention

[0004] The embodiments of the present application provide an information prompt method, device, electronic device and computer-readable storage medium, which can determine target prompt information that the user should not miss in the current user scenario, and prompt the user to view the target prompt information, thereby ensuring that the user views the target prompt information.

[0005] In a first aspect, an embodiment of the present application provides an information prompting method, which is applied to an extended reality device, and the method includes:

[0006] Obtain at least one prompt information in the environment;

[0007] Get user scenarios;

[0008] Determining target prompt information from the at least one prompt information according to the user scenario;

[0009] Using the virtual screen of the extended reality device to display the target prompt information; and

[0010] When it is detected that the user has not viewed the target prompt information, spatial audio is played to prompt the user to view the target prompt information.

[0011] In one of the embodiments, the playing space audio prompts the user to watch the target prompt information, including:

[0012] Obtaining a display position of the target prompt information;

[0013] Obtaining the user's location and posture information;

[0014] Determine the position of the target prompt information relative to the user according to the display position of the target prompt information and the position and posture information of the user;

[0015] Determining a sound source direction of the spatial audio according to the orientation;

[0016] In the direction of the sound source, the spatial audio is played to prompt the user to watch the target prompt information.

[0017] In one embodiment, the obtaining of at least one prompt information in the environment includes:

[0018] acquiring an image of the environment;

[0019] identifying at least one object in the environment based on the image of the environment;

[0020] Obtaining prompt information corresponding to the at least one object;

[0021] The using the virtual screen of the extended reality device to display the target prompt information includes:

[0022] The target prompt information and the object corresponding to the target prompt information are superimposed and displayed using the virtual screen of the extended reality device.

[0023] In one of the embodiments, determining target prompt information from the at least one prompt information according to the user scenario includes:

[0024] Obtaining the importance of each of the prompt information in the user scenario; wherein the importance of each of the prompt information in different scenarios is different;

[0025] Get importance threshold;

[0026] The prompt information whose importance is higher than the importance threshold is determined as the target prompt information.

[0027] In one of the embodiments, before playing the spatial audio to prompt the user to watch the target prompt information when detecting that the user has not watched the target prompt information, the method further includes:

[0028] Collecting the user's posture information and eye movement information;

[0029] Obtaining a display position of the target prompt information;

[0030] Determining the visual range of the user according to the posture information and eye movement information of the user;

[0031] Whether the user views the target prompt information is detected according to the user's visual range and the display position of the target prompt information.

[0032] In one of the embodiments, the playing space audio prompts the user to watch the target prompt information, including:

[0033] collecting sounds in the environment;

[0034] Determining the volume of the spatial audio according to the volume of the sound in the environment;

[0035] The spatial audio is played according to the volume of the spatial audio to prompt the user to watch the target prompt information.

[0036] In one embodiment, the user scenario is a navigation scenario; the method further comprises:

[0037] Obtain the current location and destination location of the user;

[0038] Determine a navigation path according to the current location and the destination location;

[0039] The obtaining of at least one prompt information in the environment includes:

[0040] At least one navigation information of the navigation path in the environment is obtained.

[0041] In a second aspect, an embodiment of the present application provides an information prompting device, which is applied to an extended reality device, and the device includes:

[0042] An information acquisition module, used to acquire at least one prompt information in the environment;

[0043] The scene acquisition module is used to obtain the user scene;

[0044] An information determination module, configured to determine target prompt information from the at least one prompt information according to the user scenario;

[0045] An information display module, used to display the target prompt information using a virtual screen of the extended reality device; and

[0046] The audio playing module is used to play spatial audio to prompt the user to watch the target prompt information when it is detected that the user has not watched the target prompt information.

[0047] In one embodiment, the audio playback module includes:

[0048] A display position acquisition unit, used to acquire the display position of the target prompt information;

[0049] An information unit, used to obtain the position and posture information of the user;

[0050] A position determination unit, configured to determine the position of the target prompt information relative to the user according to the display position of the target prompt information and the position and posture information of the user;

[0051] a direction determination unit, configured to determine a sound source direction of the spatial audio according to the orientation;

[0052] The playing unit is used to play the spatial audio in the direction of the sound source to prompt the user to watch the target prompt information.

[0053] In one embodiment, the information acquisition module includes:

[0054] An image acquisition unit, used to acquire an image of the environment;

[0055] an object recognition unit, configured to recognize at least one object in the environment based on the image of the environment;

[0056] a prompt obtaining unit, configured to obtain prompt information corresponding to the at least one object;

[0057] The information display module includes:

[0058] The superimposed display unit is used to use the virtual screen of the extended reality device to superimpose and display the target prompt information and the object corresponding to the target prompt information.

[0059] In one embodiment, the information determination module includes:

[0060] A degree acquisition unit, used to acquire the importance of each of the prompt information in the user scenario; wherein the importance of each of the prompt information is different in different scenarios;

[0061] A threshold value obtaining unit, used for obtaining an importance threshold value;

[0062] The target information determination unit is used to determine the prompt information whose importance is higher than the importance threshold as the target prompt information.

[0063] In one of the embodiments, before playing the spatial audio to prompt the user to watch the target prompt information when detecting that the user has not watched the target prompt information, the device further includes:

[0064] A collection module, used to collect the user's posture information and eye movement information;

[0065] A display location acquisition module, used to acquire the display location of the target prompt information;

[0066] A range determination module, used to determine the visual range of the user according to the posture information and eye movement information of the user;

[0067] The detection module is used to detect whether the user sees the target prompt information according to the user's visual range and the display position of the target prompt information.

[0068] In one embodiment, the audio playback module includes:

[0069] A sound collection unit, used for collecting sounds in the environment;

[0070] a volume determination unit, configured to determine the volume of the spatial audio according to the volume of the sound in the environment;

[0071] The audio playing unit is used to play the spatial audio according to the volume of the spatial audio to prompt the user to watch the target prompt information.

[0072] In one embodiment, the user scenario is a navigation scenario; the device further comprises:

[0073] A location acquisition module, used to acquire the current location and destination location of the user;

[0074] A path determination module, used to determine a navigation path according to the current location and the destination location;

[0075] The information acquisition module includes:

[0076] The navigation information acquisition unit is used to acquire at least one navigation information of the navigation path in the environment.

[0077] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and which implements the steps in the above-mentioned information prompt method when the computer program is executed by the processor.

[0078] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned information prompt method are implemented.

[0079] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0080] In summary, in the embodiment of the present application, at least one prompt information in the environment can be obtained; a user scenario can be obtained; according to the user scenario, target prompt information can be determined from the at least one prompt information; the target prompt information can be displayed using the virtual screen of the extended reality device; when it is detected that the user has not viewed the target prompt information, spatial audio can be played to prompt the user to view the target prompt information. In this way, target prompt information that the user should not miss in the user scenario can be determined, and spatial audio can be used to prompt the user to view the target prompt information, thereby ensuring that the user views the target prompt information. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0082] Figure 1 It is a schematic diagram of the steps of the information prompting method provided by an embodiment of the present application;

[0083] Figure 2 is a structural diagram of a spatial audio playback module provided in an embodiment of the present application;

[0084] Figure 3 It is a structural schematic diagram of an information prompting device provided by an embodiment of the present application;

[0085] Figure 4 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0087] Extended Reality (XR) technology combines the real and the virtual through computers to create a virtual environment for human-computer interaction. Extended reality includes but is not limited to: Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR). Extended reality devices refer to electronic devices with extended reality functions.

[0088] In one embodiment, Figure 1 As shown, an information prompt method is provided. Although a logical order is shown in the step diagram, in some cases, the steps shown or described may be performed in an order different from that shown in the figure. Specifically, the information prompt method can be applied to an extended reality device. In particular, the extended reality device refers to a wearable extended reality device, and in particular, refers to optical see-through smart glasses.

[0089] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0090] according to Figure 1 The information prompt method shown in the figure includes at least steps S110 to S150, which are described in detail as follows:

[0091] In step S110, at least one prompt information in the environment is obtained.

[0092] The camera collects images of the surrounding environment to obtain the user's current location information, and determines prompt information in the environment based on the user's current location information and the image of the environment. The prompt information may include but is not limited to: navigation prompt information, scenic spot prompt information and / or store prompt information.

[0093] Navigation prompt information may include, but is not limited to: action direction information, action information, obstacle information, road information and / or road condition information. Action direction information may include information such as straight, turning and U-turn. Action information may include information such as current action speed. Obstacle information is used to prompt the user that there are obstacles. Road information may include information such as the name of the current road, whether it is a one-way road and whether it is a tidal lane. Road condition information may include information such as whether the road is congested, whether a traffic accident has occurred and whether it is under maintenance.

[0094] The scenic spot prompt information may include, but is not limited to: the name of the scenic spot, the introduction of the scenic spot, the opening hours of the scenic spot and / or the camera position of the scenic spot.

[0095] Store prompt information may include, but is not limited to: store name, product categories sold, store ratings and / or store business hours.

[0096] In step S120, the user scenario is obtained.

[0097] User scenarios characterize the purpose of user behavior and may include, but are not limited to, navigation scenarios, travel scenarios, and / or shopping scenarios.

[0098] In one embodiment, the user scenario may be selected by the user, and a plurality of user scenarios are preset, and the user selects the current user scenario.

[0099] In another embodiment, the user scenario may be determined by analyzing the user behavior and the surrounding environment.

[0100] For example, if a user presets a workplace and a home address, and the user travels from the home address to the workplace at a fixed time every weekday morning, the user scenario can be identified as a commuting scenario.

[0101] For another example, if a user leaves his or her permanent city during holidays and travels to another city, the user scenario can be identified as a travel scenario.

[0102] Optionally, user scenarios corresponding to different user behaviors and different environments may be determined in advance according to big data analysis; when the user behavior and the surrounding environment are obtained, the user scenario is determined based on the user behavior and the surrounding environment in combination with the corresponding relationship determined by big data.

[0103] In step S130, target prompt information is determined from the at least one prompt information according to the user scenario.

[0104] The importance of each prompt information in each user scenario is preset; different prompt information has different importance in different user scenarios. For example, in the navigation scenario, the store rating of the surrounding stores is less important; while in the shopping scenario, the store rating of the surrounding stores is more important.

[0105] The importance of each prompt information in each user scenario can be determined by using a large language model, or by using big data analysis, or can be pre-configured in the system, or can be set or modified by the user himself.

[0106] According to the current user scenario, the importance of each prompt information in the current environment is obtained. The prompt information whose importance is higher than the importance threshold is determined as the target prompt information. The importance threshold can be set according to actual needs.

[0107] In step S140, the target prompt information is displayed using the virtual screen of the extended reality device.

[0108] The virtual screen of the extended reality device is not a real physical screen in the traditional sense, but a display area that is simulated and generated through technical means, using elements such as light and images to allow users to perceive various virtual information, images, videos, interactive interfaces and other content.

[0109] The extended reality device can display the target prompt information on the virtual screen. Because the extended reality device has an extended reality function, the target prompt information can be displayed by superimposing it on the relevant objects in the environment.

[0110] For example, when displaying store information, the store information may be displayed in the store's front area; when displaying obstacle information, the obstacle information may be displayed on the obstacle.

[0111] Optionally, each prompt information of the non-target prompt information can also be displayed. When the target prompt information and the non-target prompt information are displayed simultaneously, the target prompt information and the non-target prompt information can be distinguished and displayed using different colors, different font sizes, etc., so as to make the target prompt information more obvious.

[0112] In step S150, when it is detected that the user has not viewed the target prompt information, spatial audio is played to prompt the user to view the target prompt information.

[0113] Whether the user has viewed the target prompt information can be determined through an eye tracker and user posture information. When the user has not viewed the target prompt information, the spatial audio is played to prompt the user to view the target prompt information. The sound source direction of the spatial audio is determined according to the display position of the target prompt information relative to the user's position.

[0114] Spatial audio is an audio technology that uses immersive audio technology, takes people as the core of the sound field, and can accurately present the vast sound field effect of three-dimensional space. It can restore an audio experience with greater sense of presence, distance, direction and movement.

[0115] The extended reality device includes a spatial audio playback module, which is used to play spatial audio. Figure 2 is a schematic diagram of the structure of a spatial audio playback module provided in an embodiment of the present application. Figure 2 As shown, the spatial audio playback module can have multiple sound source directions. When audio in a certain direction needs to be played, the sound source corresponding to the direction will play the audio.

[0116] In one of the embodiments, if the target prompt information that the user has not viewed is on the left side of the user, the sound source on the left side of the spatial audio playback module can play audio to prompt the user to view the target prompt information. The user can determine that the target prompt information that the user has not viewed is on his or her left side based on the direction of the sound source.

[0117] In one embodiment, during the spatial audio playback, the sound can change with the user's movements and angles, bringing a truly immersive experience. For example, if the target prompt information that the user has not viewed is on the left side of the user, the sound source on the left side of the spatial audio playback module can play audio to prompt the user to view the target prompt information. At this time, the user turns to the right, and the location of the target prompt information that was not originally viewed is currently directly behind the user. In this case, the spatial audio playback module can switch the sound source on the left to the sound source directly behind to continue playing the spatial audio prompt.

[0118] By adopting the technical solution of the embodiment of the present application, at least one prompt information in the environment can be obtained; a user scenario can be obtained; according to the user scenario, target prompt information can be determined from the at least one prompt information; the target prompt information can be displayed using the virtual screen of the extended reality device; when it is detected that the user has not viewed the target prompt information, spatial audio can be played to prompt the user to view the target prompt information. In this way, the target prompt information that the user should not miss in the user scenario can be determined, and the user can be prompted to view the target prompt information using spatial audio, thereby ensuring that the user views the target prompt information.

[0119] Based on the above technical solution, as an embodiment, the playing of the spatial audio to prompt the user to watch the target prompt information may include: obtaining the display position of the target prompt information; obtaining the position and posture information of the user; determining the orientation of the target prompt information relative to the user according to the display position of the target prompt information and the position and posture information of the user; determining the sound source direction of the spatial audio according to the orientation; and playing the spatial audio in the sound source direction to prompt the user to watch the target prompt information;

[0120] The sound source direction of the spatial audio is determined based on the display position of the target prompt information relative to the user. Therefore, when playing the spatial audio, it is necessary to determine the position of the target prompt information relative to the user.

[0121] When displaying the target prompt information, the display position of the target prompt information is determined. For example, the target prompt information may be displayed by being superimposed on a related object in the environment. Therefore, the display position of the target prompt information may be directly obtained.

[0122] The user's position can be determined based on the positioning information of the extended reality device. The extended reality device may include a camera, which can collect the user's posture information. Based on the user's posture information, the user's body orientation and facial orientation can be determined. Based on the display position of the target prompt information, and the user's position and posture information, the orientation of the target prompt information relative to the user can be determined.

[0123] According to the orientation of the target prompt information relative to the user, the sound source direction of the spatial audio can be determined; in the sound source direction, the spatial audio is played to prompt the user to watch the target prompt information.

[0124] For example, when the target prompt information is located on the left side of the user, the sound source direction of the spatial audio can be determined to be on the left side; when the target prompt information is located behind the user, the sound source direction of the spatial audio can be determined to be directly behind.

[0125] By adopting the technical solution of the embodiment of the present application, the sound source direction of the spatial audio is determined according to the position of the target prompt information relative to the user. Therefore, when the user hears the spatial audio, the position of the target prompt information can be determined according to the sound source direction of the spatial audio, so that the target prompt information can be quickly viewed.

[0126] Based on the above technical solution, as an embodiment, the playing of spatial audio to prompt the user to watch the target prompt information may include: collecting sounds in the environment; determining the volume of the spatial audio according to the volume of the sounds in the environment; and playing the spatial audio according to the volume of the spatial audio to prompt the user to watch the target prompt information.

[0127] If the spatial audio is played at a fixed volume, the spatial audio may not be heard clearly when the volume of the sound in the environment is loud. Therefore, the volume of the spatial audio may be determined according to the volume of the sound in the environment.

[0128] The extended reality device may include an audio collection device, which can collect sounds in the environment and determine the volume of the sounds in the environment based on the collected sounds in the environment.

[0129] The volume of the spatial audio is determined according to the volume of the sound in the environment. Optionally, the correspondence between the volume of the sound in each environment and the volume of the spatial audio can be preset, and the volume of the spatial audio is determined by querying the correspondence according to the volume of the sound in the environment. Optionally, the volume of the spatial audio can be set to be greater than the volume of the sound in the environment so that the spatial audio can be heard clearly in the current environment.

[0130] After the volume of the spatial audio is determined, the spatial audio may be played according to the volume of the spatial audio to prompt the user to view the target prompt information.

[0131] Optionally, the volume of the spatial audio also supports user adjustment.

[0132] By adopting the technical solution of the embodiment of the present application, the volume of the spatial audio is determined according to the volume of the sound in the environment to ensure that the user can hear the spatial audio clearly in the current environment, and thus view the target prompt information according to the prompt of the spatial audio.

[0133] Based on the above technical solution, as an embodiment, the obtaining of at least one prompt information in the environment may include: obtaining an image of the environment; identifying at least one object in the environment according to the image of the environment; and obtaining prompt information corresponding to the at least one object. The display of the target prompt information using the virtual screen of the extended reality device may include: displaying the target prompt information and the object corresponding to the target prompt information by superimposing them on the virtual screen of the extended reality device.

[0134] In one embodiment, the extended reality device may have a camera, and the camera may collect images of the environment. In another embodiment, the extended reality device may report its own location information to the cloud, thereby obtaining an image of the environment corresponding to the location information returned by the cloud.

[0135] The extended reality device can analyze and process the image of the environment to identify various objects in the environment. Optionally, for objects that change frequently, such as pedestrians and obstacles, they can be identified by analyzing and processing the environment.

[0136] Optionally, for relatively fixed objects such as buildings, scenic spots, and lanes, they can be determined from the Global Positioning System (GPS) based on the location information of the extended reality device; objects determined from the GPS are matched with objects in the image based on the image of the environment to determine the objects in the environment.

[0137] Analyze and identify the objects to determine the prompt information corresponding to each object. In one embodiment, the prompt information corresponding to the object may include information obtained by analyzing the object. For example, by analyzing a stone on the road, it can be determined that the stone is an obstacle.

[0138] In another embodiment, the prompt information corresponding to the object may include information obtained from a data source corresponding to the object. For example, the rating information of a department store may be obtained from a store rating system; the opening hours information of a scenic spot may be obtained from an official website of the scenic spot.

[0139] After the prompt information is obtained, the target prompt information can be determined, and the target prompt information and the object corresponding to the target prompt information can be superimposed and displayed.

[0140] The extended reality device can display the real environment and virtual elements at the same time through the virtual screen, and the prompt information can also be regarded as a virtual element, so the real environment and the prompt information can be displayed together. In order to allow users to more intuitively determine what object the prompt information is specifically prompting, the prompt information can be superimposed and displayed with the object corresponding to the prompt information in the real environment.

[0141] For example, when displaying store information, virtual elements of the store information may be superimposed on the storefront area of ​​the actual store and displayed; when displaying obstacle information, the obstacle information may be superimposed on the real obstacle and displayed.

[0142] Optionally, each prompt information of the non-target prompt information can also be displayed. When the target prompt information and the non-target prompt information are displayed simultaneously, the target prompt information and the non-target prompt information can be distinguished and displayed using different colors, different font sizes, etc., so as to make the target prompt information more obvious.

[0143] By adopting the technical solution of the embodiment of the present application, prompt information of objects in the environment can be obtained. When displaying the target prompt information, the target prompt information and the object corresponding to the target prompt information can be superimposed and displayed, so that the user can more intuitively determine what object the target prompt information is specifically prompting, so that the user can quickly understand the target prompt information.

[0144] On the basis of the above technical solution, as an embodiment, determining the target prompt information from the at least one prompt information according to the user scenario may include: obtaining the importance of each prompt information in the user scenario; wherein the importance of each prompt information is different in different scenarios; obtaining an importance threshold; and determining the prompt information whose importance is higher than the importance threshold as the target prompt information.

[0145] Different prompt information has different importance in different user scenarios. For example, in the navigation scenario, the store ratings of surrounding stores are less important, while in the shopping scenario, the store ratings of surrounding stores are more important.

[0146] The importance of different prompt information in different user scenarios can be preset. The importance of different prompt information in different user scenarios can be determined using a large language model, or determined using big data analysis, or pre-configured in the system, or set or modified by the user.

[0147] According to the current user scenario, the importance of each prompt information in the environment is obtained. The importance threshold is obtained, and the importance threshold can be set according to actual needs. Among the prompt information in the environment, the prompt information with an importance higher than the importance threshold is determined as the target prompt information.

[0148] By adopting the technical solution of the embodiment of the present application, target prompt information with higher importance can be determined from the prompt information according to the importance of each prompt information. When the user has not viewed the target prompt information, spatial audio can be played to prompt the user to view the target prompt information, thereby ensuring that the user does not miss important target prompt information.

[0149] On the basis of the above technical solution, as an embodiment, before playing spatial audio to prompt the user to watch the target prompt information when it is detected that the user has not watched the target prompt information, the method may also include: collecting the user's posture information and eye movement information; obtaining the display position of the target prompt information; determining the user's line of sight based on the user's posture information and eye movement information; and detecting whether the user has watched the target prompt information based on the user's line of sight and the display position of the target prompt information.

[0150] Before playing spatial audio, it is necessary to detect whether the user is looking at the target prompt information. Whether the user is looking at the target prompt information can be determined through the user's posture information and eye movement information.

[0151] The extended reality device may include a camera, which may collect the user's posture information, and the user's body orientation and facial orientation may be determined based on the user's posture information. The extended reality device may include an eye tracker, which may collect the user's eye movement information. Based on the user's posture information and eye movement information, the user's visual range may be determined.

[0152] According to the user's visual range and the display position of the target prompt information, it can be determined whether the user has viewed the target prompt information. When the target prompt information is within the user's visual range, it can be considered that the user has viewed the target prompt information. When the target prompt information is always out of the user's visual range, it can be considered that the user has not viewed the target prompt information.

[0153] By adopting the technical solution of the embodiment of the present application, it is possible to determine whether the user has viewed the target prompt information based on the user's posture information and eye movement information, so that when the user has not viewed the target prompt information, spatial audio is played to prompt the user to view the target prompt information, thereby ensuring that the user does not miss the target prompt information.

[0154] Based on the above technical solution, as an embodiment, the user scenario can be a navigation scenario. The method can also include: obtaining the current location and the destination location of the user; and determining the navigation path according to the current location and the destination location. The obtaining of at least one prompt information in the environment can include: obtaining at least one navigation information of the navigation path in the environment.

[0155] When the user inputs or selects the destination location in advance, the current user scenario can be determined to be a navigation scenario. In the navigation scenario, the user's current location and the destination location can be obtained. The user's current location can be the current location of the extended reality device. According to the user's current location and the destination location, a navigation path from the current location to the destination location can be planned.

[0156] The prompt information may include navigation information of the navigation path in the current environment. For example, the navigation path is to go straight from road A to road B, and then turn left at road B to road C. If the user is currently on road A, the navigation information may be determined to be straight, and if the user is currently at the junction of road B and road C, the navigation information may be determined to be a left turn.

[0157] In non-navigation scenarios, the prompt information may only display the prompt information of surrounding objects, while in navigation scenarios, the prompt information includes navigation information. For example, facing the same road, in non-navigation scenarios, the prompt information may include the name of the road, while in navigation scenarios, the prompt information may include navigation information of "go straight along this road".

[0158] By adopting the technical solution of the embodiment of the present application, in the navigation scene, navigation information can be determined according to the navigation path and the current position of the user, and the navigation information can instruct the user how to move. Navigation of the user can be achieved by displaying the navigation information or playing spatial audio to prompt the user to watch the navigation information.

[0159] Based on the above technical solution, as an embodiment, spatial audio can not only be used to prompt the user to watch the target prompt information, but also directly play the target prompt information. Optionally, the target prompt information is converted into voice and then played using spatial audio. The sound source direction of the spatial audio is determined according to the orientation of the target prompt information relative to the user.

[0160] Based on the above technical solution, as an embodiment, when the extended reality device is a wearable extended reality device, the user can also be prompted to view the target prompt information through tactile feedback. The direction of triggering feedback is determined according to the direction of the target prompt information relative to the user.

[0161] In one of the embodiments, when the wearable extended reality device is a vest with an extended reality function, the vest may also have a virtual screen, a spatial audio playback module, and a tactile feedback prompt module. The user's back is divided into multiple areas, each of which corresponds to an orientation. For example, the upper part of the user's back corresponds to the front, the lower part of the user's back corresponds to the rear, the left side of the user's back corresponds to the front side, the right side of the user's back corresponds to the right side, the upper left side of the user's back corresponds to the left front, the lower left side of the user's back corresponds to the rear, the upper side of the user's back corresponds to the right front, and the lower right side of the user's back corresponds to the right rear.

[0162] The position of the target prompt information that the user has not viewed relative to the user is obtained, and according to the position, the vest sends a vibration prompt at the corresponding back area of ​​the user. For example, when the target prompt information that the user has not viewed is located in the left front of the user, the vest sends a vibration prompt at the upper left area of ​​the back of the user to prompt the user to view the target prompt information in the left front.

[0163] In this way, the user can be prompted to view the target prompt information through tactile feedback and / or spatial audio, ensuring that the user can view the target prompt information.

[0164] In order to better implement the information prompting method of the present application, the present application also provides an information prompting device based on the above information prompting method. The meanings of the terms are the same as those in the above information prompting method, and the specific implementation details can refer to the description in the method embodiment.

[0165] See also Figure 3 , Figure 3 : is a structural schematic diagram of an information prompting device provided in an embodiment of the present application, wherein the information prompting device is applied to an extended reality device, and the information prompting device includes:

[0166] The information acquisition module 301 is used to acquire at least one prompt information in the environment;

[0167] A scene acquisition module 302 is used to acquire a user scene;

[0168] An information determination module 303, configured to determine target prompt information from the at least one prompt information according to the user scenario;

[0169] An information display module 304, configured to display the target prompt information using a virtual screen of the extended reality device; and

[0170] The audio playing module 305 is used to play spatial audio to prompt the user to watch the target prompt information when it is detected that the user has not watched the target prompt information.

[0171] In one embodiment, the audio playback module 305 includes:

[0172] A display position acquisition unit, used to acquire the display position of the target prompt information;

[0173] An information unit, used to obtain the position and posture information of the user;

[0174] A position determination unit, configured to determine the position of the target prompt information relative to the user according to the display position of the target prompt information and the position and posture information of the user;

[0175] a direction determination unit, configured to determine a sound source direction of the spatial audio according to the orientation;

[0176] The playing unit is used to play the spatial audio in the direction of the sound source to prompt the user to watch the target prompt information.

[0177] In one embodiment, the information acquisition module 301 includes:

[0178] An image acquisition unit, used to acquire an image of the environment;

[0179] an object recognition unit, configured to recognize at least one object in the environment based on the image of the environment;

[0180] a prompt obtaining unit, configured to obtain prompt information corresponding to the at least one object;

[0181] The information display module 304 includes:

[0182] The superimposed display unit is used to use the virtual screen of the extended reality device to superimpose and display the target prompt information and the object corresponding to the target prompt information.

[0183] In one embodiment, the information determination module 303 includes:

[0184] A degree acquisition unit, used to acquire the importance of each of the prompt information in the user scenario; wherein the importance of each of the prompt information is different in different scenarios;

[0185] A threshold value obtaining unit, used for obtaining an importance threshold value;

[0186] The target information determination unit is used to determine the prompt information whose importance is higher than the importance threshold as the target prompt information.

[0187] In one of the embodiments, before playing the spatial audio to prompt the user to watch the target prompt information when detecting that the user has not watched the target prompt information, the device further includes:

[0188] A collection module, used to collect the user's posture information and eye movement information;

[0189] A display location acquisition module, used to acquire the display location of the target prompt information;

[0190] A range determination module, used to determine the visual range of the user according to the posture information and eye movement information of the user;

[0191] The detection module is used to detect whether the user sees the target prompt information according to the user's visual range and the display position of the target prompt information.

[0192] In one embodiment, the audio playback module 305 includes:

[0193] A sound collection unit, used for collecting sounds in the environment;

[0194] a volume determination unit, configured to determine the volume of the spatial audio according to the volume of the sound in the environment;

[0195] The audio playing unit is used to play the spatial audio according to the volume of the spatial audio to prompt the user to watch the target prompt information.

[0196] In one embodiment, the user scenario is a navigation scenario; the device further comprises:

[0197] A location acquisition module, used to acquire the current location and destination location of the user;

[0198] A path determination module, used to determine a navigation path according to the current location and the destination location;

[0199] The information acquisition module 301 includes:

[0200] The navigation information acquisition unit is used to acquire at least one navigation information of the navigation path in the environment.

[0201] By adopting the technical solution of the embodiment of the present application, at least one prompt information in the environment can be obtained; a user scenario can be obtained; according to the user scenario, target prompt information can be determined from the at least one prompt information; the target prompt information can be displayed using the virtual screen of the extended reality device; when it is detected that the user has not viewed the target prompt information, spatial audio can be played to prompt the user to view the target prompt information. In this way, the target prompt information that the user should not miss in the user scenario can be determined, and the user can be prompted to view the target prompt information using spatial audio, thereby ensuring that the user views the target prompt information.

[0202] For the specific definition of the information prompting device, please refer to the definition of the information prompting method above, which will not be repeated here. Each module in the above-mentioned information prompting device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0203] In addition, the present application also provides an electronic device, which may be an extended reality glasses. Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0204] The electronic device may include one or more processors 401 of processing cores and one or more computer-readable storage media memories 402 and other components. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0205] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0206] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0207] In one embodiment, the electronic device further includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption management and other functions through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0208] In one embodiment, the electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0209] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application program stored in the memory 402, thereby implementing the steps in any one of the information prompt methods provided in the embodiments of the present application.

[0210] When the specific electronic device is wearable optical see-through smart glasses, in addition to the above structure, it also includes at least a glasses body frame, an optical display component, an electronic circuit component, a sensor, etc. The sensors built into the glasses include a heart rate monitor, a blood glucose tester, a microphone, a camera and / or an eye tracker.

[0211] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0212] In one embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in any embodiment of the present application is implemented.

[0213] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the present application is implemented.

[0214] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which implement the method described in any embodiment of the present application when executed by a processor.

[0215] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0216] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0217] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any one of the information prompt methods provided in the present application.

[0218] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0219] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0220] Since the instructions stored in the computer-readable storage medium can execute the steps in any information prompting method provided in the present application, the beneficial effects that can be achieved by any information prompting method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0221] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0222] The above is a detailed introduction to an information prompt method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An information prompting method, characterized in that: Applied to an extended reality device, the method comprises: Obtain at least one prompt information in the environment; Get user scenarios; Determining target prompt information from the at least one prompt information according to the user scenario; Using the virtual screen of the extended reality device to display the target prompt information; and When it is detected that the user has not viewed the target prompt information, spatial audio is played to prompt the user to view the target prompt information.

2. The method according to claim 1, characterized in that The playing space audio prompts the user to watch the target prompt information, including: Obtaining a display position of the target prompt information; Obtaining the user's location and posture information; Determine the position of the target prompt information relative to the user according to the display position of the target prompt information and the position and posture information of the user; Determining a sound source direction of the spatial audio according to the orientation; In the direction of the sound source, the spatial audio is played to prompt the user to watch the target prompt information.

3. The method according to claim 1, characterized in that The obtaining of at least one prompt information in the environment includes: acquiring an image of the environment; identifying at least one object in the environment based on the image of the environment; Obtaining prompt information corresponding to the at least one object; The using the virtual screen of the extended reality device to display the target prompt information includes: The target prompt information and the object corresponding to the target prompt information are superimposed and displayed using the virtual screen of the extended reality device.

4. The method according to claim 1, characterized in that: The determining target prompt information from the at least one prompt information according to the user scenario includes: Obtaining the importance of each of the prompt information in the user scenario; wherein the importance of each of the prompt information in different scenarios is different; Get importance threshold; The prompt information whose importance is higher than the importance threshold is determined as the target prompt information.

5. The method according to claim 1, characterized in that Before playing the spatial audio to prompt the user to watch the target prompt information when it is detected that the user has not watched the target prompt information, the method further includes: Collecting the user's posture information and eye movement information; Obtaining a display position of the target prompt information; Determining the visual range of the user according to the posture information and eye movement information of the user; Whether the user views the target prompt information is detected according to the user's visual range and the display position of the target prompt information.

6. The method according to claim 1, characterized in that The playing space audio prompts the user to watch the target prompt information, including: collecting sounds in the environment; Determining the volume of the spatial audio according to the volume of the sound in the environment; The spatial audio is played according to the volume of the spatial audio to prompt the user to watch the target prompt information.

7. The method according to claim 1, characterized in that The user scenario is a navigation scenario; the method further includes: Obtain the current location and destination location of the user; Determine a navigation path according to the current location and the destination location; The obtaining of at least one prompt information in the environment includes: At least one navigation information of the navigation path in the environment is obtained.

8. An information prompting device, characterized in that: Applied to an extended reality device, the device comprises: An information acquisition module, used to acquire at least one prompt information in the environment; The scene acquisition module is used to obtain the user scene; An information determination module, configured to determine target prompt information from the at least one prompt information according to the user scenario; An information display module, used to display the target prompt information using a virtual screen of the extended reality device; and The audio playing module is used to play spatial audio to prompt the user to watch the target prompt information when it is detected that the user has not watched the target prompt information.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the information prompt method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the information prompt method according to any one of claims 1 to 7 are implemented.

Images