Object information display method and device, storage medium and program product

Abstract

One or more embodiments of the invention provide an object information display method and device, a storage medium and a program product, the method is applied to AR glasses, and the AR glasses are configured with at least one environment sensor; the method comprises the following steps: determining surrounding environment information according to a target environment sensor; determining a view blind area of the user and object information of a risk object according to the environment information; and displaying the object information according to the position relationship between the view blind area and the risk object.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of augmented reality (AR) technology, and in particular, to a method, device, storage medium, and program product for displaying object information. Background Art

[0002] In scenarios such as sports, driving, and industrial operations, blind spots refer to areas where users cannot see surrounding objects or pedestrians due to body posture, vehicle structure, or equipment limitations. These blind spots are particularly dangerous in complex environments because even if users consciously observe, it is difficult for them to detect potential risks in time, which can easily lead to accidents. For example, during driving, blind spots caused by vehicle body design may hide pedestrians or obstacles; in industrial operations, the layout of equipment or the user's posture may also make it impossible to see critical areas. Therefore, the industry is in urgent need of new technologies and methods to reduce or eliminate blind spots and the impact they cause. Summary of the invention

[0003] In view of this, one or more embodiments of this specification provide the following technical solutions:

[0004] According to a first aspect of one or more embodiments of this specification, a method for displaying object information is provided, which is applied to AR glasses, wherein the AR glasses are configured with at least one environmental sensor; the method includes:

[0005] Determine surrounding environmental information based on target environmental sensors;

[0006] Determine the user's visual blind spot and object information of the risk object according to the environmental information;

[0007] The object information is displayed according to the positional relationship between the blind area and the risk object.

[0008] Optionally, there are multiple types of target environment sensors; and determining surrounding environment information according to the target environment sensors includes:

[0009] Obtain the original information collected by various target environment sensors;

[0010] Each raw information is processed according to a sensor fusion algorithm to obtain the environmental information.

[0011] Optionally, the AR glasses are connected to auxiliary environmental sensors on other devices; the obtaining of original information respectively collected by multiple target environmental sensors includes:

[0012] When the auxiliary sensing conditions are met, original information collected by the environmental sensor of the AR glasses and the auxiliary environmental sensor are obtained.

[0013] Optionally, the auxiliary sensing condition includes at least any one of the following: determining that the user is in a high-risk scenario based on the environmental information, and the user triggering a collection instruction for the auxiliary environmental sensor.

[0014] Optionally, the AR glasses are further configured with a posture sensor for determining the user's posture; the environmental information is used to characterize surrounding objects and their positions; and the object information of the user's blind spot and the risk object determined according to the environmental information includes:

[0015] Determine the user's line of sight direction according to the posture sensor, and determine the visual field blind area according to the line of sight direction and the environmental information;

[0016] The objects represented by the environmental information are matched with the risk object set respectively, so as to use the matched objects as the risk objects, and the object information used to represent the risk objects and their positions is determined according to the environmental information.

[0017] Optionally, the displaying of the object information according to the positional relationship between the blind area of ​​vision and the risk object includes:

[0018] determining the location of the risk object according to the object information;

[0019] In the case where the risk object is located in the blind area of ​​vision, displaying the object information based on a map display area corresponding to the environmental information;

[0020] In the case that the risk object is located outside the blind area of ​​the visual field, the object information is displayed based on the visual field coverage area occupied by the risk object in the visual area of ​​the user's visual field.

[0021] Optionally, the method further includes:

[0022] Determine whether the distance between each risk object and the user is less than the minimum safety distance threshold corresponding to the risk object;

[0023] If the judgment result is yes, the corresponding risk warning information is sent to the user.

[0024] According to a second aspect of one or more embodiments of this specification, an electronic device is proposed, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method described in the first aspect by running the executable instructions.

[0025] According to a third aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0026] According to a fourth aspect of one or more embodiments of the present specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0027] In the technical solution provided in this specification, the target environment sensor in the AR glasses can be used to determine the surrounding environmental information, determine the user's blind spot and the object information of the risk object, and then display the above object information according to the positional relationship between the blind spot and the risk object. Among them, AR glasses can help users discover blind spots and perceive potential risk objects in the surrounding area, thereby enhancing the user's perception of the surrounding environment. At the same time, the object information in the AR glasses can be reasonably displayed according to the positional relationship between the blind spot and the risk object without affecting the user's normal line of sight, thereby improving the efficiency of the user in obtaining the above object information and enabling them to respond to risk objects in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 is a schematic diagram of the architecture of a system for displaying object information provided by an exemplary embodiment;

[0030] Figure 2 is a flowchart of a method for displaying object information provided by an exemplary embodiment;

[0031] Figure 3 is a flowchart of another method for displaying object information provided by an exemplary embodiment;

[0032] Figure 4 is a schematic structural diagram of an electronic device provided by an exemplary embodiment;

[0033] Figure 5 It is a structural diagram of a device for displaying object information provided by an exemplary embodiment. DETAILED DESCRIPTION

[0034] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples of devices and methods consistent with some aspects of the present invention.

[0035] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0036] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0037] Figure 1 FIG. 1 is a schematic diagram of the architecture of a display system for object information provided by an exemplary embodiment. Figure 1 As shown, the system includes AR glasses 10 and an environmental sensor 11 configured therein.

[0038] As a wearable device, AR glasses 10 can combine virtual content with the real world by superimposing digital information in the user's field of view. Such a device can be equipped with at least one environmental sensor 11, such as Figure 1 Environmental sensor 1, environmental sensor 2...environmental sensor n, where n is a positive integer. During system operation, the AR glasses 10 can determine the environmental information corresponding to the surrounding environment in real time based on the target environmental sensor and display the relevant information in the corresponding micro display, thereby enhancing the user's perception of the surrounding environment.

[0039] It should be noted that the environmental sensor 11 in this specification specifically refers to a device that collects environmental spatial features to build an environmental model, such as an image sensor, a lidar sensor, or an ultrasonic sensor, etc. The above environmental spatial features can be used to characterize the surrounding terrain, buildings, objects, and obstacles of the user's environment. In addition to the environmental sensor 11, the AR glasses 10 can also be equipped with basic sensors for detecting and measuring physical quantities or chemical properties of the surrounding environment, such as temperature and humidity sensors, light intensity sensors, sound sensors, etc.; biometric sensors for identifying the identity or physiological state of the user, such as iris scanners, facial recognition cameras, and heart rate monitors, etc., which are not limited in this specification.

[0040] Figure 2 The present invention is a flowchart of a method for displaying object information provided by an exemplary embodiment. The method is applied to AR glasses, and the AR glasses are equipped with at least one environmental sensor. The method may include the following steps:

[0041] Step S202: determining surrounding environment information according to the target environment sensor.

[0042] When a user wants to understand the blind spots in the field of vision and detect potential risk objects in the surrounding area while wearing AR glasses, he can control the AR glasses to start the target environment sensor to determine the surrounding environmental information. Among them, the data directly collected by the target environment sensor from the surrounding environment is called raw information. Since the raw information is usually difficult to intuitively describe the characteristics of the environmental space, it needs to be further processed to obtain the corresponding environmental information. The environmental information can be map information corresponding to the user's surrounding environment generated based on Simultaneous Localization And Mapping (SLAM) technology, or it can be object detection and recognition information, distance and obstacle information, three-dimensional modeling information, environmental classification information, and dynamic object information, etc., which can be used to describe the characteristics of the environmental space.

[0043] For example, when the target environment sensor is an image sensor, the AR glasses can use the image captured by the image sensor as raw information, and perform object recognition on the content in the image through image processing algorithms such as convolutional neural networks to determine the objects in the surrounding environment. Furthermore, the raw information can also determine the distance between the user and the real object in the image through binocular stereo vision, monocular ranging and other technologies, which is not limited in this specification.

[0044] For another example, when the target environment sensor is a LiDAR (Light Detection and Ranging), the AR glasses can use a series of point cloud data collected by the LiDAR as raw information. This raw information reflects the time from the emission to the return of the laser beam. Through feature extraction, three-dimensional modeling and other processing, the point cloud data can be transformed into a corresponding environmental model, and objects in the surrounding environment can be identified from it.

[0045] In short, no matter what kind of target environment sensor is used, the corresponding environment information can be obtained. At the same time, in this specification, when the AR glasses are equipped with multiple environment sensors, multiple target environment sensors can be used to obtain more accurate environment information.

[0046] In one embodiment, AR glasses can obtain raw information collected by multiple target environment sensors respectively, and process each raw information according to the sensor fusion algorithm to obtain the above-mentioned environment information. Among them, the sensor fusion algorithm is a method of comprehensively processing data from multiple sensors, such as Kalman filter, particle filter and deep learning model, which aims to improve the accuracy and reliability of data. By fusing information from different target environment sensors, the sensor fusion algorithm can provide more comprehensive and accurate environmental perception. For example: when the target environment sensor is an image sensor and a lidar sensor, and the environment is noisy, the lidar sensor detects a pedestrian but the image sensor does not capture it, the detection results of the same object obtained from the two can be fused, and the detection results of the two can be combined by, for example, weighted average or confidence scoring, and finally the noise in each detection result is eliminated and the recognition accuracy is improved. Furthermore, the above-mentioned deep learning can be used to train and iteratively optimize the fused data to improve the object detection and classification model so as to better adapt to complex environments and scene changes.

[0047] Of course, AR glasses can choose to use a single sensor fusion algorithm to ensure the speed of obtaining environmental information, or use a combination of multiple sensor fusion algorithms to ensure the accuracy of environmental information according to the application scenario. This manual does not go into details about this.

[0048] Among the above-mentioned multiple target environmental sensors, in addition to the environmental sensors configured in the AR glasses, there can also be auxiliary environmental sensors from other devices, thereby enriching the diversity of the above-mentioned original information.

[0049] In one embodiment, the AR glasses can be connected to auxiliary environmental sensors on other devices; and when the auxiliary sensing conditions are met, the environmental sensors of the AR glasses and the original information respectively collected by the auxiliary environmental sensors are obtained. Among them, the AR glasses and the auxiliary environmental sensors can be connected via Bluetooth, wireless LAN, etc. The auxiliary environmental sensors can obtain original information that is difficult to obtain from the environmental sensors in the AR glasses by virtue of their collection positions or collection capabilities that are different from those of the AR glasses. For example, in a vehicle driving scenario, the AR glasses worn by the user are limited by the cockpit space and it is difficult to obtain environmental information similar to that near the bottom of the vehicle. At this time, the AR glasses can be connected to the camera of the vehicle chassis through the vehicle computer, and the image taken by the camera is used as a piece of original information to be fused with the original information collected by other environmental sensors to obtain complete original information near the vehicle body.

[0050] The above-mentioned auxiliary sensing conditions can be used to control and manage the reception status of the AR glasses for the original information collected by the auxiliary environmental sensors, so as to avoid the AR glasses consuming too many extra resources to receive the original information of the auxiliary environmental sensors. The auxiliary sensing conditions can include at least any one of the following conditions: determining that the user is in a high-risk scenario based on the above-mentioned environmental information, and the user triggers a collection instruction for the auxiliary environmental sensor.

[0051] Among them, the so-called high-risk scenario refers to the situation where the AR glasses cannot determine the complete environmental information only by their own environmental sensors, which makes the user prone to risky situations. At this time, the AR glasses can identify the scene where the user is based on the target environmental sensor to determine whether the user is in a high-risk scene, such as the vehicle driving scene or construction scene mentioned above. Corresponding to the high-risk scene may be a low-risk scene, such as a scene of walking on the sidewalk. At this time, even if the AR glasses are connected to other auxiliary environmental sensors, the AR glasses do not need to receive the original information from the auxiliary environmental sensors. Of course, if the user wants to determine the above environmental information in combination with the original information of the above-mentioned auxiliary environmental sensors, he can actively trigger the collection instructions for the above-mentioned auxiliary environmental sensors to obtain the corresponding original information.

[0052] Step S204: determining the user's visual blind spot and object information of the risk object according to the environmental information.

[0053] After obtaining environmental information, AR glasses can determine the user's blind spots based on the environmental information, thereby helping the user to find the blind spots and enhancing the user's perception of the surrounding environment. At the same time, the environmental information can also determine the risk objects with a higher risk level in the corresponding environment, so as to remind the user to pay attention to them later. In addition, this manual does not limit the order of obtaining the above-mentioned blind spots and the above-mentioned object information. AR glasses can process them one by one or in parallel according to user configuration or preset processing rules.

[0054] In addition to the above-mentioned environmental information, the above-mentioned blind area of ​​vision needs to be accurately obtained in combination with the posture sensor used to determine the user's posture in the AR glasses. Considering that the blind area of ​​vision corresponds to the visible area of ​​the user's field of vision, the process of determining the blind area of ​​vision can essentially be regarded as determining the area that does not belong to the above-mentioned visible area of ​​vision. Specifically, the AR glasses can determine the user's line of sight direction according to the above-mentioned posture sensor, and determine the above-mentioned blind area of ​​vision according to the above-mentioned line of sight direction and the above-mentioned environmental information. Among them, the user's current viewing angle can be determined according to the posture information provided by the posture sensor, and then the user's line of sight direction can be determined. Then, based on the position of the object represented by the environmental information and the user's line of sight direction, three-dimensional modeling, computer vision and other technologies are used to analyze which objects may block the user's line of sight, and finally the visible area boundary in the direction of the user's line of sight is calculated, and the area blocked by the object in the current viewing angle and the area outside the current viewing angle are used as the blind area of ​​vision. Of course, in the case where the AR glasses are not equipped with a posture sensor, it can also be replaced by other target environmental sensors at the cost of reducing a certain accuracy, such as the deployment direction of the image sensor to determine the user's current viewing angle.

[0055] The above-mentioned risk objects can be selected from the objects represented by the above-mentioned environmental information, and the selection rules can be implemented based on the risk object set. Specifically, the AR glasses can first match the objects represented by the environmental information with the above-mentioned risk object set respectively, so as to use the matching objects as risk objects, and determine the above-mentioned risk objects and their locations according to the above-mentioned environmental information, and use them as the above-mentioned object information. Among them, the objects in the above-mentioned risk object set can be pre-defined or dynamically changed according to the environmental information. For example, when it is determined according to the environmental information that the user is riding a bicycle, children and the elderly can be used as risk objects in the risk object set, or when it is determined that the user is camping in the wild, wild animals can be used as risk objects in the risk object set. It can be understood that the dynamic nature of the risk object set can ensure that each risk object in the risk object set has a greater probability of appearing among the objects represented by the above-mentioned environmental information, thereby improving the overall matching efficiency.

[0056] In addition to the information about the risk object and its location, other information may be added to the above object information according to actual needs, such as the probability of risk occurrence and the degree of risk impact determined based on the distance between the risk object and the user and the type of the risk object.

[0057] In addition to the above-mentioned risk objects, object information of warning objects with lower risk levels and safety objects with no risk level can also be determined and displayed to users in subsequent steps. The implementation principle is basically the same as the above-mentioned risk objects and will not be elaborated in this manual.

[0058] Step S206: display the object information according to the positional relationship between the blind area and the risk object.

[0059] After the blind area and the object information are determined, the object information can be displayed on the display screen of the AR glasses to prompt the user to pay attention to the risk object in time. The display form and display position of the object information on the display screen can be determined based on the positional relationship between the blind area and the risk object.

[0060] In one embodiment, the AR glasses may first determine the location of the risk object according to the object information, and then, if the risk object is located in the blind spot, display the object information based on the map display area corresponding to the environmental information. Or, if the risk object is located outside the blind spot, display the object information based on the visual coverage area occupied by the risk object in the user's visual field.

[0061] For the first case, it is equivalent to that the risk object does not exist in the visible area of ​​the field of view. At this time, the above object information can be displayed in the map display area corresponding to the above environmental information, so that the user can more conveniently determine the risk object and its location. The map display area can display the map information generated based on the SLAM technology mentioned above. The above map display area itself can be displayed in the form of a floating panel at any position in the display screen of the AR glasses, and the size, shape, transparency and other parameters can be adjusted according to the user's map operation.

[0062] For the second case, it is equivalent to the risk object existing in the visible area of ​​the field of view. At this time, the above object information can be displayed at the location of the above risk object. For example, when the AR glasses recognize that there is a pedestrian 30 meters ahead, the AR glasses can superimpose the corresponding object information on the pedestrian, and provide the object name "pedestrian" and the distance from the user in the object information, and the position of the object information can change as the corresponding risk object moves in the visible area of ​​the field of view. This can help users intuitively determine the risk object and its location. At the same time, the information of any object in the above map display area can be identified by different colors, icons, and arrows to identify different levels of risk or object characteristics.

[0063] As mentioned above, in addition to risk objects, object information can also correspond to other objects such as warning objects or case objects. In the above two cases, whether it is the object information in the above-mentioned map display area or the object information displayed at the location of the above-mentioned risk object, different colors, icons, and arrows can be used to identify different levels of risks or object characteristics, thereby improving the user's recognition speed.

[0064] Of course, different users have different sensitivities to the object information of risk objects. It is possible that a user has already subjectively discovered the object information and the corresponding risk object, but has not responded to the risk in time due to lack of corresponding risk handling awareness. In this regard, AR glasses can send corresponding risk warning information to users based on the distance between the risk object and the user, so as to prompt the user to handle it in time.

[0065] In one embodiment, the AR glasses can first determine whether the distance between each risk object and the user is less than the minimum safety distance threshold corresponding to the risk object. If the judgment result is yes, the corresponding risk warning information is sent to the user. If the judgment result is no, no additional operation is performed, or the risk warning information previously sent is withdrawn. Among them, the risk warning information can remind the user of the potential risk of collision with the risk object by at least one of sound, vibration, or displaying a visual warning signal on the display screen. For example, when the AR glasses recognize that a pedestrian is 30 meters behind and is approaching quickly to within 5 meters of the user, the AR glasses can play a clear warning sound through a speaker or other associated playback device, generate vibration to the body through a vibration motor or a smart bracelet or other associated portable device, or display an obvious warning icon on the corresponding display screen until the user actively confirms the risk warning information, or moves away from the pedestrian approaching from behind by adjusting the route.

[0066] It is understandable that the above-mentioned risk warning information may not cause actual risks in specific scenarios due to misjudgment and other reasons, resulting in users still choosing to ignore the risk warning information after discovering it. AR glasses can record the triggering of the above-mentioned risk warning information and perform learning optimization based on the deep model, ultimately achieving the goal of optimizing the user's experience of using AR glasses while ensuring user safety. For example, if the user repeatedly ignores the risk warning information of a certain risk object, the sending frequency or sensitivity of the risk warning information can be reduced, such as further reducing the above-mentioned minimum safety distance threshold.

[0067] In addition, the warning intensity of the above-mentioned risk warning information may be affected by the type of surrounding environment. For example, if user A is determined to be indoors and user B is in a traffic lane based on different environmental information, then the risk warning information received by the above two users for the same risk object may also be different. Taking the risk object of a pet dog as an example, the risk warning information received by user A may only correspond to one vibration of the AR glasses, and the risk warning information received by user B may be continuous vibrations and alarm sounds. Of course, the above-mentioned warning intensity can also adjust the subsequent feedback frequency and warning intensity based on the user's feedback behavior on the risk warning information.

[0068] Combine the following Figure 3 The overall process of the method for displaying object information is introduced. As shown in the figure, the method includes the following steps:

[0069] Step S302, obtaining the original information collected by the camera and the laser radar respectively.

[0070] In one embodiment, suppose a user is walking on a busy street wearing AR glasses, which are equipped with multiple sensors, including cameras, laser radars, and gesture sensors. At this time, the user turns on the AR glasses and the above sensors in advance, so that the camera starts to capture images of the street, and the laser radar starts to scan the street environment and collect point cloud data. These data are raw information, including image data and point cloud data.

[0071] Step S304, comprehensively processing the multiple original data to obtain environmental information.

[0072] In one embodiment, the AR glasses can process the images captured by the camera and the point cloud data collected by the lidar through a sensor fusion algorithm. For example, a deep learning model is used to identify objects in the image and combined with the point cloud data for three-dimensional modeling to generate environmental information of the street, including the location, distance and type of the object.

[0073] Step S306: determining the user's visual blind spot and object information of the risk object according to the environmental information.

[0074] In one embodiment, AR glasses can combine with gesture sensors to determine the user's line of sight and analyze environmental information to determine blind spots, that is, areas where the user's line of sight is blocked. At the same time, it will also identify risky objects, such as vehicles or pedestrians approaching the user, and determine the location and type of these objects.

[0075] Step S308, determining whether the risk object is located in the blind area of ​​vision.

[0076] In one embodiment, the AR glasses can analyze the location of the risk objects and determine whether they are located in the user's blind spot. If the risk objects are in the blind spot, the user may not be able to see them directly, so step S310 needs to be executed for prompting. If the risk objects are outside the blind spot, step S312 can be executed for prompting.

[0077] Step S310: displaying object information based on the map display area.

[0078] In one embodiment, if the risk objects are located in the blind spot of the field of vision, the AR glasses will display the information of these objects in the map display area of ​​the display screen, such as marking the location and type of the risk objects with icons or text, to help users understand the situation in the blind spot.

[0079] Step S312: display object information based on the field of view coverage area.

[0080] In one embodiment, if the risk object is outside the blind spot, the AR glasses will directly display the object information in the user's field of view, that is, where the user can directly see it. For example, if there is a motor vehicle in front, the AR glasses may display "motor vehicle" and distance information over the motor vehicle.

[0081] Step S314: Whether the distance between the risk object and the user is less than the minimum safety distance threshold.

[0082] In one embodiment, the AR glasses can continuously monitor the distance between the risk object and the user. Taking a motor vehicle as an example, if the distance is less than the preset minimum safety distance threshold of 30 meters, it means that the user may face the risk of collision and needs to execute step S316; otherwise, it can be said that the user is relatively safe and no additional operation is required on the premise of displaying the object information.

[0083] Step S316, sending corresponding risk warning information to the user.

[0084] In one embodiment, once the AR glasses detect that the distance between the risk object and the user is less than the minimum safety distance threshold, the AR glasses will send risk warning information to the user through sound, vibration or visual warning, reminding the user to pay attention and take action to avoid potential collision.

[0085] Figure 4 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0086] Corresponding to the above-mentioned embodiment of the method for displaying object information, this specification also provides an embodiment of a device for displaying object information. Figure 5 The object information display device is applied to AR glasses, and the AR glasses are equipped with at least one environmental sensor; the device includes:

[0087] An environmental information determination unit 502 is used to determine surrounding environmental information according to a target environmental sensor;

[0088] An object information determination unit 504 is used to determine the user's visual blind spot and the object information of the risk object according to the environmental information;

[0089] The object information display unit 506 is used to display the object information according to the positional relationship between the blind area and the risk object.

[0090] Optionally, there are multiple types of target environment sensors; and the device further includes:

[0091] The original information acquisition unit is used to acquire the original information respectively collected by various target environment sensors;

[0092] The environmental information fusion unit is used to process each raw information according to a sensor fusion algorithm to obtain the environmental information.

[0093] Optionally, the AR glasses are connected to auxiliary environmental sensors on other devices; the original information acquisition unit is specifically used to:

[0094] When the auxiliary sensing conditions are met, original information collected by the environmental sensor of the AR glasses and the auxiliary environmental sensor are obtained.

[0095] Optionally, the auxiliary sensing condition includes at least any one of the following: determining that the user is in a high-risk scenario based on the environmental information, and the user triggering a collection instruction for the auxiliary environmental sensor.

[0096] Optionally, the AR glasses are further configured with a posture sensor for determining the user's posture; the environmental information is used to characterize surrounding objects and their positions; the object information determination unit 504 is specifically used to:

[0097] Determine the user's line of sight direction according to the posture sensor, and determine the visual field blind area according to the line of sight direction and the environmental information;

[0098] The objects represented by the environmental information are matched with the risk object set respectively, so as to use the matched objects as the risk objects, and the object information used to represent the risk objects and their positions is determined according to the environmental information.

[0099] Optionally, the object information display unit 506 is specifically configured to:

[0100] determining the location of the risk object according to the object information;

[0101] In the case where the risk object is located in the blind area of ​​vision, displaying the object information based on a map display area corresponding to the environmental information;

[0102] In the case that the risk object is located outside the blind area of ​​the visual field, the object information is displayed based on the visual field coverage area occupied by the risk object in the visual area of ​​the user's visual field.

[0103] Optionally, the device further comprises:

[0104] Determine whether the distance between each risk object and the user is less than the minimum safety distance threshold corresponding to the risk object;

[0105] If the judgment result is yes, the corresponding risk warning information is sent to the user.

[0106] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0107] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0108] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0109] Although the present invention includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments of the present invention may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0110] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0111] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for displaying object information, characterized in that: Applied to AR glasses, the AR glasses are equipped with at least one environmental sensor; the method comprises: Determine surrounding environmental information based on target environmental sensors; Determine the user's visual blind spot and object information of the risk object according to the environmental information; The object information is displayed according to the positional relationship between the blind area and the risk object.

2. The method according to claim 1, characterized in that There are multiple types of target environment sensors; determining surrounding environment information according to the target environment sensors includes: Obtain the original information collected by various target environment sensors; Each raw information is processed according to a sensor fusion algorithm to obtain the environmental information.

3. The method according to claim 2, characterized in that The AR glasses are connected to auxiliary environmental sensors on other devices; the original information collected by multiple target environmental sensors is obtained, including: When the auxiliary sensing conditions are met, original information collected by the environmental sensor of the AR glasses and the auxiliary environmental sensor are obtained.

4. The method according to claim 3, characterized in that: The auxiliary sensing condition includes at least any one of the following: determining that the user is in a high-risk scenario based on the environmental information, and the user triggering a collection instruction for the auxiliary environmental sensor.

5. The method according to claim 1, characterized in that The AR glasses are also equipped with a posture sensor for determining the user's posture; the environmental information is used to characterize surrounding objects and their positions; the object information of the user's blind spot and the risk object determined according to the environmental information includes: Determine the user's line of sight direction according to the posture sensor, and determine the visual field blind area according to the line of sight direction and the environmental information; The objects represented by the environmental information are matched with the risk object set respectively, so as to use the matched objects as the risk objects, and the object information used to represent the risk objects and their positions is determined according to the environmental information.

6. The method according to claim 1, characterized in that The displaying of the object information according to the positional relationship between the blind area and the risk object includes: determining the location of the risk object according to the object information; In the case where the risk object is located in the blind area of ​​vision, displaying the object information based on a map display area corresponding to the environmental information; In the case that the risk object is located outside the blind area of ​​the visual field, the object information is displayed based on the visual field coverage area occupied by the risk object in the visual area of ​​the user's visual field.

7. The method according to claim 1, characterized in that The method further comprises: Determine whether the distance between each risk object and the user is less than the minimum safety distance threshold corresponding to the risk object; If the judgment result is yes, the corresponding risk warning information is sent to the user.

8. An electronic device, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 7 by executing the executable instructions.

9. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images