Head-mounted device risk early warning method, controller, head-mounted device and medium

Abstract

The invention discloses a head-mounted device risk early warning method, a controller, a head-mounted device and a storage medium. The method comprises the following steps: acquiring infrared distance data and visual image data acquired by head-mounted equipment; performing fusion analysis processing on the infrared distance data and the visual image data, and determining a target obstacle and an obstacle attribute corresponding to the target obstacle; determining a three-dimensional safety space model corresponding to the target obstacle based on the target obstacle and the obstacle attribute corresponding to the target obstacle; based on the three-dimensional safety space model, determining a collision risk level corresponding to the target obstacle; and carrying out risk early warning based on the collision risk level. According to the method, risk early warning can be effectively carried out in real time, so that a user is guided to avoid the collision risk of the target obstacle in time, the risks of damage to the head-mounted equipment and injury to the user are reduced, the safety and reliability of the head-mounted equipment in the using process are effectively improved, and the method has high application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of virtual reality, and in particular to a head-mounted device risk early warning method, a controller, a head-mounted device and a storage medium. BACKGROUND

[0002] With the continuous development of virtual reality technology, virtual reality technology is widely used in game entertainment, education and training, industrial design and medical health and other fields, and shows great application potential. However, in the virtual reality scene, when the user uses the head-mounted device, the vision is occupied by the virtual environment, and it is often difficult to perceive the obstacles (such as walls, furniture and animals, etc.) in the real physical environment, which may cause collision risks between the user and the obstacles. If a collision occurs, not only will the device used by the user be damaged, but also the user will be injured. Therefore, how to timely perform collision risk early warning to reduce the collision risk with obstacles is a technical problem to be solved at present. SUMMARY

[0003] The embodiments of the present application provide a head-mounted device risk early warning method, a controller, a head-mounted device and a storage medium to solve the technical problem of how to timely perform collision risk early warning to reduce the collision risk of the head-mounted device and the user with obstacles.

[0004] A head-mounted device risk early warning method, comprising: acquiring infrared distance data and visual image data collected by a head-mounted device; performing fusion analysis and processing on the infrared distance data and the visual image data to determine a target obstacle and an obstacle attribute corresponding to the target obstacle; determining a three-dimensional safety space model corresponding to the target obstacle based on the target obstacle and the obstacle attribute corresponding to the target obstacle; determining a collision risk level corresponding to the target obstacle based on the three-dimensional safety space model; performing risk early warning based on the collision risk level.

[0005] Preferably, the fusion analysis and processing of the infrared distance data and the visual image data to determine the target obstacle and the obstacle attribute corresponding to the target obstacle comprises: performing fusion processing on the infrared distance data and the visual image data to determine target fusion data; adopting a pre-trained deep learning model to identify and analyze the target fusion data to determine the target obstacle and the obstacle attribute corresponding to the target obstacle.

[0006] Preferably, the obstacle attribute comprises an obstacle distance, an obstacle profile, and an obstacle category; The method comprises: Determining a relative position relationship between the target obstacle and the head-mounted device based on the obstacle distance; Determining a three-dimensional obstacle model corresponding to the target obstacle based on the obstacle category and the obstacle profile; Determining a three-dimensional safety space model corresponding to the target obstacle based on the relative position relationship and the three-dimensional obstacle model corresponding to the target obstacle.

[0007] Preferably, the method further comprises: Determining obstacle motion data corresponding to the target obstacle and user motion data corresponding to the head-mounted device based on the three-dimensional obstacle model; Determining a distance factor corresponding to the target obstacle based on the obstacle distance in the three-dimensional safety space model; Determining a speed factor corresponding to the target obstacle and a predicted trajectory intersection probability based on the obstacle motion data and the user motion data; Determining a danger degree of the obstacle based on the obstacle category in the three-dimensional safety space model; Determining a collision risk level corresponding to the target obstacle based on the distance factor, the speed factor, the predicted trajectory intersection probability, and the danger degree of the obstacle.

[0008] Preferably, the obstacle motion data comprises an obstacle speed, and the user motion data comprises a user speed; The method further comprises: Analyzing and processing the obstacle speed and the user speed to determine a speed factor corresponding to the target obstacle; Performing trajectory prediction based on the obstacle motion data and the user motion data to determine a predicted user motion trajectory and a predicted obstacle motion trajectory, and determining a predicted trajectory intersection probability based on the predicted user motion trajectory and the predicted obstacle motion trajectory.

[0009] Preferably, the method further comprises: weighting processing is performed on the distance factor, the speed factor, the predicted trajectory intersection probability and the obstacle danger degree to determine a collision risk score corresponding to the target obstacle; If the collision risk score is between a first preset threshold and a second preset threshold, it is determined that a collision risk level corresponding to the target obstacle is a low risk level; If the collision risk score is between the second preset threshold and a third preset threshold, it is determined that the collision risk level corresponding to the target obstacle is a medium risk level; If the collision risk score is greater than the third preset threshold, it is determined that the collision risk level corresponding to the target obstacle is a high risk level. The first preset threshold is less than the second preset threshold, and the second preset threshold is less than the third preset threshold.

[0010] Preferably, the risk warning is performed based on the collision risk level, including: If the collision risk level is a low risk level, at least one of a primary visual warning, a primary auditory warning and a primary tactile warning is performed; If the collision risk level is a medium risk level, at least one of a medium visual warning, a medium auditory warning and a medium tactile warning is performed on the user; If the collision risk level is a high risk level, at least one of a high visual warning, a high auditory warning and a high tactile warning is performed on the user.

[0011] A controller, including a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the above-mentioned head-mounted device risk warning method when executing the computer program.

[0012] A head-mounted device, including an infrared sensor, a camera device, a reminder device and the above-mentioned controller, the controller being connected to the infrared sensor for acquiring infrared distance data, the controller being connected to the camera device for acquiring visual image data, the controller being connected to the reminder device for controlling the reminder device to perform risk warning according to the infrared distance data and the visual image data.

[0013] A computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the above-mentioned head-mounted device risk warning method.

[0014] The head-mounted device risk early warning method, the controller, the head-mounted device and the storage medium comprehensively consider infrared distance data between objects in the surrounding environment and the head-mounted device and visual image data in the surrounding environment, can accurately determine target obstacles in the surrounding environment and obstacle attributes corresponding to each target obstacle, determine a three-dimensional safety space model according to the target obstacles and the obstacle attributes, and can directly and efficiently determine a collision risk level between each target obstacle and a user using the head-mounted device according to the three-dimensional safety space model, so as to perform risk early warning in real time and effectively according to the collision risk level, guide the user to avoid the collision risk of the target obstacle in time, reduce the risk of damage to the head-mounted device and injury to the user, effectively improve the safety and reliability of the head-mounted device in use, and have high application value. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 is a flowchart of a head-mounted device risk early warning method in an embodiment of the present application; Figure 2 is another flowchart of a head-mounted device risk early warning method in an embodiment of the present application; Figure 3 is another flowchart of a head-mounted device risk early warning method in an embodiment of the present application; Figure 4 is another flowchart of a head-mounted device risk early warning method in an embodiment of the present application; Figure 5 is another flowchart of a head-mounted device risk early warning method in an embodiment of the present application; Figure 6 is another flowchart of a head-mounted device risk early warning method in an embodiment of the present application; Figure 7 is another flowchart of a head-mounted device risk early warning method in an embodiment of the present application; Figure 8 is another flowchart of a head-mounted device risk early warning method in an embodiment of the present application; Figure 9 is a schematic diagram of a controller in an embodiment of the present application; Figure 10 is a schematic diagram of part of the hardware layout in the VR glasses in an embodiment of the application; Figure 11 is a schematic diagram of a low-risk level early warning of the head-mounted device in an embodiment of the application; Figure 12 is a schematic diagram of a medium-risk level early warning of the head-mounted device in an embodiment of the application; Figure 13 is a schematic diagram of a high-risk level early warning of the head-mounted device in an embodiment of the application; Among them, 1001 is an infrared sensor; 1002 is a camera device; 1003 is a controller; and 1004 is an inertial measurement unit. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some but not all of the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the application.

[0018] The embodiment of the application provides a head-mounted device risk early warning method. The head-mounted device risk early warning method determines a target obstacle in a surrounding physical environment by acquiring infrared distance data and visual image data collected by a head-mounted device, and determines a collision risk level between the target obstacle and the head-mounted device. According to the collision risk level, risk early warning is performed to guide a user using the head-mounted device to avoid collision with the target obstacle in time, reduce the collision risk between the target obstacle and the head-mounted device, and reduce the collision risk between the target obstacle and the user using the head-mounted device. The head-mounted device risk early warning method is used to perform collision risk early warning in time to reduce the collision risk between the user and the obstacle, improve the safety performance of the user when using the head-mounted device, and also has the effect of protecting the head-mounted device.

[0019] In an embodiment, as shown in Figure 1 , a head-mounted device risk early warning method is provided. The controller in Figure 9 is taken as an example to illustrate the method, which includes the following steps: S101: acquiring infrared distance data and visual image data collected by a head-mounted device; S102: performing fusion analysis and processing on the infrared distance data and the visual image data to determine a target obstacle and an obstacle attribute corresponding to the target obstacle; S103: determining a three-dimensional safety space model corresponding to the target obstacle based on the target obstacle and the obstacle attribute corresponding to the target obstacle; S104: Determine the collision risk level corresponding to the target obstacle based on the three-dimensional safety space model; S105: Issue a risk warning based on the collision risk level.

[0020] Among them, head-mounted devices refer to wearable virtual reality hardware in the field of virtual reality technology, such as VR (Virtual Reality) glasses. Infrared distance data refers to the distance data between objects around the head-mounted device and the head-mounted device, which is collected by the infrared sensor installed on the head-mounted device. Visual image data refers to the image data of the physical environment around the head-mounted device, which is collected by the camera device installed on the head-mounted device. Figure 10 The following is a schematic diagram of the hardware layout of a VR glasses. Figure 10 In the figure, the infrared sensor 1001 is used to collect infrared distance data, and the camera device 1002 is used to collect visual image data.

[0021] As an example, in step S101, the controller obtains infrared distance data collected by a head-mounted device (HMD) between each object in the user's surrounding physical environment and the HMD, as well as visual image data corresponding to the surrounding physical environment. In this example, a user uses a HMD that includes an infrared sensor and a camera. During use of the HMD, the infrared sensor collects distance data between each object in the user's surrounding physical environment and the user, and transmits this distance data as infrared distance data to the controller. The camera collects image data from the user's surrounding physical environment and transmits this data as visual image data to the controller. The visual image data reflects information such as the shape, texture, and category of each object in the surrounding physical environment, and is used to identify target obstacles in the user's surrounding physical environment. In this example, infrared distance data and visual image data collected by the HMD that reflect the conditions of the surrounding physical environment are obtained to facilitate the subsequent identification of target obstacles and obstacle attributes in the surrounding physical environment based on the infrared distance data and visual image data.

[0022] Among them, target obstacles refer to objects that pose a risk of collision with the head-mounted device. It is understandable that when a user uses a head-mounted device, due to the movement of the user and / or the movement of objects in the surrounding physical environment, there is a risk of collision between the head-mounted device and the user using the head-mounted device and objects in the surrounding physical environment. Objects that may collide in the surrounding physical environment are identified and determined as target obstacles, so as to monitor the target obstacles in real time, provide timely warnings to users using the head-mounted device, and reduce the risk of collision. Obstacle attributes refer to the attributes of the target obstacles, including but not limited to the location, type, and outline of the target obstacle.

[0023] As an example, in step S102, the controller uses the pre-trained deep learning model to perform fusion analysis and processing on the collected infrared distance data and visual image data, identifies at least one target obstacle in the surrounding physical environment that has a potential collision risk with the head-mounted device according to the infrared distance data and the visual image data, and identifies the attributes of each target obstacle through fusion analysis and processing of the infrared distance data and the visual image data to determine the obstacle attributes corresponding to each target obstacle. Understandably, after the user uses the head-mounted device to enter the virtual environment, in order to determine the collision risk level between each target obstacle in the surrounding physical environment of the head-mounted device and the head-mounted device, it is necessary to obtain the obstacle attributes corresponding to each target obstacle, such as whether the category of the target obstacle is a stationary obstacle, whether the position of the target obstacle is close to the head-mounted device, and the size of the outline of the target obstacle, so as to timely warn the user using the head-mounted device according to the above obstacle attributes, so as to guide the user to effectively avoid the target obstacle and reduce the collision risk.

[0024] Among them, the three-dimensional safety space model refers to a space model containing the target obstacle and the user using the head-mounted device, which is used to reflect the collision risk level between the user and the target obstacle in real time.

[0025] As an example, in step S103, the controller constructs a three-dimensional safety space model between the user using the head-mounted device and the target obstacle according to the identified obstacle attributes corresponding to the target obstacle. In this example, the number of target obstacles is at least one. In this example, the controller obtains all target obstacles in the surrounding environment and all obstacle attributes corresponding to the target obstacles in real time, and constructs a three-dimensional safety space model in real time, which is used to reflect the relationship between the user and each target obstacle in real time, so as to determine the collision risk level between each target obstacle and the user in real time. Understandably, since the three-dimensional safety space model is determined based on the target obstacle and the obstacle attributes corresponding to the target obstacle, the three-dimensional safety space model contains the obstacle attributes corresponding to each target obstacle, and the three-dimensional safety space model is a model of the three-dimensional space established with the user using the head-mounted device as the center, so the three-dimensional safety space model can more intuitively identify the environment around the user using the head-mounted device, and more intuitively reflect the relationship between each target obstacle and the user, so that the subsequent collision risk prediction of each target obstacle based on the three-dimensional safety space model is efficient.

[0026] Among them, the collision risk level is used to represent the degree of collision risk between the target obstacle and the user using the head-mounted device.

[0027] As an example, in step S104, the controller analyzes the obstacle attribute corresponding to the target obstacle in the three-dimensional safety space model, determines the distance between the target obstacle and the user using the head-mounted device and the speed of the target obstacle, and other factors affecting the collision risk, analyzes the influencing factors of the above collision risk, and determines the collision risk level between the target obstacle and the user using the head-mounted device. For example, if the distance between the target obstacle and the user is greater than the preset distance threshold, and the speed of the target obstacle is greater than the preset speed threshold, the collision risk level is determined as a high risk level; if the distance between the target obstacle and the user is greater than the preset distance threshold, or the speed of the target obstacle is greater than the preset speed threshold, the collision risk level is determined as a medium risk level, and if the distance between the target obstacle and the user is not greater than the preset distance threshold, and the speed of the target obstacle is not greater than the preset speed threshold, the collision risk level is determined as a low risk level. In this example, according to the three-dimensional safety space model, the collision risk level between the target obstacle and the user using the head-mounted device can be predicted in real time, intuitively and efficiently, and since the three-dimensional safety space model contains real-time determined obstacle attributes, the collision risk level between the target obstacle and the user using the head-mounted device can be predicted more accurately through the three-dimensional safety space model.

[0028] As an example, in step S105, after determining the collision risk level corresponding to the target obstacle, the controller warns the user using the head-mounted device of different levels of risk through the head-mounted device according to the size of the collision risk level, to prompt the user that there is different degree of collision risk between the user and the target obstacle, to guide the user to avoid collision with the target obstacle in time. In this example, if there are multiple target obstacles in the three-dimensional safety space model, steps S101 to S104 are used to determine the collision risk level between each target obstacle and the user using the head-mounted device in real time and accurately, and when the controller determines the collision risk level corresponding to multiple target obstacles, the risk warning is performed according to the highest collision risk level, for example, the highest collision risk level among the collision risk levels corresponding to multiple target obstacles is a medium risk level, and the risk warning is performed on the user using the head-mounted device according to the medium risk level. In this example, the user using the head-mounted device is warned of the risk according to the collision risk level between the target obstacle and the user using the head-mounted device, which helps the user using the head-mounted device to avoid collision risk in time, reduces the risk of damage to the head-mounted device and injury to the user, and effectively improves the safety of the head-mounted device during use.

[0029] In this embodiment, the infrared distance data and the visual image data collected by the head-mounted device are fused and analyzed, the infrared distance data between the objects in the surrounding environment and the head-mounted device and the visual image data in the surrounding environment are comprehensively considered, the target obstacles in the surrounding environment and the obstacle attributes corresponding to each target obstacle can be accurately determined, the three-dimensional safety space model is determined according to the target obstacles and the obstacle attributes, since the three-dimensional safety space model includes the obstacle attributes corresponding to all target obstacles in the surrounding environment, the collision risk level between each target obstacle and the user using the head-mounted device can be directly and efficiently determined according to the three-dimensional safety space model, so as to realize real-time and effective risk warning according to the collision risk level, to guide the user to avoid the collision risk of the target obstacle in time, to reduce the risk of damage of the head-mounted device and injury of the user, to effectively improve the safety and reliability of the head-mounted device in use, and to have high application value.

[0030] In an embodiment, as shown in Figure 2 Step S102, i.e., the fusion analysis and processing of the infrared distance data and the visual image data to determine the target obstacles and the obstacle attributes corresponding to the target obstacles, includes: S201: performing fusion processing on the infrared distance data and the visual image data to determine target fusion data; S202: identifying and analyzing the target fusion data by using a pre-trained deep learning model to determine the target obstacles and the obstacle attributes corresponding to the target obstacles.

[0031] The target fusion data refers to the data fused from the infrared distance data and the visual image data.

[0032] As an example, in step S201, the controller fuses the infrared distance data and the visual image data to obtain the fused data, and determines the fused data as the target fusion data. It can be known that Figure 10 The head-mounted device includes a plurality of camera devices and a plurality of infrared sensors for simultaneously collecting a plurality of visual image data and a plurality of infrared distance data in the surrounding environment.

[0033] For example, the controller can fuse the plurality of visual image data by using decision-level fusion or feature-level fusion to determine image fusion data corresponding to the visual image data, and fuse the image fusion data and the plurality of infrared distance data by using early fusion or late fusion to determine the target fusion data. For another example, the controller directly processes the plurality of visual image data and the plurality of infrared distance data by using early fusion or late fusion to determine the target fusion data.

[0034] In this example, the target fusion data is determined by fusing the infrared distance data and the visual image data, the advantages of the infrared distance data collected by the infrared sensor and the visual image data collected by the camera are comprehensively utilized, the comprehensiveness and accuracy of the physical environment perception of the head-mounted device are improved, and the target fusion data which can more comprehensively and accurately reflect the physical environment is obtained.

[0035] The pre-trained deep learning model is used to determine the model of the target obstacle in the target fusion data.

[0036] As an example, in step S202, the controller inputs the real-time collected target fusion data into the pre-trained deep learning model, performs obstacle recognition and obstacle attribute recognition on the target fusion data based on the deep learning model, determines the target obstacle, and outputs the obstacle attribute corresponding to the target obstacle. In this example, the pre-trained deep learning model includes at least one of a pre-trained convolutional neural network, a pre-trained recurrent neural network, a pre-trained long short-term memory network, and a pre-trained Transformer model. The pre-trained convolutional neural network is used for image feature extraction, the pre-trained recurrent neural network and the pre-trained long short-term memory network are used for processing sequence data to predict the motion trend of the target obstacle, and the pre-trained Transformer model is used to capture the global dependency of the target fusion data. Through the pre-trained deep learning model, the target obstacle in the target fusion data and the obstacle attribute corresponding to the target obstacle can be accurately identified. In this example, the main task of the pre-trained convolutional neural network is to detect the target obstacle in the target fusion data, determine the target obstacle, and perform obstacle segmentation, classification, positioning, and ranging on the target obstacle to more accurately determine the obstacle attribute. The obstacle attribute includes but is not limited to the category of the target obstacle, the contour of the target obstacle, the distance between the target obstacle and the head-mounted device, and the speed of the target obstacle.

[0037] In this embodiment, the infrared distance data and the visual image data are fused to determine the target fusion data which can more comprehensively perceive the physical environment. The pre-trained deep learning model is used to identify and analyze the more comprehensive target fusion data, which can more accurately identify the target obstacle and the obstacle attribute in the physical environment, so as to subsequently construct a more accurate and reliable three-dimensional safety space model based on the target obstacle and the obstacle attribute.

[0038] In an embodiment, as shown in Figure 3 Step S201 of fusing the infrared distance data and the visual image data to determine the target fusion data includes: S301: pre-process the infrared distance data and the visual image data to determine target distance data corresponding to the infrared distance data and target image data corresponding to the visual image data; S302: perform fusion processing on the target distance data and the target image data to determine target fusion data.

[0039] The pre-processing refers to the data processing manner of the infrared distance data and the visual image data, including but not limited to noise filtering, data calibration, and timestamp alignment of the infrared distance data and the visual image data. The target distance data refers to the data obtained after pre-processing the infrared distance data. The target image data refers to the data obtained after pre-processing the visual image data.

[0040] As an example, in step S301, the controller pre-processes the infrared distance data in the manner of noise filtering, data calibration, and timestamp alignment to obtain target distance data corresponding to the infrared distance data, and pre-processes the visual image data in the manner of noise filtering, data calibration, and timestamp alignment to obtain target image data corresponding to the visual image data. In this example, pre-processing the infrared distance data and the visual image data can effectively reduce the noise data in the infrared distance data and the visual image data, so that the target distance data and the target image data can accurately reflect the actual situation of the physical environment, to facilitate subsequent obtaining of more comprehensive target fusion data based on the target distance data and the target image data.

[0041] As an example, in step S302, the controller performs fusion processing on the target distance data and the target image data obtained by pre-processing to obtain fusion target fusion data. In this example, as shown in FIG. 5, since the VR device is provided with multiple infrared sensors 1001 and multiple camera devices 1002, the number of infrared distance data and visual image data collected in real time is also multiple. This method performs fusion processing on the target distance data corresponding to the multiple infrared distance data and the target image data corresponding to the multiple visual image data to obtain fusion data that can comprehensively reflect the surrounding physical environment. Figure 10

[0042] In this embodiment, pre-processing the infrared distance data and the visual image data can obtain target distance data and target image data that accurately reflect the actual situation of the physical environment, and performing fusion processing on the target distance data and the target image data can obtain target fusion data that can comprehensively reflect the surrounding physical environment, to facilitate subsequent accurate identification of target obstacles and obstacle attributes in the surrounding physical environment based on the target fusion data.

[0043] In an embodiment, the obstacle attribute includes obstacle distance, obstacle contour, and obstacle category. ​

[0044] wherein the obstacle distance refers to a distance between the target obstacle and the head-mounted device. The obstacle profile refers to a profile of the target obstacle. The obstacle category refers to a category to which the target obstacle belongs, such as a category of furniture, animal, and wall.

[0045] In an embodiment, as shown in FIG. 1, the step S103 of determining the three-dimensional safety space model corresponding to the target obstacle based on the target obstacle and the obstacle attribute corresponding to the target obstacle comprises: Figure 4 S401: determining a relative position relationship between the target obstacle and the head-mounted device based on the obstacle distance; S402: determining a three-dimensional obstacle model corresponding to the target obstacle based on the obstacle category and the obstacle profile; S403: determining the three-dimensional safety space model corresponding to the target obstacle based on the relative position relationship and the three-dimensional obstacle model corresponding to the target obstacle.

[0046] wherein the relative position relationship refers to a position relationship between the target obstacle and the head-mounted device in the three-dimensional safety space model.

[0047] As an example, in the step S401, the controller determines the relative position relationship between each target obstacle and the head-mounted device in a center of the head-mounted device and a user using the head-mounted device after determining the obstacle distance between each target obstacle and the head-mounted device. For example, if the obstacle distance between a target obstacle and the head-mounted device is 5 m, an X-O-Y coordinate system with the head-mounted device and the user using the head-mounted device as a coordinate origin (0, 0) is established, and the relative position relationship between the target obstacle and the head-mounted device is (3, 4) in units of meters (m). The controller determines the relative position relationship between each target obstacle and the head-mounted device in the above manner.

[0048] wherein the three-dimensional obstacle model refers to a model of the target obstacle in a three-dimensional space.

[0049] As an example, in the step S402, the controller determines a three-dimensional obstacle model of the target obstacle in a three-dimensional space according to the obstacle category and the obstacle profile of the target obstacle. For example, for a target obstacle with an obstacle category of furniture, the controller draws a three-dimensional obstacle model of the target obstacle as furniture in a three-dimensional space according to the obstacle profile.

[0050] ​As an example, in step S403, the controller places the three-dimensional obstacle model corresponding to each target obstacle at the relative position corresponding to each target obstacle, to obtain a three-dimensional safety space model centered on the head-mounted device and the user using the head-mounted device, in which the three-dimensional obstacle model corresponding to each target obstacle is arranged at the relative position. In this example, the controller places the center of gravity of the three-dimensional obstacle model corresponding to each target obstacle at the relative position of the target obstacle in the X-O-Y coordinate system, in which the three-dimensional space coordinate system is expanded to obtain the three-dimensional obstacle model corresponding to each target obstacle in the three-dimensional space coordinate system, forming a three-dimensional safety space model centered on the head-mounted device and the user using the head-mounted device, containing the three-dimensional obstacle model of each target obstacle, which can more intuitively reflect the obstacle attribute corresponding to each target obstacle. In this example, the three-dimensional safety space model includes but is not limited to point cloud map, voxel grid map or other real-time update data structure. In this example, as shown in Figure 10 As shown in the figure, the controller determines whether the user using the head-mounted device (VR glasses) rotates the head posture or generates motion in real time through the inertial measurement unit 1004, and when it is determined that the user rotates the head posture or generates motion, the infrared sensor 1001 and the camera 1002 are used to repeatedly execute steps S101 to S102 in real time, to obtain infrared distance data and visual image data in real time, and to determine the target obstacle and the obstacle attribute corresponding to the target obstacle in real time, and to update the three-dimensional safety space model in real time according to the real-time determination of the target obstacle and the obstacle attribute corresponding to the target obstacle, so as to subsequently determine the collision risk level efficiently and accurately according to the real-time three-dimensional safety space model. Understandably, when predicting the collision risk, the higher the real-time of the data used for risk prediction, the more accurate the risk prediction, therefore, when the user rotates the head posture or generates motion, the infrared distance data and the visual image data are obtained in real time, and the three-dimensional safety space model is updated in real time, so that the collision risk level can be accurately determined according to the real-time updated three-dimensional safety space model subsequently. This method can automatically update the three-dimensional safety space model in real time without human intervention, which can effectively improve the ease of use of the head-mounted device and the degree of freedom of the user's virtual reality experience.

[0051] In this embodiment, a three-dimensional safety space model capable of intuitively reflecting the obstacle attribute corresponding to the target obstacle is constructed, which does not limit the physical environment in terms of area, and can intuitively reflect the safety area in which there is no target obstacle in the physical environment while intuitively representing the target obstacle and the obstacle attribute corresponding to the target obstacle, and can effectively guide the user using the head-mounted device to move to the safety area while efficiently determining the collision risk level according to the intuitively represented target obstacle and the obstacle attribute corresponding to the target obstacle, to achieve the purpose of avoiding the target obstacle.

[0052] In an embodiment, as shown in Figure 5 step S104, determining the collision risk level corresponding to the target obstacle based on the three-dimensional safety space model comprises: S501: determining the obstacle motion data corresponding to the target obstacle and the user motion data corresponding to the head-mounted device based on the three-dimensional obstacle model; S502: determining the distance factor corresponding to the target obstacle based on the obstacle distance in the three-dimensional safety space model; S503: determining the speed factor and the predicted trajectory intersection probability corresponding to the target obstacle based on the obstacle motion data and the user motion data; S504: determining the obstacle danger degree based on the obstacle category in the three-dimensional safety space model; S505: determining the collision risk level corresponding to the target obstacle based on the distance factor, the speed factor, the predicted trajectory intersection probability, and the obstacle danger degree.

[0053] Wherein, the obstacle motion data refers to the data generated by the target obstacle in the process of motion. The user motion data refers to the data generated by the user using the head-mounted device in the process of motion.

[0054] As an example, step S501, the controller determines the obstacle motion data such as obstacle speed, obstacle acceleration, and current obstacle position generated by the target obstacle in the process of motion according to the three-dimensional obstacle model in the real-time updated three-dimensional safety space model, and determines the user motion data such as user speed, user acceleration, and current user position generated by the user using the head-mounted device in the process of motion in the three-dimensional safety space model through the inertial measurement unit 1004 in real time. Wherein, the obstacle speed refers to the speed of the target obstacle in the process of motion. The obstacle acceleration refers to the acceleration of the target obstacle in the process of motion. The current obstacle position refers to the position of the target obstacle at each moment in the process of motion. If the target obstacle is a static obstacle, the obstacle speed and the obstacle acceleration are both 0, and the current obstacle position generally does not change. The user speed refers to the speed of the user in the process of motion. The user acceleration refers to the acceleration of the user in the process of motion. The current user position refers to the position of the user at each moment in the process of motion. Understandably, since the target obstacle and the user using the head-mounted device may generate different motion data at different moments, the obstacle motion data and the user motion data in the three-dimensional safety space model are acquired in real time, so as to accurately determine the collision risk level of the target obstacle and the head-mounted device and the user using the head-mounted device.

[0055] Wherein, the distance factor refers to the factor affecting the collision risk level and related to the obstacle distance.

[0056] As an example, at step S502, the controller inversely processes the obstacle distance of each target obstacle in the three-dimensional safety space model to determine a distance factor corresponding to each target obstacle. For example, the distance factor corresponding to the target obstacle is Alternatively, the distance factor corresponding to the target obstacle is where s is the obstacle distance of the target obstacle. Understandably, the greater the obstacle distance between the target obstacle and the user using the head-mounted device, the less likely the collision between the target obstacle and the user, and thus, the distance factor obtained by inversely processing the obstacle distance of the target obstacle can effectively represent the collision risk between the target obstacle and the user.

[0057] where the speed factor refers to a factor affecting the collision risk level and related to the speed. The predicted trajectory intersection probability refers to the probability that the future position of the target obstacle intersects with the future position of the user using the head-mounted device. The future position refers to the position at any time in the future.

[0058] As an example, at step S503, the controller processes the obstacle motion data corresponding to each target obstacle and the user motion data corresponding to the user using the head-mounted device to determine the speed factor corresponding to each target obstacle, processes the obstacle motion data corresponding to each target obstacle and the user motion data corresponding to the user using the head-mounted device to predict the future position of each target obstacle and the future position of the user using the head-mounted device, and analyzes the future position of each target obstacle and the future position of the user using the head-mounted device to determine the predicted trajectory intersection probability between each target obstacle and the user using the head-mounted device. Understandably, since the speed affects the risk of collision, the speed factor is determined according to the obstacle motion speed and the user motion speed, so as to accurately determine the collision risk level according to the speed factor. When the user using the head-mounted device is in the virtual scene, if the positions of the user and the target obstacle in the physical scene are relatively close, it indicates that the user and the target obstacle are more likely to collide, and thus, the predicted trajectory intersection probability between the user and the target obstacle is predicted according to the obstacle motion speed and the user motion speed, so as to accurately determine the collision risk level between the user and the target obstacle according to the predicted trajectory intersection probability.

[0059] where the obstacle danger degree refers to the influence degree of the target obstacle on the collision risk.

[0060] As an example, in step S504, the controller queries a pre-stored mapping table of obstacle categories and obstacle hazard levels in the system database based on the obstacle category corresponding to each target obstacle in the three-dimensional safe space model, and determines the obstacle hazard level corresponding to the target obstacle. For example, if the obstacle category is a wall, the obstacle hazard level is 1; if the obstacle category is furniture, the obstacle hazard level is 0.7; and if the obstacle category is a cushion, the obstacle hazard level is 0.3. It is understandable that target obstacles of different obstacle categories will cause different levels of damage to the head-mounted device and the user using the head-mounted device when a collision occurs. Therefore, different obstacle hazard levels are set for target obstacles of different obstacle categories to facilitate accurate determination of the collision risk level corresponding to the target obstacle based on the collision risk level corresponding to the target obstacle.

[0061] As an example, in step S505, the controller uses a preset algorithm to process the distance factor, speed factor, predicted trajectory intersection probability, and obstacle hazard level corresponding to each target obstacle to determine the collision risk level corresponding to each target obstacle. In this example, the distance factor, speed factor, predicted trajectory intersection probability, and obstacle hazard level, which all influence the collision risk level, are taken into account to accurately and comprehensively determine the collision risk level corresponding to the target obstacle.

[0062] In this embodiment, the distance factor, speed factor, predicted trajectory intersection probability, and obstacle danger level corresponding to each target obstacle are processed to determine the collision risk level corresponding to each target obstacle. Factors affecting the collision risk level are taken into account, which is more accurate and comprehensive.

[0063] In one embodiment, the obstacle motion data includes obstacle speed, and the user motion data includes user speed.

[0064] It can be understood that the speed factor affecting the collision risk level is related to the obstacle speed and the user speed. Therefore, the obstacle speed and the user speed are obtained to accurately determine the speed factor.

[0065] In one embodiment, if Figure 6 As shown, step S503, i.e., determining the speed factor corresponding to the target obstacle and the predicted trajectory intersection probability based on the obstacle motion data and the user motion data, includes: S601: Analyze and process the obstacle speed and the user speed to determine the speed factor corresponding to the target obstacle; S602: performing trajectory prediction based on the obstacle motion data and the user motion data to determine the user predicted motion trajectory and the obstacle predicted motion trajectory, and determining a predicted trajectory intersection probability based on the user predicted motion trajectory and the obstacle predicted motion trajectory.

[0066] As an example, in step S601, the controller differentiates the user speed from the obstacle speed, determines the relative speed between the target obstacle and the user using the head-mounted device, obtains the projection of the relative speed in the direction of the line between the user using the head-mounted device and the target obstacle, determines the projection as the approach speed between the target obstacle and the user, and processes the approach speed in a proportional relationship to determine the speed factor corresponding to the target obstacle. The approach speed refers to the speed used to represent the approach of the target obstacle to the user using the head-mounted device. In this example, the user speed and the obstacle speed are considered to accurately and comprehensively determine the speed factor.

[0067] The obstacle predicted motion trajectory refers to the motion trajectory of the target obstacle in a preset future time period. The user predicted motion trajectory refers to the motion trajectory of the user using the head-mounted device in a preset future time period.

[0068] As an example, in step S602, the controller processes the obstacle motion data to predict the future position corresponding to each time in a preset future time period of the target obstacle, determines the obstacle predicted motion trajectory corresponding to the future time period of the target obstacle according to the future position corresponding to each time, processes the user motion data to predict the future position corresponding to each time in a preset future time period of the user using the head-mounted device, determines the user predicted motion trajectory corresponding to the future time period of the user according to the future position corresponding to each time, and analyzes the obstacle predicted motion trajectory and the user predicted motion trajectory to determine the predicted trajectory intersection probability between the target obstacle and the user using the head-mounted device. In this example, the controller can determine the predicted trajectory intersection probability as the ratio of the number of position intersections in the obstacle predicted motion trajectory and the user predicted motion trajectory to the total number of position points. For example, if the total number of position points in the obstacle predicted motion trajectory and the user predicted motion trajectory is N, and the number of position intersections is n, the predicted trajectory intersection probability is . The total number of position points refers to the number of points corresponding to the future positions in the future time period.

[0069] In this example, the method of predicting the future position corresponding to each time in a preset future time period of the target obstacle and the method of predicting the future position corresponding to each time in a preset future time period of the user using the head-mounted device include but are not limited to linear extrapolation, acceleration model, Kalman filter, and machine learning model prediction. In this example, the method of predicting the future position corresponding to each time in a preset future time period of the target obstacle is taken as an example to illustrate that the linear extrapolation specifically includes: the controller obtains the current position of the target obstacle at the current time and the obstacle speed in the obstacle motion data , obtains the future position of the target obstacle at each time in a preset future time period by

[0070]

[0071] Figure 7 S701: The distance factor, the speed factor, the predicted trajectory intersection probability and the obstacle danger degree are weighted to determine a collision risk score corresponding to the target obstacle.S702: If the collision risk score is between a first preset threshold and a second preset threshold, it is determined that the collision risk level corresponding to the target obstacle is a low risk level.S703: If the collision risk score is between the second preset threshold and a third preset threshold, it is determined that the collision risk level corresponding to the target obstacle is a medium risk level.​​​​​​​​​​​​​​​​​​​​​​​ S704: determining that the collision risk level corresponding to the target obstacle is a high risk level if the collision risk score is greater than a third preset threshold value; wherein the first preset threshold value is less than the second preset threshold value, and the second preset threshold value is less than the third preset threshold value.

[0072] wherein the collision risk score refers to a score for representing the collision risk level between the target obstacle and the user using the head-mounted device. Understandably, the greater the collision risk score, the higher the collision risk level between the target obstacle and the user using the head-mounted device.

[0073] As an example, in step S701, the controller adopts preset coefficients to weight the distance factor , the speed factor , the predicted trajectory intersection probability , and the obstacle danger degree to obtain the collision risk score corresponding to the target obstacle. In this example, the collision risk score corresponding to the target obstacle is , wherein , , and are preset coefficients.

[0074] wherein the first preset threshold value, the second preset threshold value, and the third preset threshold value are all preset score values for judging the size of the collision risk score.

[0075] As an example, in step S702, the controller determines that the collision risk between the target obstacle and the user using the head-mounted device is low when it is determined that the collision risk score is between the first preset threshold value and the second preset threshold value, and at this time, it is determined that the collision risk level corresponding to the target obstacle is a low risk level.

[0076] As an example, in step S703, the controller determines that the collision risk level between the target obstacle and the user using the head-mounted device is higher than the low risk level when it is determined that the collision risk score is between the second preset threshold value and the third preset threshold value, and at this time, it is determined that the collision risk level corresponding to the target obstacle is a medium risk level.

[0077] As an example, in step S704, the controller determines that the collision risk level between the target obstacle and the user using the head-mounted device is higher when it is determined that the collision risk score is greater than the third preset threshold value, and at this time, it is determined that the collision risk level corresponding to the target obstacle is a high risk level.

[0078] In this embodiment, the first preset threshold value is less than the second preset threshold value, and the second preset threshold value is less than the third preset threshold value. For example, the first preset threshold value is 0.3, the second preset threshold value is 0.6, and the third preset threshold value is 0.8.

[0079] In this embodiment, the distance factor, the speed factor, the predicted trajectory intersection probability and the obstacle danger degree are weighted, the collision risk score corresponding to the target obstacle is determined more comprehensively and accurately, and the collision risk level corresponding to the target obstacle is determined according to the size of the collision risk score. The method considers the distance factor, the speed factor, the predicted trajectory intersection probability and the obstacle danger degree, so that the collision risk level corresponding to the target obstacle is more accurate.

[0080] In an embodiment, as shown in Figure 8 Step S105, i.e., based on the collision risk level, the risk warning includes: S801: If the collision risk level is a low risk level, at least one of the primary visual warning, the primary auditory warning and the primary tactile warning is performed; S802: If the collision risk level is a medium risk level, at least one of the medium visual warning, the medium auditory warning and the medium tactile warning is performed on the user; S803: If the collision risk level is a high risk level, at least one of the high visual warning, the high auditory warning and the high tactile warning is performed on the user.

[0081] The primary visual warning means that the head-mounted device performs a low-intensity warning on the user visually. The primary auditory warning means that the head-mounted device performs a low-intensity warning on the user audibly. The primary tactile warning means that the head-mounted device performs a low-intensity warning on the user tactilely.

[0082] As an example, in step S801, when the controller determines that the collision risk level is a low risk level, the head-mounted device performs at least one of the primary visual warning, the primary auditory warning and the primary tactile warning on the user visually, audibly and tactilely, respectively, to warn the user of the low risk level of collision risk between the user and the target obstacle. In this example, the head-mounted device displays a soft and semi-transparent obstacle outline corresponding to the target obstacle in the virtual field of view of the user as the primary visual warning, and the prompting device is the display of the head-mounted device. The earphone or the sound device in the head-mounted device emits a small sound as the primary auditory warning, and the prompting device is the earphone or the sound device. The primary auditory warning includes but is not limited to a low-frequency and low-intensity beep sound and a prompt sound with a prompt content of “pay attention to safety”. The head-mounted device performs a slight vibration as the primary tactile warning, and the prompting device is the user handle or the vibration device in the head-mounted device. In this example, as shown in Figure 11 the low-risk level warning schematic diagram of the head-mounted device is shown.Figure 11 It can be seen that when the controller determines that the collision risk level is a low risk level, it displays a soft and translucent obstacle outline corresponding to the target obstacle in the user's virtual field of view through the head-mounted device as a risk warning method for primary visual warning, and issues a risk warning to the user using the head-mounted device, so that the user can avoid the low risk level collision risk corresponding to the target obstacle in time.

[0083] Among them, the intermediate visual warning refers to a warning of moderate intensity to the user through the head-mounted device. The intermediate auditory warning refers to a warning of moderate intensity to the user through the head-mounted device. The intermediate tactile warning refers to a warning of moderate intensity to the user through the head-mounted device.

[0084] As an example, in step S802, when the controller determines that the collision risk level is a medium risk level, it provides the user with at least one of the following warning methods: a medium visual warning, a medium auditory warning, and a medium tactile warning, and provides the user with a risk warning to remind the user to avoid the medium risk level collision with the target obstacle in time. In this example, the controller controls the display of the head-mounted device to display the obvious obstacle outline corresponding to the target obstacle in the user's virtual field of view as a medium visual warning. In this case, the reminder device is the display of the head-mounted device; the earphones or audio equipment in the head-mounted device emit a prompt sound with a sound louder than the primary auditory warning as a medium auditory warning. In this case, the reminder device is the earphones or audio equipment, wherein the prompt sound corresponding to the medium auditory warning includes but is not limited to a beep with a frequency and loudness higher than the primary auditory warning and a prompt sound with the content of "danger"; the head-mounted device performs an obvious vibration on the user's handle or the user's head with an intensity greater than the primary tactile warning as a medium tactile warning. In this case, the reminder device is the user handle or vibration device in the head-mounted device. In this example, Figure 12 The following is a schematic diagram of the risk level warning in the head-mounted device. Figure 12 It can be seen that when the controller determines that the collision risk level is a medium risk level, it uses two risk warning methods: displaying a clear obstacle outline in the user's virtual field of view through the head-mounted device as a medium visual warning, and emitting a prompt sound louder than the primary auditory warning through the headphones or audio equipment in the head-mounted device as a medium auditory warning. These two risk warning methods provide risk warnings to users using head-mounted devices, so that users can avoid the medium risk level collision risk corresponding to the target obstacle in time.

[0085] Advanced visual warnings refer to warnings of a high intensity through the user's vision using a head-mounted device. Advanced auditory warnings refer to warnings of a high intensity through the user's hearing using a head-mounted device. Advanced tactile warnings refer to warnings of a high intensity through the user's tactile sense using a head-mounted device.

[0086] As an example, in step S803, when the controller determines that the collision risk level is a high risk level, it provides the user with at least one of the following warning methods: advanced visual warning, advanced auditory warning, and advanced tactile warning, and provides the user with a risk warning to remind the user to avoid the high risk level of collision with the target obstacle in time. In this example, the highlighted border and cross-hatching of the target obstacle are displayed in the user's virtual field of view through the head-mounted device as an advanced visual warning. In this case, the reminder device is the display of the head-mounted device; the earphones or audio equipment in the head-mounted device emit a prompt sound with a sound intensity greater than the intermediate auditory warning as an advanced auditory warning. In this case, the reminder device is the earphones or audio equipment, wherein the prompt sound corresponding to the advanced auditory warning includes but is not limited to a beep with a frequency and loudness higher than the intermediate auditory warning and a prompt sound with the prompt content of "impending collision, please stop"; the vibration intensity greater than the intermediate tactile warning is used as an advanced tactile warning through the head-mounted device. In this case, the reminder device is the user handle or vibration device in the head-mounted device. In this example, Figure 13 The following is a schematic diagram of a high-risk warning for head-mounted devices. Figure 13 It can be seen that when the controller determines that the collision risk level is a high risk level, it uses three warning methods to warn users using head-mounted devices: displaying the highlighted border and cross-hatched lines of the target obstacle in the user's virtual field of view through the head-mounted device as an advanced visual warning; emitting a prompt sound with a sound intensity greater than the intermediate auditory warning through the headphones or audio equipment in the head-mounted device as an advanced auditory warning; and vibrating the head-mounted device with a vibration intensity greater than the intermediate tactile warning as an advanced tactile warning. These three warning methods provide risk warnings to users using head-mounted devices, so that users can avoid the high-risk level of collision with the target obstacle in time.

[0087] In this embodiment, according to the level of collision risk, different levels of risk warning methods are used to timely issue collision risk warnings to users through head-mounted devices, so that users can intuitively determine the severity of the collision risk between them and the target obstacles, and guide users using head-mounted devices to timely avoid different levels of collision risks between them and the target obstacles, effectively reducing the possibility of collision between the head-mounted device and the user using the head-mounted device and the target obstacle, and can effectively avoid damage to the head-mounted device and injury to the user using the head-mounted device, and has high application value.

[0088] In one embodiment, after step S104, that is, after determining the collision risk level corresponding to the target obstacle, the head mounted device risk warning method further includes: Freezing the movement of a virtual object in the virtual environment, slowing down the movement of a virtual object in the virtual environment, or displaying at least one safe path in the virtual environment.

[0089] wherein the virtual object refers to an object in the virtual environment. The safe path refers to a path with less collision risk.

[0090] As an example, after determining the collision risk level, the controller determines that there is a collision risk between the target obstacle and the user using the head-mounted device, freezes the movement of the virtual object in the virtual environment displayed by the head-mounted device, or slows down the movement of the virtual object in the virtual environment displayed by the head-mounted device, so that the user pays more attention to avoiding the collision risk of the target obstacle. Alternatively, the controller displays at least one safe path for the user to avoid the target obstacle in the display of the head-mounted device to guide the user to avoid the target obstacle.

[0091] In this embodiment, after determining the collision risk level, freezing the movement of the virtual object in the virtual environment, slowing down the movement of the virtual object in the virtual environment, or displaying at least one safe path in the virtual environment can effectively guide the user using the head-mounted device to avoid the collision risk of the target obstacle, and effectively reduce the possibility of collision between the target obstacle and the head-mounted device and the user.

[0092] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0093] In one embodiment, a controller is provided, which can be a server, and its internal structure diagram can be as shown in Figure 9 The controller includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the controller is used to provide computing and control capabilities. The memory of the controller includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the controller is used to store data used or generated in the process of executing the head-mounted device risk warning method. The network interface of the controller is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a head-mounted device risk warning method.

[0094] In an embodiment, a controller is provided, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the head-mounted device risk warning method in the above embodiment, such as Figure 1 S101-S105 shown in Figures 2 to 8 To avoid repetition, they will not be described here.

[0095] In one embodiment, a head-mounted device is provided, including an infrared sensor, a camera device, a reminder device and the above-mentioned controller, wherein the controller is connected to the infrared sensor for obtaining infrared distance data, the controller is connected to the camera device for obtaining visual image data, and the controller is connected to the reminder device for controlling the reminder device to issue a risk warning based on the infrared distance data and the visual image data.

[0096] As an example, Figure 10 As shown, when the head-mounted device is a VR glasses, the VR glasses include an infrared sensor 1001, a camera device 1002, and a reminder device ( Figure 10 ), controller 1003 and inertial measurement unit 1004. In this example, the controller 1003 is connected to the infrared sensor 1001, the camera device 1002, the reminder device and the inertial measurement unit 1004 respectively. The inertial measurement unit 1004 collects the head posture data and the data related to the user's movement of the user using VR glasses (head-mounted device) in real time, and uploads the user's head posture data and the data related to the user's movement to the controller. The controller 1003 monitors whether the user's head posture rotates or whether the user moves according to the user's head posture data and the data related to the user's movement. When it is determined that the user's head posture rotates or the user moves, the controller 1003 3. Real-time control of the infrared sensor 1001 to collect infrared distance data from the surrounding physical environment, and control of the camera device 1002 to collect visual image data from the surrounding physical environment, performing fusion analysis and processing on the infrared distance data and the visual image data, determining the target obstacle and the obstacle attributes corresponding to the target obstacle, and based on the target obstacle and the obstacle attributes corresponding to the target obstacle, determining the three-dimensional safe space model corresponding to the target obstacle, and based on the three-dimensional safe space model, determining the collision risk level corresponding to the target obstacle; based on the collision risk level, controlling the display device to issue a risk warning corresponding to the collision risk level to the user wearing the VR glasses.

[0097] In this embodiment, a head-mounted device including an infrared sensor, a camera device, a reminder device and a controller is used to predict the collision risk level between the target obstacle and the user using the head-mounted device in real time, and to issue a risk warning corresponding to the collision risk level, so as to guide the user using the head-mounted device to avoid the collision risk corresponding to the collision risk level in a timely manner, thereby reducing the possibility of damage to the head-mounted device and injury to the user.

[0098] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the head-mounted device risk warning method in the above embodiment is implemented, for example Figure 1S101-S105 are shown, or Figures 2 to 8 For brevity, details are not repeated here.

[0099] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0101] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A head-mounted device risk warning method, characterized in that: include: Obtain infrared distance data and visual image data collected by the head-mounted device; Performing fusion analysis on the infrared distance data and the visual image data to determine a target obstacle and obstacle attributes corresponding to the target obstacle; Determining a three-dimensional safety space model corresponding to the target obstacle based on the target obstacle and the obstacle attributes corresponding to the target obstacle; Determining a collision risk level corresponding to a target obstacle based on the three-dimensional safety space model; Based on the collision risk level, a risk warning is issued.

2. The head-mounted device risk warning method according to claim 1, characterized in that: The performing fusion analysis on the infrared distance data and the visual image data to determine the target obstacle and the obstacle attributes corresponding to the target obstacle includes: fusing the infrared distance data and the visual image data to determine target fusion data; A pre-trained deep learning model is used to identify and analyze the target fusion data to determine the target obstacle and the obstacle attributes corresponding to the target obstacle.

3. The head-mounted device risk warning method according to claim 1, characterized in that: The obstacle attributes include obstacle distance, obstacle outline and obstacle category; The determining, based on the target obstacle and the obstacle attributes corresponding to the target obstacle, a three-dimensional safe space model corresponding to the target obstacle includes: Determining a relative positional relationship between the target obstacle and the head mounted device based on the obstacle distance; Determining a three-dimensional obstacle model corresponding to the target obstacle based on the obstacle category and the obstacle outline; Based on the relative position relationship and the three-dimensional obstacle model corresponding to the target obstacle, a three-dimensional safety space model corresponding to the target obstacle is determined.

4. The head-mounted device risk warning method according to claim 3, characterized in that: Determining the collision risk level corresponding to the target obstacle based on the three-dimensional safety space model includes: Determining obstacle motion data corresponding to the target obstacle and user motion data corresponding to the head-mounted device based on the three-dimensional obstacle model; Determining a distance factor corresponding to the target obstacle based on the obstacle distance in the three-dimensional safety space model; Determining a velocity factor and a predicted trajectory intersection probability corresponding to the target obstacle based on the obstacle motion data and the user motion data; Determine the degree of obstacle danger based on the obstacle category in the 3D safety space model; A collision risk level corresponding to the target obstacle is determined based on the distance factor, the speed factor, the predicted trajectory intersection probability, and the obstacle danger level.

5. The head-mounted device risk warning method according to claim 4, characterized in that: The obstacle motion data includes obstacle speed, and the user motion data includes user speed; The determining, based on the obstacle motion data and the user motion data, a speed factor corresponding to the target obstacle and a predicted trajectory intersection probability, includes: Analyzing and processing the obstacle speed and the user speed to determine a speed factor corresponding to the target obstacle; Trajectory prediction is performed based on the obstacle motion data and the user motion data to determine a user predicted motion trajectory and an obstacle predicted motion trajectory, and a predicted trajectory intersection probability is determined based on the user predicted motion trajectory and the obstacle predicted motion trajectory.

6. The head-mounted device risk warning method according to claim 4, characterized in that: The determining, based on the distance factor, the speed factor, the predicted trajectory intersection probability, and the obstacle danger level, of a collision risk level corresponding to the target obstacle includes: performing weighted processing on the distance factor, the speed factor, the predicted trajectory intersection probability, and the obstacle danger level to determine a collision risk score corresponding to the target obstacle; If the collision risk score is between the first preset threshold and the second preset threshold, determining that the collision risk level corresponding to the target obstacle is a low risk level; If the collision risk score is between the second preset threshold and the third preset threshold, determining that the collision risk level corresponding to the target obstacle is a medium risk level; If the collision risk score is greater than a third preset threshold, determining that the collision risk level corresponding to the target obstacle is a high risk level; The first preset threshold is smaller than the second preset threshold, and the second preset threshold is smaller than the third preset threshold.

7. The head-mounted device risk warning method according to claim 1, characterized in that: The risk warning based on the collision risk level includes: If the collision risk level is a low risk level, performing at least one of a primary visual warning, a primary auditory warning, and a primary tactile warning; If the collision risk level is a medium risk level, at least one of a medium visual warning, a medium auditory warning, and a medium tactile warning is given to the user; If the collision risk level is a high risk level, at least one of an advanced visual warning, an advanced auditory warning, and an advanced tactile warning is provided to the user.

8. A controller comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the head-mounted device risk warning method according to any one of claims 1 to 7 is implemented.

9. A head-mounted device, characterized in that: It includes an infrared sensor, a camera device, a reminder device and the controller according to claim 8, wherein the controller is connected to the infrared sensor for obtaining infrared distance data, the controller is connected to the camera device for obtaining visual image data, and the controller is connected to the reminder device for controlling the reminder device to issue a risk warning based on the infrared distance data and the visual image data.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the head-mounted device risk warning method according to any one of claims 1 to 7 is implemented.

Images