Obstacle detection method, device, equipment and readable storage medium

Abstract

The present application discloses an obstacle detection method, device, equipment and readable storage medium. The present application relates to the field of obstacle detection technology. The obstacle detection method is applied to a head-mounted device, including: collecting an environmental image; if an obstacle is detected in the environment based on the environmental image, determining the actual distance between the obstacle and a target user, wherein the target user is a user wearing the head-mounted device; if the actual distance is less than a preset distance threshold, determining the actual angle between the obstacle and the target user, wherein the actual angle is the angle between the direction from the target user to the obstacle and the estimated movement direction of the target user; if the actual angle is in a preset dangerous angle interval, outputting a prompt message. The present application improves the safety of head-mounted devices.

Description

Technical Field

[0001] The present application relates to the technical field of obstacle detection, and in particular to an obstacle detection method, device, equipment and readable storage medium. Background Art

[0002] The head-mounted device presents images to the user in a way that can be perceived as a virtual immersive environment, but the head-mounted device blocks the user's observation of the real-world environment in which he or she is located. That is, when the user puts on the head-mounted device, he or she cannot observe the actual physical environment in which he or she is located. At this time, if there are obstacles around the wearer of the head-mounted device (such as walls, tables, cabinets, etc.), collisions may occur, resulting in danger, resulting in low safety in the use of the head-mounted device.

[0003] Therefore, how to improve the safety of head-mounted devices is a technical problem that needs to be solved urgently in this technical field. Summary of the invention

[0004] The main purpose of this application is to provide an obstacle detection method, device, equipment and readable storage medium, aiming at solving the technical problem of how to improve the safety of head-mounted devices.

[0005] To achieve the above object, the present application provides an obstacle detection method, which is applied to a head mounted device and comprises the following steps:

[0006] Collect environmental images;

[0007] If an obstacle is detected in the environment based on the environment image, determining an actual distance between the obstacle and a target user, wherein the target user is a user wearing the head mounted device;

[0008] If the actual distance is less than a preset distance threshold, determining an actual angle between the obstacle and the target user, wherein the actual angle is an angle between a direction from the target user to the obstacle and an estimated movement direction of the target user;

[0009] If the actual angle is within the preset dangerous angle range, a prompt message is output.

[0010] In one embodiment, the step of determining the actual angle between the obstacle and the target user comprises:

[0011] Determine the direction from the target user to the obstacle as a simulated direction, and determine an estimated movement direction of the target user, wherein the estimated movement direction is a sight direction or a current movement direction of the target user;

[0012] The angle between the simulated direction and the estimated movement direction is used as the actual angle between the obstacle and the target user.

[0013] In one embodiment, the step of determining the direction from the target user to the obstacle as a simulated direction includes:

[0014] Constructing a three-dimensional human body model of the target user and a three-dimensional obstacle model of the obstacle;

[0015] Determine, among the point-to-point distances between each model point in the three-dimensional model of the human body and each model point in the three-dimensional model of the obstacle, a reference distance whose sorting is in a preset tail interval;

[0016] The direction in which the model point of the three-dimensional model of the human body corresponding to the reference distance points to the model point of the three-dimensional model of the obstacle is determined as a simulation direction.

[0017] In one embodiment, the head mounted device includes an external camera sensor, and the step of determining the actual distance between the obstacle and the target user includes:

[0018] Acquire a scene image captured by the epitaxial imaging sensor;

[0019] When it is detected that there is a target user in the scene image, locating a target position of the target user based on the scene image;

[0020] An actual distance between the obstacle and the target user is determined based on the target position.

[0021] In one embodiment, the head mounted device includes an inertial sensor, and the step of determining the actual distance between the obstacle and the target user comprises:

[0022] Acquiring inertial data collected by the inertial sensor, wherein the inertial data includes acceleration and / or angular velocity;

[0023] Locating a target position of a target user based on the inertial data;

[0024] An actual distance between the obstacle and the target user is determined based on the target position.

[0025] In one embodiment, after the step of collecting the environment image, the method further includes:

[0026] Inputting the environment image into a pre-trained obstacle recognition model to obtain an obstacle recognition result, wherein the obstacle recognition model is a deep learning model based on computer vision;

[0027] If the obstacle recognition result indicates that an obstacle exists, it is determined that an obstacle is detected in the environment image.

[0028] In one embodiment, after the step of outputting the prompt information, the method further includes:

[0029] Acquire characteristic information of the obstacle, wherein the characteristic information includes shape, size and position;

[0030] A virtual substitute is generated in a virtual scene based on the feature information, the actual distance and the actual angle, wherein the virtual distance between the virtual character of the target user in the virtual scene and the virtual substitute is consistent with the actual distance, and the virtual angle between the virtual character of the target user in the virtual scene and the virtual substitute is consistent with the actual angle.

[0031] In addition, to achieve the above-mentioned purpose, the present application also provides an obstacle detection device, the obstacle detection device comprising:

[0032] A collection module, used for collecting environmental images;

[0033] a distance detection module, configured to determine an actual distance between an obstacle and a target user if an obstacle is detected in the environment based on the environment image, wherein the target user is a user wearing a head mounted device;

[0034] An angle detection module, configured to determine an actual angle between the obstacle and the target user if the actual distance is less than a preset distance threshold, wherein the actual angle is an angle between a direction from the target user to the obstacle and an estimated movement direction of the target user;

[0035] The prompt module is used to output a prompt message if the actual angle is within a preset dangerous angle interval.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a head-mounted device, which includes a memory, a processor, and a collision detection program stored in the memory and executable on the processor, and the collision detection program implements the steps of the above-mentioned collision detection method when executed by the processor.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a readable storage medium, which is a computer-readable storage medium, and a program for implementing the obstacle detection method is stored on the computer-readable storage medium. The program for implementing the obstacle detection method is executed by a processor to implement the steps of the obstacle detection method as described above.

[0038] The present application also provides a computer program product, including a computer program, which implements the steps of the obstacle detection method as described above when executed by a processor.

[0039] In this application, an environmental image is collected; if an obstacle is detected in the environment based on the environmental image, the actual distance between the obstacle and the target user is determined, wherein the target user is a user wearing a head-mounted device; if the actual distance is less than a preset distance threshold, the actual angle between the obstacle and the target user is determined, wherein the actual angle is the angle between the direction from the target user to the obstacle and the estimated movement direction of the target user; if the actual angle is in a preset dangerous angle range, a prompt message is output. In this way, in the embodiment of the present application, when it is detected that the distance between the target user and the obstacle is less than the preset distance threshold, and the angle between the two falls into the dangerous angle range, it is considered that the target user is close to the obstacle, and a prompt message is output at this time to remind the target user to avoid the obstacle (such as stopping moving forward, changing the direction of movement, etc.), thereby improving the safety of the head-mounted device. Furthermore, considering that there is a specific angular relationship between the target user and the obstacle, even if the distance between the user and the obstacle is very close, the user will not collide with the obstacle. Therefore, when the embodiment of the present application detects that the distance between the target user and the obstacle is less than a preset distance threshold, it further detects the actual angle between the target user and the obstacle. Only when the actual angle is also within the preset dangerous angle range, an early warning prompt is issued. This can reduce the phenomenon of false warnings and improve the accuracy of early warnings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 This is a flow chart of the first embodiment of the obstacle detection method of the present application;

[0043] Figure 2 This is a schematic diagram of a first scene at an actual angle involved in an embodiment of an obstacle detection method of the present application;

[0044] Figure 3 This is a schematic diagram of a second scene at an actual angle involved in an embodiment of an obstacle detection method of the present application;

[0045] Figure 4 This is a schematic diagram of a third scene of actual angle involved in an embodiment of the obstacle detection method of the present application;

[0046] Figure 5 This is a schematic diagram of a simplified process of an obstacle detection method according to an embodiment of the present application;

[0047] Figure 6 This is a schematic diagram of the device structure of the obstacle detection device of the present application.

[0048] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0050] With the rapid development of virtual reality and augmented reality technologies, head-mounted devices have become an important tool for people to experience the virtual world. Head-mounted devices block people's vision and hearing from the outside world, and guide users to feel like they are in a virtual environment. The display principle is that the left and right eye screens display the left and right eye images respectively, and the human eye obtains this different information and creates a three-dimensional sense in the mind.

[0051] However, an important problem when using head-mounted devices is that users may be injured due to their inability to perceive obstacles in the actual environment.

[0052] Based on this, the main solution of the present application is: to collect environmental images; if an obstacle is detected in the environment based on the environmental image, then determine the actual distance between the obstacle and the target user, wherein the target user is a user wearing a head-mounted device; if the actual distance is less than a preset distance threshold, then determine the actual angle between the obstacle and the target user, wherein the actual angle is the angle between the direction from the target user to the obstacle and the estimated movement direction of the target user; if the actual angle is within a preset dangerous angle range, then output a prompt message.

[0053] In this application, when it is detected that the distance between the target user and the obstacle is less than the preset distance threshold, and the angle between the two falls into the dangerous angle range, the target user is considered to be close to the obstacle, and a prompt message is output at this time to remind the target user to avoid the obstacle (such as stopping moving forward, changing the direction of movement, etc.), thereby improving the safety of the head-mounted device. In addition, considering that there is a specific angle relationship between the target user and the obstacle, even if the distance between the user and the obstacle is very close, there will be no collision with the obstacle. Therefore, when the distance between the target user and the obstacle is detected to be less than the preset distance threshold, the application further detects the actual angle between the target user and the obstacle. When the actual angle is also in the preset dangerous angle range, an early warning prompt is issued, which can reduce the phenomenon of false warnings and improve the accuracy of warnings.

[0054] It should be noted that the execution subject of each embodiment of the obstacle detection method of the present application is a head-mounted device that can realize the above-mentioned functions, such as AR (Augmented Reality) glasses, AR helmets and other AR devices and VR (Virtual Reality) glasses, VR helmets and other VR devices. By way of example, VR glasses are used as the execution subject to describe each embodiment of the obstacle detection method of the present application.

[0055] Based on this, the present application proposes an obstacle detection method of the first embodiment, please refer to Figure 1 The obstacle detection method includes steps S10 to S40:

[0056] Step S10, collecting environmental images;

[0057] The environmental image is specifically an image in the line of sight direction collected by the camera sensor of the VR glasses. For example, a camera with a shooting direction in the line of sight direction can be set at the frame or temple of the VR glasses. After the user wears the VR glasses, the camera can collect images in the user's line of sight to obtain the environmental image. The line of sight direction is the line of sight direction of the wearer when the VR glasses are worn.

[0058] Step S20, if an obstacle is detected in the environment based on the environment image, determining the actual distance between the obstacle and the target user, wherein the target user is a user wearing a head mounted device;

[0059] The obstacle recognition algorithm can be preset in advance, and whether there are obstacles in the environment image is detected based on the preset obstacle recognition algorithm. If there are obstacles in the environment image, it is determined that there are obstacles in the environment. The obstacle recognition algorithm can be an algorithm based on deep learning (such as using a convolutional neural network for image recognition, extracting features from the image and identifying obstacles), an algorithm based on computer vision (such as detecting moving obstacles by establishing a background model and subtracting the background from the current frame, using an edge detection algorithm to identify the outline of an object and then identify obstacles, etc.), an algorithm based on semantics, etc. (classifying each pixel in the image into a specific category, such as roads, pedestrians, vehicles, etc., to identify obstacles), and this embodiment does not impose specific restrictions on this.

[0060] The actual distance is used to indicate the distance between the target user and the obstacle in the real world. Specifically, the positions of the obstacle and the target user in the same coordinate system (such as the world coordinate system) can be located respectively, so as to determine the actual distance between the two based on their positions in the same coordinate system. Alternatively, a digital model of the real environment can be created and a collision detection algorithm can be run on the model to predict the possible contact points between the target user and the obstacle, and the distance between the possible contact points between the target user and the obstacle can be used as the actual distance between the target user and the obstacle.

[0061] It should be noted that the above are only two feasible implementation methods for determining the actual distance provided in this embodiment, and this embodiment does not specifically limit the specific implementation method for determining the actual distance.

[0062] Step S30, if the actual distance is less than the preset distance threshold, determining the actual angle between the obstacle and the target user, wherein the actual angle is the angle between the direction from the target user to the obstacle and the estimated movement direction of the target user;

[0063] The preset distance threshold may be an arbitrary value set in advance. When the actual distance between the target user and the obstacle is less than the preset distance threshold, it is considered that there may be a risk of collision between the target user and the obstacle. At this time, the actual angle between the obstacle and the target user is further determined.

[0064] When the actual distance is greater than or equal to the preset distance threshold, it can be considered that there is no collision risk between the current target user and the obstacle, and it can be considered that there is no need to further determine the actual angle between the obstacle and the target user. The obstacle detection process is directly ended and the initial step is returned to enter the next obstacle detection process. In this way, unnecessary detection of the actual angle can be reduced, thereby reducing the amount of calculation.

[0065] The estimated movement direction is used to indicate the possible movement direction of the target user at the next moment.

[0066] Step S40: if the actual angle is within the preset dangerous angle range, a prompt message is output.

[0067] The preset danger angle interval may be an angle range set in advance. If the actual angle is within the preset danger angle interval, it indicates that the target user is approaching an obstacle. If the actual angle is not within the preset danger angle interval, it indicates that the target user is moving away from the obstacle. The preset danger angle interval may be less than 90 degrees, preferably less than 30 degrees. When the actual angle falls within the preset danger angle interval, a prompt message is output; when the actual angle does not fall within the preset danger angle interval, it is considered that there is no collision risk between the current target user and the obstacle, and the obstacle detection process ends, and returns to the initial step to enter the next obstacle detection process.

[0068] When the actual distance between the target user and the obstacle is less than the preset threshold, taking the preset danger angle interval as less than 90 degrees as an example, refer to Figures 2 to 4 As shown, θ represents the actual angle between the obstacle and the target user. Figures 2 to 4 The solid line in the middle indicates the direction from the target user 200 to the obstacle 100, and the dotted line indicates the estimated moving direction of the target user 200 (the current moving direction of the target user is the estimated moving direction of the target user). It can be seen that Figure 2 In the example, the actual angle between the obstacle 100 and the target user 200 is greater than 90 degrees. Figure 3 The actual angle between the obstacle 100 and the target user 200 is equal to 90 degrees. The actual angle between the obstacle 100 and the target user 200 is not within the preset dangerous angle range. It is determined that there is no collision risk. In fact, the target user is also moving away from the obstacle, and there is no collision risk. Figure 4 The actual angle between the obstacle 100 and the target user 200 is less than 90 degrees. At this time, the actual angle is in the preset dangerous angle range, and a prompt message is output. In fact, the target user is also moving close to the obstacle, and the user needs to be reminded to avoid the obstacle.

[0069] The output prompt information may specifically be displaying the outline of the obstacle on the display interface of the VR glasses, a voice prompt, or generating vibration feedback, etc. This embodiment does not impose any specific limitation on this.

[0070] In this embodiment, an environmental image is collected; if an obstacle is detected in the environment based on the environmental image, the actual distance between the obstacle and the target user is determined, wherein the target user is a user wearing a head-mounted device; if the actual distance is less than a preset distance threshold, the actual angle between the obstacle and the target user is determined, wherein the actual angle is the angle between the direction from the target user to the obstacle and the estimated direction of movement of the target user; if the actual angle is within a preset dangerous angle range, a prompt message is output. In this way, in this embodiment, when it is detected that the distance between the target user and the obstacle is less than the preset distance threshold, and the angle between the two falls into the dangerous angle range, it is considered that the target user is close to the obstacle, and a prompt message is output at this time to remind the target user to avoid the obstacle (such as stopping, changing the direction of movement, etc.), thereby improving the safety of the head-mounted device. Furthermore, considering that there is a specific angular relationship between the target user and the obstacle, even if the distance between the user and the obstacle is very close, the user will not collide with the obstacle. For example, when the angle between the direction vectors of the user and the obstacle is 90 degrees, even if the distance between the user and the obstacle is very close, the user will not collide with the obstacle. Therefore, when the embodiment detects that the distance between the target user and the obstacle is less than a preset distance threshold, the actual angle between the target user and the obstacle is further detected. When the actual angle is also within the preset dangerous angle range, an early warning prompt is issued. This can reduce the phenomenon of false warnings and improve the accuracy of early warnings.

[0071] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be repeated in the following. On this basis, the step of determining the actual angle between the obstacle and the target user includes:

[0072] Step A10, determining the direction from the target user to the obstacle as a simulated direction, and determining an estimated movement direction of the target user, wherein the estimated movement direction is a sight direction or a current movement direction of the target user;

[0073] Step A20: taking the angle between the simulated direction and the estimated movement direction as the actual angle between the obstacle and the target user.

[0074] Regarding the simulated direction of the obstacle, specifically, the simulated direction can be the orientation of the target user relative to the obstacle, can be the normal of the surface where the target user is closest to the obstacle, or can be defined as the direction from a certain point of the target user to a specific point on the obstacle. This embodiment does not impose any specific restrictions on this.

[0075] The possible movement direction of the target user at the next moment is usually the line of sight direction. Therefore, for the estimated movement direction of the target user, as one implementation, if the target user is in motion, the current movement direction of the target user can be determined as the estimated movement direction of the target user. If the target user is stationary, the line of sight direction of the target user can be determined as the estimated movement direction of the target user. As another implementation, considering that the user usually moves in the line of sight direction, regardless of whether the target user is moving, the line of sight direction of the target user can be used as the estimated movement direction of the target user, so that there is no need to detect the current movement direction of the target user, which reduces the amount of calculation and the calculation complexity of the direction vector.

[0076] In a possible implementation, the step of determining the direction from the target user to the obstacle as the simulated direction includes:

[0077] Step B10, constructing a human body three-dimensional model of the target user and a three-dimensional obstacle model of the obstacle;

[0078] Step B20, determining a reference distance whose sorting is in a preset tail interval among the point-to-point distances between each model point in the three-dimensional model of the human body and each model point in the three-dimensional model of the obstacle;

[0079] As one implementation method of setting the preset tail interval, all point-to-point distances between each model point in the human body three-dimensional model (hereinafter referred to as human body model points) and each model point in the obstacle three-dimensional model (hereinafter referred to as obstacle model points) can be sorted from large to small to obtain a distance queue, and the last preset number of point-to-point distances in the distance queue are used as the preset tail interval. For example, assuming that there are three human body model points [A, B, C] and two obstacle model points [a, b], the distance between A and a is d1, the distance between A and b is d2, the distance between B and a is d3, the distance between B and b is d4, the distance between C and a is d5, and the distance between C and b is d6, then the distance queue obtained by arranging the distance from large to small is [d1, d2, d3, d5, d4, d6], and assuming that the preset number is three, the preset tail space is [d5, d4, d6].

[0080] As another implementation, for each human body model point, the distance between each obstacle model point and the human body model point is determined, the shortest distance among all distances is determined, and the shortest distances corresponding to all human body model points are combined to obtain a preset tail interval. For example, assuming that the human body model points have three human body model points [A, B, C] and two obstacle model points [a, b], the distance between A and a is d1, the distance between A and b is d2, the distance between B and a is d3, the distance between B and b is d4, the distance between C and a is d5, the distance between C and b is d6, and the minimum value between d2 and d1 is d2, the minimum value between d3 and d4 is d4, and the minimum value between d5 and d6 is d6, then the preset tail space is [d2, d4, d6].

[0081] Furthermore, in order to reduce the amount of calculation, the model points (including the model points in the three-dimensional model of the obstacle and the three-dimensional model of the human body) may preferably be contour points of the model contour.

[0082] Step B30, determining the direction from the model point of the three-dimensional model of the human body corresponding to the reference distance to the model point of the three-dimensional model of the obstacle as the simulation direction.

[0083] It should be noted that if there are multiple simulated directions, as one implementation method, the average direction of all simulated directions may be calculated, and the angle between the average direction and the estimated motion direction may be used as the actual angle between the target user and the obstacle.

[0084] As another implementation, for each simulated direction, the angle between the simulated direction and the estimated direction of movement is determined to obtain an actual angle. After multiple actual angles are calculated based on multiple simulated directions, a prompt message may be output when any actual angle is in a preset dangerous angle interval, or a prompt message may be output when the total number of actual angles in the preset dangerous angle interval is greater than a preset threshold, or a prompt message may be output when all actual angles are in the preset dangerous angle interval. This embodiment does not impose specific restrictions on this. Preferably, a prompt message is output when any actual angle is in a preset dangerous angle interval, so that the sensitivity of VR glasses to collision risks can be increased.

[0085] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first and second embodiments can be referred to the above description, and will not be repeated in the following. On this basis, the head-mounted device includes an extended camera sensor, and the steps of determining the actual distance between the obstacle and the target user include:

[0086] Step C10, acquiring a scene image captured by the external camera sensor;

[0087] The extended camera sensor is used to collect images, and may specifically be a camera, etc. This embodiment does not impose any specific limitation on this. The extended camera sensor is an external sensor, that is, when the user wears VR glasses, the extended camera sensor is away from the wearer.

[0088] Furthermore, the extended camera sensor can also be a position-adjustable sensor, so that the target user appears within the field of view of the extended camera sensor by adjusting the position of the sensor. The position adjustment specifically refers to the adjustable sensor angle, which includes one or more of the pitch angle, yaw angle, and roll angle, that is, the extended camera sensor has multiple degrees of freedom, such as 2 degrees of freedom, 4 degrees of freedom, or 6 degrees of freedom, etc., and this embodiment does not impose specific restrictions on this.

[0089] Step C20, when it is detected that there is a target user in the scene image, locate the target position of the target user based on the scene image, and determine the actual distance between the obstacle and the target user based on the target position.

[0090] The target position of the target user is located based on the scene image, and specifically the target position of the target user in the world coordinate system can be located.

[0091] In this embodiment, the target position of the target user is located by the external camera sensor, which provides a data basis for determining the actual distance between the obstacle and the target user. And positioning by collecting images by the external camera sensor can achieve rapid positioning of the target user.

[0092] In a possible implementation, the head mounted device includes an inertial sensor, and the step of determining the actual distance between the obstacle and the target user includes:

[0093] Step D10, acquiring inertial data collected by the inertial sensor, wherein the inertial data includes acceleration and / or angular velocity;

[0094] Step D20, locating the target position of the target user based on the inertial data, and determining the actual distance between the obstacle and the target user based on the target position.

[0095] It can be understood that the positioning method based on inertial data has a higher positioning accuracy of the target position, and thus, the target user can be accurately positioned based on inertial data. Considering that VR glasses are worn on the user's head, the inertial data collected is usually the user's head. Therefore, the extended camera sensor can be used to preliminarily locate the initial position of the target user (including the head position and the non-head position), and then the head position in the initial position can be accurately located using the inertial data, that is, the head position in the initial position is updated to the head position located based on the inertial data, and the initial position after the head position is updated is used as the final target position of the target user. In this way, the positioning method combining the inertial sensor and the extended camera sensor can take into account both positioning accuracy and positioning speed.

[0096] In a possible implementation manner, after the step of collecting the environment image, the method further includes:

[0097] Step E10, inputting the environment image into a pre-trained obstacle recognition model to obtain an obstacle recognition result, wherein the obstacle recognition model is a deep learning model based on computer vision;

[0098] In a feasible manner, before inputting the environment image into the pre-trained obstacle recognition model, the environment image may be preprocessed, and the image preprocessing includes but is not limited to image denoising, contrast enhancement, etc., so as to improve the image quality of the environment image and thereby improve the accuracy of subsequent obstacle recognition.

[0099] The obstacle recognition model can be trained based on the training data to learn to recognize various common obstacles (such as walls, furniture, people, etc.) to obtain a pre-trained obstacle recognition model.

[0100] Step E20: If the obstacle recognition result indicates that an obstacle exists, it is determined that an obstacle is detected in the environment image.

[0101] Deep learning models can automatically learn feature representations from data without the need for manual feature design. They show extremely high accuracy in image recognition, classification, and detection tasks, can process large-scale images, have certain robustness to noise, occlusion, and other transformations in images, and can maintain high performance in complex environments. Computer vision can achieve or exceed human-level accuracy in image recognition tasks, can quickly analyze images and provide consistency in image recognition, and can be easily expanded to process large amounts of image data. Therefore, this embodiment selects a deep learning model based on computer vision for obstacle recognition, which realizes real-time detection and high-accuracy recognition of obstacles in VR environments, and has the advantages of good real-time performance and high accuracy.

[0102] Based on the first embodiment, the second embodiment and / or the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment, the second embodiment and the third embodiment can be referred to the above description, and will not be described in detail later. On this basis, after the step of outputting the prompt information, the method further includes:

[0103] Step F10, obtaining characteristic information of the obstacle, wherein the characteristic information includes shape, size and position;

[0104] The feature information includes but is not limited to shape, size and position, and may include the outline of an obstacle. Relevant personnel may set the feature information to be extracted according to actual conditions, and this implementation does not impose specific restrictions on this.

[0105] It should be noted that the position may be the position of the obstacle relative to the ground, similar to the height of the obstacle. By acquiring the characteristic information of the obstacle, the generated virtual substitute is basically consistent with the obstacle, thereby realizing high-precision restoration of the obstacle.

[0106] Step F20, generating a virtual substitute in the virtual scene based on the feature information, the actual distance and the actual angle, wherein the virtual distance between the virtual character of the target user in the virtual scene and the virtual substitute is consistent with the actual distance, and the virtual angle between the virtual character of the target user in the virtual scene and the virtual substitute is consistent with the actual angle.

[0107] The virtual distance and virtual angle are specifically the distance and angle between the virtual character of the target user and the virtual substitute in the virtual scene displayed by the VR glasses. The shape, size, position and other information of the generated virtual substitute are consistent with the obstacle to achieve high-precision restoration of the obstacle. At the same time, the virtual substitute is generated in the virtual scene, which allows the user to intuitively perceive the existence of the obstacle, provide guidance for the user to avoid the obstacle, and thus effectively avoid the obstacle.

[0108] For example, to help understand the technical concept or technical principle of obstacle detection after the present embodiment is combined with the above-mentioned embodiment 1, embodiment 2 and embodiment 3, please refer to Figure 5 , Figure 5 A brief flowchart of an obstacle detection method is provided, as follows:

[0109] Step 1: Image acquisition and preprocessing: Use the high-definition camera on the VR glasses to capture high-definition images of the user's surrounding environment in real time.

[0110] Step 2: Obstacle recognition: Use an obstacle recognition model based on deep learning and computer vision technology to identify obstacles in the collected environment images.

[0111] Step 3: Spatial positioning and distance calculation, using the built-in inertial sensors of VR glasses (such as accelerometers, gyroscopes, etc.) combined with the images collected by the external camera sensor to achieve accurate spatial positioning of the user's head and non-head;

[0112] Step 4: Prompts and feedback. When the user approaches an obstacle, the VR glasses will promptly issue warnings to the user through the display interface, sound prompts or vibration feedback.

[0113] In the above steps, the image acquisition and preprocessing steps include using the high-definition camera equipped on the VR glasses to capture the surrounding environment images in real time, and using image processing technology to preprocess the images, such as image denoising, contrast enhancement, etc., to improve the accuracy of subsequent obstacle recognition.

[0114] In the above steps, the obstacle recognition step is implemented based on deep learning and computer vision technology, which can accurately identify obstacles in the image and extract key information.

[0115] In the above steps, the spatial positioning and distance calculation step uses the built-in inertial sensors of the VR glasses to achieve accurate spatial positioning of the user's head and body, and determines whether the user is close to the obstacle by calculating the actual distance and actual angle between the user and the obstacle. Specifically, if the actual distance is less than the preset distance threshold and the actual angle is in the preset dangerous angle range, it is determined that the user is close to the obstacle.

[0116] In the above steps, the prompt and feedback step promptly warns the user through the display interface of the VR glasses, sound prompts or vibration feedback.

[0117] By integrating computer vision algorithms, sensor technology and physical interactive feedback mechanisms, it achieves real-time detection, accurate identification and timely prompts of obstacles in the actual environment. It has the advantages of good real-time performance, high accuracy and good user experience.

[0118] It should be noted that the above specific embodiments are only used to understand the present application and do not constitute a limitation on the obstacle detection process of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0119] In addition, the present application also provides an obstacle detection device, referring to Figure 6 , the obstacle detection device comprises:

[0120] The acquisition module 10 is used to acquire environmental images;

[0121] A distance detection module 20, configured to determine an actual distance between an obstacle and a target user if an obstacle is detected in the environment based on the environment image, wherein the target user is a user wearing a head mounted device;

[0122] An angle detection module 30, for determining an actual angle between the obstacle and the target user if the actual distance is less than a preset distance threshold, wherein the actual angle is the angle between the direction from the target user to the obstacle and the estimated movement direction of the target user;

[0123] The prompt module 40 is used to output a prompt message if the actual angle is within a preset dangerous angle interval.

[0124] In one embodiment, the angle detection module 30 is further configured to:

[0125] Determine the direction from the target user to the obstacle as a simulated direction, and determine an estimated movement direction of the target user, wherein the estimated movement direction is a sight direction or a current movement direction of the target user;

[0126] The angle between the simulated direction and the estimated motion direction is taken as the actual angle between the obstacle and the target user.

[0127] In one embodiment, the angle detection module 30 is further configured to:

[0128] Constructing a 3D human body model of the target user and a 3D obstacle model of the obstacle;

[0129] Determine, among the point-to-point distances between each model point in the three-dimensional model of the human body and each model point in the three-dimensional model of the obstacle, a reference distance whose sorting is in a preset tail interval;

[0130] The direction in which the model point of the three-dimensional model of the human body corresponding to the reference distance points to the model point of the three-dimensional model of the obstacle is determined as the simulation direction.

[0131] In one embodiment, the head mounted device includes an external camera sensor, a distance detection module 20, and is further configured to:

[0132] Acquire a scene image captured by an external camera sensor;

[0133] When it is detected that there is a target user in the scene image, a target position of the target user is located based on the scene image, and an actual distance between the obstacle and the target user is determined based on the target position.

[0134] In one embodiment, the head mounted device includes an inertial sensor, a distance detection module 20, and is further configured to:

[0135] Acquiring inertial data collected by an inertial sensor, wherein the inertial data includes acceleration and / or angular velocity;

[0136] The target position of the target user is located based on the inertial data, and the actual distance between the obstacle and the target user is determined based on the target position.

[0137] In one embodiment, the obstacle detection device further includes an identification module, the identification module being configured to:

[0138] Inputting the environment image into a pre-trained obstacle recognition model to obtain an obstacle recognition result, wherein the obstacle recognition model is a deep learning model based on computer vision;

[0139] If the obstacle recognition result indicates that an obstacle exists, it is determined that the obstacle is detected in the environment image.

[0140] In one embodiment, the obstacle detection device further includes a generation module, the generation module being configured to:

[0141] Obtaining characteristic information of the obstacle, wherein the characteristic information includes shape, size and position;

[0142] A virtual substitute is generated in a virtual scene based on the feature information, the actual distance and the actual angle, wherein the virtual distance between the virtual character of the target user in the virtual scene and the virtual substitute is consistent with the actual distance, and the virtual angle between the virtual character of the target user in the virtual scene and the virtual substitute is consistent with the actual angle.

[0143] In addition, an embodiment of the present application also proposes a head-mounted device, which includes a memory, a processor, and an obstacle detection program stored in the memory and executable on the processor. When the obstacle detection program is executed by the processor, the steps of the obstacle detection method as described above are implemented.

[0144] The head-mounted device may also include a processing device (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the head-mounted device are also stored. The processing device, ROM, and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus. Generally, the following systems can be connected to the I / O interface: input devices such as touch screens, touch pads, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices such as liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices such as tapes, hard disks, etc.; and communication devices. The communication device can allow the head-mounted device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a head-mounted device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0145] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0146] The head-mounted device provided in the embodiment of the present application adopts the obstacle detection method in the above embodiment to solve the technical problem of obstacle detection. Compared with the prior art, the beneficial effects of the head-mounted device provided in the present application are the same as the beneficial effects of the obstacle detection method provided in the above embodiment, and the other technical features in the head-mounted device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0147] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0148] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0149] In addition, to achieve the above-mentioned purpose, an embodiment of the present application further provides a readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the obstacle detection method in the above-mentioned embodiment.

[0150] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0151] The computer-readable storage medium may be included in the head-mounted device; or may exist independently without being assembled into the head-mounted device.

[0152] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the head-mounted device, the head-mounted device: collects environmental images; if an obstacle is detected in the environment based on the environmental image, determines the actual distance between the obstacle and the target user, wherein the target user is the user wearing the head-mounted device; if the actual distance is less than a preset distance threshold, determines the actual angle between the obstacle and the target user, wherein the actual angle is the angle between the direction from the target user to the obstacle and the estimated movement direction of the target user; if the actual angle is in a preset danger angle interval, outputs a prompt message.

[0153] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0155] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0156] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned obstacle detection method, and can solve the technical problem of obstacle detection. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the obstacle detection method provided in the above-mentioned embodiment, and will not be elaborated here.

[0157] In addition, an embodiment of the present application further provides a computer program product, including an obstacle detection program, which implements the steps of the above obstacle detection method when executed by a processor.

[0158] The specific implementation of the computer program product of the present application is basically the same as the above-mentioned embodiments of the obstacle detection method, and will not be repeated here.

[0159] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0160] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0161] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software sensor, which is stored in a storage medium (such as ROM / RAM, disk, CD) as above, including a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of each embodiment of the present application.

[0162] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An obstacle detection method, characterized in that: Applied to a head mounted device, the obstacle detection method comprises the following steps: Collect environmental images; If an obstacle is detected in the environment based on the environment image, determining an actual distance between the obstacle and a target user, wherein the target user is a user wearing the head mounted device; If the actual distance is less than a preset distance threshold, constructing a human body three-dimensional model of the target user and a three-dimensional obstacle model of the obstacle; Determine, among the point-to-point distances between each human body model point and each obstacle model point, a reference distance whose sorting is in a preset tail interval, wherein the human body model point is a model point of the three-dimensional human body model, and the obstacle model point is a model point of the three-dimensional obstacle model; Determine the direction in which the human body model point corresponding to the reference distance points to the obstacle model point as a simulation direction; Determining an estimated movement direction of the target user; wherein the estimated movement direction is a sight line direction or a current movement direction of the target user; For each of the simulated directions, determining the angle between the simulated direction and the estimated motion direction to obtain an actual angle; If any of the actual angles is within the preset dangerous angle range, output a prompt message; The preset tail interval is obtained by sorting all the point-to-point distances from large to small to obtain a distance queue, and combining the last preset number of point-to-point distances in the distance queue to obtain an interval; or, The preset tail interval is an interval obtained by combining the shortest distances corresponding to the human body model points. For each human body model point, the minimum value of the distances from the human body model point to the obstacle model points is the shortest distance corresponding to the human body model point.

2. The method according to claim 1, characterized in that The head mounted device includes an external camera sensor, and the step of determining the actual distance between the obstacle and the target user includes: Acquire a scene image captured by the epitaxial camera sensor; When it is detected that there is a target user in the scene image, locating a target position of the target user based on the scene image; An actual distance between the obstacle and the target user is determined based on the target position.

3. The method according to claim 1, characterized in that The head mounted device includes an inertial sensor, and the step of determining the actual distance between the obstacle and the target user includes: Acquiring inertial data collected by the inertial sensor, wherein the inertial data includes acceleration and / or angular velocity; Locating a target position of a target user based on the inertial data; An actual distance between the obstacle and the target user is determined based on the target position.

4. The method according to any one of claims 1 to 3, characterized in that After the step of collecting the environment image, the method further includes: Inputting the environment image into a pre-trained obstacle recognition model to obtain an obstacle recognition result, wherein the obstacle recognition model is a deep learning model based on computer vision; If the obstacle recognition result indicates that an obstacle exists, it is determined that an obstacle exists in the environment.

5. The method according to any one of claims 1 to 3, characterized in that: After the step of outputting the prompt information, the method further includes: Acquire characteristic information of the obstacle, wherein the characteristic information includes shape, size and position; generating a virtual substitute in a virtual scene of the head mounted device based on the feature information, the actual distance and the actual angle; The virtual distance between the virtual character of the target user in the virtual scene and the virtual substitute is consistent with the actual distance, and the virtual angle between the virtual character of the target user in the virtual scene and the virtual substitute is consistent with the actual angle.

6. An obstacle detection device, characterized in that: The obstacle detection device includes a head mounted device, and further includes: A collection module, used for collecting environmental images; a distance detection module, configured to determine an actual distance between a target user and the obstacle if an obstacle is detected in the environment based on the environment image, wherein the target user is the area where the target user wearing the head mounted device is currently located; An angle detection module, configured to construct a human body three-dimensional model of the target user and a three-dimensional obstacle model of the obstacle if the actual distance is less than a preset distance threshold; Determine, among the point-to-point distances between each human body model point and each obstacle model point, a reference distance whose sorting is in a preset tail interval, wherein the human body model point is a model point of the three-dimensional human body model, and the obstacle model point is a model point of the three-dimensional obstacle model; Determine the direction in which the human body model point corresponding to the reference distance points to the obstacle model point as a simulation direction; Determining an estimated movement direction of the target user; wherein the estimated movement direction is a sight line direction or a current movement direction of the target user; For each of the simulated directions, determining the angle between the simulated direction and the estimated motion direction to obtain an actual angle; A prompt module, configured to output a prompt message if any of the actual angles is within a preset dangerous angle interval; The preset tail interval is obtained by sorting all the point-to-point distances from large to small to obtain a distance queue, and combining the last preset number of point-to-point distances in the distance queue to obtain an interval; or, The preset tail interval is an interval obtained by combining the shortest distances corresponding to the human body model points. For each human body model point, the minimum value of the distances from the human body model point to the obstacle model points is the shortest distance corresponding to the human body model point.

7. A head mounted device, characterized in that: The head mounted device comprises: a memory, a processor, and an obstacle detection program stored in the memory and executable on the processor, wherein the obstacle detection program implements the steps of the obstacle detection method according to any one of claims 1 to 5 when executed by the processor.

8. A readable storage medium, characterized in that: The readable storage medium is a computer-readable storage medium, on which is stored a program for implementing obstacle detection, and the program for implementing obstacle detection is executed by a processor to implement the steps of the obstacle detection method according to any one of claims 1 to 5.

Images