Obstacle detection methods, devices, terminal equipment, and media for head-mounted devices

By constructing obstacle bounding boxes for dynamic obstacles and utilizing visual SLAM algorithms and frame difference processing technology, the pose data of dynamic obstacles can be monitored and identified in real time, solving the problem that head-mounted devices cannot effectively detect dynamic obstacles, thus improving user safety and detection performance.

CN117369646BActive Publication Date: 2026-11-17GEER TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202311432859.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-31
Publication Date
2026-11-17
Estimated Expiration
2043-10-31

AI Technical Summary

Technical Problem

Existing head-mounted devices cannot effectively detect dynamic obstacles when worn by users, leading to an increased risk of collisions. This is especially true during intense sports activities or immersive VR games, where users' observation of the real environment is obstructed, resulting in insufficient safety.

Method used

By acquiring continuous image frames of the obstacle warning area, an obstacle bounding box for dynamic obstacles is constructed. Using visual SLAM algorithm and frame difference processing technology, combined with 3D point cloud data and relative motion information, the pose data of dynamic obstacles is monitored and identified in real time, the distance between them is determined, and a warning is given when the threat distance matches.

Benefits of technology

It enables real-time tracking and accurate identification of dynamic obstacles, improves the obstacle detection performance of head-mounted devices, ensures user safety, and reduces the risk of collisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117369646B_ABST
    Figure CN117369646B_ABST
Patent Text Reader

Abstract

This application relates to the field of obstacle detection technology, and more particularly to an obstacle detection method, apparatus, terminal device, and medium for a head-mounted device. The method includes: acquiring continuous image frames in an obstacle warning area, wherein the continuous image frames include at least a current frame image; when a dynamic obstacle exists in the current frame image, constructing an obstacle bounding box for the dynamic obstacle based on the continuous image frames; determining a first pose data of the dynamic obstacle based on the relative motion information between the obstacle bounding box and the head-mounted device, and determining a distance between the obstacle bounding box and the head-mounted device based on the first pose data; and providing a warning about the dynamic obstacle when the distance matches a preset threat distance. This application improves the obstacle detection performance of head-mounted devices.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of obstacle detection technology, and in particular to an obstacle detection method, device, terminal equipment, and medium for a head-mounted device. Background Technology

[0002] With the rapid development of VR (Virtual Reality) technology, head-mounted devices are being used more and more widely in VR products, and users are also placing higher demands on the obstacle detection function of these devices.

[0003] Head-mounted displays present images to users in a way that can be perceived as a virtual immersive environment. However, these displays obstruct the user's observation of their real-world environment. In other words, when wearing a head-mounted display, users cannot observe their actual physical surroundings. For example, during vigorous physical activity or in immersive VR games, collisions are more likely, especially when a moving object enters the safe area, drastically increasing the probability of danger. Therefore, improving the obstacle detection performance of head-mounted displays is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] The main objective of this application is to provide an obstacle detection method, device, terminal equipment, and medium for head-mounted devices, aiming to improve the obstacle detection performance of head-mounted devices.

[0005] To achieve the above objectives, this application provides an obstacle detection method for a head-mounted device, the obstacle detection method for the head-mounted device comprising:

[0006] Acquire consecutive image frames in the obstacle warning area, wherein the consecutive image frames include at least the current frame image;

[0007] When there are dynamic obstacles in the current frame image, an obstacle bounding box for the dynamic obstacles is constructed based on the consecutive image frames;

[0008] The first pose data of the dynamic obstacle is determined based on the relative motion information between the obstacle enclosure and the head-mounted device, and the distance between the obstacle enclosure and the head-mounted device is determined based on the first pose data.

[0009] When the distance between the points matches the preset threat distance, a warning is given for the dynamic obstacle.

[0010] Optionally, the continuous image frames further include target frame images, and after the step of acquiring the continuous image frames in the obstacle cues region, the obstacle detection method of the head-mounted device further includes:

[0011] Frame difference processing is performed on the current frame image and the target frame image to obtain a frame difference image, wherein the target frame image is the frame image preceding the current frame image;

[0012] Detect whether the grayscale value of the frame difference image exceeds a preset grayscale threshold;

[0013] If the grayscale value does not exceed the grayscale threshold, the current frame image is used as the next target frame image, and the step of performing frame difference processing based on the current frame image and the target frame image is returned to be executed until the grayscale value exceeds the grayscale threshold, at which point it is determined that there is a dynamic obstacle in the current frame image.

[0014] Optionally, the consecutive image frames also include N image frames following the current image frame, where N is a natural number greater than 0;

[0015] The step of constructing the obstacle bounding box of the dynamic obstacle based on the consecutive image frames includes:

[0016] Obstacle features are extracted based on the current frame image and the N frame images to obtain the three-dimensional point cloud data of the dynamic obstacle;

[0017] An obstacle bounding box is constructed based on the parallax characteristics of the head-mounted device and the three-dimensional point cloud data.

[0018] Optionally, the step of extracting obstacle features based on the current frame image and the N frame images to obtain the point cloud data of the dynamic obstacle includes:

[0019] The current frame image is divided into regions according to a preset grid area to obtain the processed region and the ideal image frame corresponding to the current frame image is determined.

[0020] In the ideal image frame, an ideal block region matching the segmented block region is obtained, and a gray value comparison is performed based on the first gray value of the segmented block region and the second gray value of the matching ideal block region.

[0021] When the first gray value is inconsistent with the second gray value, the three-dimensional point cloud data of the dynamic obstacle in the current frame image is determined based on the divided block region;

[0022] The next frame of the current frame is taken as the next current frame, and the process of dividing the current frame according to the preset grid region is repeated until the three-dimensional point cloud data of the dynamic obstacle in the N frames is obtained.

[0023] Optionally, the first pose data includes the first coordinate data and the first direction of motion of the dynamic obstacle, and the step of determining the distance between the obstacle bounding box and the head-mounted device based on the first pose data includes:

[0024] The second motion direction of the head-mounted device in the frame image to be played is determined based on the second pose data of the head-mounted device, wherein the second pose data also includes the second coordinate data of the head-mounted device;

[0025] Determine whether the first direction of motion and the second direction of motion are moving in opposite directions;

[0026] If the first direction of motion and the second direction of motion are moving in opposite directions, then the distance between the obstacle enclosure and the head-mounted device is determined based on the first coordinate data and the second coordinate data.

[0027] Optionally, the first pose data further includes a first motion velocity of the dynamic obstacle, and the second pose data further includes a second motion velocity of the head-mounted device. After the step of determining whether the first motion direction and the second motion direction are moving in opposite directions, the obstacle detection method of the head-mounted device further includes:

[0028] If the first direction of motion and the second direction of motion are moving in the same direction, the time difference data is determined based on the time node of the frame image to be played and the time node of the object frame image, wherein the object frame image is the frame image preceding the frame image to be played.

[0029] The first movement trajectory of the dynamic obstacle is determined based on the time difference data and the first movement speed, and the second movement trajectory of the head-mounted device is determined based on the time difference data and the second movement speed.

[0030] Detect whether the first moving trajectory and the second moving trajectory have an intersection point in the image to be played, wherein the trajectory intersection point is the coordinate of the overlapping position between the first moving trajectory and the second moving trajectory;

[0031] If the first movement trajectory and the second movement trajectory intersect in the frame image to be played, then a prompt is given regarding the dynamic obstacle.

[0032] Optionally, prior to the step of acquiring consecutive image frames in the obstacle cues region, the obstacle detection method for the head-mounted device further includes:

[0033] In response to the safety boundary delineation command of the head-mounted device, the position coordinates of the head-mounted device in the preset virtual navigation field of view are obtained;

[0034] Based on the preset safety boundary, a length is defined with the location coordinates as the center to determine the obstacle warning area.

[0035] Furthermore, to achieve the above objectives, this application also provides an obstacle detection device for a head-mounted device, which includes:

[0036] The acquisition module is used to acquire consecutive image frames in the obstacle prompt area, wherein the consecutive image frames include at least the current frame image;

[0037] The construction module is used to construct the obstacle bounding box of the dynamic obstacle based on the consecutive image frames when the current frame image contains a dynamic obstacle;

[0038] The motion module is used to determine the first pose data of the dynamic obstacle based on the relative motion information between the obstacle enclosure and the head-mounted device, and to determine the distance between the obstacle enclosure and the head-mounted device based on the first pose data.

[0039] The prompting module is used to provide a prompt to the dynamic obstacle when the distance between the points matches a preset threat distance.

[0040] Each functional module of the obstacle detection device of the head-mounted device of this application implements the steps of the obstacle detection method of the head-mounted device of this application as described above during operation.

[0041] In addition, to achieve the above objectives, this application also provides a terminal device, which includes a memory, a processor, and an obstacle detection program for a head-mounted device stored in the memory and executable on the processor. When the obstacle detection program for the head-mounted device is executed by the processor, it implements the steps of the obstacle detection method for the head-mounted device described above.

[0042] In addition, to achieve the above objectives, this application also provides a medium, which is a computer-readable storage medium, on which an obstacle detection program for a head-mounted device is stored. When the obstacle detection program for the head-mounted device is executed by a processor, it implements the steps of the obstacle detection method for the head-mounted device described above.

[0043] This application provides an obstacle detection method for head-mounted devices that optimizes the obstacle avoidance function of existing head-mounted devices. Specifically, this application can monitor continuous image frames in an obstacle warning area in real time, and the continuous image frames include at least the current frame image. When a dynamic obstacle is detected in the current frame image, an obstacle bounding box for the dynamic obstacle is constructed based on the continuous image frames, thereby achieving real-time tracking of the dynamic obstacle. Then, based on the relative motion information between the obstacle bounding box and the head-mounted device, the first pose data of the dynamic obstacle can be accurately located and identified. Then, based on the first pose data, the distance between the obstacle bounding box and the head-mounted device is determined, improving the accuracy of obtaining the distance. Finally, the distance is compared with a preset threat distance. When the distance matches the preset threat distance, a warning is given for the dynamic obstacle, enabling the user to avoid the dynamic obstacle in time, thereby improving the obstacle detection performance of the head-mounted device and ensuring the safety of the user wearing the head-mounted device. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the first embodiment of the obstacle detection method for the head-mounted device of this application;

[0045] Figure 2 This is a flowchart illustrating the second embodiment of the obstacle detection method for the head-mounted device of this application;

[0046] Figure 3 This is a schematic diagram of a scenario involving an embodiment of the present application;

[0047] Figure 4 This is a schematic diagram of the obstacle detection device for a head-mounted device according to the embodiments of this application;

[0048] Figure 5 This is a schematic diagram of the structure of the terminal device involved in the embodiments of this application;

[0049] Figure 6 This is a schematic diagram of the structure of the storage medium involved in the embodiments of this application.

[0050] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] This application provides an obstacle detection method for a head-mounted device, referring to... Figure 1 As shown, Figure 1 This is a flowchart illustrating the first embodiment of the obstacle detection method for the head-mounted device of this application.

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0053] In this embodiment, the obstacle detection method of the head-mounted device of this application is executed by a terminal device that locates and identifies dynamic obstacles. The terminal device includes, but is not limited to, the head-mounted device and the computer associated with the head-mounted device. Specifically, the obstacle detection method of the head-mounted device described below is executed by the control center in the terminal device. The executing entity of this application will not be described again in the following embodiments.

[0054] The obstacle detection method for the head-mounted device in this application includes:

[0055] Step S10: Obtain consecutive image frames in the obstacle prompt area, wherein the consecutive image frames include at least the current frame image.

[0056] In this embodiment, a communication connection is established between the head-mounted device and the game controller. After the head-mounted device receives a safety boundary delineation command through the human-computer interaction module of the game controller, the terminal device is triggered to automatically obtain the position coordinates of the head-mounted device in the preset virtual navigation field of view based on the head-mounted device's response to the safety boundary delineation command. Then, based on the preset safety boundary delineation length and the position coordinates as the center, the obstacle prompting area can be accurately obtained, thereby effectively improving the intelligence of the head-mounted device. Then, the image acquisition device of the head-mounted device acquires continuous image frames in the obstacle prompting area in real time, so as to achieve the purpose of real-time monitoring of continuous image frames in the obstacle prompting area.

[0057] It should be noted that the human-computer interaction module includes, but is not limited to, the touch sensor built into the game controller and the button components of the game controller; the obstacle prompt area can be understood as the user-defined jurisdiction of monitoring obstacles within a preset virtual navigation field of view, and the shape of the jurisdiction includes, but is not limited to, a matrix, a triangle or a circle, and this application does not impose any restrictions here; continuous image frames can be understood as multiple images played sequentially in chronological order within a continuous time period, and continuous image frames include at least the current frame image, which can be understood as an image played at the current time point; the image acquisition device includes, but is not limited to, a binocular lens, which is two 360° panoramic cameras.

[0058] Step S20: When there are dynamic obstacles in the current frame image, construct the obstacle bounding box of the dynamic obstacle based on the continuous image frames.

[0059] In this embodiment, during the execution of the visual SLAM (Simultaneous Localization and Mapping) algorithm, the terminal device performs frame difference processing on two consecutive frames within a continuous time period according to a preset frame difference algorithm to obtain a frame difference image between the two consecutive frames. When the grayscale value of the frame difference image exceeds a preset grayscale threshold, it is determined that there is a dynamic obstacle in the current frame image, thereby effectively improving the accuracy of identifying dynamic obstacles in the obstacle warning area. Then, the continuous image frames are processed according to the RIO (Region of Interest) algorithm, which can accurately obtain the three-dimensional point cloud data of the dynamic obstacle in each frame image within a continuous time period. Then, based on the binocular vision mode of the binocular lens, local three-dimensional dense reconstruction is performed on the three-dimensional point cloud data in each frame image, which can accurately obtain the obstacle bounding box of the dynamic obstacle, thereby effectively improving the reliability of the head-mounted device in real-time tracking of dynamic obstacles.

[0060] It should be noted that the obstacle bounding box is also called the obstacle OBB (Oriented Bounding Box), and the size and orientation of the obstacle bounding box are determined based on the shape, size, and direction of movement of the dynamic obstacle.

[0061] Step S30: Determine the first pose data of the dynamic obstacle based on the relative motion information between the obstacle enclosure and the head-mounted device, and determine the distance between the obstacle enclosure and the head-mounted device based on the first pose data.

[0062] In this embodiment, the terminal device can accurately locate and identify the first pose data of the dynamic obstacle based on the relative motion information from the obstacle bounding box to the head-mounted device, and at the same time collect the second pose data of the head-mounted device through the inertial sensor in the head-mounted device, thereby eliminating the pose error caused by time delay; when the first motion direction in the first pose data and the second motion direction in the second pose data are moving in opposite directions, the distance between the obstacle bounding box and the head-mounted device can be accurately obtained based on the first coordinate data in the first pose data and the second coordinate data in the second pose data, thereby effectively improving the accuracy of obtaining the distance.

[0063] It should be noted that relative motion information can be understood as the continuous change in the relative position of the head-mounted device with respect to the obstacle bounding box.

[0064] Additionally, it should be noted that, in order to distinguish the pose data of the dynamic obstacle from the pose data of the head-mounted device in this application, the pose data of the dynamic obstacle is named the first pose data, and the pose data of the head-mounted device is named the second pose data. The first pose data includes the motion direction of the dynamic obstacle (i.e., the first motion direction), coordinate data (i.e., the first coordinate data), and motion speed (i.e., the first motion speed); the second pose data can be understood as the human body pose data representing the user, and the second pose data includes the motion direction of the head-mounted device (i.e., the second motion direction), coordinate data (i.e., the second coordinate data), and motion speed (i.e., the second motion speed).

[0065] Step S40: When the distance between the points matches the preset threat distance, a warning is given to the dynamic obstacle.

[0066] In this embodiment, the distance between the obstacle surround box and the head-mounted device is detected to be greater than or equal to a preset threat distance. If the distance is greater than or equal to the preset threat distance, it is determined that the distance matches the preset threat distance, and a preset warning command is automatically triggered to alert the user to the dynamic obstacle. This allows the user to avoid the dynamic obstacle in time, thereby improving the obstacle detection performance of the head-mounted device and ensuring the safety of the user when wearing the head-mounted device.

[0067] It should be noted that the preset threat distance can be customized according to the user's needs, and this application does not impose any restrictions on it; the preset warning commands include, but are not limited to, volume adjustment warnings, voice warnings, or vibration warnings.

[0068] In summary, this application provides an obstacle detection method for head-mounted devices that optimizes the obstacle avoidance function of existing head-mounted devices. Specifically, this application can monitor continuous image frames in the obstacle warning area in real time, and the continuous image frames include at least the current frame image. When a dynamic obstacle is detected in the current frame image, an obstacle bounding box for the dynamic obstacle is constructed based on the continuous image frames, thereby achieving real-time tracking of the dynamic obstacle. Then, based on the relative motion information between the obstacle bounding box and the head-mounted device, the first pose data of the dynamic obstacle can be accurately located and identified. Then, based on the first pose data, the distance between the obstacle bounding box and the head-mounted device is determined, improving the accuracy of obtaining the distance. Finally, the distance is compared with a preset threat distance. When the distance matches the preset threat distance, a warning is given for the dynamic obstacle, enabling the user to avoid the dynamic obstacle in time, thereby improving the obstacle detection performance of the head-mounted device and ensuring the safety of the user wearing the head-mounted device.

[0069] Furthermore, based on the first embodiment of the obstacle detection method for the head-mounted device of this application, a second embodiment of the obstacle detection method for the head-mounted device of this application is proposed, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the obstacle detection method for the head-mounted device of this application.

[0070] Furthermore, in some feasible embodiments, the continuous image frames also include target frame images. After step S20 above: acquiring continuous image frames in the obstacle cues region, the obstacle detection method of the head-mounted device further includes the following implementation steps:

[0071] Step A10: Perform frame difference processing on the current frame image and the target frame image to obtain a frame difference image, wherein the target frame image is the previous frame image of the current frame image;

[0072] In this embodiment, refer to Figure 3 , Figure 3 This is a schematic diagram of a scenario involved in an embodiment of this application. The terminal device determines the first grayscale value of the current frame image and the second grayscale value of the target frame image, and then inputs the first grayscale value and the second grayscale value to the frame difference processing module. In the frame difference processing module, the grayscale difference between the first grayscale value and the second grayscale value is determined according to a preset frame difference algorithm. Based on the absolute value of the grayscale difference, the frame difference image can be accurately obtained, and then dynamic obstacles can be effectively identified based on the grayscale values ​​of the frame difference image.

[0073] It should be noted that the target frame image is the frame image preceding the current frame image; the expression of the preset frame difference algorithm is shown in the following formula (1):

[0074] D n (x0, y0) = |F n (x i y i )-F n-1 (x j y j )|

[0075] Among them, D n For frame difference images, D n (x0, y0) are the grayscale values ​​of the frame difference image, F n For the current frame image, F n (x i y i F is the first grayscale value of the current frame image. n-1 For the target frame image, F n-1 (x j y j(x0, y0) represents the second grayscale value of the target frame image, and (x0, y0) represents the coordinate data of each pixel in the frame difference image. i y i (x) represents the coordinate data of each pixel in the current frame image. j y j ) represents the coordinate data of each pixel in the target frame image. In addition, the gray value is used to represent the absolute value of the gray value difference between the first gray value and the second gray value.

[0076] Step A20: Detect whether the grayscale value of the frame difference image exceeds a preset grayscale threshold;

[0077] Step A30: If the grayscale value does not exceed the grayscale threshold, then the current frame image is used as the next target frame image, and the step of performing frame difference processing based on the current frame image and the target frame image is returned to be executed until the grayscale value exceeds the grayscale threshold, then it is determined that there is a dynamic obstacle in the current frame image.

[0078] In this embodiment, the terminal device detects whether the grayscale value of the frame difference image exceeds a preset grayscale threshold. If the grayscale value does not exceed the preset grayscale threshold (i.e., the grayscale value is less than the preset grayscale threshold), the current frame image is used as the next target frame image, and the step of performing frame difference processing based on the current frame image and the target frame image is returned to be executed until the grayscale value exceeds the grayscale threshold (i.e., the grayscale value is greater than or equal to the preset grayscale threshold). Then, it is determined that there is a dynamic obstacle in the current frame image, thereby enabling the head-mounted device to accurately identify obstacles and effectively improve the obstacle detection performance of the head-mounted device.

[0079] Furthermore, in some other feasible embodiments, the continuous image frames also include N frames following the current frame image, where N is a natural number greater than 0. Step S20 above, constructing the obstacle bounding box of the dynamic obstacle based on the continuous image frames, may further include the following implementation steps:

[0080] Step S201: Extract obstacle features based on the current frame image and the N frame images to obtain the three-dimensional point cloud data of the dynamic obstacle;

[0081] In this embodiment, during the RIO algorithm operation, the terminal device divides the current frame image according to a preset grid region to obtain the processed segmented block region, and determines the ideal image frame corresponding to the current frame image. It accurately obtains the ideal block region matching the segmented block region within the ideal image frame, and compares the gray values ​​of the first gray value of the segmented block region with the second gray value of the matching ideal block region. When the first gray value and the second gray value are inconsistent, the three-dimensional point cloud data of the dynamic obstacle in the current frame image can be determined promptly based on the segmented block region. Then, the next frame image is used as the next current frame image, and the process of dividing the current frame image according to the preset grid region is repeated until the three-dimensional point cloud data of the dynamic obstacle in N frames is obtained. This effectively improves the extraction efficiency of the three-dimensional point cloud data of the dynamic obstacle, thereby effectively improving the obstacle detection performance of the head-mounted device.

[0082] It should be noted that 3D point cloud data can be understood as a set of vectors used to represent dynamic obstacles in a preset 3D coordinate system, which can intuitively represent information such as the shape, surface and texture of objects in space.

[0083] Additionally, it should be noted that the binocular lens designed in this application includes two cameras. The terminal device can simultaneously acquire continuous image frames captured by each of the two cameras in the obstacle indication area. That is, the three-dimensional point cloud data includes first three-dimensional point cloud data and second three-dimensional point cloud data. There are multiple first three-dimensional point cloud data and multiple second three-dimensional point cloud data. The first three-dimensional point cloud data is obtained by extracting obstacle features from the current frame image captured by one camera in the binocular lens and the N frames following the current frame image. The second three-dimensional point cloud data is obtained by extracting obstacle features from the current frame image captured by the other camera in the binocular lens and the N frames following the current frame image.

[0084] Step S202: Construct an obstacle bounding box based on the parallax features of the head-mounted device and the three-dimensional point cloud data.

[0085] In this embodiment, the terminal device first determines the calibration parameters of the binocular lens, and then accurately obtains the parallax characteristics of the head-mounted device based on the calibration parameters. Then, based on the parallax characteristics, it performs local dense reconstruction of each first three-dimensional point cloud data and the second three-dimensional point cloud data matched by each first three-dimensional point cloud data. Based on the reconstructed three-dimensional point cloud data, it constructs an obstacle bounding box so that the movement direction and speed of dynamic obstacles in the preset virtual navigation field of view can be tracked in real time through the obstacle bounding box.

[0086] Furthermore, in some feasible embodiments, step S201 above: extracting obstacle features based on the current frame image and the N frame images to obtain the three-dimensional point cloud data of the dynamic obstacle, may also include the following implementation steps:

[0087] Step S2011: Divide the current frame image into regions according to a preset grid area to obtain the processed partitioned regions, and determine the ideal image frame corresponding to the current frame image.

[0088] In this embodiment, the terminal device inputs the current frame image into the RIO algorithm model. In the RIO algorithm model, the current frame image is divided according to the preset grid area, so that the processed block area can be obtained in time and the ideal image frame corresponding to the current frame image can be determined.

[0089] It should be noted that the preset grid areas include, but are not limited to, 3*3 matrices and 5*5 matrices, which can be customized according to user needs. This application does not impose any restrictions on them. The processed segmented block area can be understood as a segmented image frame with the same size as the 3*3 matrix or 5*5 matrix, and there are multiple segmented block areas. The ideal image frame can be understood as the initial image frame, that is, the image frame that has not entered the dynamic obstacle.

[0090] Step S2012: Obtain an ideal block region that matches the segmented block region in the ideal image frame, and compare the gray values ​​based on the first gray value of the segmented block region and the second gray value of the matching ideal block region.

[0091] In this embodiment, the terminal device obtains ideal block regions that match the segmented block regions in the ideal image frame. That is, the terminal device first determines the region coordinates of each segmented block region, and then obtains multiple ideal block regions that match the segmented block regions from the ideal image frame according to the region coordinates. Then, the first gray value of the segmented block region and the second gray value of the matching ideal block region are compared. Specifically, it detects whether each first gray value is the same as its corresponding second gray value. When there is a target gray value that is different from the matching second gray value, it is determined that the target gray value and the matching second gray value are inconsistent. In other words, by comparing the first gray value of the segmented block region and the second gray value of the matching ideal block region, the three-dimensional point cloud data of dynamic obstacles in each frame of the image can be quickly located and identified.

[0092] It should be noted that the target grayscale value is any one of the multiple first grayscale values.

[0093] Step S2013: When the first gray value is inconsistent with the second gray value, the three-dimensional point cloud data of the dynamic obstacle in the current frame image is determined according to the divided block region.

[0094] In this embodiment, when the target gray value is different from the matching second gray value, it is determined that the target gray value and the matching second gray value are inconsistent. Then, the division block region specified by the target gray value is determined, and the three-dimensional point cloud data in the division block region is used as the three-dimensional point cloud data of the dynamic obstacle in the current frame image, thereby effectively improving the efficiency of extracting the three-dimensional point cloud data of the dynamic obstacle.

[0095] Step S2014: Take the next frame image of the current frame image as the next current frame image, and return to execute the step of dividing the current frame image according to the preset grid area until the three-dimensional point cloud data of the dynamic obstacle in the N frames image is obtained.

[0096] In this embodiment, the terminal device takes the next frame image after the current frame image as the next current frame image and returns to execute the step of dividing the current frame image according to the preset grid area until the three-dimensional point cloud data of the dynamic obstacle in the N frames after the current frame image is obtained, thereby ensuring the integrity of the extracted three-dimensional point cloud data of the dynamic obstacle.

[0097] The N frames following the current frame can be customized according to the user's needs. This application does not impose any restrictions on this, and N is a natural number greater than 0.

[0098] Furthermore, in some other feasible embodiments, the first pose data includes the first coordinate data and the first direction of motion of the dynamic obstacle. Step S30 above, which determines the distance between the obstacle bounding box and the head-mounted device based on the first pose data, may further include the following implementation steps:

[0099] Step S301: Determine the second motion direction of the head-mounted device in the frame image to be played based on the second pose data of the head-mounted device, wherein the second pose data also includes the second coordinate data of the head-mounted device.

[0100] In this embodiment, at the same time the terminal device determines the first pose data of the dynamic obstacle, it also collects the second pose data of the head-mounted device through the inertial sensor in the head-mounted device, thereby eliminating the pose error caused by time delay. Then, based on the second pose data of the head-mounted device, the movement direction of the head-mounted device in the frame image to be played is predicted to obtain the second movement direction of the head-mounted device in the frame image to be played.

[0101] It should be noted that the second pose data also includes the second coordinate data and the second motion velocity of the head-mounted device.

[0102] Step S302: Determine whether the first direction of motion and the second direction of motion are moving in opposite directions;

[0103] Step S303: If the first direction of motion and the second direction of motion are moving in opposite directions, then determine the distance between the obstacle enclosure and the head-mounted device based on the first coordinate data and the second coordinate data.

[0104] In this embodiment, it is determined whether the first motion direction and the second motion direction are moving in opposite directions. When the first motion direction in the first pose data and the second motion direction in the second pose data are moving in opposite directions, the distance between the obstacle bounding box and the head-mounted device can be accurately obtained based on the first coordinate data in the first pose data and the second coordinate data in the second pose data, thereby effectively improving the accuracy of obtaining the distance.

[0105] It should be noted that "opposite movement" can be understood as relative movement, that is, the obstacle enclosure and the head-mounted device moving face to face.

[0106] Furthermore, in some other feasible embodiments, the first pose data further includes the first motion velocity of the dynamic obstacle, and the second pose data further includes the second motion velocity of the head-mounted device. After step S302 above: determining whether the first motion direction and the second motion direction are moving in opposite directions, the obstacle detection method of the head-mounted device may further include the following implementation steps:

[0107] Step B10: If the first motion direction and the second motion direction are moving in the same direction, determine the time difference data based on the time node of the frame image to be played and the time node of the object frame image, wherein the object frame image is the frame image preceding the frame image to be played.

[0108] In this embodiment, if the first direction of motion and the second direction of motion are moving in the same direction, the terminal device determines the time difference data based on the time node of the frame image to be played and the time node of the object frame image, so as to subsequently obtain the obstacle bounding box and the corresponding movement trajectory of the head-mounted device.

[0109] It should be noted that the object frame image is the frame image preceding the frame image to be played; running in the same direction can be understood as the obstacle bounding box and the head-mounted device moving in the same direction.

[0110] Step B20: Determine the first movement trajectory of the dynamic obstacle based on the time difference data and the first movement speed, and determine the second movement trajectory of the head-mounted device based on the time difference data and the second movement speed.

[0111] In this embodiment, the terminal device can accurately obtain the first movement trajectory of the dynamic obstacle based on the product of the time difference data and the first movement speed, and then accurately obtain the second movement trajectory of the head-mounted device based on the product of the time difference data and the second movement speed.

[0112] Step B30: Detect whether the first moving trajectory and the second moving trajectory have an intersection point in the image to be played, wherein the intersection point is the coordinate of the overlapping position between the first moving trajectory and the second moving trajectory;

[0113] In this embodiment, the terminal device detects whether the first moving trajectory and the second moving trajectory have an intersection point in the image to be played, wherein the trajectory intersection point is the coordinate of the overlapping position between the first moving trajectory and the second moving trajectory.

[0114] Step B40: If the first movement trajectory and the second movement trajectory intersect in the image to be played, then a prompt is given regarding the dynamic obstacle.

[0115] In this embodiment, if the first movement trajectory and the second movement trajectory intersect in the image to be played, a preset warning command is automatically triggered to alert the user to the dynamic obstacle, thereby enabling the user to avoid the dynamic obstacle in time, thus improving the obstacle detection performance of the head-mounted device and ensuring the safety of the user wearing the head-mounted device.

[0116] Furthermore, in some feasible embodiments, before step S10: acquiring consecutive image frames in the obstacle cues region, the obstacle detection method for the head-mounted device further includes:

[0117] Step C10: In response to the safety boundary delineation command of the head-mounted device, obtain the position coordinates of the head-mounted device in the preset virtual navigation field of view.

[0118] In this embodiment, a communication connection is established between the head-mounted device and the game controller. After the head-mounted device receives a safety boundary delineation instruction through the human-computer interaction module of the game controller, the terminal device is triggered to automatically obtain the position coordinates of the head-mounted device in the preset virtual navigation field of view based on the head-mounted device's response to the safety boundary delineation instruction, which effectively improves the intelligence of the head-mounted device.

[0119] Step C20: Determine the obstacle warning area based on the preset safety boundary and the location coordinates.

[0120] In this embodiment, the terminal device collects the position coordinates of the head-mounted device in real time. That is, the position coordinates of the head-mounted device are real-time data, namely real-time position coordinates. Then, with the real-time position coordinates as the center and a preset safety boundary as the radius, the obstacle prompt area in the preset virtual navigation field of view can be accurately located, thereby effectively improving the head-mounted device's targeted detection of dynamic obstacles in the preset virtual navigation field of view.

[0121] In summary, this application, during the operation of the visual SLAM algorithm, acquires continuous image frames in the obstacle warning area to monitor unpredictable dynamic obstacles in real time. Then, by estimating the relationship between the dynamic obstacles and the pose of the head-mounted device in real time, it utilizes collision detection methods to further improve the safety of the head-mounted device during use.

[0122] In addition, this application also provides an obstacle detection device for a head-mounted device, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the obstacle detection device for a head-mounted device involved in the embodiments of this application.

[0123] The obstacle detection device for the head-mounted device of this application includes:

[0124] The acquisition module H10 is used to acquire continuous image frames in the obstacle prompt area, wherein the continuous image frames include at least the current frame image;

[0125] The construction module H20 is used to construct the obstacle bounding box of the dynamic obstacle based on the consecutive image frames when the current frame image contains a dynamic obstacle;

[0126] The motion module H30 is used to determine the first pose data of the dynamic obstacle based on the relative motion information between the obstacle enclosure box and the head-mounted device, and to determine the distance between the obstacle enclosure box and the head-mounted device based on the first pose data.

[0127] The prompting module H40 is used to provide a prompt to the dynamic obstacle when the distance between the points matches the preset threat distance.

[0128] Optionally, the obstacle detection device of the head-mounted device further includes a detection module, which includes:

[0129] A frame difference processing unit is used to perform frame difference processing based on the current frame image and the target frame image to obtain a frame difference image, wherein the target frame image is the previous frame image of the current frame image;

[0130] The detection unit is used to detect whether the grayscale value of the frame difference image exceeds a preset grayscale threshold.

[0131] The execution unit is returned to the next execution unit. If the grayscale value does not exceed the grayscale threshold, the current frame image is used as the next target frame image, and the execution of the frame difference processing based on the current frame image and the target frame image is returned until the grayscale value exceeds the grayscale threshold, at which point it is determined that there is a dynamic obstacle in the current frame image.

[0132] Optionally, the building block H20 also includes:

[0133] The extraction unit is used to extract obstacle features based on the current frame image and the N frame images to obtain the three-dimensional point cloud data of the dynamic obstacle;

[0134] A construction unit is used to construct an obstacle bounding box based on the parallax features of the head-mounted device and the three-dimensional point cloud data.

[0135] Optionally, the building block H20 also includes:

[0136] The segmentation unit is used to segment the current frame image according to a preset grid region to obtain the processed segmented block region and determine the ideal image frame corresponding to the current frame image.

[0137] The comparison unit is used to obtain an ideal block region that matches the segmented block region in the ideal image frame, and to compare the gray values ​​based on the first gray value of the segmented block region and the second gray value of the matching ideal block region.

[0138] A point cloud data determination unit is used to determine the three-dimensional point cloud data of the dynamic obstacle in the current frame image based on the divided block region when the first gray value and the second gray value are inconsistent.

[0139] The return unit is used to take the next frame image of the current frame image as the next current frame image and return to execute the step of dividing the current frame image according to the preset grid area until the three-dimensional point cloud data of the dynamic obstacle in the N frames image is obtained.

[0140] Optionally, the motion module H30 also includes:

[0141] The unit to be played is used to determine the second motion direction of the head-mounted device in the frame image to be played based on the second pose data of the head-mounted device, wherein the second pose data also includes the second coordinate data of the head-mounted device;

[0142] The judgment unit is used to determine whether the first direction of motion and the second direction of motion are moving towards each other;

[0143] An opposing movement unit is used to determine the distance between the obstacle enclosure and the head-mounted device based on the first coordinate data and the second coordinate data if the first movement direction and the second movement direction are moving in opposite directions.

[0144] Optionally, the motion module H30 also includes:

[0145] A co-directional running unit is used to determine time difference data based on the time node of the frame image to be played and the time node of the object frame image if the first motion direction and the second motion direction are co-directional. The object frame image is the frame image preceding the frame image to be played.

[0146] A mobile unit is configured to determine a first movement trajectory of the dynamic obstacle based on the time difference data and the first movement speed, and to determine a second movement trajectory of the head-mounted device based on the time difference data and the second movement speed.

[0147] The detection unit is used to detect whether the first moving trajectory and the second moving trajectory have an intersection point in the image to be played, wherein the intersection point is the coordinate of the overlapping position between the first moving trajectory and the second moving trajectory;

[0148] The prompting unit is used to prompt the dynamic obstacle if the first movement trajectory and the second movement trajectory have an intersection point in the image to be played.

[0149] Optionally, the obstacle detection device of the head-mounted device further includes a region delineation module, which includes:

[0150] A response unit is used to respond to a safety boundary delineation command of the head-mounted device and obtain the position coordinates of the head-mounted device in a preset virtual navigation field of view;

[0151] The boundary delineation unit is used to determine the obstacle warning area centered on the position coordinates based on the preset safety boundary delineation length.

[0152] Each functional module of the obstacle detection device of the head-mounted device of this application implements the steps of the obstacle detection method of the head-mounted device of this application as described above during operation.

[0153] In addition, this application also provides a terminal device. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of the terminal device involved in the embodiments of this application. Specifically, the terminal device in this application embodiment may be an obstacle detection device for a locally running head-mounted device.

[0154] like Figure 5As shown, the terminal device in this embodiment may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0155] The memory 1005 is disposed on the main body of the terminal device. The memory 1005 stores a program that performs corresponding operations when executed by the processor 1001. The memory 1005 is also used to store parameters for use by the terminal device. The memory 1005 can be a high-speed RAM or a stable, non-volatile memory, such as a disk drive. Optionally, the memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0156] Those skilled in the art will understand that Figure 5 The terminal device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0157] like Figure 5 As shown, the memory 1005, which serves as a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and an obstacle detection program for the head-mounted device of the terminal equipment.

[0158] exist Figure 5 In the terminal device shown, the processor 1001 can be used to call the obstacle detection program of the head-mounted device stored in the memory 1005 and execute the steps of the obstacle detection method of the head-mounted device described above.

[0159] In addition, please refer to Figure 6 , Figure 6 This is a schematic diagram of the storage medium involved in the embodiments of this application. This application provides a medium, which is a computer-readable storage medium. The computer-readable storage medium stores an obstacle detection program for a head-mounted device. When the obstacle detection program for the head-mounted device is executed by a processor, it implements the steps of the obstacle detection method for the head-mounted device described above.

[0160] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0161] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0163] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method of obstacle detection for a head-mounted device, the method comprising: The obstacle detection method for the head-mounted device includes: Acquire consecutive image frames in the obstacle warning area, wherein the consecutive image frames include at least the current frame image; When there are dynamic obstacles in the current frame image, an obstacle bounding box for the dynamic obstacles is constructed based on the consecutive image frames; The first pose data of the dynamic obstacle is determined based on the relative motion information between the obstacle bounding box and the head-mounted device. The first pose data includes the first coordinate data, the first direction of motion, and the first velocity of the dynamic obstacle. At the same time as determining the first pose data of the dynamic obstacle, the second pose data output by the inertial sensor built into the head-mounted device is simultaneously acquired to eliminate the time delay error caused by the asynchronous acquisition of the first and second poses. Then, the second direction of motion of the head-mounted device in the frame image to be played is determined based on the second pose data of the head-mounted device. The second pose data also includes the second coordinate data and the second velocity of the head-mounted device. The system determines whether the first and second movement directions are moving in opposite directions. If they are, it determines the distance between the obstacle bounding box and the head-mounted device based on the first and second coordinate data. If they are moving in the same direction, it determines the time difference data based on the time node of the frame image to be played and the time node of the object frame image, where the object frame image is the frame image preceding the frame image to be played. The system then determines the first movement trajectory of the dynamic obstacle based on the time difference data and the first movement speed, and the second movement trajectory of the head-mounted device based on the time difference data and the second movement speed. It detects whether the first and second movement trajectories intersect in the frame image to be played, where the intersection point is the coordinate of the overlapping position between the first and second movement trajectories. If the first and second movement trajectories intersect in the frame image to be played, the system provides a prompt regarding the dynamic obstacle. When the distance between the points matches the preset threat distance, a warning is given for the dynamic obstacle.

2. The method of claim 1, wherein the head-mounted device is a head-mounted display (HMD) device. The continuous image frames also include target frame images. After the step of acquiring the continuous image frames in the obstacle cues region, the obstacle detection method for the head-mounted device further includes: Frame difference processing is performed on the current frame image and the target frame image to obtain a frame difference image, wherein the target frame image is the frame image preceding the current frame image; Detect whether the grayscale value of the frame difference image exceeds a preset grayscale threshold; If the grayscale value does not exceed the grayscale threshold, the current frame image is used as the next target frame image, and the step of performing frame difference processing based on the current frame image and the target frame image is returned to be executed until the grayscale value exceeds the grayscale threshold, at which point it is determined that there is a dynamic obstacle in the current frame image.

3. The obstacle detection method for a head-mounted device as described in claim 1, characterized in that, The continuous image frames also include N image frames following the current image frame, where N is a natural number greater than 0; The step of constructing the obstacle bounding box of the dynamic obstacle based on the consecutive image frames includes: Obstacle features are extracted based on the current frame image and the N frame images to obtain the three-dimensional point cloud data of the dynamic obstacle; An obstacle bounding box is constructed based on the parallax characteristics of the head-mounted device and the three-dimensional point cloud data.

4. The obstacle detection method for a head-mounted device as described in claim 3, characterized in that, The step of extracting obstacle features based on the current frame image and the N frame images to obtain the point cloud data of the dynamic obstacle includes: The current frame image is divided into regions according to a preset grid area to obtain the processed region and the ideal image frame corresponding to the current frame image is determined. In the ideal image frame, an ideal block region matching the segmented block region is obtained, and a gray value comparison is performed based on the first gray value of the segmented block region and the second gray value of the matching ideal block region. When the first gray value is inconsistent with the second gray value, the three-dimensional point cloud data of the dynamic obstacle in the current frame image is determined based on the divided block region; The next frame of the current frame is taken as the next current frame, and the process of dividing the current frame according to the preset grid region is repeated until the three-dimensional point cloud data of the dynamic obstacle in the N frames is obtained.

5. The obstacle detection method for a head-mounted device as described in claim 1, characterized in that, Prior to the step of acquiring consecutive image frames in the obstacle cue region, the obstacle detection method for the head-mounted device further includes: In response to the safety boundary delineation command of the head-mounted device, the position coordinates of the head-mounted device in the preset virtual navigation field of view are obtained; Based on the preset safety boundary, a length is defined with the location coordinates as the center to determine the obstacle warning area.

6. An obstacle detection device for a head-mounted device, characterized in that, The obstacle detection device of the head-mounted device includes: The acquisition module is used to acquire consecutive image frames in the obstacle prompt area, wherein the consecutive image frames include at least the current frame image; The construction module is used to construct the obstacle bounding box of the dynamic obstacle based on the consecutive image frames when the current frame image contains a dynamic obstacle; A motion module is used to determine the first pose data of the dynamic obstacle based on the relative motion information from the obstacle bounding box to the head-mounted device, and to determine the distance between the obstacle bounding box and the head-mounted device based on the first pose data. The first pose data includes the first coordinate data, the first direction of motion, and the first velocity of the dynamic obstacle. Specifically, the motion module is used to simultaneously acquire the second pose data output by the inertial sensor built into the head-mounted device at the same time as determining the first pose data of the dynamic obstacle, eliminating the time delay error caused by the asynchronous acquisition of the first and second poses. Then, based on the second pose data of the head-mounted device, it determines the second direction of motion of the head-mounted device in the frame image to be played. The second pose data also includes the second coordinate data and the second velocity of the head-mounted device. It also determines whether the first direction of motion and the second direction of motion are moving in opposite directions. If the first direction of motion and the second direction of motion are moving in opposite directions, the module determines the second direction of motion of the head-mounted device in the frame image to be played. If the second movement direction is opposite to the first, then the distance between the obstacle bounding box and the head-mounted device is determined based on the first and second coordinate data. If the first and second movement directions are in the same direction, time difference data is determined based on the time node of the frame image to be played and the time node of the object frame image, where the object frame image is the frame image preceding the frame image to be played. The first movement trajectory of the dynamic obstacle is determined based on the time difference data and the first movement speed, and the second movement trajectory of the head-mounted device is determined based on the time difference data and the second movement speed. It is detected whether the first and second movement trajectories intersect in the frame image to be played, where the intersection point is the coordinate of the overlapping position between the first and second movement trajectories. If the first and second movement trajectories intersect in the frame image to be played, then a prompt is given regarding the dynamic obstacle. The prompting module is used to provide a prompt to the dynamic obstacle when the distance between the points matches a preset threat distance.

7. A terminal device, characterized in that, The terminal device includes a memory, a processor, and an obstacle detection program for the head-mounted device stored in the memory and executable on the processor. When the processor executes the obstacle detection program for the head-mounted device, it implements the steps of the obstacle detection method for the head-mounted device as described in any one of claims 1 to 5.

8. A medium, said medium being a computer-readable storage medium, characterized in that, The computer-readable storage medium stores an obstacle detection program for a head-mounted device, which, when executed by a processor, implements the steps of the obstacle detection method for a head-mounted device as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • System and method for detection and tracking of moving objects

    CN103679698A

  • Safety prompt information display method and virtual display equipment

    CN115147747A

  • XR equipment and obstacle information sensing method of XR equipment

    CN116912403A