Methods and devices for early warning of obstacles in virtual reality scenes

Abstract

This application relates to the field of virtual reality technology, providing a method and device for early warning of obstacles in virtual reality scenes. For each obstacle detected by a lightweight deep learning network, the method predicts and smooths the obstacle's position information in the current image frame. Simultaneously, it matches obstacles in different image frames captured by the same camera, enabling cross-frame propagation of the same obstacle. This ensures that the subsequently calculated 3D positions of obstacle key points change continuously over time, mitigating the problem of obstacle position jitter or abrupt changes in the camera coordinate system and improving the accuracy and stability of obstacle detection. Furthermore, based on the depth information determined by both binocular and monocular images, a fine-tuning coefficient is calculated to fine-tune the 3D positions of obstacle key points, further improving the accuracy of obstacle detection, preventing collisions between obstacles and the user, and thus enhancing user safety.

Description

Technical Field

[0001] This application relates to the field of virtual reality (VR) technology, and provides a method and device for early warning of obstacles in a virtual reality scene. Background Technology

[0002] Virtual reality (VR) technology is popular among users, especially in the gaming industry, because it provides an immersive experience. When users wear VR devices, they are isolated from the real world. Therefore, to ensure a safe and immersive experience, VR devices typically require a designated safety zone. Once a user leaves this zone, the VR device stops displaying the virtual world and switches to showing the real world to prevent accidents.

[0003] In practical applications, during immersive experiences, the complex real-world environment often fails to recognize existing static obstacles (such as chairs, coffee tables, etc.) within a designated safe zone, making these obstacles appear as if they were already within the safe zone. This creates a risk of collision for users moving within the safe zone. Furthermore, because the user's field of vision is completely covered by the virtual screen, they are unable to perceive dynamic obstacles (such as people, pets, etc.) that intrude into the safe zone, again posing a risk of collision.

[0004] Therefore, obstacle detection and timely warning are essential when users wear VR devices for an immersive experience. Summary of the Invention

[0005] This application provides a method and device for detecting obstacles in a virtual reality scene to improve the safety of VR users.

[0006] On one hand, embodiments of this application provide a method for early warning of obstacles in a virtual reality scene, applied to a VR device including a stereo camera, comprising:

[0007] Obstacle detection is performed on the RGB images captured by the camera. When at least one obstacle is detected, the two-dimensional position information of each obstacle in consecutive image frames is output.

[0008] For each obstacle, the predicted position information of the obstacle in the current image frame is determined based on the target position information of the obstacle in the historical image frames.

[0009] The obstacles in different image frames captured by the same camera are matched, and the predicted position information of the obstacles is smoothed to obtain the target position information;

[0010] Based on the target 2D coordinates of the obstacle key points contained in the target location information in the current image frame captured by the main camera, the initial 3D coordinates of the obstacle key points are determined.

[0011] Based on the depth information between the obstacle key points and the main camera, the initial 3D coordinates of the obstacle key points are fine-tuned to obtain the target 3D coordinates;

[0012] Safety warnings are issued based on the target 3D coordinates of the key points of the obstacle.

[0013] On the other hand, this application provides a virtual reality device, including a processor, a memory, and a binocular camera, wherein the binocular camera, the memory, and the processor are connected via a bus;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] Obstacle detection is performed on the RGB images captured by the camera. When at least one obstacle is detected, the two-dimensional position information of each obstacle in consecutive image frames is output.

[0016] For each obstacle, the predicted position information of the obstacle in the current image frame is determined based on the target position information of the obstacle in the historical image frames.

[0017] The obstacles in different image frames captured by the same camera are matched, and the predicted position information of the obstacles is smoothed to obtain the target position information;

[0018] Based on the target 2D coordinates of the obstacle key points contained in the target location information in the current image frame captured by the main camera, the initial 3D coordinates of the obstacle key points are determined.

[0019] Based on the depth information between the obstacle key points and the main camera, the initial 3D coordinates of the obstacle key points are fine-tuned to obtain the target 3D coordinates;

[0020] Safety warnings are issued based on the target 3D coordinates of the key points of the obstacle.

[0021] On the other hand, this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer device to execute the obstacle detection method in a virtual reality scene provided in the embodiments of this application.

[0022] The obstacle detection method and device in the virtual reality scene provided in this application embodiment detect obstacles by using RGB images captured by the VR device camera. For each detected obstacle, the position information of the obstacle in the current image frame is predicted based on the target position information of the obstacle in historical image frames. After matching obstacles in different image frames captured by the same camera, the predicted position information is smoothed. By predicting and smoothing the position information of the obstacle, the three-dimensional position of the subsequently calculated obstacle key points achieves a continuous change in time sequence, alleviating the problem of position jitter or abrupt change of the obstacle in the camera coordinate system, and improving the accuracy and stability of obstacle detection. Furthermore, based on the target 2D coordinates of the obstacle key points contained in the target position information of the obstacle in the current image frame captured by the smoothed main camera, the initial 3D coordinates of the obstacle key points are determined. Based on the depth information between the obstacle key points and the main camera, the initial 3D coordinates of the obstacle key points are fine-tuned to obtain more accurate target 3D coordinates. Thus, a safety warning is given based on the target 3D coordinates to prevent users from colliding with obstacles and improve user safety. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A structural diagram of a lightweight 3D object detection network provided in an embodiment of this application;

[0025] Figure 2A A schematic diagram of a VR device including a binocular camera, provided for an embodiment of this application;

[0026] Figure 2B A schematic diagram of another VR device including a binocular camera provided in an embodiment of this application;

[0027] Figure 3 A flowchart of an obstacle warning method provided in an embodiment of this application;

[0028] Figure 4 A flowchart for smoothing the 2D coordinates of key points of an obstacle provided in this application embodiment;

[0029] Figure 5 A flowchart for smoothing the absolute scale of obstacles provided in this application embodiment;

[0030] Figure 6A flowchart illustrating a method for fine-tuning the 3D coordinates of an obstacle, as provided in an embodiment of this application.

[0031] Figure 7 A flowchart illustrating the fine-tuning coefficients for determining the critical 3D coordinates of an obstacle, as provided in this embodiment of the application.

[0032] Figure 8 A schematic diagram of a security warning provided in an embodiment of this application;

[0033] Figure 9 Another security warning diagram provided for an embodiment of this application;

[0034] Figure 10 A structural diagram of a virtual reality device provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0036] With the development of computer vision, especially object detection, object tracking, key point estimation and image processing technologies, obstacle detection is essential in virtual reality scenarios to avoid the dangers posed to VR users by existing static obstacles or later-entering dynamic obstacles within the safe area.

[0037] In obstacle detection tasks, 3D object detection technology plays a crucial role. Existing visual 3D object detection technologies are mainly divided into image-based object detection methods and point cloud-based object detection methods.

[0038] Image-based target detection methods use RGB or depth images to estimate the 3D position of a target. These methods require sophisticated 3D target detection algorithms, typically necessitating the design of intricate neural network models. Furthermore, depth images require VR devices to be equipped with depth cameras, resulting in high hardware requirements.

[0039] Point cloud-based target detection methods use 3D point clouds to estimate the three-dimensional position of a target. Typically, VR devices need to be equipped with sensors (such as LiDAR, scanners, etc.) to acquire 3D point clouds, which increases development costs and device power consumption.

[0040] Meanwhile, neither image-based nor point cloud-based object detection methods can meet the real-time operation requirements in scenarios with low computing resources.

[0041] In view of this, this application provides a method or device for early warning of obstacles in a virtual reality scene based on deep learning algorithms and PnP algorithms. It uses a lightweight deep learning network to perform 3D target detection on images captured by at least one camera, improving target detection efficiency and providing strong real-time performance. Furthermore, it performs temporal smoothing on the detection results of monocular images, ensuring that the 3D positions of obstacle key points calculated by the PnP algorithm change continuously over time, mitigating the problem of position jitter or abrupt changes in the camera coordinate system and improving the accuracy of obstacle detection. Simultaneously, it fine-tunes the 3D positions of obstacle key points calculated by the PnP algorithm based on the depth information determined by binocular images, further improving the accuracy of obstacle detection.

[0042] See Figure 1 This is a structural diagram of a lightweight 3D object detection network provided in an embodiment of this application. The network structure is built based on a deep learning algorithm and includes nine modules, D1 to D9. Wherein:

[0043] Module D1 performs convolution and ReLU activation function processing on the input data. Modules D2 to D5 perform downsampling processing through residual units. The structure of each residual unit is as follows: Figure 1 As shown in (a), D1 to D5 constitute the backbone network.

[0044] Modules D6 to D8 are upsampling units. Each upsampling unit upsamples the width and height of the input feature map by a factor of 2 and reduces the number of channels in the input feature map. The structure of the upsampling unit is as follows: Figure 1 As shown in (b) of the diagram.

[0045] The D9 module is a multi-head network, consisting of 7 identical head modules, such as... Figure 1 As shown in (c), the D9 module is used to regress: the target's two-dimensional center point, the target's local offset, the two-dimensional target key points (heatmap), the key point's local offset, the key point's offset relative to the center point, the width and height of the target's rectangular region, and the target's absolute scale (unit: meters).

[0046] Compared with existing network structures, the structure of the above-mentioned 3D object detection network includes at least one of the following optimizations:

[0047] In multiple residual units (ResBlock) of the backbone network, depthwise separable convolutional kernels (DwConv) are used to replace conventional convolutional kernels. In deformable convolutions, depthwise separable convolutional kernels are used to generate offset predictions and mask predictions. Specifically, the depthwise separable convolutional kernel uses a 3×3 channel-wise convolutional kernel and a 1×1 ordinary convolutional kernel instead of a conventional 3×3 convolutional kernel. This effectively reduces the number of parameters and computational load without significantly affecting the detection performance.

[0048] In the backbone network, multiple residual units used for downsampling are interconnected with the previous sampling unit (UpSample);

[0049] The upsampling unit upsamples the width and height of the input feature map by a factor of 2 and reduces the number of channels in the input feature map.

[0050] This reduces the number of residual units in the backbone network.

[0051] The 3D target detection network provided in this application embodiment has achieved lightweight modification, and can ensure the overall algorithm's running efficiency and meet real-time requirements while meeting detection accuracy requirements.

[0052] Generally, to achieve perception of the real world, VR devices are equipped with multi-view cameras, such as... Figure 2A and Figure 2B As shown, where, Figure 2A This is a schematic diagram of a VR device that includes a stereo camera. Figure 2B This is a schematic diagram of a VR device containing four cameras.

[0053] It should be noted that the embodiments of this application can be implemented based on a binocular camera; for a quad-camera system, any set of binocular cameras can be selected. For example, selecting... Figure 2B Images are captured by the two cameras located above or below the center.

[0054] Based on the above Figure 1 A lightweight 3D object detection network detects obstacles in images captured by VR device cameras and provides obstacle warnings based on the detection results, thereby protecting the user's safety during immersive experiences.

[0055] See Figure 3 The following is a flowchart of an obstacle warning method provided in an embodiment of this application. This process is executed by a VR device and mainly includes the following steps:

[0056] S301: Perform obstacle detection on the RGB images captured by the camera. When at least one obstacle is detected, output the two-dimensional position information of each obstacle in consecutive image frames.

[0057] In one alternative implementation, in S301, based on Figure 1 The lightweight 3D object detection network shown performs obstacle detection on the RGB image captured by the main camera in the VR device's binocular camera system. When at least one obstacle is detected, it also performs obstacle detection on the RGB image captured by the secondary camera. This can save the computational load of real-time obstacle detection and improve the operating efficiency of the VR device.

[0058] In another alternative implementation, in S301, based on Figure 1 The lightweight 3D object detection network shown performs obstacle detection on RGB images captured by binocular cameras (i.e., the main camera and the secondary camera) to improve the accuracy of obstacle detection.

[0059] Regardless of whether the obstacle detection method is based on monocular or binocular images, the processing method for the input RGB image is the same. Specifically, after acquiring the RGB image from a monocular or binocular camera, the RGB image is preprocessed (including dynamic filling, scaling, normalization, etc.) to transform it into a uniform floating-point (float32) image of size 512×512×3. Then, it is input into a pre-trained 3D object detection network for obstacle detection.

[0060] When the 3D object detection network detects at least one obstacle, it processes the data and outputs the two-dimensional position information of each obstacle in the binocular image. Considering that dynamic obstacles may exist in practical applications, in step S301, when at least one obstacle is detected, the two-dimensional position information of each obstacle in consecutive image frames is output and recorded.

[0061] The two-dimensional location information of each obstacle includes, but is not limited to, the 2D coordinates of the obstacle's center point, the local offset of the obstacle's center point, the 2D coordinates of the obstacle's key points, the local offset of the obstacle's key points, the offset of the obstacle's key points relative to the obstacle's center point, the size of the obstacle's rectangular area, the absolute scale of the obstacle, and the obstacle's category.

[0062] The S301 employs a lightweight 3D object detection network built upon a specific neural network for obstacle detection, accelerating obstacle inference and ensuring the overall algorithm's high efficiency. Furthermore, compared to 3D object detection algorithms based on 3D point clouds or depth images, 3D object detection based on monocular or binocular RGB images avoids reliance on specialized acquisition equipment and reduces the hardware resource requirements for development.

[0063] It should be noted that the main camera in the stereo camera system is the one that captures the richest RGB image content, while the other is the secondary camera. Furthermore, for each camera's consecutive image frames, operations S302-S303 are performed, using Kalman filtering and the Hungarian algorithm to achieve multi-target tracking and temporal smoothing, effectively improving the stability of 3D target detection.

[0064] S302: For each obstacle, determine the predicted position information of the obstacle in the current image frame based on the target position information of the obstacle in the historical image frames.

[0065] In the embodiments of this application, in order to alleviate the jitter problem of the two-dimensional position of obstacle key points and the three-dimensional position of obstacle key points determined based on the two-dimensional position of obstacle key points that may occur when using only a 3D target detection network, a multi-target tracking algorithm based on Kalman filtering is introduced in S302. Kalman filtering predicts the position of each obstacle in the next image frame by modeling the position change of each obstacle in consecutive image frames.

[0066] In specific implementation, in S302, for each obstacle, the two-dimensional position information of the obstacle in historical image frames output by the 3D target detection network is smoothed to obtain the target position information of the obstacle in the historical image frames. Then, based on the target position information of the obstacle in the historical image frames, the predicted position information of the obstacle in the current image frame is determined. The smoothing process for obtaining the target position information is described in the following process and will not be elaborated here.

[0067] It should be noted that when an obstacle is detected for the first time, there are no historical image frames available, so prediction is not possible. Therefore, the two-dimensional position information of the obstacle in the first image frame is used as the target position information.

[0068] S303: Match obstacles in different image frames captured by the same camera and smooth the predicted position information of the obstacles to obtain the target position information.

[0069] In the embodiments of this application, considering that multiple obstacles may exist or newly enter within the safe area, or that other objects outside the safe area may also be classified as obstacles, in order to achieve the tracking of multiple targets in different image frames, the Hungarian algorithm is used in S303 to perform linear matching of obstacles in different image frames acquired by the same camera, so as to achieve cross-frame propagation of different obstacles and thus ensure the temporal continuity of the two-dimensional position information of the same obstacle.

[0070] For the temporally continuous two-dimensional position information {I0, I1, ..., I...} of the same obstacle nThe obstacle (where n is the number of RGB image frames) can be stored using a dual-queue structure. One queue records the target position information of the obstacle, and the other queue records the predicted position information of the obstacle.

[0071] It should be noted that the length of the dual queues can be set according to actual needs, and this application embodiment does not impose any limiting requirements. For example, this application embodiment sets the length of the dual queues to 30.

[0072] In S303, the predicted position information of obstacles in the current image frame is smoothed based on the target position information of obstacles in the historical image frames recorded by the dual queues, thereby improving the stability of 3D target detection.

[0073] An optional implementation involves directly smoothing the 2D coordinates of key obstacle points in the 2D location information using an exponential mean. The smoothing process is described in [link to relevant documentation]. Figure 4 It mainly includes the following steps:

[0074] S303_11: For each obstacle key point, calculate the mean 2D coordinate of the obstacle key point based on the target 2D coordinate of the obstacle key point in the historical image frame.

[0075] In S303_11, based on the target position information of obstacles in historical image frames recorded in the dual queues, the target 2D coordinates of each obstacle key point are obtained. For each obstacle key point, the mean 2D coordinate of the obstacle key point is calculated based on its target 2D coordinates in historical image frames, as shown in the following formula:

[0076]

[0077] Among them, K i (i = 0, 1, ..., m) represents the target 2D coordinates of the obstacle keypoint in the i-th frame image, and m represents the historical image frame number. This represents the average 2D coordinates of the obstacle's key points in historical image frames.

[0078] S303_12: Smooth the predicted 2D coordinates of the obstacle key point in the current image frame based on the mean of the 2D coordinates to obtain the target 2D coordinates of the obstacle key point contained in the target position information in the current image frame.

[0079] In S303_12, the 2D coordinate smoothing formula is as follows:

[0080]

[0081] Where α represents the prediction confidence of the obstacle's key points, and K predThis represents the predicted 2D coordinates of the key points of the obstacle. The value represents the mean 2D coordinates of the key point of the obstacle, and K represents the target 2D coordinates of the key point of the obstacle.

[0082] One optional implementation involves smoothing the absolute scale of each obstacle in its two-dimensional location information. The smoothing process is described in [reference needed]. Figure 5 It mainly includes the following steps:

[0083] S303_21: For each obstacle, calculate the absolute scale mean and absolute scale standard deviation of the obstacle based on the target absolute scale of the obstacle in historical image frames.

[0084] In S303_21, for each obstacle, the target absolute scale of the obstacle is obtained based on the target position information of the obstacle in the historical image frames recorded in the dual queues, and the absolute scale mean and absolute scale standard deviation of the obstacle are calculated based on the target absolute scale of the obstacle in the historical image frames.

[0085] S303_22: Determine the Euclidean distance between the predicted absolute scale and the mean absolute scale of the obstacle in the current image frame.

[0086] S303_23: Determine whether the Euclidean distance is less than the absolute scale standard deviation. If yes, proceed to S303_24; otherwise, proceed to S303_25.

[0087] In S303_23, the difference between the predicted absolute scale and the absolute scale mean can be determined by comparing the Euclidean distance between the predicted absolute scale and the absolute scale mean and the absolute scale standard deviation.

[0088] S303_24: Smooth the predicted absolute scale of the obstacle in the current image frame based on the absolute scale mean to obtain the target absolute scale of the obstacle contained in the target location information in the current image frame.

[0089] In S303_24, when the Euclidean distance between the predicted absolute scale of the obstacle in the current image frame and the mean absolute scale is less than the standard deviation of the absolute scale, it indicates that the difference between the predicted absolute scale of the obstacle in the current image frame and the mean absolute scale of the obstacle in historical image frames is small. Therefore, the exponential mean method can be used to smooth the predicted absolute scale of the obstacle in the current image frame based on the mean absolute scale of the obstacle in historical image frames. The smoothing formula is as follows:

[0090]

[0091] Where β represents the weighting coefficient, S pred This indicates the predicted absolute scale of the obstacle in the current image frame. β represents the average absolute scale of the obstacle in historical image frames, and S represents the target absolute scale of the obstacle in the current image frame. Optional, β = 0.1.

[0092] S303_25: Use the absolute target scale of the obstacle in the previous image frame as the absolute target scale of the obstacle contained in the target location information of the current image frame.

[0093] In S303_25, when the Euclidean distance between the predicted absolute scale of the obstacle in the current image frame and the mean absolute scale is greater than or equal to the standard deviation of the absolute scale, it indicates that the difference between the predicted absolute scale of the obstacle in the current image frame and the mean absolute scale of the obstacle in historical image frames is large, which may bring a large smoothing error. In this case, the target absolute scale of the obstacle in the previous image frame is directly used as the target absolute scale of the obstacle in the current image frame.

[0094] It should be noted that other predicted parameters in the predicted location information of obstacles that have not been smoothed can be directly used as the corresponding target parameters in the target location information of obstacles.

[0095] In this embodiment, after obtaining the target position information of each obstacle in consecutive image frames acquired by the binocular camera, the Kalman filtering algorithm and the Hungarian algorithm are used to linearly allocate the detection results and the temporal sequence of each obstacle in the consecutive image frames. Then, based on the target absolute scale of the obstacles in the historical image frames recorded by the dual queues, the difference between the predicted absolute scale and the mean absolute scale of the obstacles in the current image frame is estimated. The predicted absolute scale is then exponentially averaged according to the distribution of the difference. Simultaneously, the predicted 2D coordinates of the obstacle key points in the current image frame are exponentially averaged according to the prediction confidence of the obstacle key points. By temporally smoothing the predicted absolute scale of each obstacle and the predicted 2D coordinates of each obstacle key point, the subsequent calculation of the 3D position of the obstacle key points achieves a continuous temporal variation, alleviating the problem of position jitter or abrupt changes of obstacles in the camera coordinate system.

[0096] S304: Determine the initial 3D coordinates of the obstacle key points based on the target 2D coordinates contained in the target position information of the obstacle key points in the current image frame captured by the main camera.

[0097] In S304, for each obstacle, the PnP algorithm is used to determine the relative pose between the camera and the obstacle based on the target 2D coordinates of the obstacle key points and the transformation relationship between the camera coordinate system, image coordinate system and world coordinate system, and obtain the initial 3D coordinates of the obstacle key points.

[0098] It should be noted that the PnP algorithm is a commonly used method in computer vision for solving the pose of a camera relative to a 3D object using a monocular image. It is quite mature and will not be described in detail here.

[0099] To improve the accuracy of the 3D position of key points of obstacles, this application provides a depth fine-tuning technique based on binocular RGB images and key point confidence, which enables the overall algorithm to maintain real-time stability even in low computing resource scenarios. For details, please refer to S305.

[0100] S305: Based on the depth information between the obstacle key points and the main camera, fine-tune the initial 3D coordinates of the obstacle key points to obtain the target 3D coordinates.

[0101] In S305, the depth information between obstacle keypoints and the main camera includes: first depth information calculated based on monocular images and second depth information calculated based on binocular images. The initial 3D coordinates of the obstacle keypoints are fine-tuned based on the first and second depth information to obtain the target 3D coordinates. For details of the fine-tuning process, see [link to relevant documentation]. Figure 6 It mainly includes the following steps:

[0102] S3051: Determine the first depth information between the obstacle key points and the main camera based on the target 2D coordinates of the obstacle key points in the current image frame captured by the main camera.

[0103] In S3051, the PnP algorithm is used to determine the first depth information between the obstacle key points and the main camera based on the target 2D coordinates of the obstacle key points in the current image frame acquired by the main camera, and the projection relationship between the 2D points and 3D points. This first depth information is denoted as d. pred .

[0104] S3052: Determine the second depth information of the key points and the main camera based on the target 2D coordinates of the key points of the obstacle in the current image frame acquired by the main camera and the secondary camera respectively.

[0105] Since the first depth information is calculated based on a monocular camera, its accuracy is relatively low. Because the obstacle keypoints are ordered, to improve the accuracy of the depth information, the depth information between the obstacle and the main camera can be calibrated using the RGB images captured by the secondary camera. Specifically, in S2052, a triangulation calculation method is used to determine the second depth information between the obstacle keypoints and the main camera based on the target 2D coordinates of the obstacle keypoints in the current image frame captured by the main and secondary cameras, and the correspondence between the obstacle keypoints in the images captured by the binocular cameras. This second depth information is denoted as d. tri .

[0106] S3053: Determine the fine-tuning coefficients based on the first depth information, the second depth information, and the confidence level of the obstacle key points. The confidence level is determined during the obstacle detection process.

[0107] In the embodiments of this application, during obstacle detection using a lightweight 3D object detection network, when an obstacle is detected, both the obstacle key points and their confidence levels are obtained simultaneously. Assume that the obstacle key points detected in the RGB images acquired by the main and secondary cameras are denoted as K... main and K pub The confidence scores of obstacle key points are denoted as p. main and p sub Therefore, in S305, the fine-tuning coefficients can be determined using the confidence levels of obstacle key points and the depth information between the obstacle and the main camera. For details on the process of determining the fine-tuning coefficients, please refer to [link to relevant documentation]. Figure 7 It mainly includes the following steps:

[0108] S3053_1: Determine the target depth information based on the first depth information, the second depth information, and the confidence level of the obstacle key points.

[0109] In S3053_1, the formula for calculating the target depth information between obstacle key points and the main camera is as follows:

[0110] d main =p main ·d pred +(1-p mian )·d tri Formula 4

[0111] Where, d main p represents the target depth information between obstacle key points and the main camera. main d represents the confidence level of obstacle keypoints in the current image frame captured by the main camera. pred d represents the first depth information between obstacle keypoints calculated based on monocular images and the main camera. tri This represents the second depth information between obstacle keypoints calculated based on binocular images and the main camera.

[0112] S3053_2: Determine the fine-tuning coefficients based on the target depth information and the second depth information.

[0113] In S3053_2, the fine-tuning coefficients for the 3D positions of each obstacle keypoint can be calculated based on the similarity of the camera projections. The formula for calculating the fine-tuning coefficients is as follows:

[0114] k = d main / d pred Formula 5

[0115] Where k represents the fine-tuning coefficient.

[0116] S3054: Based on the fine-tuning coefficient, fine-tune the initial 3D coordinates of the obstacle's key points to obtain the target's 3D coordinates.

[0117] In S3054, the fine-tuning formula is as follows:

[0118] W refine =k*W Formula 6

[0119] Among them, W refine W represents the target 3D coordinates of the obstacle key points after fine-tuning, and W represents the target 3D coordinates of the obstacle key points before fine-tuning.

[0120] In this embodiment, the depth information of the obstacle key points relative to the main camera is obtained by the PnP algorithm and triangulation calculation, respectively. Then, the fine-tuning coefficient of the obstacle's three-dimensional position is calculated based on the two depth information. Finally, the initial 3D coordinates obtained by the PnP calculation are adjusted based on the fine-tuning coefficient. By using a stereo camera for depth fine-tuning, a more accurate three-dimensional position can be obtained.

[0121] In some embodiments, to improve fine-tuning efficiency, one obstacle key point of the same obstacle in the binocular image can be selected as a reference point, and the fine-tuning coefficient of each obstacle key point can be calculated by using the depth information between the reference point and the main camera. The selected reference point can be the first obstacle key point or the center point of each obstacle contained in an obstacle.

[0122] In other embodiments, the target 3D coordinates of each obstacle key point can be obtained by using the average of the fine-tuning coefficients of each obstacle key point in the binocular image as the final fine-tuning coefficient.

[0123] S306: Provide safety warnings based on the target's 3D coordinates of key points on obstacles.

[0124] In S306, for each obstacle, the three-dimensional position of the obstacle relative to the VR device can be located by using the target 3D coordinates of the obstacle's key points. Based on the three-dimensional position, a warning can be issued to avoid collisions with the obstacle and protect user safety.

[0125] One possible implementation is that, before the VR experience, there may be static obstacles around the user. After determining the three-dimensional position of the obstacle relative to the VR device through obstacle detection and positioning, when setting the safe zone, the distance between the obstacle and the VR device can be used to determine whether the obstacle is within the safe zone. If it is not, the current configuration parameters of the safe zone are maintained. If it is, a safety prompt is given through the VR device's display screen, and the size of the safe zone is adjusted according to the distance between the obstacle and the VR device so that the obstacle is located outside the safe zone to avoid collisions.

[0126] For example, such as Figure 8 As shown, the dashed line represents the size of the safe area before adjustment, the solid line represents the size of the safe area after adjustment, the cube represents an obstacle, and the trapezoid represents a virtual display screen used to display safety prompts.

[0127] Another optional implementation involves addressing the possibility of dynamic obstacles entering the safe zone during the VR experience. Obstacle detection and localization can accurately identify these obstacles and determine their distance from the VR device based on the obstacle's 3D position relative to the VR device. By comparing this distance with a distance threshold, it can be determined whether the obstacle poses a threat to user safety, thus preventing accidents. Specifically, when the distance is less than the distance threshold, a safety warning is displayed on the VR device's screen; when the distance is greater than or equal to the distance threshold, a safety warning can be given via voice, or no warning may be given.

[0128] For example, such as Figure 9 As shown, when the distance between the obstacle and the VR device is less than the distance threshold, it enters the safety warning range (within...). Figure 9 (Used as a dashed line in the middle), the VR device exits the virtual screen and displays the real screen.

[0129] It should be noted that this application does not impose restrictive requirements on the display method of safety prompts. For example, the virtual screen can be exited and the real screen containing obstacles can be displayed directly; alternatively, the real screen of obstacles can be overlaid on the virtual screen to ensure the smooth operation of the game.

[0130] The obstacle detection method in a virtual reality scene provided in this application embodiment has two aspects. First, a lightweight 3D target detection network built with a deep learning network is used to detect obstacles in RGB images captured by at least one camera of the VR device. For each detected obstacle, the two-dimensional position information of the obstacle in consecutive image frames captured by the binocular camera is output. This improves the detection speed while reducing the storage resource consumption, which is conducive to deployment on wearable devices. Second, a Kalman multi-object tracking algorithm is used to predict the target position information of obstacles in the current image frame, and a Hungarian algorithm is used to match obstacles in different image frames captured by the same camera, realizing cross-target tracking of the same obstacle. Frame propagation ensures the temporal continuity of obstacle key points. Furthermore, based on the target position information of obstacles in historical image frames, the predicted position information of obstacles in the current image frame is smoothed, enabling the subsequently calculated 3D positions of obstacle key points to change continuously in time. This alleviates the problem of obstacle position jitter or abrupt changes in the camera coordinate system, improving the accuracy and stability of obstacle detection. Simultaneously, based on the depth information determined by both binocular and monocular images, fine-tuning coefficients are calculated to fine-tune the 3D positions of obstacle key points, further improving the accuracy and robustness of obstacle detection, preventing collisions between obstacles and users, and thus enhancing user safety.

[0131] Based on the same technical concept, this application provides a virtual reality device that can implement the obstacle warning method steps in the virtual reality scene described above, and achieve the same technical effect.

[0132] See Figure 10 The virtual reality device includes a processor 1001, a memory 1002, and a binocular camera 1003. The binocular camera 1003 includes a main camera and a secondary camera. The binocular camera 1003, the memory 1002, and the processor 1001 are connected via a bus 1004.

[0133] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0134] Obstacle detection is performed on the RGB images captured by the camera. When at least one obstacle is detected, the two-dimensional position information of each obstacle in consecutive image frames is output.

[0135] For each obstacle, the predicted position information of the obstacle in the current image frame is determined based on the target position information of the obstacle in the historical image frames.

[0136] The obstacles in different image frames captured by the same camera are matched, and the predicted position information of the obstacles is smoothed to obtain the target position information;

[0137] Based on the target 2D coordinates of the obstacle key points contained in the target location information in the current image frame captured by the main camera, the initial 3D coordinates of the obstacle key points are determined.

[0138] Based on the depth information between the obstacle key points and the main camera, the initial 3D coordinates of the obstacle key points are fine-tuned to obtain the target 3D coordinates;

[0139] Safety warnings are issued based on the target 3D coordinates of the key points of the obstacle.

[0140] Optionally, the two-dimensional location information includes at least the 2D coordinates of the key points of the obstacle;

[0141] The processor 1001 smooths the predicted location information of the obstacle to obtain the target location information. The specific operation is as follows:

[0142] For each obstacle key point, the mean 2D coordinate of the obstacle key point is calculated based on the target 2D coordinate of the obstacle key point in the historical image frame;

[0143] The predicted 2D coordinates of the obstacle key points in the current image frame are smoothed based on the mean of the 2D coordinates to obtain the target 2D coordinates of the obstacle key points contained in the target location information in the current image frame.

[0144] Optionally, the two-dimensional location information may include at least the absolute scale of the obstacle;

[0145] The processor 1001 smooths the predicted position information of the obstacle to obtain the target position information, and also executes:

[0146] For each obstacle, the mean and standard deviation of the absolute scale of the obstacle are calculated based on the target absolute scale of the obstacle in historical image frames.

[0147] Determine the Euclidean distance between the predicted absolute scale of the obstacle in the current image frame and the mean of the absolute scale;

[0148] If the Euclidean distance is less than the absolute scale standard deviation, then the predicted absolute scale of the obstacle in the current image frame is smoothed according to the absolute scale mean to obtain the target absolute scale of the obstacle contained in the target location information in the current image frame.

[0149] If the Euclidean distance is not less than the absolute scale standard deviation, then the target absolute scale of the obstacle in the previous image frame is taken as the target absolute scale of the obstacle contained in the target location information in the current image frame.

[0150] Optionally, the processor 1001 fine-tunes the initial 3D coordinates of the obstacle key points based on the depth information between the obstacle key points and the main camera to obtain the target 3D coordinates. Specifically, the operation is as follows:

[0151] Based on the target 2D coordinates of the obstacle key points in the current image frame captured by the main camera, the first depth information between the obstacle key points and the main camera is determined;

[0152] Based on the target 2D coordinates of the obstacle key points in the current image frame acquired by the main camera and the secondary camera respectively, the second depth information between the obstacle key points and the main camera is determined;

[0153] Based on the first depth information, the second depth information, and the confidence level of obstacle key points, a fine-tuning coefficient is determined, wherein the confidence level of obstacle key points is determined during the detection of the obstacle;

[0154] Based on the fine-tuning coefficient, the initial 3D coordinates of the key points of the obstacle are fine-tuned to obtain the target 3D coordinates.

[0155] Optionally, the processor determines the first depth information between the obstacle keypoint and the main camera based on the target 2D coordinates of the obstacle keypoint in the current image frame captured by the main camera. Specifically, the operation is as follows:

[0156] Using the PnP algorithm, the first depth information between the obstacle key points and the main camera is determined based on the target 2D coordinates of the obstacle key points in the current image frame captured by the main camera, and the projection relationship between the 2D points and the 3D points.

[0157] The processor 1001 determines the second depth information between the obstacle key points and the main camera based on the target 2D coordinates of the obstacle key points in the current image frame acquired by the main camera and the secondary camera respectively. The specific operation is as follows:

[0158] Using a triangulation calculation method, based on the target 2D coordinates of the obstacle key points in the current image frame acquired by the main camera and the secondary camera respectively, and the correspondence between the obstacle key points in the image acquired by the binocular camera, the second depth information between the obstacle key points and the main camera is determined.

[0159] Optionally, the processor 1001 determines the fine-tuning coefficients based on the first depth information, the second depth information, and the confidence level of obstacle key points. Specifically, the operation is as follows:

[0160] The target depth information is determined based on the first depth information, the second depth information, and the confidence level of the obstacle key points;

[0161] The fine-tuning coefficient is determined based on the target depth information and the second depth information.

[0162] Optionally, the formula for calculating the target depth information is:

[0163] d main =p main ·d pred +(1-p mian )·d tri

[0164] Where, d main p represents the target depth information. main d represents the confidence level of the obstacle keypoints in the current image frame captured by the main camera. pred Represents the first depth information, d tri This represents the second depth information.

[0165] Optionally, the network for obstacle detection in the RGB images captured by the camera is a lightweight 3D object detection network, which includes at least one of the following structures:

[0166] Multiple residual units use depthwise separable convolution kernels;

[0167] Multiple residual units used for downsampling are interconnected with multiple upsampling units;

[0168] The upsampling unit upsamples the width and height of the input feature map by a factor of 2 and reduces the number of channels in the input feature map.

[0169] It should be noted that, Figure 10 This is merely an example, showing the hardware necessary to implement the obstacle warning method steps in the virtual reality scene provided in the embodiments of this application. Not shown, the virtual reality device also includes conventional hardware such as left and right glasses, a display screen, a speaker, and a speaker.

[0170] Examples of this application Figure 10The processor involved can be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0171] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform the obstacle warning method in the virtual reality scene described in the foregoing embodiments.

[0172] This application also provides a computer program product for storing a computer program that executes the obstacle warning method in the virtual reality scene described in the foregoing embodiments.

[0173] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0175] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0176] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0177] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An obstacle warning method, characterized in that, VR devices that include stereo cameras include: Obstacle detection is performed on the RGB images captured by the camera. When at least one obstacle is detected, the two-dimensional position information of each obstacle in consecutive image frames is output. For each obstacle, the predicted position information of the obstacle in the current image frame is determined based on the target position information of the obstacle in the historical image frames. The obstacles in different image frames captured by the same camera are matched, and the predicted position information of the obstacles is smoothed to obtain the target position information; Based on the target 2D coordinates of the obstacle key points contained in the target location information in the current image frame captured by the main camera, the initial 3D coordinates of the obstacle key points are determined. Based on the first depth information, the second depth information, and the confidence level of the obstacle key points, a fine-tuning coefficient is determined to fine-tune the initial 3D coordinates of the obstacle key points to obtain the target 3D coordinates. The first depth information is calculated based on a monocular image, and the second depth information is calculated based on a binocular image. Safety warnings are issued based on the target 3D coordinates of the key points of the obstacle.

2. The method as described in claim 1, characterized in that, The two-dimensional location information includes at least the 2D coordinates of the key points of the obstacle; The process of smoothing the predicted location information of the obstacle to obtain the target location information includes: For each obstacle key point, the mean 2D coordinate of the obstacle key point is calculated based on the target 2D coordinate of the obstacle key point in the historical image frame; The predicted 2D coordinates of the obstacle key points in the current image frame are smoothed based on the mean of the 2D coordinates to obtain the target 2D coordinates of the obstacle key points contained in the target location information in the current image frame.

3. The method as described in claim 1, characterized in that, The two-dimensional location information also includes at least the absolute dimensions of the obstacle; The step of smoothing the predicted location information of the obstacle to obtain the target location information further includes: For each obstacle, the mean and standard deviation of the absolute scale of the obstacle are calculated based on the target absolute scale of the obstacle in historical image frames. Determine the Euclidean distance between the predicted absolute scale of the obstacle in the current image frame and the mean of the absolute scale; If the Euclidean distance is less than the absolute scale standard deviation, then the predicted absolute scale of the obstacle in the current image frame is smoothed according to the absolute scale mean to obtain the target absolute scale of the obstacle contained in the target location information in the current image frame. If the Euclidean distance is not less than the absolute scale standard deviation, then the target absolute scale of the obstacle in the previous image frame is taken as the target absolute scale of the obstacle contained in the target location information in the current image frame.

4. The method as described in claim 1, characterized in that, The step of determining fine-tuning coefficients based on first depth information, second depth information, and obstacle key point confidence to fine-tune the initial 3D coordinates of the obstacle key points, thereby obtaining the target 3D coordinates, includes: Based on the target 2D coordinates of the obstacle key points in the current image frame captured by the main camera, the first depth information between the obstacle key points and the main camera is determined; Based on the target 2D coordinates of the obstacle key points in the current image frame acquired by the main camera and the secondary camera respectively, the second depth information between the obstacle key points and the main camera is determined; Based on the first depth information, the second depth information, and the confidence level of obstacle key points, a fine-tuning coefficient is determined, wherein the confidence level of obstacle key points is determined during the detection of the obstacle; Based on the fine-tuning coefficient, the initial 3D coordinates of the key points of the obstacle are fine-tuned to obtain the target 3D coordinates.

5. The method as described in claim 4, characterized in that, The step of determining the first depth information between the obstacle key points and the main camera based on the target 2D coordinates of the obstacle key points in the current image frame captured by the main camera includes: Using the PnP algorithm, the first depth information between the obstacle key points and the main camera is determined based on the target 2D coordinates of the obstacle key points in the current image frame captured by the main camera, and the projection relationship between the 2D points and the 3D points. The step of determining the second depth information between the obstacle key points and the main camera based on the target 2D coordinates of the obstacle key points in the current image frame acquired by the main camera and the secondary camera respectively includes: Using a triangulation calculation method, based on the target 2D coordinates of the obstacle key points in the current image frame acquired by the main camera and the secondary camera respectively, and the correspondence between the obstacle key points in the image acquired by the binocular camera, the second depth information between the obstacle key points and the main camera is determined.

6. The method as described in claim 4, characterized in that, The step of determining the fine-tuning coefficient based on the first depth information, the second depth information, and the confidence level of obstacle key points includes: The target depth information is determined based on the first depth information, the second depth information, and the confidence level of the obstacle key points; The fine-tuning coefficient is determined based on the target depth information and the second depth information.

7. The method as described in claim 6, characterized in that, The formula for calculating the target depth information is: in, This indicates the target depth information. This represents the confidence level of the obstacle key points in the current image frame captured by the main camera. This represents the first depth information. This represents the second depth information.

8. The method as described in claim 1, characterized in that, The network for obstacle detection in RGB images captured by the camera is a lightweight 3D object detection network, which includes at least one of the following structures: Multiple residual units use depthwise separable convolution kernels; Multiple residual units used for downsampling are interconnected with multiple upsampling units; The upsampling unit upsamples the width and height of the input feature map by a factor of 2 and reduces the number of channels in the input feature map.

9. A virtual reality device, characterized in that, It includes a processor, a memory, and a stereo camera, wherein the stereo camera, the memory, and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: Obstacle detection is performed on the RGB images captured by the camera. When at least one obstacle is detected, the two-dimensional position information of each obstacle in consecutive image frames is output. For each obstacle, the predicted position information of the obstacle in the current image frame is determined based on the target position information of the obstacle in the historical image frames. The obstacles in different image frames captured by the same camera are matched, and the predicted position information of the obstacles is smoothed to obtain the target position information; Based on the target 2D coordinates of the obstacle key points contained in the target location information in the current image frame captured by the main camera, the initial 3D coordinates of the obstacle key points are determined. Based on the first depth information, the second depth information, and the confidence level of the obstacle key points, a fine-tuning coefficient is determined to fine-tune the initial 3D coordinates of the obstacle key points to obtain the target 3D coordinates. The first depth information is calculated based on a monocular image, and the second depth information is calculated based on a binocular image. Safety warnings are issued based on the target 3D coordinates of the key points of the obstacle.

10. The virtual reality device as described in claim 9, characterized in that, The process of determining fine-tuning coefficients based on first depth information, second depth information, and obstacle key point confidence levels to fine-tune the initial 3D coordinates of the obstacle key points, thereby obtaining the target 3D coordinates, is as follows: Based on the target 2D coordinates of the obstacle key points in the current image frame captured by the main camera, the first depth information between the obstacle key points and the main camera is determined; Based on the target 2D coordinates of the obstacle key points in the current image frame acquired by the main camera and the secondary camera respectively, the second depth information between the obstacle key points and the main camera is determined; Based on the first depth information, the second depth information, and the confidence level of obstacle key points, a fine-tuning coefficient is determined, wherein the confidence level of obstacle key points is determined during the detection of the obstacle; Based on the fine-tuning coefficient, the initial 3D coordinates of the key points of the obstacle are fine-tuned to obtain the target 3D coordinates.

Images