AR film watching method for vehicle end mobile scene

Abstract

The invention discloses an AR (Augmented Reality) film watching method for a vehicle end mobile scene, which comprises the following steps of: capturing environment information through a binocular camera of AR equipment, filtering an environment outside a window by utilizing a depth estimation network and a depth filtering method, and constructing a map of an environment in a vehicle through a space positioning algorithm. And finally, the streaming media content is projected to a designated area of the space in the vehicle. According to the invention, the interference of the moving scene of the vehicle end on space positioning is solved, stable space positioning and projection are realized, better film watching experience is brought to a user, meanwhile, the perception of the surrounding environment is not influenced, and a film watching method in which virtual film and television content and a real environment are fused in the moving scene of the vehicle is provided.

Description

Technical Field

[0001] The present invention belongs to the field of augmented reality, and in particular relates to a method for positioning and projecting augmented reality for a vehicle-side mobile scene. Background Art

[0002] Augmented Reality (AR) is a technology that calculates the position and angle of a camera in real time and adds corresponding images, videos, and 3D models. The goal of this technology is to put the virtual world on the screen and interact with the real world. With the improvement of the computing performance of portable electronic products, it is expected that the use of augmented reality will become more and more extensive.

[0003] With the continuous development of augmented reality technology, its in-depth applications in various fields are becoming more and more numerous, especially in the field of AR viewing, which brings a more visually impactful experience. However, the current spatial positioning methods are all based on static scenes. For mobile scenes such as vehicles, the existing technical methods cannot provide accurate positioning effects, resulting in the inability to project streaming media content to a fixed perspective in the car. Summary of the invention

[0004] One of the purposes of the present invention is to solve the problem that the environmental information collected by the vehicle side is affected by the moving scene outside the vehicle, and to propose an AR viewing method for the moving scene on the vehicle side.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] An AR viewing technology for a vehicle-side mobile scene, the AR viewing technology for a vehicle-side mobile scene comprising:

[0007] Use a binocular camera to obtain left and right eye images, estimate their depth, and obtain depth image data;

[0008] According to the depth image data, a depth threshold is set to obtain depth mask data;

[0009] Performing depth filtering on the binocular image according to the depth mask data to obtain image data of only static scenes;

[0010] Perform spatial positioning according to the static scene image data to obtain spatial coordinate data of any point in the static scene inside the vehicle;

[0011] According to the spatial coordinate data of the interior space of the vehicle, the movie data to be projected is projected to a designated position.

[0012] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0013] Preferably, the method of using a binocular camera to obtain left and right eye images, performing depth estimation on the images, and obtaining depth image data includes:

[0014] Through grayscale transformation, the left and right eye images are converted into grayscale image data;

[0015] The left and right grayscale images are converted into cost volume data through the spatial pyramid pooling network;

[0016] The cost volume network is converted into depth image data through a 3D convolutional network.

[0017] Preferably, setting a depth threshold according to the depth image data to obtain depth mask data includes:

[0018] A depth threshold is set, and data in the depth map data that is greater than the depth threshold is set to 0, and data that is less than the depth threshold is set to 1, to obtain depth mask data.

[0019] Preferably, the depth mask data is binary data with a zero-one distribution, and the distribution of the image data of the static scene inside the vehicle is that the image data of the part with a depth mask of 0 is assigned a value of 0, and the image data of the part with a depth mask of 1 is the original image value.

[0020] Preferably, spatial positioning is performed according to the static scene image data to obtain spatial coordinate data of any point in the static scene inside the vehicle, and spatial coordinate data of any position in the space inside the vehicle is obtained, including:

[0021] Set the position of the left camera at the first frame of image acquisition as the starting position, and set the pose to the unit matrix;

[0022] For any subsequent frame of image data of a static scene inside the car, the position and posture of the current frame relative to the first frame data are outputted by spatial positioning;

[0023] The feature points obtained by feature extraction of the image data of the static scene inside the car are converted into coordinate data in space by calibrating the internal parameters of the image acquisition device and the pose obtained by the above calculation.

[0024] Preferably, projecting the movie data to be projected to a designated position according to the spatial coordinate data of the interior space of the vehicle comprises:

[0025] Based on the projection matrix, the spatial coordinates specified in the space are converted into the coordinates of the rendering screen;

[0026] Using the jME3 rendering engine, the movie data is rendered to the above screen coordinates.

[0027] Preferably, the depth threshold is 2-5m.

[0028] Compared with the prior art, the advantages of the present invention are:

[0029] Compared with the traditional flat viewing method, the method provided by the present invention greatly improves the user's mobile viewing experience on the vehicle side. At the same time, the method uses a deep filtering method to filter the moving scene outside the window, thereby improving the stability and accuracy of spatial positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the specific implementation methods of the present invention or the technical methods in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 It is an overall flow chart of the AR viewing method of the vehicle-side mobile scene of the present invention;

[0032] Figure 2 This is a flow chart of the binocular depth estimation algorithm in the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical methods in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0035] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0036] like Figure 1As shown, the AR viewing method for a vehicle-side mobile scene provided in this embodiment includes the following steps:

[0037] Step S1: Use AR glasses to collect surrounding environment information, input the image (one RGB image and two grayscale images) as the current frame, and use the binocular depth estimation algorithm to estimate the depth map of the current frame. The binocular depth estimation algorithm uses a binocular pyramid matching network to achieve feature encoding, and then uses a three-dimensional convolutional neural network structure to achieve depth estimation;

[0038] Step S2: Set a depth threshold according to the actual use scenario of a family car, usually set to 5m. Set pixels greater than the threshold to 0, and pixels less than the threshold to 1. Based on this rule, a binary distribution Mask image is generated. The design principle of the depth threshold is based on the distance between the AR glasses and the scene outside the car, and is generally set within the range of 2-5m;

[0039] Step S3: Apply the Mask generated in step S2 to the current image frame to perform deep filtering, filter out the dynamic scene outside the car window, and retain the static scene inside the car;

[0040] Step S4: Based on the static scene image inside the car obtained in step S3, the spatial positioning module is used to initialize the scene inside the car, and a spatial map of the scene inside the car and an initial pose are obtained. The spatial positioning module is mainly based on the binocular vision SLAM algorithm, and based on the static scene image after dynamic scene filtering, pose calculation and map generation are realized;

[0041] Step S5: Based on the map of the in-vehicle scene reconstructed in step S4, a fixed coordinate point in the space is set as a projection coordinate, the video stream data is projected to a specified position, and the projection image of the video stream can be maintained at the position.

[0042] Among them, the binocular depth estimation algorithm in this embodiment adopts a binocular pyramid matching network, such as Figure 2 As shown in Figure 2, the network is mainly divided into two modules: spatial pyramid pooling module and three-dimensional convolutional neural network module.

[0043] The spatial pyramid pooling network module adopts a multi-scale context enhancement method. It uses multiple fixed-size average pooling blocks, connects convolution and upsampling, and finally splices multiple feature maps together. In this way, the features of the extracted image contain rich context information, while increasing the size of the receptive field in the network, allowing for better depth estimation. Another module is the 3D convolution network module. This module mainly adopts the idea of ​​encoding-decoding. It is a multi-layer cross-connected 3D convolution with a convolution kernel of (3*3*3). It can be seen that the spatial information of the neighborhood around each pixel is utilized, and the information of multiple disparities is also utilized, so it is more robust than comparing only one disparity. Finally, a linear interpolation upsampling is used to restore the original resolution of the image, thereby calculating the depth value of each pixel and obtaining a depth map.

[0044] When in use, the AR device's binocular camera is used to capture environmental information, and the environment outside the window is filtered using a depth estimation network and depth filtering method. Then, a spatial positioning algorithm is used to construct a map of the interior environment, and finally, the streaming media content is projected to a designated area inside the vehicle.

[0045] The vehicle-side mobile scene AR viewing technology method described in an embodiment of the present invention aims to present two-dimensional images in a fixed area in three-dimensional space to bring users a better viewing experience without affecting the perception of the surrounding environment. It provides a viewing technology that integrates virtual film and television content with the real environment in a vehicle mobile scene.

[0046] The above-described embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the attached claims.

Claims

1. An AR viewing method for a vehicle-side mobile scene, characterized in that: include: Use a binocular camera to obtain left and right eye images, estimate their depth, and obtain depth image data; According to the depth image data, a depth threshold is set to obtain depth mask data; Performing depth filtering on the binocular image according to the depth mask data to obtain image data of only static scenes; Perform spatial positioning according to the static scene image data to obtain spatial coordinate data of any point in the static scene inside the vehicle; According to the spatial coordinate data of the interior space of the vehicle, the movie data to be projected is projected to a designated position.

2. The AR viewing method for a vehicle-side mobile scene according to claim 1, characterized in that: The method of using a binocular camera to obtain left and right eye images, performing depth estimation on the images, and obtaining depth image data includes: Through grayscale transformation, the left and right eye images are converted into grayscale image data; The left and right grayscale images are converted into cost volume data through the spatial pyramid pooling network; The cost volume network is converted into depth image data through a 3D convolutional network.

3. The AR viewing method for a vehicle-side mobile scene according to claim 1, characterized in that: The step of setting a depth threshold according to the depth image data and acquiring depth mask data comprises: A depth threshold is set, and data in the depth map data that is greater than the depth threshold is set to 0, and data that is less than the depth threshold is set to 1, to obtain depth mask data.

4. The AR viewing method for a vehicle-side mobile scene as claimed in claim 3, characterized in that: The depth mask data is binary data with a zero-one distribution, and the distribution of the image data of the static scene inside the car is that the image data of the part with a depth mask of 0 is assigned a value of 0, and the image data of the part with a depth mask of 1 is the original image value.

5. The AR viewing method for vehicle-side mobile scenes according to claim 1, characterized in that: The method of performing spatial positioning according to the static scene image data to obtain spatial coordinate data of any point in the static scene in the vehicle includes: Set the position of the left camera at the first frame of image acquisition as the starting position, and set the pose to the unit matrix; For any subsequent frame of image data of a static scene inside the car, the position and posture of the current frame relative to the first frame data are outputted by spatial positioning; The feature points obtained by feature extraction of the image data of the static scene inside the car are converted into coordinate data in space by calibrating the internal parameters of the image acquisition device and the pose obtained by the above calculation.

6. The AR viewing method for vehicle-side mobile scenes according to claim 1, characterized in that: The step of projecting the movie data to be projected to a designated position according to the spatial coordinate data of the interior space of the vehicle comprises: Based on the projection matrix, the spatial coordinates specified in the space are converted into the coordinates of the rendering screen; Using the jME3 rendering engine, the movie data is rendered to the above screen coordinates.

7. The AR viewing method for a vehicle-side mobile scene as claimed in claim 3, characterized in that: The depth threshold is 2-5m.

Images