Three-dimensional laser point cloud VR display method and apparatus

By using the target symbol distance field model in VR equipment for pose solving and incremental reconstruction, the problem of low laser point cloud pose solving accuracy in the prior art is solved, significantly improving the image display effect and improving the virtual reality experience.

WO2024244310A1PCT designated stage expired Publication Date: 2024-12-05BEIJING TIANMA INTELLIGENT CONTROL TECHNOLOGY CO LTD +1

Patent Information

Application Number
PCT/CN2023/130547
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-01
Filing Date
2023-11-08
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing VR devices have low accuracy when solving laser point cloud postures, resulting in poor image display effect.

Method used

The target symbol distance field model is used for pose solving and incremental reconstruction. By obtaining the target laser point cloud, generating the symbol distance field model, correcting the distortion, performing grid processing, calculating the symbol distance function, adjusting the pose, and transforming the global symbol distance function , and render a stereo projection image to display the image.

Benefits of technology

Improves the accuracy of posture solution and image display effect, enhancing the virtual reality experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023130547_05122024_PF_FP_ABST
    Figure CN2023130547_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Provided are a three-dimensional laser point cloud VR display method and apparatus, an electronic device, a non-transitory computer-readable storage medium, a computer program product, and a computer program. The three-dimensional laser point cloud VR display method comprises: acquiring a target laser point cloud, and generating a target symbol distance field model corresponding to the target laser point cloud; on the basis of the target symbol distance field model, carrying out pose solving to obtain a target pose corresponding to the target laser point cloud; and carrying out incremental reconstruction on the target symbol distance field model according to the target pose to obtain a target global symbol distance field model, and displaying an image corresponding to the target global symbol distance field model.
Need to check novelty before this filing date? Find Prior Art

Description

Three-dimensional laser point cloud VR display method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 2023106440131 filed in China on June 1, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of virtual display technology, and in particular to a three-dimensional laser point cloud VR display method and device, electronic equipment, non-transitory computer-readable storage medium, computer program product, and computer program. Background Art

[0004] With the continuous development and improvement of science and technology, virtual reality (VR) devices have made great progress and progress, and have gradually become popular in people's daily lives. In related technologies, VR devices need to solve the posture of the input laser point cloud or the feature points corresponding to the laser point cloud, and display the image corresponding to the laser point cloud to the user based on the posture obtained. However, the posture obtained by this posture solution method is low in accuracy, resulting in poor image display effect.

[0005] Summary of the Invention

[0006] The present disclosure provides a method and apparatus for displaying a three-dimensional laser point cloud in VR, an electronic device, a non-transitory computer-readable storage medium, a computer program product, and a computer program to at least address the technical problem of poor image display quality in related technologies. The technical solutions of the present disclosure are as follows:

[0007] According to a first aspect of an embodiment of the present disclosure, a three-dimensional laser point cloud VR display method is provided, comprising:

[0008] Obtain the target laser point cloud and generate a target symbol distance field model corresponding to the target laser point cloud;

[0009] The target posture is solved based on the target signed distance field model to obtain the target posture corresponding to the target laser point cloud;

[0010] The target signed distance field model is incrementally reconstructed according to the target posture to obtain the target global signed distance field model, and the image corresponding to the target global signed distance field model is displayed.

[0011] In some embodiments, generating a target signed distance field model corresponding to the target laser point cloud includes:

[0012] When the target laser point cloud is not the first laser point cloud acquired, obtaining a timestamp corresponding to the target laser point cloud;

[0013] Determine the inertial navigation data corresponding to the timestamp, and perform distortion correction on the target laser point cloud according to the inertial navigation data to obtain a corrected target laser point cloud;

[0014] The corrected target laser point cloud is gridded to obtain the target symbol distance field model corresponding to the target laser point cloud.

[0015] In some embodiments, determining the inertial navigation data corresponding to the timestamp, and performing distortion correction on the target laser point cloud according to the inertial navigation data to obtain a corrected target laser point cloud includes:

[0016] Retrieve a nearby inertial navigation data set corresponding to the timestamp, wherein the nearby inertial navigation data set includes nearby inertial navigation data corresponding to each point in the target laser point cloud;

[0017] Use the neighboring inertial navigation data as interpolation to calculate the inertial navigation data corresponding to each point in the target laser point cloud to obtain the inertial navigation data set;

[0018] According to the inertial navigation data set, the first point coordinates corresponding to the target laser point cloud are transformed into the second point coordinates corresponding to the laser point cloud acquired for the first time, so as to obtain the corrected target laser point cloud.

[0019] In some embodiments, performing posture solving based on the target signed distance field model to obtain the target posture corresponding to the target laser point cloud includes:

[0020] Calculating a first signed distance function corresponding to the target signed distance field model;

[0021] Determine a historical signed distance field model corresponding to the target laser point cloud, and calculate a second signed distance function corresponding to the historical signed distance field model, wherein the historical signed distance field model is a signed distance field model corresponding to the laser point cloud acquired for the first time;

[0022] When the first signed distance function and the second signed distance function satisfy the posture acquisition condition, the target posture corresponding to the target laser point cloud is determined.

[0023] In some embodiments, calculating a first signed distance function corresponding to the target signed distance field model includes:

[0024] Perform grid filtering on any point in a grid in the target signed distance field model to obtain the mean value corresponding to any point in the grid;

[0025] Replace the value of any point in any grid with the mean value to obtain any grid after filtering;

[0026] The signed distance function corresponding to any filtered grid is calculated to obtain the first signed distance function corresponding to the target signed distance field model.

[0027] In some embodiments, when the first signed distance function and the second signed distance function satisfy a posture acquisition condition, determining a target posture corresponding to the target laser point cloud includes:

[0028] Obtaining a difference between the first signed distance function and the second signed distance function, and determining an initial pose corresponding to the target laser point cloud;

[0029] Based on the difference being higher than the difference threshold, the initial posture is adjusted until the difference is no higher than the difference threshold, and the target posture corresponding to the target laser point cloud is obtained.

[0030] In some embodiments, incrementally reconstructing the target signed distance field model according to the target pose to obtain the target global signed distance field model includes:

[0031] According to the target posture, the first signed distance function is transformed into a global signed distance function to obtain the target global signed distance field model.

[0032] In some embodiments, displaying an image corresponding to a target global signed distance field model includes:

[0033] Obtaining a first stereoscopic projection image corresponding to a target global signed distance field model, and generating a second stereoscopic projection image based on the first stereoscopic projection image;

[0034] Rendering the first stereoscopic projection image and the second stereoscopic projection image according to the calibration parameters corresponding to the target laser point cloud and the target posture to obtain a rendered first stereoscopic projection image and a rendered second stereoscopic projection image;

[0035] The rendered first stereoscopic projection image is projected onto a first lens in the virtual reality glasses, and the rendered second stereoscopic projection image is projected onto a second lens in the virtual reality glasses to display the image corresponding to the target global signed distance field model.

[0036] In some embodiments, rendering the first stereoscopic projection image and the second stereoscopic projection image includes:

[0037] When a color image corresponding to the target laser point cloud is obtained, color rendering is performed on the first stereoscopic projection image and the second stereoscopic projection image;

[0038] When a color image corresponding to the target laser point cloud is not obtained, pseudo-color rendering is performed on the first stereoscopic projection image and the second stereoscopic projection image.

[0039] According to a second aspect of an embodiment of the present disclosure, a three-dimensional laser point cloud VR display device is provided, comprising:

[0040] A model generation unit is used to obtain a target laser point cloud and generate a target symbol distance field model corresponding to the target laser point cloud;

[0041] An attitude solving unit is used to solve the attitude based on the target signed distance field model to obtain the target attitude corresponding to the target laser point cloud;

[0042] The image display unit is used to incrementally reconstruct the target signed distance field model according to the target posture, obtain the target global signed distance field model, and display the image corresponding to the target global signed distance field model.

[0043] According to a third aspect of an embodiment of the present disclosure, a VR device is provided, including:

[0044] at least one processor; and

[0045] a memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods described in the above aspect.

[0047] According to a fourth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is proposed, wherein the computer instructions are used to enable the computer to execute any one of the methods described in the aforementioned aspect.

[0048] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is proposed, including a computer program, which implements the method described in any one of the aforementioned aspects when executed by a processor.

[0049] According to a sixth aspect of an embodiment of the present disclosure, a computer program is proposed, characterized in that the computer program includes computer program code, and when the computer program code is run on a computer, the computer executes the method described in the aforementioned aspect.

[0050] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0051] In some or related embodiments, a three-dimensional laser point cloud VR display method includes: acquiring a target laser point cloud and generating a target signed distance field model corresponding to the target laser point cloud; performing posture solution based on the target signed distance field model to obtain a target posture corresponding to the target laser point cloud; incrementally reconstructing the target signed distance field model based on the target posture to obtain a target global signed distance field model, and displaying an image corresponding to the target global signed distance field model. Therefore, by directly using the target signed distance field model for posture solution, the posture solution is more accurate than using laser point clouds or feature points, thereby improving the image display effect.

[0052] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0054] FIG1 is a schematic diagram showing a flow chart of a three-dimensional laser point cloud VR display method provided by an embodiment of the present disclosure;

[0055] FIG2 is a flow chart showing another method for displaying three-dimensional laser point cloud VR images according to an embodiment of the present disclosure;

[0056] FIG3 shows a schematic structural diagram of a three-dimensional laser point cloud VR display device provided by an embodiment of the present disclosure;

[0057] Fig. 4 is a block diagram showing a VR device according to an exemplary embodiment. DETAILED DESCRIPTION

[0058] In order to enable ordinary persons in the art to better understand the technical solutions of the embodiments of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0059] It should be noted that the terms "first," "second," and the like in the description and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0060] In the first embodiment, as shown in FIG1 , FIG1 illustrates a flow chart of a three-dimensional laser point cloud VR display method provided by an embodiment of the present disclosure. The method can be implemented by a computer program and can be run on a device that performs a multi-round dialogue recommendation method. The computer program can be integrated into an application or run as a standalone tool application. For example, the method can be run on a virtual reality (VR) device. The VR device can be, for example, VR glasses.

[0061] In some specific embodiments, the three-dimensional laser point cloud VR display method includes steps S101 to S103.

[0062] S101, obtaining a target laser point cloud and generating a target symbolic distance field model corresponding to the target laser point cloud.

[0063] According to some embodiments, the target laser point cloud refers to the laser point cloud input into the VR device by a device generating the laser point cloud. Specifically, the target laser point cloud refers to the laser point cloud of the working surface collected by a real-world 3D mapping device. The real-world 3D mapping device can be installed on dynamically operating equipment such as inspection robots, coal mining machines, and hydraulic supports, or it can be installed in a fixed location such as a roadway.

[0064] In some embodiments, the signed distance field model itself is a very simple structure. It can be represented by a signed distance field function or a volume map. Simply put, it is a representation of space that stores the distance from a point in space to the nearest triangle. If the point is inside the physical space, the distance is negative.

[0065] It is easy to understand that when a VR device performs a three-dimensional laser point cloud VR display, the VR device can obtain a target laser point cloud and generate a target symbolic distance field model corresponding to the target laser point cloud.

[0066] S102, performing posture solution based on the target signed distance field model to obtain the target posture corresponding to the target laser point cloud.

[0067] According to some embodiments, the target pose is used to describe the pose of the coordinate system corresponding to the target signed distance field model relative to the initial coordinate system, where the initial coordinate system refers to the coordinate system corresponding to the laser point cloud acquired for the first time.

[0068] It is easy to understand that when the VR device obtains the target signed distance field model, it can perform posture solution based on the target signed distance field model to obtain the target posture corresponding to the target laser point cloud.

[0069] S103 , incrementally reconstructing the target signed distance field model according to the target posture to obtain a target global signed distance field model, and displaying an image corresponding to the target global signed distance field model.

[0070] According to some embodiments, incremental reconstruction is a process of gradually increasing the viewing angle and iteratively optimizing the reprojection error. The goal is to calculate the camera parameters of different views, obtain a sparse point cloud of the scene, and determine the visual relationship between different views and the point cloud.

[0071] In some embodiments, the global signed distance field model refers to a signed distance field model obtained by incrementally reconstructing the target signed distance field model.

[0072] It is easy to understand that when the VR device obtains the target posture, it can incrementally reconstruct the target signed distance field model according to the target posture to obtain the target global signed distance field model, and display the image corresponding to the target global signed distance field model.

[0073] In summary, the method provided by the embodiments of the present disclosure obtains a target laser point cloud and generates a target signed distance field model corresponding to the target laser point cloud; performs posture solution based on the target signed distance field model to obtain the target posture corresponding to the target laser point cloud; incrementally reconstructs the target signed distance field model based on the target posture to obtain a target global signed distance field model, and displays the image corresponding to the target global signed distance field model. Therefore, by directly using the target signed distance field model for posture solution, the posture solution is more accurate than using laser point clouds or feature points, thereby improving image display quality.

[0074] Please refer to Figure 2, which shows a flow chart of a 3D laser point cloud VR display method provided by an embodiment of the present disclosure. In some specific embodiments, the 3D laser point cloud VR display method includes steps S201 to S210.

[0075] S201, obtain the target laser point cloud, and if the target laser point cloud is not the first laser point cloud obtained, obtain the timestamp corresponding to the target laser point cloud.

[0076] According to some embodiments, when a VR device performs a 3D laser point cloud VR display, the VR device can obtain a laser point cloud stream input by a real-world 3D mapping device. In the laser point cloud stream, the laser point cloud read at the first moment in the laser point cloud stream, in chronological order, is the first laser point cloud acquired.

[0077] In some embodiments, when the target laser point cloud is the first laser point cloud acquired, the signed distance field model corresponding to the laser point cloud is the global signed distance field model. Subsequently, when laser point clouds are read from the laser point cloud stream, the signed distance field models corresponding to the subsequently read laser point clouds are registered with the signed distance field model corresponding to the first laser point cloud acquired. The registration result is the pose of the subsequently read laser point cloud.

[0078] It is easy to understand that when the VR device performs three-dimensional laser point cloud VR display, the VR device can obtain the target laser point cloud and, if the target laser point cloud is not the first laser point cloud obtained, obtain the timestamp corresponding to the target laser point cloud.

[0079] S202 , determining the inertial navigation data corresponding to the timestamp, and performing distortion correction on the target laser point cloud according to the inertial navigation data to obtain a corrected target laser point cloud.

[0080] According to some embodiments, inertial navigation data refers to information such as velocity, yaw angle, and position in the navigation coordinate system obtained by measuring the acceleration of the carrier in the inertial reference system, integrating it with time, and transforming it into the navigation coordinate system based on Newton's laws of mechanics.

[0081] In some embodiments, the device for generating the laser point cloud and the device for acquiring the inertial navigation data need to be rigidly connected to the real-scene three-dimensional mapping device, thereby improving the accuracy of acquiring the inertial navigation data.

[0082] In some embodiments, when the target laser point cloud is a laser point cloud acquired for the first time, distortion correction is not required for the target laser point cloud. When the target laser point cloud is not a laser point cloud acquired for the first time, distortion correction can be performed on the target laser point cloud based on the coordinate system corresponding to the laser point cloud acquired for the first time.

[0083] According to some embodiments, to determine the inertial navigation data corresponding to a timestamp and perform distortion correction on the target laser point cloud based on the inertial navigation data to obtain a corrected target laser point cloud, first, a set of adjacent inertial navigation data corresponding to the timestamp can be retrieved. Next, the adjacent inertial navigation data can be used as interpolation to calculate the inertial navigation data corresponding to each point in the target laser point cloud to obtain an inertial navigation data set. Finally, based on the inertial navigation data set, the coordinates of the first point corresponding to the target laser point cloud can be transformed to the coordinates of the second point corresponding to the initially acquired laser point cloud to obtain a corrected target laser point cloud.

[0084] In some embodiments, when retrieving the inertial navigation data corresponding to the timestamp, since the laser point cloud and the inertial navigation data may not completely correspond, it is necessary to obtain the closest inertial navigation data corresponding to any point in the laser point cloud, that is, the adjacent inertial navigation data.

[0085] In some embodiments, the near-inertial navigation data set refers to a set formed by gathering at least one near-inertial navigation data, and includes near-inertial navigation data corresponding to each point in the target laser point cloud.

[0086] In some embodiments, the first point coordinates corresponding to the target laser point cloud are transformed to the second point coordinates corresponding to the laser point cloud acquired for the first time, and the point coordinates at the moment corresponding to the target laser point cloud can be transformed to the coordinates at the initial moment corresponding to the laser point cloud acquired for the first time, thereby completing the distortion correction.

[0087] It is easy to understand that when the VR device obtains the timestamp, it can determine the inertial navigation data corresponding to the timestamp, and perform distortion correction on the target laser point cloud based on the inertial navigation data to obtain the corrected target laser point cloud.

[0088] S203 , performing gridding processing on the corrected target laser point cloud to obtain a target symbolic distance field model corresponding to the target laser point cloud.

[0089] According to some embodiments, when meshing the corrected target laser point cloud, the grid size can be set as required.

[0090] It is easy to understand that when the VR device obtains the corrected target laser point cloud, the VR device can perform grid processing on the corrected target laser point cloud to obtain a target symbolic distance field model corresponding to the target laser point cloud.

[0091] S204: Calculate a first signed distance function corresponding to the target signed distance field model.

[0092] According to some embodiments, when calculating the first signed distance function corresponding to the target signed distance field model, the first signed distance function can be obtained by calculating the signed distance function for each grid in the target signed distance field model. Specifically, first, grid filtering can be performed on the points within any grid in the target signed distance field model to obtain the mean value corresponding to the points within any grid. Next, the mean value can be substituted for the value of the points within any grid to obtain any filtered grid. Finally, the signed distance function corresponding to any filtered grid can be calculated to obtain the first signed distance function corresponding to the target signed distance field model. Therefore, grid filtering can reduce the impact of noise generated during the process of obtaining the signed distance function.

[0093] In some embodiments, the grid filtering process specifically refers to mean filtering, also known as Gaussian filtering. The pixel value obtained after mean filtering is the average value of the pixel values ​​corresponding to any point in the grid.

[0094] It is easy to understand that when the VR device obtains the target signed distance field model, it can calculate the first signed distance function corresponding to the target signed distance field model.

[0095] S205 , determining a historical signed distance field model corresponding to the target laser point cloud, and calculating a second signed distance function corresponding to the historical signed distance field model.

[0096] According to some embodiments, the historical signed distance field model refers to a signed distance field model corresponding to a laser point cloud acquired for the first time.

[0097] In some embodiments, the second signed distance function refers to a signed distance function corresponding to the historical signed distance field model. The calculation method of the second signed distance function is as shown in S204 and will not be repeated here.

[0098] It is easy to understand that when the VR device obtains the target signed distance field model, it can determine the historical signed distance field model corresponding to the target laser point cloud and calculate the second signed distance function corresponding to the historical signed distance field model.

[0099] S206 , when the first signed distance function and the second signed distance function meet the posture acquisition condition, determining the target posture corresponding to the target laser point cloud.

[0100] According to some embodiments, the posture acquisition condition refers to a condition used when determining whether the target posture can be determined. The posture acquisition condition may be, for example, that the difference between the first signed distance function and the second signed distance function is not greater than a difference threshold.

[0101] In some embodiments, the difference threshold does not specifically refer to a fixed threshold and can be determined according to needs.

[0102] According to some embodiments, when determining the target pose corresponding to the target laser point cloud, if the first signed distance function and the second signed distance function meet the pose acquisition condition, the difference between the first signed distance function and the second signed distance function can be obtained to determine the initial pose corresponding to the target laser point cloud. If the difference is greater than a difference threshold, the initial pose is adjusted until the difference is no greater than the difference threshold, thereby obtaining the target pose corresponding to the target laser point cloud. Therefore, the accuracy of target pose acquisition can be improved.

[0103] In some embodiments, the difference between the first signed distance function and the second signed distance function refers to the difference in signed distances obtained between a first grid corresponding to the first signed distance function and a second grid corresponding to the second signed distance function.

[0104] It is easy to understand that when the VR device obtains the first signed distance function and the second signed distance function, the target posture corresponding to the target laser point cloud can be determined if the first signed distance function and the second signed distance function meet the posture acquisition condition.

[0105] S207 : transforming the first signed distance function into a global signed distance function according to the target posture to obtain a target global signed distance field model.

[0106] According to some embodiments, since the signed distance field model corresponding to the first acquired laser point cloud is the global signed distance field model, the global signed distance function is the signed distance function corresponding to the historical signed distance field model, that is, the second signed distance function.

[0107] It is easy to understand that when the VR device obtains the target posture, the first signed distance function can be transformed into a global signed distance function according to the target posture to obtain the target global signed distance field model to complete the incremental reconstruction.

[0108] S208 : Acquire a first stereoscopic projection picture corresponding to the target global signed distance field model, and generate a second stereoscopic projection picture according to the first stereoscopic projection picture.

[0109] According to some embodiments, when displaying an image corresponding to a target global signed distance field model, a virtual binocular camera can be placed in the 3D model coordinate system. The binocular camera includes two cameras: a first camera and a second camera. Next, the first camera can be used as a point light source to draw a ray toward the target global signed distance field model. The intersection of this ray and the target global signed distance field model is the scene point to be rendered at that moment. Finally, this scene point can be projected onto the focal plane to generate a first stereoscopic projection image.

[0110] In some embodiments, the first camera is a left camera or a right camera. When the first camera is a left camera, the second camera is a right camera. When the first camera is a right camera, the second camera is a left camera.

[0111] According to some embodiments, when acquiring the second stereoscopic projection image, the second camera does not perform ray tracing. Instead, it directly uses the position parameters between the first and second cameras to perform coordinate transformation on the first stereoscopic projection image to obtain the second stereoscopic projection image. This ensures real-time image acquisition.

[0112] It is easy to understand that when the VR device obtains the target global signed distance field model, it can obtain a first stereoscopic projection image corresponding to the target global signed distance field model, and generate a second stereoscopic projection image according to the first stereoscopic projection image.

[0113] S209 , rendering the first stereoscopic projection image and the second stereoscopic projection image according to the calibration parameters corresponding to the target laser point cloud and the target posture, to obtain a rendered first stereoscopic projection image and a rendered second stereoscopic projection image.

[0114] According to some embodiments, the calibration parameters include camera parameters of a camera in the real-scene 3D mapping device, radar parameters of a radar, and related parameters between the camera and the radar.

[0115] In some embodiments, when the VR device obtains the laser point cloud stream input by the real-scene 3D mapping device, it can simultaneously obtain the color video stream input by the real-scene 3D mapping device. At the same time, the device that generates the laser point cloud and the device that obtains the color video stream (such as a camera) need to be rigidly connected to the real-scene 3D mapping device, so as to ensure that the laser point cloud corresponding to any timestamp in the laser point cloud stream corresponds to the color picture corresponding to any timestamp in the color video stream. Therefore, when rendering the first stereoscopic projection picture and the second stereoscopic projection picture, if the color picture corresponding to the target laser point cloud is obtained, the first stereoscopic projection picture and the second stereoscopic projection picture can be rendered in color.

[0116] According to some embodiments, if the VR device does not synchronously obtain the color video stream input by the real-scene 3D mapping device when obtaining the laser point cloud stream input by the real-scene 3D mapping device, then if the color image corresponding to the target laser point cloud is not obtained, the first stereoscopic projection image and the second stereoscopic projection image can be rendered in pseudo color.

[0117] In some embodiments, when performing pseudo color rendering, pseudo color rendering may be performed according to the coordinates of scene points to be rendered in the first stereoscopic projection image and the second stereoscopic projection image.

[0118] Specifically, (r, g, b) can be set as the pixel value of the color image, with the pixel value range being 0 to 255. The maximum value of the z-axis data of the target global signed distance field model is set to m, and the minimum value is set to n. Then, during pseudo-color rendering, the pixel value of the scene point to be rendered in the first stereo projection image and the second stereo projection image can be determined according to the following formula: r = abs(255-z*255 / (mn)) g = abs(127-z*255 / (mn)) b = z*255 / (ba)

[0119] It is easy to understand that when the VR device obtains the first stereoscopic projection image and the second stereoscopic projection image, it can render the first stereoscopic projection image and the second stereoscopic projection image according to the calibration parameters and target posture corresponding to the target laser point cloud to obtain the rendered first stereoscopic projection image and the rendered second stereoscopic projection image.

[0120] S210: Projecting the rendered first stereoscopic projection image onto a first lens in the virtual reality glasses, and projecting the rendered second stereoscopic projection image onto a second lens in the virtual reality glasses, to display an image corresponding to the target global signed distance field model.

[0121] According to some embodiments, the first lens corresponds to the first camera in a one-to-one manner, and the second lens corresponds to the second camera in a one-to-one manner.

[0122] It is easy to understand that when the VR device obtains the rendered first stereoscopic projection image and the rendered second stereoscopic projection image, the rendered first stereoscopic projection image can be projected onto the first lens in the virtual reality glasses, and the rendered second stereoscopic projection image can be projected onto the second lens in the virtual reality glasses to complete the stereoscopic display of the image corresponding to the target global signed distance field model.

[0123] In summary, the method provided by the embodiment of the present disclosure first obtains the target laser point cloud, and when the target laser point cloud is not the first laser point cloud obtained, obtains the timestamp corresponding to the target laser point cloud; determines the inertial navigation data corresponding to the timestamp, and performs distortion correction on the target laser point cloud based on the inertial navigation data to obtain the corrected target laser point cloud; grids the corrected target laser point cloud to obtain the target symbol distance field model corresponding to the target laser point cloud; therefore, the accuracy of obtaining the target symbol distance field model can be improved. Then, calculate the first symbol distance function corresponding to the target symbol distance field model; determine the historical symbol distance field model corresponding to the target laser point cloud, and calculate the second symbol distance function corresponding to the historical symbol distance field model; when the first symbol distance function and the second symbol distance function meet the posture acquisition conditions, determine the target posture corresponding to the target laser point cloud; therefore, the accuracy of obtaining the target posture can be improved. Finally, according to the target posture, the first signed distance function is transformed into a global signed distance function to obtain the target global signed distance field model; the first stereo projection image corresponding to the target global signed distance field model is obtained, and the second stereo projection image is generated based on the first stereo projection image; according to the calibration parameters corresponding to the target laser point cloud and the target posture, the first stereo projection image and the second stereo projection image are rendered to obtain the rendered first stereo projection image and the rendered second stereo projection image; the rendered first stereo projection image is projected onto the first lens in the virtual reality glasses, and the rendered second stereo projection image is projected onto the second lens in the virtual reality glasses to display the image corresponding to the target global signed distance field model; therefore, by directly using the target signed distance field model for posture solution, compared with using laser point clouds or feature points, the accuracy of the solved posture is higher, thereby improving the image display effect.

[0124] The following are embodiments of the apparatus disclosed herein, which can be used to implement the method embodiments disclosed herein. For details not disclosed in the apparatus embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0125] Please refer to Figure 3, which shows a schematic diagram of the structure of a 3D laser point cloud VR display device provided by an embodiment of the present disclosure. The 3D laser point cloud VR display device can be implemented as all or part of a device through software, hardware, or a combination of both. The 3D laser point cloud VR display device 300 includes a model generation unit 301, a posture solving unit 302, and an image display unit 303.

[0126] The model generation unit 301 is used to obtain a target laser point cloud and generate a target signed distance field model corresponding to the target laser point cloud.

[0127] The attitude solving unit 302 is used to perform attitude solving based on the target signed distance field model to obtain the target attitude corresponding to the target laser point cloud.

[0128] The image display unit 303 is configured to incrementally reconstruct the target signed distance field model according to the target posture, obtain a target global signed distance field model, and display an image corresponding to the target global signed distance field model.

[0129] According to some embodiments, when the model generation unit 301 is used to generate a target signed distance field model corresponding to a target laser point cloud, it is specifically used to:

[0130] When the target laser point cloud is not the first laser point cloud acquired, obtaining a timestamp corresponding to the target laser point cloud;

[0131] Determine the inertial navigation data corresponding to the timestamp, and perform distortion correction on the target laser point cloud according to the inertial navigation data to obtain a corrected target laser point cloud;

[0132] The corrected target laser point cloud is gridded to obtain the target symbol distance field model corresponding to the target laser point cloud.

[0133] According to some embodiments, the model generation unit 301 is used to determine the inertial navigation data corresponding to the timestamp, and perform distortion correction on the target laser point cloud based on the inertial navigation data. When obtaining the corrected target laser point cloud, the model generation unit 301 is specifically used to:

[0134] Retrieve a nearby inertial navigation data set corresponding to the timestamp, wherein the nearby inertial navigation data set includes nearby inertial navigation data corresponding to each point in the target laser point cloud;

[0135] Use the neighboring inertial navigation data as interpolation to calculate the inertial navigation data corresponding to each point in the target laser point cloud to obtain the inertial navigation data set;

[0136] According to the inertial navigation data set, the first point coordinates corresponding to the target laser point cloud are transformed to the second point coordinates corresponding to the first acquired laser point cloud to obtain the corrected target laser point cloud.

[0137] According to some embodiments, the attitude solving unit 302 is used to perform attitude solving based on the target signed distance field model to obtain the target attitude corresponding to the target laser point cloud, specifically for:

[0138] Calculating a first signed distance function corresponding to the target signed distance field model;

[0139] Determine a historical signed distance field model corresponding to the target laser point cloud, and calculate a second signed distance function corresponding to the historical signed distance field model, wherein the historical signed distance field model is a signed distance field model corresponding to the laser point cloud acquired for the first time;

[0140] When the first signed distance function and the second signed distance function satisfy the posture acquisition condition, the target posture corresponding to the target laser point cloud is determined.

[0141] According to some embodiments, when the posture solving unit 302 is used to calculate the first signed distance function corresponding to the target signed distance field model, it is specifically used to:

[0142] Perform grid filtering on any point in a grid in the target signed distance field model to obtain the mean value corresponding to any point in the grid;

[0143] Replace the value of any point in any grid with the mean value to obtain any grid after filtering;

[0144] The signed distance function corresponding to any filtered grid is calculated to obtain a first signed distance function corresponding to the target signed distance field model.

[0145] According to some embodiments, the posture solving unit 302 is used to determine the target posture corresponding to the target laser point cloud when the first signed distance function and the second signed distance function meet the posture acquisition condition, specifically to:

[0146] Obtaining a difference between the first signed distance function and the second signed distance function, and determining an initial pose corresponding to the target laser point cloud;

[0147] If the difference is higher than the difference threshold, the initial posture is adjusted until the difference is no higher than the difference threshold, and the target posture corresponding to the target laser point cloud is obtained.

[0148] According to some embodiments, the image display unit 303 is configured to incrementally reconstruct the target signed distance field model according to the target posture to obtain the target global signed distance field model, specifically for:

[0149] According to the target posture, the first signed distance function is transformed into a global signed distance function to obtain a target global signed distance field model.

[0150] According to some embodiments, when the image display unit 303 is used to display an image corresponding to the target global signed distance field model, it is specifically used to:

[0151] Obtaining a first stereoscopic projection image corresponding to a target global signed distance field model, and generating a second stereoscopic projection image based on the first stereoscopic projection image;

[0152] Rendering the first stereoscopic projection image and the second stereoscopic projection image according to the calibration parameters corresponding to the target laser point cloud and the target posture to obtain a rendered first stereoscopic projection image and a rendered second stereoscopic projection image;

[0153] The rendered first stereoscopic projection image is projected onto a first lens in the virtual reality glasses, and the rendered second stereoscopic projection image is projected onto a second lens in the virtual reality glasses to display an image corresponding to the target global signed distance field model.

[0154] According to some embodiments, when the image display unit 303 is used to render the first stereoscopic projection image and the second stereoscopic projection image, it is specifically used to:

[0155] When a color image corresponding to the target laser point cloud is obtained, color rendering is performed on the first stereoscopic projection image and the second stereoscopic projection image;

[0156] When a color image corresponding to the target laser point cloud is not obtained, pseudo-color rendering is performed on the first stereoscopic projection image and the second stereoscopic projection image.

[0157] It should be noted that the three-dimensional laser point cloud VR display device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when executing the three-dimensional laser point cloud VR display method. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the three-dimensional laser point cloud VR display device provided in the above embodiment and the three-dimensional laser point cloud VR display method embodiment are of the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0158] In summary, the device provided by the embodiments of the present disclosure uses a model generation unit to acquire a target laser point cloud and generate a target signed distance field model corresponding to the target laser point cloud. A posture solving unit then performs posture solving based on the target signed distance field model to obtain a target posture corresponding to the target laser point cloud. An image display unit then incrementally reconstructs the target signed distance field model based on the target posture to obtain a target global signed distance field model, and displays an image corresponding to the target global signed distance field model. Therefore, by directly using the target signed distance field model for posture solving, the posture solved is more accurate than using a laser point cloud or feature points, thereby improving image display quality.

[0159] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0160] According to an embodiment of the present disclosure, the present disclosure also provides a VR device, a readable storage medium, a computer program product, and a computer program.

[0161] According to a third aspect of an embodiment of the present disclosure, a VR device is provided, including:

[0162] at least one processor; and

[0163] a memory communicatively connected to the at least one processor; wherein,

[0164] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods described in the above aspect.

[0165] FIG4 shows a schematic block diagram of an example VR device 400 that can be used to implement embodiments of the present disclosure. The components shown herein, their connections and relationships, and their functions are provided for example purposes only and are not intended to limit the implementation of the embodiments of the present disclosure described and / or claimed herein.

[0166] As shown in Figure 4, VR device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. RAM 403 can also store various programs and data required for the operation of VR device 400. Computing unit 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0167] Multiple components in VR device 400 are connected to I / O interface 405, including: input unit 406, such as a keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as a magnetic disk, optical disk, etc.; and communication unit 409, such as a network card, modem, wireless communication transceiver, etc. Communication unit 409 allows VR device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0168] The computing unit 401 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the 3D laser point cloud VR display method. For example, in some embodiments, the 3D laser point cloud VR display method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed into the VR device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the 3D laser point cloud VR display method described above can be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the three-dimensional laser point cloud VR display method in any other appropriate manner (for example, by means of firmware).

[0169] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0170] The program code for implementing the method of the embodiment of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or VR device.

[0171] According to a fourth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is proposed, wherein the computer instructions are used to enable the computer to execute any one of the methods described in the aforementioned aspect.

[0172] In the context of the embodiments of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0173] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0174] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data VR device), or a computing system that includes middleware components (e.g., an application VR device), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0175] A computer system may include a client and a VR device. The client and VR device are generally remote from each other and typically interact via a communication network. The relationship between the client and VR device is established by computer programs running on the respective computers and having a client-VR device relationship with each other. The VR device may be a cloud VR device, also known as a cloud computing VR device or cloud host, which is a host product in the cloud computing service system. It solves the management difficulties and poor business scalability of traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"). The VR device may also be a VR device for a distributed system, or a VR device integrated with blockchain.

[0176] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is proposed, including a computer program, which implements the method described in any one of the aforementioned aspects when executed by a processor.

[0177] According to a sixth aspect of an embodiment of the present disclosure, a computer program is proposed, characterized in that the computer program includes computer program code, and when the computer program code is run on a computer, the computer executes the method described in the aforementioned aspect.

[0178] It should be noted that the aforementioned explanations of the method and system embodiments also apply to the electronic device, computer-readable storage medium, computer program product, and computer program of the above embodiments, and will not be repeated here.

[0179] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the embodiments of the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the embodiments of the present disclosure can be achieved, and this document is not limited here.

[0180] The above specific implementations do not constitute a limitation on the scope of protection of the embodiments of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

[0181] All embodiments of the present disclosure may be implemented individually or in combination with other embodiments, and are all considered to be within the scope of protection claimed by the present disclosure.

Claims

1. A three-dimensional laser point cloud VR display method, characterized in that: include: Acquire a target laser point cloud, and generate a target symbol distance field model corresponding to the target laser point cloud; Performing posture solving based on the target signed distance field model to obtain the target posture corresponding to the target laser point cloud; The target signed distance field model is incrementally reconstructed according to the target posture to obtain a target global signed distance field model, and an image corresponding to the target global signed distance field model is displayed.

2. The method according to claim 1, characterized in that The generating of the target symbol distance field model corresponding to the target laser point cloud comprises: When the target laser point cloud is not a laser point cloud acquired for the first time, acquiring a timestamp corresponding to the target laser point cloud; Determine the inertial navigation data corresponding to the timestamp, and perform distortion correction on the target laser point cloud according to the inertial navigation data to obtain a corrected target laser point cloud; The corrected target laser point cloud is gridded to obtain the target symbol distance field model corresponding to the target laser point cloud.

3. The method according to claim 2, characterized in that The determining the inertial navigation data corresponding to the timestamp, and performing distortion correction on the target laser point cloud according to the inertial navigation data to obtain a corrected target laser point cloud, comprises: Retrieving a nearby inertial navigation data set corresponding to the timestamp, wherein the nearby inertial navigation data set includes nearby inertial navigation data corresponding to each point in the target laser point cloud; Using the adjacent inertial navigation data as interpolation, calculating the inertial navigation data corresponding to each point in the target laser point cloud, and obtaining an inertial navigation data set; According to the inertial navigation data set, the first point coordinates corresponding to the target laser point cloud are transformed to the second point coordinates corresponding to the laser point cloud acquired for the first time, so as to obtain the corrected target laser point cloud.

4. The method according to claim 2 or 3, characterized in that: The performing posture solving based on the target signed distance field model to obtain the target posture corresponding to the target laser point cloud includes: Calculating a first signed distance function corresponding to the target signed distance field model; Determine a historical signed distance field model corresponding to the target laser point cloud, and calculate a second signed distance function corresponding to the historical signed distance field model, wherein the historical signed distance field model is a signed distance field model corresponding to the laser point cloud acquired for the first time; When the first signed distance function and the second signed distance function satisfy a posture acquisition condition, the target posture corresponding to the target laser point cloud is determined.

5. The method according to claim 4, characterized in that The calculating a first signed distance function corresponding to the target signed distance field model includes: Performing grid filtering processing on points in any grid in the target signed distance field model to obtain a mean value corresponding to the points in any grid; Substituting the mean value for the point in any grid to obtain any grid after filtering; The signed distance function corresponding to any of the filtered grids is calculated to obtain the first signed distance function corresponding to the target signed distance field model.

6. The method according to claim 4 or 5, characterized in that: The step of determining the target posture corresponding to the target laser point cloud when the first signed distance function and the second signed distance function satisfy a posture acquisition condition comprises: Obtaining a difference between the first signed distance function and the second signed distance function, and determining an initial posture corresponding to the target laser point cloud; Based on the difference being higher than a difference threshold, the initial posture is adjusted until the difference is no higher than the difference threshold, thereby obtaining the target posture corresponding to the target laser point cloud.

7. The method according to any one of claims 4 to 6, characterized in that The incrementally reconstructing the target signed distance field model according to the target posture to obtain a target global signed distance field model includes: According to the target posture, the first signed distance function is transformed into a global signed distance function to obtain the target global signed distance field model.

8. The method according to any one of claims 1 to 7, characterized in that The displaying of the image corresponding to the target global signed distance field model includes: Acquire a first stereoscopic projection picture corresponding to the target global signed distance field model, and generate a second stereoscopic projection picture according to the first stereoscopic projection picture; Rendering the first stereoscopic projection image and the second stereoscopic projection image according to the calibration parameters corresponding to the target laser point cloud and the target posture to obtain a rendered first stereoscopic projection image and a rendered second stereoscopic projection image; The rendered first stereoscopic projection image is projected onto a first lens in a virtual reality pair of glasses, and the rendered second stereoscopic projection image is projected onto a second lens in the virtual reality pair of glasses to display the image corresponding to the target global signed distance field model.

9. The method according to claim 8, characterized in that The rendering of the first stereoscopic projection picture and the second stereoscopic projection picture includes: When a color image corresponding to the target laser point cloud is obtained, color rendering is performed on the first stereoscopic projection image and the second stereoscopic projection image; In the case where a color picture corresponding to the target laser point cloud is not obtained, pseudo-color rendering is performed on the first stereoscopic projection picture and the second stereoscopic projection picture.

10. A three-dimensional laser point cloud VR display device, characterized in that: include: A model generation unit, used to obtain a target laser point cloud and generate a target symbol distance field model corresponding to the target laser point cloud; A posture solving unit, used for performing posture solving based on the target signed distance field model to obtain the target posture corresponding to the target laser point cloud; An image display unit is used to incrementally reconstruct the target signed distance field model according to the target posture to obtain a target global signed distance field model, and display an image corresponding to the target global signed distance field model.

11. A VR device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the three-dimensional laser point cloud VR display method according to any one of claims 1 to 9 is implemented.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the three-dimensional laser point cloud VR display method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the three-dimensional laser point cloud VR display method according to any one of claims 1 to 9.

14. A computer program, characterized in that The computer program includes computer program code, and when the computer program code is run on a computer, the computer executes the three-dimensional laser point cloud VR display method as described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Clothes processing method and device based on double three-dimensional distance fields

    CN112966390A

  • Super-resolution method and device of laser radar, electronic equipment and storage medium

    CN114764746A

  • Grid model reconstruction method, system and device based on two-dimensional image and medium

    CN115731365A

  • Three-dimensional reconstruction method and device and readable storage medium

    CN115797561A

  • Three-dimensional laser point cloud VR display method and device

    CN116740313A

Cited By

  • Method and device for calculating deformation of coal mine tunnel

    CN120182384A