A three-dimensional scene simulation system and method based on VR technology

Abstract

The application relates to the technical field of three-dimensional image reconstruction, and discloses a three-dimensional scene simulation system and method based on VR technology, a three-dimensional scene simulation system based on VR technology, which comprises a linear brightness graph generation module, a minimum view angle change calculation module, a visible brightness step amplitude calculation module, a global safe translation radius calculation module and a continuous appearance color calculation module, and a three-dimensional scene presentation module; the linear brightness graph generation module is used for correcting and generating a linear brightness graph; the minimum view angle change calculation module is used for calculating a minimum view angle change; the visible brightness step amplitude calculation module is used for calculating a visible brightness step amplitude; the global safe translation radius calculation module is used for calculating a global safe translation radius; the continuous appearance color calculation module is used for calculating a continuous appearance color; and the three-dimensional scene presentation module is used for outputting an image to a head-mounted display. The application determines a candidate source image, a nearest competitive source image and a brightness difference, avoids source image switching misjudgment caused by area averaging, combines with a current line-of-sight direction to calculate an angular domain normalization variable, weights the linear brightness through a smoothing function to realize non-mutation transition of the source image, and completely eliminates map jumping and brightness mutation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional image reconstruction, more particularly, it relates to a three-dimensional scene simulation system and method based on VR technology. BACKGROUND

[0002] With the rapid application of VR technology in virtual display, education and training, and industrial simulation, users' requirements for immersive experience and visual continuity of three-dimensional scenes continue to improve. Existing VR three-dimensional scene simulation relies on multiple perspective source images to construct a virtual environment, and different source images are switched to adapt to changes in user perspective. However, many defects in the technical implementation result in prominent problems of visual discontinuity and unbalanced experience.

[0003] Firstly, when the camera collects source images, nonlinear encoding is performed to adapt to human visual perception or compress data, so that the pixel value of the source image is not proportional to the true brightness of the scene. At the same time, the difference in exposure time of different source images and the lens vignetting effect further cause distortion of the brightness data of the source image. If the original source image is directly used for switching, it will lay hidden dangers for subsequent brightness mutations.

[0004] Secondly, when switching the source image, the existing technology does not analyze the perspective difference and brightness relationship of each surface point of the reconstruction model point by point. The geometric position, normal and depth of the surface point are different, and there are differences in the visibility and perspective deviation of each source image. Directly using the regional average method to determine the switching time is easy to cause the surface point to appear map jump. And the brightness difference between the source images is not quantified, and the brightness mutation that can be perceived by the human eye is easy to occur when switching, which destroys the visual continuity of the scene.

[0005] In addition, when the user moves in the VR scene, the existing technology does not calculate the safe translation range of each surface point, and only uses fixed movement constraints. When the user moves too much, it is easy to break the perspective safety threshold and increase the jump probability. At the same time, the optical lens of the head-mounted display has radial distortion, and the existing correction is mostly simple linear compensation. It does not accurately pre-distort the left and right eyes respectively, which causes the deformation of the scene edge and further destroys the visual reality, making it difficult to meet the demand for high immersion experience. SUMMARY

[0006] The present application provides a three-dimensional scene simulation system and method based on VR technology to solve the technical problems in the background art.

[0007] The present application provides a three-dimensional scene simulation system based on VR technology, comprising:

[0008] A linear brightness map generation module for camera response correction, exposure time correction and vignetting correction of the source image to generate a linear brightness map;

[0009] a minimum view angle change calculation module configured to determine an angle between a view direction of each source image and the reference view direction according to the camera extrinsic parameters, and determine a candidate source image and a nearest competing source image according to the angle, and calculate a view angle difference between the candidate source image and the nearest competing source image to obtain a minimum view angle change;

[0010] a visible brightness step amplitude calculation module configured to map each surface point of the reconstruction model to a linear brightness map through a projection function and the camera intrinsic and extrinsic parameters, and calculate a brightness difference between the candidate source image and the nearest competing source image to obtain a visible brightness step amplitude;

[0011] a global safe translation radius calculation module configured to calculate a safe translation budget according to the minimum view angle change and the visible brightness step amplitude, and take a minimum value of the safe translation budget in the user activity area as a global safe translation radius;

[0012] a continuous appearance color calculation module configured to calculate an angular domain normalization variable according to the view angle difference between the candidate source image and the nearest competing source image, and a current view direction, and determine a non-negative weight through a smoothing function, and weight the linear brightness of the candidate source image and the nearest competing source image according to the non-negative weight to obtain a continuous appearance color, and constrain a translation amplitude of the user according to the global safe translation radius;

[0013] a three-dimensional scene rendering module configured to output the continuous appearance color processed through a lens pre-distortion function to a left eye display panel and a right eye display panel of a head-mounted display.

[0014] Further, a linear response quantity is obtained by processing a pixel value of each pixel position of the source image through a camera response inverse function; an exposure normalized response quantity is obtained by dividing the linear response quantity corresponding to each pixel position of the source image by an exposure time of the source image; a linear brightness value of the linear brightness map at the pixel position is obtained by dividing the exposure normalized response quantity corresponding to each pixel position of the source image by a vignetting model decay factor of the pixel position, thereby generating the linear brightness map.

[0015] Further, the minimum view angle change is calculated, including the following steps:

[0016] In step S201, a vector is obtained by subtracting each surface point of the reconstruction model from a camera optical center, and a view direction vector is obtained by dividing the vector by a length of the vector, wherein the camera optical center is determined by the camera extrinsic parameters;

[0017] In step S202, a dot product of the reference view direction and each view direction vector is calculated, and an angle between the reference view direction and each view direction vector is obtained by taking an inverse cosine value of the dot product;

[0018] Step S203: Select the source image with the smallest included angle among the same surface points as the candidate source image, and select the source image with the smallest difference in included angle with the candidate source image as the nearest competing source image.

[0019] Step S204: Calculate half of the angle difference between the nearest competing source image and the candidate source image for each surface point as the minimum viewpoint change.

[0020] Furthermore, the surface point is transformed to the camera coordinate system through camera extrinsic parameters, then transformed to image coordinates through the camera intrinsic parameter matrix, and finally the pixel coordinates of the surface point on each source image are obtained through the projection function. The linear brightness value of the pixel coordinates is obtained on the linear brightness map corresponding to the source image and used as the linear brightness sample value. The absolute value of the difference between the linear brightness sample value of the candidate source image and the linear brightness sample value of the other source images is calculated, and the largest value is taken as the visible brightness step amplitude value.

[0021] Furthermore, the visible brightness step amplitude of each surface point is divided by the corresponding minimum viewing angle change to obtain the jump risk field point value of that surface point. Then, the product of the depth of that surface point and the preset human eye perceived brightness difference is divided by the jump risk field point value to obtain the safe translation budget of that surface point. Finally, among all surface points in the user's activity area, the minimum value of the safe translation budget is selected as the global safe translation radius.

[0022] Further, calculating the continuous appearance color includes the following steps:

[0023] Step S301: Calculate the dot product of the current viewing direction with the viewing vectors of the candidate source image and the nearest competing source image, and then take the inverse cosine of the two dot products to obtain the two included angles.

[0024] Step S302: Calculate the ratio between the difference between the two included angles and twice the minimum angle change, and truncate the ratio to obtain the angle domain normalized variable.

[0025] Step S303: Calculate the square of the angle domain normalized variable multiplied by 3, and subtract the cube of the angle domain normalized variable multiplied by 2 to obtain the function value of the cubic Hermite smoothing function.

[0026] Step S304: Use the function value as the non-negative weight of the nearest competing source image, subtract the function value from 1 to obtain the non-negative weight of the candidate source image, and the sum of the two non-negative weights is 1;

[0027] Step S305: Multiply the linear brightness value of the candidate source image by its non-negative weight, multiply the linear brightness value of the most recent competing source image by its non-negative weight, and add the two products to obtain the continuous appearance color.

[0028] Further, the position vectors of two adjacent frames of the user are acquired, the Euclidean distance between the two position vectors is calculated as the actual translation distance of the user, and the actual translation distance of the user is restricted to not exceed the global safe translation radius.

[0029] Further, the lens pre-distortion function and the parameter set of the left and right eyes are acquired, the parameter set includes the first radial distortion coefficient and the second radial distortion coefficient of the lens of the eye, the non-distortion normalized image plane coordinates of each eye are acquired, the coordinates are composed of x-direction coordinate values and y-direction coordinate values, and the radial distance of the non-distortion normalized image plane coordinates is calculated; 1 is added to the first radial distortion coefficient, the square of the radial distance is multiplied, and the second radial distortion coefficient is multiplied by the fourth power of the radial distance, and then the intermediate calculation result is obtained; the intermediate calculation result is multiplied by the x-direction coordinate value and the y-direction coordinate value of the non-distortion normalized image plane coordinates respectively, and the x-direction coordinate value and the y-direction coordinate value of the display panel coordinates are obtained.

[0030] Further, the non-distortion normalized image plane coordinates of the left and right eyes are subjected to a ray projection operation, the surface points of the reconstructed model that are hit by the ray projection operation are obtained, and the continuous appearance color of the surface points is written into the corresponding display panel coordinates to generate the final image of each eye; the final image of the left eye is output to the left eye display panel of the head-mounted display, and the final image of the right eye is output to the right eye display panel of the head-mounted display.

[0031] The application provides a three-dimensional scene simulation method based on VR technology, comprising the following steps:

[0032] Step S401, camera response correction, exposure time correction and vignetting correction are performed on the source image to generate a linear luminance map;

[0033] Step S402, the angle between the view direction of each source image and the reference line-of-sight direction is determined according to the camera external parameters, and the candidate source image and the nearest competing source image are determined accordingly, and the minimum view angle change is obtained by calculating the view angle difference between the two;

[0034] Step S403, each surface point of the reconstructed model is mapped to the linear luminance map through a projection function and camera internal and external parameters, and the visible luminance step amplitude is obtained by calculating the luminance difference between the candidate source image and the nearest competing source image;

[0035] Step S404, the safe translation budget is calculated according to the minimum view angle change and the visible luminance step amplitude, and the minimum value of the safe translation budget in the user activity area is taken as the global safe translation radius;

[0036] Step S405, according to the angle difference between the candidate source image and the nearest competing source image, combined with the current line of sight direction, the angular domain normalization variable is obtained, and the non-negative weight is determined through the smoothing function, according to which the linear brightness of the candidate source image and the nearest competing source image is weighted to obtain the continuous appearance color, and the user's translation amplitude is constrained according to the global safe translation radius;

[0037] Step S406, the continuous appearance color is processed by the lens pre-distortion function and output to the left eye display panel and the right eye display panel of the head-mounted display.

[0038] The beneficial effects of the present application are that the present application performs camera response correction, exposure time correction and vignetting correction on the source image, eliminates the brightness distortion caused by the camera non-linear characteristics, exposure difference and lens vignetting, and ensures that the brightness data accurately reflects the real radiance of the scene; for each surface point of the reconstruction model, the angle between the view direction of each source image and the reference line of sight direction is calculated point by point, the candidate source image and the nearest competing source image and the brightness difference are determined, and the misjudgment of source image switching caused by regional average is avoided; the angular domain normalization variable is calculated combined with the current line of sight direction, the weight is determined through the smoothing function, and the linear brightness is weighted, the source image is transitioned without mutation, and the map jump and brightness mutation are completely eliminated; the safe translation budget of each surface point is calculated and the minimum value is taken as the global safe translation radius, the user's moving amplitude is constrained, and the visual imbalance caused by breaking through the safe range of the angle of view is prevented; the left and right eyes are respectively processed by the lens pre-distortion, the radial distortion of the head-mounted display lens is offset, and the deformation of the scene edge is avoided; thereby realizing the source image switching without perception and the stable effect of the scene when the user moves, and significantly improving the visual continuity and immersion of the VR three-dimensional scene. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a schematic diagram of a three-dimensional scene simulation system based on VR technology of the present application;

[0040] Figure 2 is a flowchart of calculating the minimum angle of view change of the present application;

[0041] Figure 3 is a flowchart of calculating the continuous appearance color of the present application;

[0042] Figure 4 is a flowchart of a three-dimensional scene simulation method based on VR technology of the present application.

[0043] In the figure: linear brightness map generation module 101, minimum angle of view change calculation module 102, visible brightness step amplitude calculation module 103, global safe translation radius calculation module 104, continuous appearance color calculation module 105, three-dimensional scene presentation module 106. DETAILED DESCRIPTION

[0044] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0045] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of the present invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in one or more embodiments of the present invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" indicate that the element or object preceding the term encompasses the elements or objects listed following the term and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0046] like Figures 1-4 As shown, a 3D scene simulation system based on VR technology includes:

[0047] The linear brightness map generation module 101 is used to perform camera response correction, exposure time correction and vignetting correction on the source image to generate a linear brightness map.

[0048] The minimum viewing angle change calculation module 102 is used to determine the angle between the viewing direction of each source image and the reference viewing direction based on the camera's external parameters, and to determine the candidate source image and the nearest competing source image accordingly, and calculate the viewing angle difference between the two to obtain the minimum viewing angle change.

[0049] The visible brightness step amplitude calculation module 103 is used to map each surface point of the reconstructed model to a linear brightness map through a projection function and camera intrinsic and extrinsic parameters, and to calculate the brightness difference between the candidate source image and the most recent competing source image to obtain the visible brightness step amplitude.

[0050] The global safe translation radius calculation module 104 is used to calculate the safe translation budget based on the minimum viewing angle change and the visible brightness step amplitude, and take the minimum value of the safe translation budget as the global safe translation radius within the user's active area.

[0051] a continuous appearance color calculation module 105, configured to obtain an angular domain normalization variable according to a perspective difference between the candidate source image and the nearest competing source image, in combination with the current line-of-sight direction, and determine a non-negative weight through a smoothing function, according to which the linear luminance of the candidate source image and the nearest competing source image is weighted to obtain a continuous appearance color, while the translation amplitude of the user is constrained according to a global safe translation radius;

[0052] a three-dimensional scene rendering module 106, configured to output the continuous appearance color processed by the lens pre-distortion function to the left-eye display panel and the right-eye display panel of the head-mounted display.

[0053] It should be noted that the reconstruction model is a virtual geometric model obtained by three-dimensional digitization of a real scene, containing the surface shape, position and spatial relationship of all objects in the scene, for example, multi-view images or laser scanning data of red cultural scenes such as memorial halls and historical sites are collected, and the multi-view images or laser scanning data are converted into a three-dimensional digital model containing a set of surface points through a three-dimensional reconstruction algorithm, which is used to reproduce the spatial structure of the scene in VR (Virtual Reality).

[0054] In an embodiment of the present application, the pixel value of each pixel position of the source image is processed through the camera response inverse function to obtain a linear response quantity; the linear response quantity corresponding to each pixel position of the source image is divided by the exposure time of the source image to obtain an exposure-normalized response quantity; the exposure-normalized response quantity corresponding to each pixel position of the source image is divided by the attenuation factor of the vignetting model at the pixel position to obtain the linear luminance value of the linear luminance map at the pixel position, and then the linear luminance map is generated.

[0055] Specifically, the linear luminance value of the u-th pixel position of the linear luminance map is calculated as follows: , wherein represents the pixel value of the u-th pixel position of the source image, represents the camera response inverse function, and t represents the exposure time of the source image, represents the attenuation factor of the vignetting model at the u-th pixel position of the source image.

[0056] ​It should be noted that when collecting images, the consumer camera will perform nonlinear coding on the real light intensity of the scene to adapt to the visual perception of the human eye or to compress the image data, for example, common gamma compression, which causes the read pixel value to be not proportional to the actual brightness of the scene, so that the source image pixel value is processed by the camera response inverse function, the nonlinear coded pixel value can be restored to a linear response quantity proportional to the real radiance of the scene, thereby eliminating the interference of the nonlinear characteristics of the camera itself on the brightness quantization, and the camera response inverse function can be an inverse function based on gamma correction or an inverse function based on polynomial fitting, and the like, which will not be described here.

[0057] It should be noted that different source images can be taken with different exposure times, and even if the same camera takes the same scene, the difference in exposure time will result in different linear response quantities, that is, the longer the exposure time, the more light energy the camera sensor receives, and the larger the linear response quantity value. This difference only comes from the shooting parameters and is unrelated to the brightness of the scene, so dividing the linear response quantity by the exposure time of the source image eliminates the problem of inconsistent brightness dimensions caused by different exposure times, so that source images taken with different exposure times have a unified numerical reference when reflecting the brightness of the same scene, ensuring that subsequent processing is not disturbed by shooting parameters.

[0058] It should be noted that due to the limitation of the optical structure of the camera lens, light passing through the edge area of the lens will be refracted and lost or blocked, resulting in a higher brightness of the same object point in the center area of the image than in the edge area. This phenomenon is called vignetting effect, and the vignetting model is a mathematical model for quantifying the vignetting effect of the lens. The attenuation factor of the vignetting model is used to quantify the brightness attenuation degree corresponding to the pixel position, and its value range is usually between 0 and 1. Therefore, dividing the exposure-normalized response quantity by the attenuation factor corresponding to the pixel position can correct the brightness deviation caused by the vignetting effect of the lens, so that the brightness value of the same object point at any position in the image remains consistent, and the final generated linear brightness map can reflect the real radiance of the scene. The polynomial vignetting model can be a polynomial vignetting model or a cosine fourth power vignetting model, and the like, which will not be described here.

[0059] In an embodiment of the present application, as shown in Figure 2 The minimum view angle change is calculated, including the following steps:

[0060] In step S201, the camera optical center is subtracted from each surface point of the reconstruction model to obtain a vector, and then the vector is divided by the length of the vector to obtain a view direction vector, wherein the camera optical center is determined by the camera external parameter;

[0061] In step S202, the dot product of the reference view direction and each view direction vector is calculated, and the inverse cosine value of the dot product is taken to obtain the included angle between the reference view direction and each view direction vector.

[0062] Specifically, the included angle between the reference line-of-sight direction and the view direction vector corresponding to the kth source image at the Pth surface point The calculation formula is as follows:

[0063] , wherein represents the camera optical center corresponding to the kth source image, represents the coordinates of the Pth surface point, represents the reference line-of-sight direction, represents the inverse cosine function, represents the modulo operation;

[0064] Step S203: selecting the source image with the smallest included angle in the same surface point as the candidate source image, and selecting the source image with the smallest included angle difference from the candidate source image as the nearest competing source image;

[0065] Step S204: calculating one-half of the included angle difference between the nearest competing source image and the candidate source image of each surface point as the minimum view angle change.

[0066] It should be noted that the camera optical center is the center of the camera optical system, i.e., the optical center point of the lens, which is the reference point for imaging after the light enters the camera. The camera extrinsic parameters include a rotation matrix and a translation vector, which together constitute the mapping relationship between the world coordinate system and the camera coordinate system. The world coordinates of the camera optical center are equal to the negative transpose matrix of the rotation matrix multiplied by the translation vector. Therefore, the position of the camera optical center can be uniquely determined by using only the camera extrinsic parameters.

[0067] It should be noted that the reference line-of-sight direction represents a unit directional vector for angle measurement, which can be taken as the optical axis direction from the observation posture of the initial virtual reality station, i.e., the initial line-of-sight direction of the user when entering the virtual scene, and remains unchanged in the entire scene range. For example, if the scene is a cultural relic exhibition hall, the reference line-of-sight direction can be set as the direction of the key cultural relics in the center of the exhibition hall, so as to ensure that the initial view angle of the user can directly observe the core display content.

[0068] It should be noted that the trigger condition of the texture jump is related to the geometric position, normal, occlusion state and depth of the surface point. The visibility and view angle difference of different surface points to each source image are different. The candidate source image, the nearest competing source image and the minimum view angle change are determined point by point, which can accurately match the unique geometric properties of each surface point, avoid the judgment deviation caused by the area average method, and ensure the accurate judgment of the source image switching condition of each surface point.

[0069] In one embodiment of the present application, the surface point is converted to the camera coordinate system through the camera external parameter, then converted to the image coordinate through the camera internal parameter matrix, and finally the pixel coordinates of the surface point on each source image are obtained through the projection function, and the linear brightness value of the pixel coordinates on the linear brightness graph corresponding to the source image is obtained as the linear brightness sampling value.

[0070] It should be noted that the world coordinates of the surface point are first multiplied by the rotation matrix to complete the pose alignment of the surface point around the camera optical center, and then the result of the multiplication is added to the translation vector to complete the position offset from the world coordinate system to the camera coordinate system, that is, the three-dimensional coordinates of the surface point in the camera coordinate system are obtained; the camera internal parameter matrix contains parameters such as focal length and principal point coordinates, the horizontal and vertical coordinates of the surface point in the camera coordinate system are first divided by the vertical coordinate (depth) to obtain the normalized image coordinates, and then multiplied by the focal length in the internal parameter matrix, and the principal point coordinates in the internal parameter matrix are added to obtain the two-dimensional coordinates of the surface point in the image coordinate system; finally, the pixel coordinates are obtained through the projection function, and the projection function is used to normalize and quantize the two-dimensional coordinates in the image coordinate system. If the image coordinate system coordinates have decimals, the projection function will convert them to integers through rounding to ensure that the surface point can be accurately mapped to the pixel position of the linear brightness graph.

[0071] In one embodiment of the present application, the absolute value of the difference between the linear brightness sampling value of the candidate source image and the linear brightness sampling value of the remaining source image is calculated, and the maximum value is taken as the visible brightness step amplitude.

[0072] It should be noted that the geometric position of each surface point is different, the pixel coordinates mapped to each source image are also different, and the linear brightness sampling values obtained on the linear brightness graph corresponding to each source image also differ, so that one surface point corresponds to one visible brightness step amplitude; the brightness difference between the candidate source image and the nearest competing source image is calculated to obtain the visible brightness step amplitude, because these two source images are the most likely objects to switch for the surface point, and the brightness difference can directly reflect the degree of brightness change most likely to occur when switching, without calculating the brightness difference between all source images, reducing redundant operations, providing a quantitative basis for photometric level for subsequent texture jump risk assessment, that is, the visible brightness step amplitude is smaller, indicating that the brightness change when switching is not obvious, and the jump risk is low, otherwise it indicates that the brightness change when switching is significant, and the jump needs to be focused on through subsequent processes to suppress the jump, and finally realize the texture switching without perception in the virtual reality scene.

[0073] In one embodiment of the present application, the visible brightness step amplitude of each surface point is divided by the corresponding minimum viewing angle change to obtain the surface point's jump risk field point value, then the depth of the surface point is multiplied by the preset human eye perceived brightness difference to obtain the surface point's safe translation budget, and finally the minimum value of the safe translation budget is selected from all surface points in the user activity area as the global safe translation radius.

[0074] Specifically, the safe translation budget of the Pth surface point is calculated as follows:

[0075] wherein represents the depth of the Pth surface point, represents the preset human eye perceived brightness difference, represents the jump risk field point value of the Pth surface point.

[0076] It should be noted that the depth represents the distance from the surface point on the reconstruction model to the reference viewpoint, wherein the reference viewpoint is a preset initial observation point in the virtual reality scene, which is consistent with the source of the above-mentioned reference line direction, i.e. the initial standing position of the user when entering the virtual scene; the preset human eye perceived brightness difference represents the minimum brightness change between two regions that can be perceived by the human eye, i.e. when the brightness difference between the two regions is less than this value, the human eye cannot distinguish the brightness difference between the two regions, and its value range is usually 0.02 to 0.05 times the scene reference brightness, wherein the scene reference brightness represents the linear brightness average of all visible surface points in the virtual reality scene, for example, in a scene with an average brightness of 100 nits, the preset human eye perceived brightness difference can be set to 2 to 5 nits, thereby ensuring that the brightness change perceived by the human eye is captured; all surface points in the user activity area represent the set of surface points on the reconstruction model that can be covered by the user's line of sight, wherein the user activity area is a spatial range preset according to the scene function requirement, for example, in a virtual scene of a red culture exhibition hall, a 10m x 10m rectangular area within the exhibition hall can be set, which can be set in combination with the actual size of the scene and the user interaction requirement, and will not be described here.

[0077] ​It should be noted that the safety translation budget of each surface point can only constrain the local movement range corresponding to the point, and if the local budget is directly used, the brightness jump risk may occur in some areas due to the difference between different point budgets, so the minimum value of the safety translation budget is selected as the global safety translation radius, which can cover the movement constraints in the scene, that is, as long as the translation amplitude does not exceed the global radius, the brightness change of all visible surface points can be ensured to be less than the human eye perceptible brightness difference, avoiding the risk of missing local risks, and the global radius can be directly used to guide the interactive design of the VR device, for example, when it is detected that the user translation approaches the global radius, the jump risk can be actively avoided by limiting the view angle movement, and finally the user movement and texture switching in the whole scene range are seamlessly connected without perception, and the stability and smoothness of the virtual reality experience are improved.

[0078] In one embodiment of the application, as shown in Figure 3 the continuous appearance color is calculated, including the following steps:

[0079] Step S301, the dot products of the current view direction and the view direction vectors of the candidate source image and the nearest competitor source image are calculated respectively, and the inverse cosine values of the two dot products are taken to obtain two included angles;

[0080] Step S302, the ratio between the difference of the two included angles and twice the minimum view angle change is calculated, and the ratio is truncated to obtain an angle domain normalization variable;

[0081] The truncation operation is that if the calculation result is less than 0, 0 is taken, if the calculation result is greater than 1, 1 is taken, and if the calculation result is between 0 and 1, it remains unchanged;

[0082] Step S303, the square of the angle domain normalization variable is multiplied by 3, and the cube of the angle domain normalization variable is subtracted by 2 to obtain the function value of the cubic Hermite smoothing function;

[0083] Step S304, the function value is taken as the non-negative weight of the nearest competitor source image, and 1 is subtracted from the function value to obtain the non-negative weight of the candidate source image, and the sum of the two non-negative weights is 1;

[0084] Step S305, the linear brightness value of the candidate source image is multiplied by its non-negative weight, the linear brightness value of the nearest competitor source image is multiplied by its non-negative weight, and the two products are added to obtain the continuous appearance color.

[0085] Specifically, the angle domain normalization variable of the Pth surface point in the current view direction is calculated as follows:

[0086] Where r represents the current view direction, and respectively represent the included angle between the current line-of-sight direction and the view direction vectors of the candidate source image and the nearest competing source image at the Pth surface point, represents the minimum view angle change of the Pth surface point, and clamp represents a clipping function.

[0087] It should be noted that one surface point corresponds to one angular domain normalization variable, which is used to quantify the position of the current line-of-sight direction in the view angle transition interval (twice the minimum view angle change) of the candidate source image and the nearest competing source image, to determine which view angle of the source image the current line-of-sight direction is closer to. The closer the value is to 0, the closer the current line-of-sight is to the view direction of the candidate source image. The closer the value is to 1, the closer the current line-of-sight is to the view direction of the nearest competing source image. The cubic Hermite smoothing function generates a weight that smoothly changes with the angular domain normalization variable, realizing the non-mutual transition of the linear luminance values of the candidate source image and the nearest competing source image. That is, this smooth weight change can avoid the luminance mutation when switching between source images, and eliminate the sense of map jumping. One surface point corresponds to one continuous appearance color, which is used to render the final display color, obtained by weighting the linear luminance values of the candidate source image and the nearest competing source image according to the non-negative weight, and can reflect the true visual appearance of the surface point under the current line-of-sight direction.

[0088] In an embodiment of the present application, the position vectors of the user in two adjacent frames are obtained, the Euclidean distance between the two position vectors is calculated as the actual translation distance of the user, and the actual translation distance of the user is constrained to be not more than the global safe translation radius.

[0089] It should be noted that the position vectors of the user in two adjacent frames are obtained by the positioning system of the virtual reality device itself. Common positioning methods include: the inertial measurement unit (IMU) built in the VR device collects the motion posture data of the user, combines with the space positioning technology (such as SLAM synchronous positioning and map construction), and calculates the three-dimensional position coordinates of the user in the virtual scene in real time. The corresponding position vectors of the user in the virtual scene at the two time points are the coordinates of the user in the virtual scene at the two time points, without relying on external camera to collect video.

[0090] In one embodiment of the present application, the lens pre-distortion function and the parameter set of the left and right eyes are obtained, the parameter set including the first radial distortion coefficient and the second radial distortion coefficient of the lens of the eye, the non-distortion normalized image plane coordinates of each eye are obtained, the coordinates consisting of the x-direction coordinate value and the y-direction coordinate value, and the radial distance of the non-distortion normalized image plane coordinates is calculated, that is, the square root of the sum of the square of the x-direction coordinate value and the square of the y-direction coordinate value; the intermediate calculation result is obtained by adding 1 to the first radial distortion coefficient multiplied by the square of the radial distance, and adding the second radial distortion coefficient multiplied by the fourth power of the radial distance, and then the intermediate calculation result is multiplied by the x-direction coordinate value and the y-direction coordinate value of the non-distortion normalized image plane coordinates respectively to obtain the x-direction coordinate value and the y-direction coordinate value of the display panel coordinates.

[0091] It should be noted that the lens pre-distortion function is an inverse function used to compensate for the distortion of the optical lens of the head-mounted display, the optical lens of the head-mounted display will cause radial distortion (such as stretching or shrinking of the edge area) of the display image, and the function pre-distorts the image by calculation in the opposite direction of the lens distortion to ensure that the image returns to normal after refraction through the lens; the non-distortion normalized image plane coordinates are two-dimensional coordinates mapped to a fixed range after removing the influence of lens distortion, which are determined by the hardware parameters of the head-mounted display, for example, first, each pixel coordinate of the display panel is converted into a physical coordinate of the image plane according to the resolution of the display panel, then the physical coordinate is divided by the focal length according to the focal length parameter of the optical system to eliminate the influence of the focal length on the coordinate range, and finally the coordinate is mapped to a normalized range of -1 to 1 or 0 to 1; therefore, the display panel coordinates are used to compensate for the distortion of the optical lens of the head-mounted display to ensure that the user sees a virtual scene without distortion; the lens pre-distortion function and the parameter set of the left and right eyes can be obtained by Zhang Zhengyou calibration method, which will not be repeated here.

[0092] In one embodiment of the present application, the non-distortion normalized image plane coordinates of the left and right eyes are subjected to ray casting operation to obtain the surface points of the reconstructed model hit by the ray casting operation, and the continuous appearance color of the surface points is written into the corresponding display panel coordinates to generate the final image of each eye, the final image of the left eye is output to the left eye display panel of the head-mounted display, and the final image of the right eye is output to the right eye display panel of the head-mounted display.

[0093] It should be noted that the light projection operation is to emit a virtual ray from the user's eye position along the direction corresponding to the non-distorted normalized image plane coordinate, find the intersection point of the ray and the surface of the reconstructed model, and the intersection point is the surface hit by the light projection operation, which can be realized by the Moore-Tromp algorithm, and will not be repeated here; in addition, the final rendering effect can be a static image or a dynamic video, and corresponding stereoscopic images are generated for the left and right eyes, and a three-dimensional visual effect is presented through a head-mounted display.

[0094] In one embodiment of the application, as shown in Figure 4 A three-dimensional scene simulation method based on VR technology, comprising the following steps:

[0095] Step S401, camera response correction, exposure time correction and vignetting correction are performed on the source image to generate a linear luminance map;

[0096] Step S402, the angle between the view direction of each source image and the reference line direction is determined according to the camera external parameters, and the candidate source image and the nearest competing source image are determined according to the angle, and the minimum view angle change is obtained by calculating the view angle difference between the two;

[0097] Step S403, each surface point of the reconstructed model is mapped to the linear luminance map by a projection function and camera internal and external parameters, and the visible luminance step amplitude is obtained by calculating the luminance difference between the candidate source image and the nearest competing source image;

[0098] Step S404, the minimum view angle change and the visible luminance step amplitude are calculated to obtain a safe translation budget, and the minimum value of the safe translation budget in the user activity area is taken as the global safe translation radius;

[0099] Step S405, the angle domain normalization variable is calculated according to the view angle difference between the candidate source image and the nearest competing source image, combined with the current line direction, and the non-negative weight is determined by a smoothing function, and the linear luminance of the candidate source image and the nearest competing source image is weighted to obtain a continuous appearance color, and the translation amplitude of the user is constrained according to the global safe translation radius;

[0100] Step S406, the continuous appearance color is processed by a lens pre-distortion function and output to the left eye display panel and the right eye display panel of the head-mounted display.

[0101] It should be noted that the interval and the threshold size are set for the convenience of comparison, wherein the size of the threshold depends on how much sample data and the base number set by the person skilled in the art for each group of sample data, as long as it does not affect the proportional relationship between the parameters and the quantized values. And the above formula is the calculation of the dimensionless value, the formula is obtained by collecting a large amount of data to simulate the formula of the nearest real situation, and the preset parameters in the formula are set by the person skilled in the art according to the actual situation.

[0102] The above describes the embodiments of the present embodiment, but the present embodiment is not limited to the specific embodiments described above, and the specific embodiments described above are only illustrative but not limiting, and a person skilled in the art can make many forms under the inspiration of the present embodiment, which all belong to the protection of the present embodiment.

Claims

1. A three-dimensional scene simulation system based on VR technology, characterized in that, The method comprises the following steps: a linear brightness map generation module for camera response correction, exposure time correction and vignetting correction on the source images to generate a linear brightness map; a minimum view angle change calculation module for determining the included angle between the view direction of each source image and the reference line of sight according to the camera external parameters, and determining the candidate source image and the nearest competing source image according to the included angle, and calculating the view angle difference between the two to obtain the minimum view angle change; a visible brightness step amplitude calculation module for mapping each surface point of the reconstruction model to the linear brightness map through the projection function and the camera internal and external parameters, and calculating the brightness difference between the candidate source image and the nearest competing source image to obtain the visible brightness step amplitude; a global safe translation radius calculation module for calculating the safe translation budget according to the minimum view angle change and the visible brightness step amplitude, and taking the minimum value of the safe translation budget in the user activity area as the global safe translation radius; a continuous appearance color calculation module for calculating the angular domain normalization variable according to the view angle difference between the candidate source image and the nearest competing source image, combining the current line of sight, and determining the non-negative weight through the smoothing function, and weighting the linear brightness of the candidate source image and the nearest competing source image to obtain the continuous appearance color, while restricting the translation amplitude of the user according to the global safe translation radius; a three-dimensional scene rendering module for processing the continuous appearance color through the lens pre-distortion function and outputting it to the left eye display panel and the right eye display panel of the head-mounted display; calculating the minimum view angle change comprises the following steps: step S201, subtracting the camera optical center from each surface point of the reconstruction model to obtain a vector, and then dividing the vector by the length of the vector to obtain a view direction vector, wherein the camera optical center is determined by the camera external parameters; step S202, calculating the dot product of the reference line of sight and each view direction vector, and then taking the inverse cosine value of the dot product to obtain the included angle between the reference line of sight and each view direction vector; step S203, selecting the source image with the smallest included angle in the same surface point as the candidate source image, and selecting the source image with the smallest included angle difference from the candidate source image as the nearest competing source image; step S204, calculating the half of the included angle difference between the nearest competing source image and the candidate source image of each surface point as the minimum view angle change; calculating the continuous appearance color comprises the following steps: step S301, calculating the dot product of the current line of sight and the view direction vector of the candidate source image and the nearest competing source image respectively, and then taking the inverse cosine value of the two dot products to obtain two included angles; step S302, calculating the ratio between the difference of the two included angles and twice the minimum view angle change, and performing a truncation operation on the ratio to obtain the angular domain normalization variable; step S303, calculating the square of the angular domain normalization variable multiplied by 3 minus the cube of the angular domain normalization variable multiplied by 2 to obtain the function value of the cubic Hermite smoothing function; step S304, taking the function value as the non-negative weight of the nearest competing source image, and taking 1 minus the function value as the non-negative weight of the candidate source image, and the sum of the two non-negative weights is 1. Step S305, multiplying the linear luminance value of the candidate source image by its non-negative weight, multiplying the linear luminance value of the nearest competing source image by its non-negative weight, adding the two products to obtain the continuous appearance color. 2.The three-dimensional scene simulation system based on VR technology of claim 1, wherein, Processing the pixel value of each pixel position of the source image through the camera response inverse function to obtain a linear response quantity; dividing the linear response quantity corresponding to each pixel position of the source image by the exposure time of the source image to obtain an exposure-normalized response quantity; Dividing the exposure-normalized response quantity corresponding to each pixel position of the source image by the attenuation factor of the vignetting model of the pixel position to obtain the linear luminance value of the linear luminance map at the pixel position, and further generating the linear luminance map. 3.The three-dimensional scene simulation system based on VR technology of claim 1, wherein, Converting the surface point to the camera coordinate system through the camera external parameter, then converting it to the image coordinate through the camera internal parameter matrix, and finally obtaining the pixel coordinates of the surface point on each source image through the projection function, and obtaining the linear luminance value of the pixel coordinates on the linear luminance map corresponding to the source image as the linear luminance sampling value; calculating the absolute value of the difference between the linear luminance sampling value of the candidate source image and the linear luminance sampling value of the remaining source images, and taking the maximum value as the visible luminance step amplitude.

4. The three-dimensional scene simulation system based on VR technology according to claim 1, characterized in that, Dividing the visible luminance step amplitude of each surface point by the corresponding minimum angle of view change to obtain the surface point's jump risk field point value, then dividing the product of the surface point's depth and the preset human eye perceived luminance difference by the jump risk field point value to obtain the surface point's safety translation budget, and finally selecting the minimum value of the safety translation budget among all surface points in the user activity area as the global safety translation radius. 5.The three-dimensional scene simulation system based on VR technology of claim 1, wherein, Obtaining the position vectors of the user in two adjacent frames, calculating the Euclidean distance between the two position vectors as the user's actual translation distance, and restricting the user's actual translation distance from exceeding the global safety translation radius. 6.The three-dimensional scene simulation system based on VR technology of claim 1, wherein, Obtaining the lens pre-distortion function and parameter set of the left and right eyes, the parameter set including the first radial distortion coefficient and the second radial distortion coefficient of the eye lens, obtaining the non-distortion normalized image plane coordinates of each eye, which consist of x-direction coordinate values and y-direction coordinate values, and calculating the radial distance of the non-distortion normalized image plane coordinates; obtaining an intermediate calculation result by adding 1 to the product of the first radial distortion coefficient and the square of the radial distance, and adding the product of the second radial distortion coefficient and the fourth power of the radial distance, then multiplying the intermediate calculation result by the x-direction coordinate values and y-direction coordinate values of the non-distortion normalized image plane coordinates respectively to obtain the x-direction coordinate values and y-direction coordinate values of the display panel coordinates. 7.The three-dimensional scene simulation system based on VR technology of claim 6, wherein, Performing ray projection operation on the non-distortion normalized image plane coordinates of the left and right eyes to obtain the surface points of the reconstructed model hit by the ray projection operation, and writing the continuous appearance color of the surface points into the corresponding display panel coordinates to generate the final image of each eye, outputting the final image of the left eye to the left eye display panel of the head-mounted display, and outputting the final image of the right eye to the right eye display panel of the head-mounted display.

8. A three-dimensional scene simulation method based on VR technology, characterized in that, Performing a three-dimensional scene simulation system based on VR technology as claimed in any one of claims 1 to 7, comprising the following steps: Step S401, camera response correction, exposure time correction and vignetting correction are performed on the source images to generate linear intensity maps; Step S402, the angle between the view direction of each source image and the reference line-of-sight direction is determined according to the camera extrinsic parameters, and the candidate source image and the nearest competing source image are determined according to the angle, the minimum view angle change is obtained by calculating the view angle difference between the two; Step S403, each surface point of the reconstruction model is mapped to the linear intensity map by the projection function and the camera intrinsic and extrinsic parameters, and the visible intensity step amplitude is obtained by calculating the intensity difference between the candidate source image and the nearest competing source image; Step S404, the safety translation budget is obtained by calculating the minimum view angle change and the visible intensity step amplitude, and the minimum value of the safety translation budget in the user activity area is taken as the global safety translation radius; Step S405, the angular domain normalization variable is obtained by calculating the view angle difference between the candidate source image and the nearest competing source image combined with the current line-of-sight direction, the non-negative weight is determined by the smoothing function, and the linear intensity of the candidate source image and the nearest competing source image is weighted to obtain the continuous appearance color, and the translation amplitude of the user is constrained according to the global safety translation radius; Step S406, the continuous appearance color is processed by the lens pre-distortion function and output to the left eye display panel and the right eye display panel of the head-mounted display.

Images