A three-dimensional virtual simulation method based on real-time cloud rendering

Abstract

The present invention relates to the technical field of three-dimensional virtual simulation, and discloses a three-dimensional virtual simulation method of real-time cloud rendering, the method comprising: collecting a target scene image and dividing the scene; constructing a balanced rendering objective function and optimizing and solving it to obtain an optimal rendering strategy, distributing the scene blocks to multiple cloud servers; the cloud server performs three-dimensional rendering and compression transmission on the scene blocks; the terminal device performs splicing and reconstruction to obtain a three-dimensional rendering result of the target scene. The present invention calculates the amount of characteristic information contained in the neighborhood pixel area of ​​a pixel, characterizes the rendering difficulty of the neighborhood pixel area, divides the target scene image into multiple scene blocks with similar rendering difficulty and rendering time, improves the rendering efficiency of the target scene image, constructs a balanced rendering objective function that characterizes the overall rendering load of the cloud server after adopting different rendering strategies, and constitutes a relatively balanced rendering strategy to perform three-dimensional rendering of the target scene.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional virtual simulation, and in particular to a three-dimensional virtual simulation method of real-time cloud rendering. Background Art

[0002] With the rapid development of computer hardware technology, network technology and virtual reality (VR) technology, 3D virtual simulation has gradually become an important tool in many industries, such as architecture, medical, military, education, entertainment, etc. 3D virtual simulation is a realistic simulation of the real world or fictional environment through computer technology, allowing users to interact through the virtual environment. Traditional 3D simulation technology usually relies on local hardware resources, such as the computing power of the graphics processing unit (GPU) and the central processing unit (CPU). However, with the complexity and refinement of the simulation environment, the demand for computing resources has increased dramatically, and ordinary terminal devices are difficult to meet the real-time rendering requirements of high-quality 3D simulation. In this context, cloud computing technology has gradually become the key to solving this problem. Cloud computing can provide efficient computing power through distributed computing, resource sharing and other features, so that users do not need high-performance local devices, but can only rely on the powerful computing power of the cloud to achieve real-time rendering of complex 3D simulations. This mode is real-time cloud rendering, which can not only greatly reduce the hardware requirements of users, but also stream the rendered images to the user terminal through network transmission, realizing a cross-platform and cross-device 3D simulation experience. High-quality three-dimensional scenes and complex physical effects will significantly increase the computational burden of rendering, thus affecting real-time performance; and in order to ensure real-time performance, reducing rendering quality will lead to a decline in user experience. How to ensure the real-time performance of rendering while ensuring high-quality visual effects is an important issue that needs to be urgently addressed in current cloud rendering technology. To address this issue, the present invention proposes a three-dimensional virtual simulation method for real-time cloud rendering, which improves rendering efficiency and simulation quality by optimizing and balancing cloud rendering and real-time performance. Summary of the invention

[0003] In view of this, the present invention provides a three-dimensional virtual simulation method of real-time cloud rendering. According to the multi-feature information of pixels and the feature change information of the neighborhood pixel area, the pixels containing the most information are screened out as the central pixels of the scene blocks, and pixel diffusion is performed to realize block processing of the image, and then the scene blocks are rendered and compressed and transmitted separately, thereby improving the three-dimensional virtual simulation efficiency of the target scene image.

[0004] To achieve the above object, the present invention provides a real-time cloud rendering three-dimensional virtual simulation method, comprising the following steps:

[0005] S1: Collect the target scene image and perform scene segmentation, dividing the target scene image into multiple scene blocks, wherein the scene segmentation adopts the scene complexity analysis method of multi-feature fusion;

[0006] S2: Obtain the scene complexity value corresponding to the scene block and the computing efficiency information of the cloud server, then construct and optimize the balanced rendering objective function of the associated scene block and the cloud server, obtain the optimal rendering strategy of the scene block, and distribute the scene block and the corresponding depth matrix to the corresponding cloud server according to the optimal rendering strategy;

[0007] S3: The cloud server performs 3D rendering on the received scene blocks in combination with the depth matrix to obtain 3D reconstruction data of the scene blocks, and transmits the rendered 3D reconstruction data to the terminal device in a compressed transmission manner, wherein the compressed transmission adopts a multi-scale fusion deep autoencoder method;

[0008] S4: The terminal device decompresses the received 3D reconstruction data compression result, and performs splicing and reconstruction to obtain a 3D rendering result of the target scene.

[0009] As a further improvement method of the present invention:

[0010] Optionally, collecting the target scene image in step S1 includes:

[0011] Collect the target scene image to be simulated in 3D virtual reality , where the target scene image The representation is:

[0012] ;

[0013] ;

[0014] in:

[0015] Represents a pixel matrix of X rows and Y columns, where X represents the target scene image The number of pixel rows, Y represents the target scene image The number of pixel columns, Represents the target scene image The pixel at row x and column y The pixel value of ;

[0016] In order, it represents pixels Color values ​​in R, G, B color channels;

[0017] Calculate the target scene image The color feature information and texture feature information of the pixels in The color feature information is :

[0018] ;

[0019] ;

[0020]

[0021] in:

[0022] Indicates the remainder symbol;

[0023] Indicates selecting the maximum value;

[0024] Indicates selecting the minimum value;

[0025] Pixel The texture feature information is :

[0026] ;

[0027] ;

[0028] ;

[0029] in:

[0030] Represents pixels Color change information in the horizontal direction;

[0031] Represents pixels Color change information in the vertical direction;

[0032] Calculate the target scene image The pixel centrality of the pixels in The pixel centrality is :

[0033] ;

[0034] in:

[0035] In pixels centered In the neighborhood pixel area, the pixel No. The color feature information of the neighboring pixels, Represents pixels No. Texture feature information of neighborhood pixels;

[0036] According to the pixel centrality, S pixels are selected as the central pixels of S scene blocks, and the target scene image Divided into S scene blocks.

[0037] In an embodiment of the present invention, Represents pixels The amount of feature information contained in the neighborhood pixel area, wherein the greater the amount of feature information, the greater the amount of calculation required for three-dimensional rendering;

[0038] Optionally, according to the pixel centrality of the pixels, S pixels are selected as the central pixels of the S scene blocks, and the target scene image Divided into S scene blocks, including:

[0039] S11: According to the pixel centrality of the pixel, the central density value of the pixel as the central pixel is calculated; wherein the target scene image Medium Pixels The central density value of the central pixel is :

[0040] ;

[0041] in:

[0042] Indicates distance in pixels The pixel set consisting of the nearest N pixels, , n represents the pixel set Any pixel in ;

[0043] represents the pixel centrality of pixel n, Represents pixels The Euclidean distance between

[0044] S12: Perform distance correction on the center density value of the pixel to obtain a corrected center density value; wherein the pixel The corrected central density value is :

[0045] ;

[0046] in:

[0047] Represents the target scene image Medium and Pixel The maximum pixel distance between them; the pixel distance is the Euclidean distance between pixels;

[0048] Indicates that the central density value is higher than Any pixel of The minimum pixel distance between

[0049] express The target scene image The maximum center density value of all pixels in;

[0050] S13: Select the S pixels with the largest center density value after correction as the center pixel, and diffuse outward with the center pixel as the center to obtain S non-overlapping diffusion image rectangles. During the diffusion process, the scene complexity value of the current diffusion image rectangle is calculated in real time, and the direction that can increase the scene complexity value is selected for diffusion until the target scene image is All pixels in are contained in the diffusion image rectangle, and the S diffusion image rectangles at this time are used as S scene blocks, and the scene complexity value of each scene block is calculated; the scene complexity value of the sth scene block is:

[0051] ;

[0052] in:

[0053] Indicates the scene complexity value of the s-th scene block;

[0054] represents the pixel set of the sth scene block, , j represents the pixel set Any pixel in ;

[0055] represents the color feature information of pixel j, Represents the texture feature information of pixel j.

[0056] Optionally, in step S2, the scene complexity value corresponding to the scene block and the computing efficiency information of the cloud server are obtained, and then a balanced rendering objective function associating the scene block with the cloud server is constructed, including:

[0057] Get the scene complexity value corresponding to the scene block and the computing efficiency information of the cloud server, where the scene complexity value of the sth scene block is , the computing efficiency information of the kth cloud server is , , construct the balanced rendering objective function of the associated scene blocks and cloud servers:

[0058] ;

[0059] ;

[0060] ;

[0061] in:

[0062] represents the balanced rendering objective function;

[0063] represents the rendering strategy of S scene blocks, Indicates the rendering method of the s-th scene block, , This means that the sth scene is divided into blocks and distributed to the kth cloud server. Indicates the rendering strategy Balanced rendering objective function value;

[0064] Indicates the number of bits of the s-th scene block, Indicates the amount of data currently stored in the kth cloud server, in bits;

[0065] represents the floating point computing capability of the kth cloud server;

[0066] Indicates the physical distance between the kth cloud server and the current scene block storage location;

[0067] Indicates data transmission speed;

[0068] represents the transmission time control weight; in the embodiment of the present invention, Set to 0.2;

[0069] Indicates that according to the rendering strategy To distribute the scene blocks, whether the k-th cloud server needs to receive the representation coefficient of the s-th scene block, Indicates that the kth cloud server needs to receive the sth scene block, It means that the k-th cloud server does not need to receive the s-th scene block;

[0070] Indicates that according to the rendering strategy Distribute the scene in blocks and the rendering load of the kth cloud server;

[0071] The balanced rendering objective function is optimized and solved to obtain the optimal rendering strategy for scene blocks, and the scene blocks are distributed to the corresponding cloud servers according to the optimal rendering strategy.

[0072] Optionally, optimizing and solving the balanced rendering objective function to obtain an optimal rendering strategy for scene blocks includes:

[0073] The balanced rendering objective function is optimized and solved to obtain the optimal rendering strategy for scene blocks. The optimization solution process of the balanced rendering objective function is as follows:

[0074] S21: Initialize and generate U groups of rendering strategies, where the u-th group of rendering strategies generated by initialization is , for any set of rendering strategies generated by initialization ,satisfy , that is, a scene block will only be distributed to one cloud server, ; Set the iteration threshold coefficient ;

[0075] S22: Set the current iteration number of the rendering strategy to t, then the tth iteration result of the uth group rendering strategy is , the initial value of t is 0 and the maximum value is Max;

[0076] S23: Using the U group rendering strategies obtained in the t-th iteration as the input of the balanced rendering objective function, and selecting the rendering strategy with the smallest value of the balanced rendering objective function as the optimal rendering strategy obtained in the t-th iteration ;

[0077] S24: Combine the optimal rendering strategy and iterate the rendering strategy, wherein the iteration formula of the u-th group of rendering strategies is:

[0078] ;in:

[0079] Represents a random number between 0 and 1;

[0080] Combined with the threshold coefficient, the rendering strategy of group u is calculated to accept the t+1th iteration result. Probability :

[0081] ;

[0082] If the uth group rendering strategy accepts the t+1th iteration result , then go to step S25, otherwise , go to step S25;

[0083] S25: Let t=t+1, go to step S23, until the maximum number of iterations is reached, and select the rendering strategy that has the smallest value of the balanced rendering objective function and satisfies that a scene block will only be distributed to one cloud server as the optimal rendering strategy;

[0084] According to the optimal rendering strategy, the scene blocks and the depth matrices corresponding to the scene blocks are distributed to the corresponding cloud servers, wherein the depth matrices corresponding to the scene blocks contain the depth values ​​of each pixel in the scene blocks.

[0085] Optionally, in step S3, the cloud server performs three-dimensional rendering on the received scene blocks in combination with the depth matrix to obtain three-dimensional reconstruction data of the scene blocks, including:

[0086] The cloud server performs 3D rendering on the received scene blocks in combination with the depth matrix to obtain 3D reconstruction data of the scene blocks. The 3D rendering process of the s-th scene block is as follows:

[0087] Get the pixel matrix and depth matrix of the sth scene block, where the pixel matrix is :

[0088] ;

[0089] in:

[0090] Represents a pixel matrix The pixel value of the pixel in the pth row and qth column, , Represents a pixel matrix the number of rows, Represents a pixel matrix The number of columns;

[0091] The depth matrix is :

[0092] ;

[0093] in:

[0094] Represents a pixel matrix The depth value of the pixel at the pth row and the qth column in;

[0095] The pixel coordinates of the pixel in the sth scene block Convert to 3D coordinates :

[0096] ;

[0097] ;

[0098] ;

[0099] in:

[0100] represents the extrinsic rotation matrix of the camera that captures the target scene image, Represents the camera's extrinsic translation matrix;

[0101] represents the coordinate value of p in the camera coordinate system, Represents the coordinate value of q in the camera coordinate system;

[0102] Indicates the horizontal focal length of the camera. Indicates the vertical focal length of the camera;

[0103] Represents the principal point coordinates of the camera;

[0104] Combining the 3D coordinates and pixel values ​​of the pixels, the 3D reconstruction data of the s-th scene block is constructed:

[0105] ;

[0106] ;

[0107] in:

[0108] represents the 3D reconstruction data of the s-th scene block, Represents 3D reconstruction data Point cloud data in Represents the coordinates of point cloud data, Represents the color value of the point cloud data in the RGB color channel;

[0109] The rendered 3D reconstruction data is transmitted to the terminal device using a compressed transmission method.

[0110] Optionally, the method of transmitting the rendered three-dimensional reconstruction data to the terminal device in a compressed transmission manner includes:

[0111] S31: The cloud server performs multi-scale convolution on the three-dimensional reconstruction data to obtain multi-scale convolution features of the three-dimensional reconstruction data; wherein the three-dimensional reconstruction data The multi-scale convolution feature is :

[0112] ;

[0113] ;

[0114] in:

[0115] Represents 3D reconstruction data In the convolution feature of scale m, , M represents the maximum scale;

[0116] Indicates converting three-dimensional reconstruction data into matrix form;

[0117] S32: compress the multi-scale convolution features into feature information; the multi-scale convolution features The corresponding feature information is :

[0118] ;

[0119] S33: Transmit the compressed feature information to the terminal device.

[0120] Optionally, in step S4, the terminal device decompresses the received 3D reconstruction data compression result, and performs splicing and reconstruction to obtain a 3D rendering result of the target scene, including:

[0121] The terminal device decompresses and splices and reconstructs the received 3D reconstruction data compression results, wherein the decompression method is the inverse processing operation of the compression processing in the compression transmission method. The splicing and reconstruction process of S groups of 3D reconstruction data is as follows:

[0122] S41: arranging the S groups of three-dimensional reconstruction data according to the positions of the corresponding scene blocks in the target scene image;

[0123] S42: Calculate the conversion angle of each three-dimensional reconstruction data in order from top to bottom and from left to right; wherein the conversion angle calculation process of the sth group of three-dimensional reconstruction data is:

[0124] S421: Selecting the sth group of 3D reconstruction data 3D reconstruction data of the previous position , that is, sorting the 3D reconstruction data in the target scene image Previous nearest neighbor 3D reconstruction data;

[0125] S422: Calculate and obtain the sth group of three-dimensional reconstruction data And 3D reconstruction data The centroid coordinates of the sth group of 3D reconstruction data The centroid coordinates are :

[0126] ;

[0127] S423: Centralize the coordinates in the three-dimensional reconstruction data based on the centroid coordinates, where the coordinates The result of the centralization process is :

[0128] ;

[0129] S424: Extract the first H coordinates after centralization, where the 3D reconstruction data And 3D reconstruction data The h-th central processing coordinates are as well as ;

[0130] S425: Construct the covariance matrix , where T represents transpose;

[0131] S426: Perform singular value decomposition on the covariance matrix A to obtain the left singular vector and the right singular vectors , construct the sth group of three-dimensional reconstruction data Conversion angle :

[0132] ;

[0133] S427: Using conversion angle 3D reconstruction data The coordinates of the point cloud data are transformed, where the coordinates The conversion formula is ;

[0134] S43: converting the three-dimensional reconstruction data according to the conversion angle until all the three-dimensional reconstruction data are converted to form a three-dimensional rendering result of the target scene.

[0135] In order to solve the above problem, the present invention provides an electronic device, the electronic device comprising:

[0136] A memory storing at least one instruction;

[0137] Communication interface, enabling electronic equipment to communicate; and

[0138] The processor executes the instructions stored in the memory to implement the above-mentioned three-dimensional virtual simulation method of real-time cloud rendering.

[0139] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the above-mentioned real-time cloud rendering three-dimensional virtual simulation method.

[0140] Compared with the prior art, the present invention proposes a real-time cloud rendering 3D virtual simulation method, which has the following advantages:

[0141] First, this scheme proposes an image block processing method, which calculates the feature information contained in the neighborhood pixel area of ​​the pixel according to the color feature information and texture feature information of the pixel. The larger the feature information, the greater the amount of calculation required for three-dimensional rendering. The feature information is used as the pixel centrality. According to the distance between the pixel centralities of different pixels, multiple pixels with a certain distance are selected as central pixels. The central pixels are used for diffusion processing, and the target scene image is divided into multiple scene blocks. The rendering difficulty of each scene block is similar, and the cloud server can complete the rendering in a similar time, thereby improving the rendering efficiency of the target scene image.

[0142] At the same time, this scheme proposes a cloud server rendering strategy generation method. According to the amount of data currently stored in the cloud server, the cloud server computing efficiency information, the size of the scene block, and the scene complexity value, a balanced rendering objective function is constructed to characterize the overall rendering load of the cloud server after adopting different rendering strategies. The balanced rendering objective function is solved to obtain a relatively balanced scene block distribution rendering strategy. The cloud server performs three-dimensional modeling on the scene blocks, and uses multi-scale convolution processing to perform compression and dimensionality reduction processing to improve the transmission efficiency of the three-dimensional reconstruction data. According to the coordinate transformation relationship between different three-dimensional reconstruction data, splicing and reconstruction are performed to achieve the three-dimensional rendering of the target scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0143] Figure 1 A flowchart of a real-time cloud-rendered three-dimensional virtual simulation method provided by an embodiment of the present invention.

[0144] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0145] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0146] The embodiment of the present application provides a three-dimensional virtual simulation method of real-time cloud rendering. The execution subject of the three-dimensional virtual simulation method of real-time cloud rendering includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the three-dimensional virtual simulation method of real-time cloud rendering can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0147] Embodiment 1:

[0148] A real-time cloud rendering three-dimensional virtual simulation method comprises the following steps:

[0149] S1: Capture a target scene image and perform scene segmentation, dividing the target scene image into multiple scene blocks.

[0150] The step S1 includes collecting the target scene image, including:

[0151] Collect the target scene image to be simulated in 3D virtual reality , where the target scene image The representation is:

[0152] ;

[0153] ;

[0154] in:

[0155] Represents a pixel matrix of X rows and Y columns, where X represents the target scene image The number of pixel rows, Y represents the target scene image The number of pixel columns, Represents the target scene image The pixel at row x and column y The pixel value of ;

[0156] In order, it represents pixels Color values ​​in R, G, B color channels;

[0157] Calculate the target scene image The color feature information and texture feature information of the pixels in The color feature information is :

[0158] ;

[0159] ;

[0160]

[0161] in:

[0162] Indicates the remainder symbol;

[0163] Indicates selecting the maximum value;

[0164] Indicates selecting the minimum value;

[0165] Pixel The texture feature information is :

[0166] ;

[0167] ;

[0168] ;

[0169] in:

[0170] Represents pixels Color change information in the horizontal direction;

[0171] Represents pixels Color change information in the vertical direction;

[0172] Calculate the target scene image The pixel centrality of the pixels in The pixel centrality is :

[0173] ;

[0174] in:

[0175] In pixels centered In the neighborhood pixel area, the pixel No. The color feature information of the neighboring pixels, Represents pixels No. Texture feature information of neighborhood pixels;

[0176] According to the pixel centrality, S pixels are selected as the central pixels of S scene blocks, and the target scene image Divided into S scene blocks.

[0177] According to the pixel centrality of the pixels, S pixels are selected as the central pixels of S scene blocks, and the target scene image Divided into S scene blocks, including:

[0178] S11: According to the pixel centrality of the pixel, the central density value of the pixel as the central pixel is calculated; wherein the target scene image Medium Pixels The central density value of the central pixel is :

[0179] ;

[0180] in:

[0181] Indicates distance in pixels The pixel set consisting of the nearest N pixels, , n represents the pixel set Any pixel in ;

[0182] represents the pixel centrality of pixel n, Represents pixels The Euclidean distance between

[0183] S12: Perform distance correction on the center density value of the pixel to obtain a corrected center density value; wherein the pixel The corrected central density value is :

[0184] ;

[0185] in:

[0186] Represents the target scene image Medium and Pixel The maximum pixel distance between them; the pixel distance is the Euclidean distance between pixels;

[0187] Indicates that the central density value is higher than Any pixel of The minimum pixel distance between

[0188] express The target scene image The maximum center density value of all pixels in;

[0189] S13: Select the S pixels with the largest center density value after correction as the center pixel, and diffuse outward with the center pixel as the center to obtain S non-overlapping diffusion image rectangles. During the diffusion process, the scene complexity value of the current diffusion image rectangle is calculated in real time, and the direction that can increase the scene complexity value is selected for diffusion until the target scene image is All pixels in are contained in the diffusion image rectangle, and the S diffusion image rectangles at this time are used as S scene blocks, and the scene complexity value of each scene block is calculated; the scene complexity value of the sth scene block is:

[0190] ;

[0191] in:

[0192] Indicates the scene complexity value of the s-th scene block;

[0193] represents the pixel set of the sth scene block, , j represents the pixel set Any pixel in ;

[0194] represents the color feature information of pixel j, Represents the texture feature information of pixel j.

[0195] S2: Obtain the scene complexity value corresponding to the scene block and the computing efficiency information of the cloud server, and then construct and optimize the balanced rendering objective function of the associated scene block and the cloud server to obtain the optimal rendering strategy of the scene block. According to the optimal rendering strategy, the scene block and the corresponding depth matrix are distributed to the corresponding cloud server.

[0196] In the step S2, the scene complexity value corresponding to the scene block and the computing efficiency information of the cloud server are obtained, and then a balanced rendering objective function of the associated scene block and the cloud server is constructed, including:

[0197] Get the scene complexity value corresponding to the scene block and the computing efficiency information of the cloud server, where the scene complexity value of the sth scene block is , the computing efficiency information of the kth cloud server is , , construct the balanced rendering objective function of the associated scene blocks and cloud servers:

[0198] ;

[0199] ;

[0200] ;

[0201] in:

[0202] represents the balanced rendering objective function;

[0203] represents the rendering strategy of S scene blocks, Indicates the rendering method of the s-th scene block, , This means that the sth scene is divided into blocks and distributed to the kth cloud server. Indicates the rendering strategy Balanced rendering objective function value;

[0204] Indicates the number of bits of the s-th scene block, Indicates the amount of data currently stored in the kth cloud server, in bits;

[0205] represents the floating point computing capability of the kth cloud server;

[0206] Indicates the physical distance between the kth cloud server and the current scene block storage location;

[0207] Indicates data transmission speed;

[0208] represents the transmission time control weight; in the embodiment of the present invention, Set to 0.2;

[0209] Indicates that according to the rendering strategy To distribute the scene blocks, whether the k-th cloud server needs to receive the representation coefficient of the s-th scene block, Indicates that the kth cloud server needs to receive the sth scene block, It means that the k-th cloud server does not need to receive the s-th scene block;

[0210] Indicates that according to the rendering strategy Distribute the scene in blocks and the rendering load of the kth cloud server;

[0211] The balanced rendering objective function is optimized and solved to obtain the optimal rendering strategy for scene blocks, and the scene blocks are distributed to the corresponding cloud servers according to the optimal rendering strategy.

[0212] The method of optimizing and solving the balanced rendering objective function to obtain the optimal rendering strategy for the scene blocks includes:

[0213] The balanced rendering objective function is optimized and solved to obtain the optimal rendering strategy for scene blocks. The optimization solution process of the balanced rendering objective function is as follows:

[0214] S21: Initialize and generate U groups of rendering strategies, where the u-th group of rendering strategies generated by initialization is , for any set of rendering strategies generated by initialization ,satisfy , that is, a scene block will only be distributed to one cloud server, ; Set the iteration threshold coefficient ;

[0215] S22: Set the current iteration number of the rendering strategy to t, then the tth iteration result of the uth group rendering strategy is , the initial value of t is 0 and the maximum value is Max;

[0216] S23: Using the U group rendering strategies obtained in the t-th iteration as the input of the balanced rendering objective function, and selecting the rendering strategy with the smallest value of the balanced rendering objective function as the optimal rendering strategy obtained in the t-th iteration ;

[0217] S24: Combine the optimal rendering strategy and iterate the rendering strategy, wherein the iteration formula of the u-th group of rendering strategies is:

[0218] ;in:

[0219] Represents a random number between 0 and 1;

[0220] Combined with the threshold coefficient, the rendering strategy of group u is calculated to accept the t+1th iteration result. Probability :

[0221] ;

[0222] If the uth group rendering strategy accepts the t+1th iteration result , then go to step S25, otherwise , go to step S25;

[0223] S25: Let t=t+1, go to step S23, until the maximum number of iterations is reached, and select the rendering strategy that has the smallest value of the balanced rendering objective function and satisfies that a scene block will only be distributed to one cloud server as the optimal rendering strategy;

[0224] According to the optimal rendering strategy, the scene blocks and the depth matrices corresponding to the scene blocks are distributed to the corresponding cloud servers, wherein the depth matrices corresponding to the scene blocks contain the depth values ​​of each pixel in the scene blocks.

[0225] S3: The cloud server performs three-dimensional rendering on the received scene blocks in combination with the depth matrix to obtain three-dimensional reconstruction data of the scene blocks, and transmits the rendered three-dimensional reconstruction data to the terminal device using a compressed transmission method.

[0226] In the step S3, the cloud server performs three-dimensional rendering on the received scene blocks in combination with the depth matrix to obtain three-dimensional reconstruction data of the scene blocks, including:

[0227] The cloud server performs 3D rendering on the received scene blocks in combination with the depth matrix to obtain 3D reconstruction data of the scene blocks. The 3D rendering process of the s-th scene block is as follows:

[0228] Get the pixel matrix and depth matrix of the sth scene block, where the pixel matrix is :

[0229] ;

[0230] in:

[0231] Represents a pixel matrix The pixel value of the pixel in the pth row and qth column, , Represents a pixel matrix the number of rows, Represents a pixel matrix The number of columns;

[0232] The depth matrix is :

[0233] ;

[0234] in:

[0235] Represents a pixel matrix The depth value of the pixel at the pth row and the qth column in;

[0236] The pixel coordinates of the pixel in the sth scene block Convert to 3D coordinates :

[0237] ;

[0238] ;

[0239] ;

[0240] in:

[0241] represents the extrinsic rotation matrix of the camera that captures the target scene image, Represents the camera's extrinsic translation matrix;

[0242] represents the coordinate value of p in the camera coordinate system, Represents the coordinate value of q in the camera coordinate system;

[0243] Indicates the horizontal focal length of the camera. Indicates the vertical focal length of the camera;

[0244] Represents the principal point coordinates of the camera;

[0245] Combining the 3D coordinates and pixel values ​​of the pixels, the 3D reconstruction data of the s-th scene block is constructed:

[0246] ;

[0247] ;

[0248] in:

[0249] represents the 3D reconstruction data of the s-th scene block, Represents 3D reconstruction data Point cloud data in Represents the coordinates of point cloud data, Represents the color value of the point cloud data in the RGB color channel;

[0250] The rendered 3D reconstruction data is transmitted to the terminal device using a compressed transmission method.

[0251] The method of transmitting the rendered 3D reconstruction data to the terminal device in a compressed transmission manner includes:

[0252] S31: The cloud server performs multi-scale convolution on the three-dimensional reconstruction data to obtain multi-scale convolution features of the three-dimensional reconstruction data; wherein the three-dimensional reconstruction data The multi-scale convolution feature is :

[0253] ;

[0254] ;

[0255] in:

[0256] Represents 3D reconstruction data In the convolution feature of scale m, , M represents the maximum scale;

[0257] Indicates converting three-dimensional reconstruction data into matrix form;

[0258] S32: compress the multi-scale convolution features into feature information; the multi-scale convolution features The corresponding feature information is :

[0259] ;

[0260] S33: Transmit the compressed feature information to the terminal device.

[0261] S4: The terminal device decompresses the received 3D reconstruction data compression result, and performs splicing and reconstruction to obtain a 3D rendering result of the target scene.

[0262] In step S4, the terminal device decompresses the received 3D reconstruction data compression result, and performs splicing and reconstruction to obtain a 3D rendering result of the target scene, including:

[0263] The terminal device decompresses and splices and reconstructs the received 3D reconstruction data compression results, wherein the decompression method is the inverse processing operation of the compression processing in the compression transmission method. The splicing and reconstruction process of S groups of 3D reconstruction data is as follows:

[0264] S41: arranging the S groups of three-dimensional reconstruction data according to the positions of the corresponding scene blocks in the target scene image;

[0265] S42: Calculate the conversion angle of each three-dimensional reconstruction data in order from top to bottom and from left to right; wherein the conversion angle calculation process of the sth group of three-dimensional reconstruction data is:

[0266] S421: Selecting the sth group of 3D reconstruction data 3D reconstruction data of the previous position , that is, sorting the 3D reconstruction data in the target scene image Previous nearest neighbor 3D reconstruction data;

[0267] S422: Calculate and obtain the sth group of three-dimensional reconstruction data And 3D reconstruction data The centroid coordinates of the sth group of 3D reconstruction data The centroid coordinates are :

[0268] ;

[0269] S423: Centralize the coordinates in the three-dimensional reconstruction data based on the centroid coordinates, where the coordinates The result of the centralization process is :

[0270] ;

[0271] S424: Extract the first H coordinates after centralization, where the 3D reconstruction data And 3D reconstruction data The h-th central processing coordinates are as well as ;

[0272] S425: Construct the covariance matrix , where T represents transpose;

[0273] S426: Perform singular value decomposition on the covariance matrix A to obtain the left singular vector and the right singular vectors , construct the sth group of three-dimensional reconstruction data Conversion angle :

[0274] ;

[0275] S427: Using conversion angle 3D reconstruction data The coordinates of the point cloud data are transformed, where the coordinates The conversion formula is ;

[0276] S43: converting the three-dimensional reconstruction data according to the conversion angle until all the three-dimensional reconstruction data are converted to form a three-dimensional rendering result of the target scene.

[0277] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0278] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0279] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0280] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A real-time cloud rendering three-dimensional virtual simulation method, characterized in that: The method comprises: S1: Collect the target scene image and perform scene division, dividing the target scene image into multiple scene blocks; S2: Obtain the scene complexity value corresponding to the scene block and the computing efficiency information of the cloud server, then construct and optimize the balanced rendering objective function of the associated scene block and the cloud server, obtain the optimal rendering strategy of the scene block, and distribute the scene block and the corresponding depth matrix to the corresponding cloud server according to the optimal rendering strategy; S3: The cloud server performs three-dimensional rendering on the received scene blocks in combination with the depth matrix to obtain three-dimensional reconstruction data of the scene blocks, and transmits the rendered three-dimensional reconstruction data to the terminal device in a compressed transmission manner; S4: The terminal device decompresses the received 3D reconstruction data compression result, and performs splicing and reconstruction to obtain a 3D rendering result of the target scene; In the step S2, the scene complexity value corresponding to the scene block and the computing efficiency information of the cloud server are obtained, and then a balanced rendering objective function of the associated scene block and the cloud server is constructed, including: Get the scene complexity value corresponding to the scene block and the computing efficiency information of the cloud server, where the scene complexity value of the sth scene block is , the computing efficiency information of the kth cloud server is , , construct the balanced rendering objective function of the associated scene blocks and cloud servers: ； ； ； in: represents the balanced rendering objective function; represents the rendering strategy of S scene blocks, Indicates the rendering method of the s-th scene block, , This means that the sth scene is divided into blocks and distributed to the kth cloud server. Indicates the rendering strategy Balanced rendering objective function value; Indicates the number of bits of the s-th scene block, Indicates the amount of data currently stored in the kth cloud server, in bits; represents the floating point computing capability of the kth cloud server; Indicates the physical distance between the kth cloud server and the current scene block storage location; Indicates data transmission speed; represents the transmission time regulation weight; Indicates that according to the rendering strategy To distribute the scene blocks, whether the k-th cloud server needs to receive the representation coefficient of the s-th scene block, Indicates that the k-th cloud server needs to receive the s-th scene block, It means that the k-th cloud server does not need to receive the s-th scene block; The balanced rendering objective function is optimized and solved to obtain the optimal rendering strategy for scene blocks, and the scene blocks are distributed to the corresponding cloud servers according to the optimal rendering strategy.

2. A real-time cloud-rendered three-dimensional virtual simulation method according to claim 1, characterized in that: The step S1 includes collecting the target scene image, including: Collect the target scene image to be simulated in 3D virtual reality , where the target scene image The representation is: ； ； in: Represents a pixel matrix of X rows and Y columns, where X represents the target scene image The number of pixel rows, Y represents the target scene image The number of pixel columns, Represents the target scene image The pixel at row x and column y The pixel value of ; In order, it represents pixels Color values ​​in R, G, B color channels; Calculate the target scene image The color feature information and texture feature information of the pixels in The color feature information is ; Pixel The texture feature information is ; Calculate the target scene image The pixel centrality of the pixels in The pixel centrality is ; According to the pixel centrality, S pixels are selected as the central pixels of S scene blocks, and the target scene image Divided into S scene blocks.

3. A real-time cloud-rendered three-dimensional virtual simulation method according to claim 2, characterized in that: According to the pixel centrality of the pixels, S pixels are selected as the central pixels of S scene blocks, and the target scene image Divided into S scene blocks, including: S11: Calculate the center density value of the pixel as the center pixel according to the pixel centrality of the pixel; S12: performing distance correction on the center density value of the pixel to obtain a corrected center density value; S13: Select the S pixels with the largest center density value after correction as the center pixel, and diffuse outward with the center pixel as the center to obtain S non-overlapping diffusion image rectangles. During the diffusion process, the scene complexity value of the current diffusion image rectangle is calculated in real time, and the direction that can increase the scene complexity value is selected for diffusion until the target scene image is All pixels in are contained in the diffusion image rectangle, and the S diffusion image rectangles at this time are used as S scene blocks, and the scene complexity value of each scene block is calculated; the scene complexity value of the sth scene block is .

4. A three-dimensional virtual simulation method of real-time cloud rendering according to claim 1, characterized in that: The method of optimizing and solving the balanced rendering objective function to obtain the optimal rendering strategy for the scene blocks includes: The balanced rendering objective function is optimized and solved to obtain the optimal rendering strategy for scene blocks. The optimization solution process of the balanced rendering objective function is as follows: S21: Initialize and generate U groups of rendering strategies, where the u-th group of rendering strategies generated by initialization is , for any set of rendering strategies generated by initialization ,satisfy , that is, a scene block will only be distributed to one cloud server, ; Set the iteration threshold coefficient ; S22: Set the current iteration number of the rendering strategy to t, then the tth iteration result of the uth group rendering strategy is , the initial value of t is 0 and the maximum value is Max; S23: Using the U group rendering strategies obtained in the t-th iteration as the input of the balanced rendering objective function, and selecting the rendering strategy with the smallest value of the balanced rendering objective function as the optimal rendering strategy obtained in the t-th iteration ; S24: combining the optimal rendering strategy, iterating the rendering strategy; S25: Let t=t+1, go to step S23, until the maximum number of iterations is reached, and select the rendering strategy that has the smallest value of the balanced rendering objective function and satisfies that a scene block will only be distributed to one cloud server as the optimal rendering strategy; According to the optimal rendering strategy, the scene blocks and the depth matrices corresponding to the scene blocks are distributed to the corresponding cloud servers, wherein the depth matrices corresponding to the scene blocks contain the depth values ​​of each pixel in the scene blocks.

5. A real-time cloud-rendered three-dimensional virtual simulation method according to claim 4, characterized in that: In the step S3, the cloud server performs three-dimensional rendering on the received scene blocks in combination with the depth matrix to obtain three-dimensional reconstruction data of the scene blocks, including: The cloud server performs 3D rendering on the received scene blocks in combination with the depth matrix to obtain 3D reconstruction data of the scene blocks. The 3D rendering process of the s-th scene block is as follows: Get the pixel matrix and depth matrix of the sth scene block, where the pixel matrix is , the depth matrix is ; Combined with the depth matrix, the pixel coordinates of the pixels in the sth scene block are Convert to 3D coordinates ; Combining the 3D coordinates and pixel values ​​of the pixels, the 3D reconstruction data of the s-th scene block is constructed: ； ； in: represents the 3D reconstruction data of the s-th scene block, Represents 3D reconstruction data Point cloud data in Represents the coordinates of point cloud data, Represents the color value of the point cloud data in the RGB color channel. Represents a pixel matrix the number of rows, Represents a pixel matrix The number of columns; The rendered 3D reconstruction data is transmitted to the terminal device using a compressed transmission method.

6. A real-time cloud-rendered three-dimensional virtual simulation method according to claim 5, characterized in that: The method of transmitting the rendered 3D reconstruction data to the terminal device in a compressed transmission manner includes: S31: The cloud server performs multi-scale convolution on the three-dimensional reconstruction data to obtain multi-scale convolution features of the three-dimensional reconstruction data; wherein the three-dimensional reconstruction data The multi-scale convolution feature is ; S32: compress the multi-scale convolution features into feature information; the multi-scale convolution features The corresponding feature information is ; S33: Transmit the compressed feature information to the terminal device.

7. A real-time cloud-rendered 3D virtual simulation method according to claim 6, characterized in that: In step S4, the terminal device decompresses the received 3D reconstruction data compression result, and performs splicing and reconstruction to obtain a 3D rendering result of the target scene, including: The terminal device decompresses and splices and reconstructs the received 3D reconstruction data compression results, wherein the decompression method is the inverse processing operation of the compression processing in the compression transmission method. The splicing and reconstruction process of S groups of 3D reconstruction data is as follows: S41: arranging the S groups of three-dimensional reconstruction data according to the positions of the corresponding scene blocks in the target scene image; S42: Calculate the conversion angle of each 3D reconstruction data in order from top to bottom and from left to right; S43: converting the three-dimensional reconstruction data according to the conversion angle until all the three-dimensional reconstruction data are converted to form a three-dimensional rendering result of the target scene.

Images