A three-dimensional model distributed cloud rendering method, system and electronic equipment

Abstract

The application discloses a three-dimensional model distributed cloud rendering method, system and electronic equipment. The three-dimensional model rendering method applied to a client comprises the following steps: controlling a model splitting server to split a target model into a plurality of sub-models, and enabling each rendering server to render a sub-model; generating a rendering instruction in response to a preset browsing operation for the target model, and simultaneously sending the rendering instruction to each rendering server, so that each rendering server respectively renders the corresponding sub-model to generate a corresponding video stream and feeds back the video stream; performing a decoding operation on each video stream to generate a corresponding video image frame set; generating a target image frame according to a plurality of video image frames at the same time in all video image frame sets; and displaying all newly generated target image frames. The server shares the rendering, has the advantages of real-time smooth rendering capability, small delay, high rendering efficiency and the like, and improves the use experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer graphics and streaming media coding, in particular to a three-dimensional model distributed cloud rendering method, a three-dimensional model rendering display device, a three-dimensional model splitting system, a three-dimensional model rendering system, a storage medium and an electronic device. BACKGROUND

[0002] The core of a building information model (BIM) is to provide a complete and consistent building engineering information database by establishing a virtual building engineering three-dimensional model using digital technology. The information database contains not only geometric information, professional attributes and state information describing building components, but also state information of non-component objects (such as space and motion behavior). With the help of this three-dimensional model containing building engineering information, the information integration level of building engineering can be greatly improved, thereby providing a platform for information exchange and sharing for related parties of building engineering projects.

[0003] In current practical applications, a BIM three-dimensional data model usually needs to be displayed in a two-dimensional way on a computer desktop. The two-dimensional display of the BIM three-dimensional data model on the computer desktop is achieved by simulating the material, texture, light and perspective of the model, as well as rotating, moving and zooming the model, so that the real effect of the three-dimensional model can be felt. The above conversion process is the rendering process of the BIM three-dimensional data model.

[0004] BIM three-dimensional data model rendering is usually divided into two modes, one is client rendering and the other is real-time cloud rendering. Client rendering refers to that a user accesses a model on a server side through a browser, the browser downloads and caches the model data locally, and accesses a local graphics card for rendering through a WebGL driver. This rendering method requires high local computing resources (graphics card, CPU, memory). Cloud rendering refers to that rendering calculation occurs on a remote server side, and a client is only an interactive display that sends rendering instructions to the server side and receives rendering results (usually real-time streaming media). This rendering method has relatively low requirements for local computing resources.

[0005] In order to promote the application of BIM three-dimensional data models and reduce the demand for local computing resources, the way of rendering BIM three-dimensional data models in the cloud has been gradually adopted. However, a single server does not have the ability to perform real-time and smooth rendering on a super large-scale model (such as an airplane or an industrial park). SUMMARY

[0006] In view of the above problems, the application provides a three-dimensional model distributed cloud rendering method, a three-dimensional model rendering display device, a three-dimensional model splitting system, a three-dimensional model rendering system, a storage medium and an electronic device, which at least solve the defect that a single server does not have the ability to perform real-time and smooth rendering on a super-large model (such as an airplane or an industrial park).

[0007] In a first aspect, the application provides a three-dimensional model distributed cloud rendering method applied to a client, the method comprising:

[0008] controlling a model splitting server to split a target model into a plurality of sub-models and respectively sending each sub-model to a rendering server corresponding to the sub-model;

[0009] controlling each rendering server to respectively perform a clock synchronization operation and determine a maximum network transmission delay between the rendering servers;

[0010] determining an instruction timestamp in response to a preset browsing operation of the target model, and generating a rendering instruction according to an operation parameter of the preset browsing operation, the instruction timestamp and the maximum network transmission delay;

[0011] simultaneously sending the rendering instruction to the rendering servers, so that the rendering servers respectively render the sub-models and generate video streams corresponding to the sub-models and feed back the video streams;

[0012] respectively performing decoding operations on the video streams to generate a video image frame set corresponding to each video stream;

[0013] performing a clock synchronization operation on all video image frame sets, and respectively generating a target image frame from a plurality of video image frames at the same time in all video image frame sets;

[0014] displaying all newly generated target image frames.

[0015] Further, the rendering instruction comprises one or more of the following information:

[0016] instruction ID, operation parameter, rendering parameter, encoding parameter, instruction timestamp and maximum network transmission delay.

[0017] Further, the generating a target image frame from a plurality of video image frames at the same time in all video image frame sets comprises:

[0018] respectively obtaining a pixel point visible distance value set of the plurality of video image frames;

[0019] determining a minimum visible distance data value set from the plurality of pixel point visible distance value sets.

[0020] determine a new pixel point set according to the minimum visible distance data value set and the plurality of video image frames;

[0021] generate the target image frame according to the new pixel point set.

[0022] In a second aspect, a three-dimensional model distributed cloud rendering method is provided, applied to a model splitting server, and the method comprises:

[0023] splitting the target model into a plurality of sub-models according to a preset splitting strategy in response to a splitting instruction sent by a client;

[0024] sending each sub-model to a rendering server corresponding to the sub-model, so that the rendering server renders the corresponding sub-model and records a model number of the corresponding sub-model.

[0025] Further, the target model is split into a plurality of sub-models according to a preset splitting strategy based on a coordinate system of the target model, and the relative coordinate positions between the plurality of sub-models after splitting remain unchanged.

[0026] In a third aspect, a three-dimensional model distributed cloud rendering method is provided, applied to a rendering server, and the method comprises:

[0027] performing clock synchronization with the client and obtaining a network transmission delay in response to a clock synchronization instruction sent by the client, and sending the network transmission delay to the client to enable the client to determine a maximum network transmission delay;

[0028] performing rendering operation on a current sub-model to generate a rendering result in response to a rendering instruction sent by the client, performing video encoding on the rendering result to generate a video stream, and sending the video stream to the client.

[0029] Further, the rendering operation on the current sub-model in response to the rendering instruction sent by the client comprises:

[0030] performing rendering operation on the current sub-model in the case where the current time reaches a preset time in response to the rendering instruction sent by the client, the preset time including a time delayed by the maximum network transmission delay on the basis of an instruction timestamp.

[0031] Further, in the pixel processing stage in the rendering process, the visible distance value of the content corresponding to each pixel point to the observation point is determined for each video image frame, and a pixel point visible distance value set of each video image frame is generated according to the visible distance values of the content corresponding to all pixel points to the observation point in the each video image frame.

[0032] Further, after the rendering result is video encoded to generate a video stream, and before the video stream is sent to the client, the method further comprises:

[0033] embedding rendering information in the video stream, the rendering information comprising one or more of the set of pixel point visibility distance values and the instruction ID of the rendering instruction.

[0034] A fourth aspect of the present application provides a three-dimensional model rendering display device, the device comprising:

[0035] a splitting module configured to control a model splitting server to split a target model into a plurality of sub-models and send each sub-model to a corresponding rendering server;

[0036] a determining module configured to control each rendering server to perform a clock synchronization operation and determine a maximum network transmission delay between the rendering servers;

[0037] an instruction generating module configured to determine an instruction timestamp in response to a preset browsing operation on the target model, and generate a rendering instruction based on operation parameters of the preset browsing operation, the instruction timestamp, and the maximum network transmission delay;

[0038] an instruction sending module configured to send the rendering instruction to each rendering server simultaneously, so that each rendering server renders its own sub-model and generates a corresponding video stream and feeds back the video stream;

[0039] a decoding module configured to decode each video stream to generate a set of video image frames corresponding to each video stream;

[0040] a merging module configured to perform a clock synchronization operation on all sets of video image frames, and generate a target image frame based on a plurality of video image frames at the same time from all sets of video image frames;

[0041] a display module configured to display all newly generated target image frames.

[0042] A fifth aspect of the present application provides a three-dimensional model splitting system, characterized in that it comprises:

[0043] a splitting module configured to split a target model into a plurality of sub-models according to a preset splitting strategy in response to a splitting instruction sent by a client;

[0044] The sending module is configured to send each sub-model to a corresponding rendering server of the sub-model, so that the corresponding rendering server renders the corresponding sub-model and records a model number of the corresponding sub-model.

[0045] In a sixth aspect, the present application provides a three-dimensional model rendering system, which comprises:

[0046] one or more rendering servers, wherein each of the one or more rendering servers comprises:

[0047] a synchronization module configured to perform a clock synchronization operation with a client and obtain a network transmission delay in response to a clock synchronization instruction sent by the client, and send the network transmission delay to the client so that the client determines a maximum network transmission delay;

[0048] a rendering module configured to perform a rendering operation on a current sub-model to generate a rendering result in response to a rendering instruction sent by the client, perform video encoding on the rendering result to generate a video stream, and send the video stream to the client.

[0049] In a seventh aspect, the present application provides a computer readable storage medium storing a computer program, which can be executed by one or more processors to implement the three-dimensional model distributed cloud rendering method applied to a rendering server, the three-dimensional model distributed cloud rendering method applied to a model splitting server, or the three-dimensional model distributed cloud rendering method applied to a client.

[0050] In an eighth aspect, the present application provides an electronic device comprising a memory and one or more processors, wherein the memory stores a computer program, and the memory and the one or more processors are communicatively connected, and the computer program is executed by the one or more processors to implement the three-dimensional model distributed cloud rendering method applied to a rendering server, the three-dimensional model distributed cloud rendering method applied to a model splitting server, or the three-dimensional model distributed cloud rendering method applied to a client.

[0051] Compared with the prior art, the technical scheme of the present application has the following advantages or beneficial effects:

[0052] The three-dimensional data model distributed cloud rendering method disclosed in the present application can be shared by multiple servers for rendering. Whether it is a small or large scale model, the method has real-time and smooth rendering capability, small delay, high rendering efficiency and other advantages, and improves the user experience of three-dimensional data model rendering. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description only need to be drawn for the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0054] Figure 1 A flow chart of a three-dimensional model rendering method applied to a client provided by an embodiment of the present application;

[0055] Figure 2 A flow chart of a three-dimensional model rendering method applied to a model splitting server provided by an embodiment of the present application;

[0056] Figure 3 A model splitting schematic diagram provided by an embodiment of the present application;

[0057] Figure 4 A model rotation occlusion relationship schematic diagram provided by an embodiment of the present application;

[0058] Figure 5 Another model rotation occlusion relationship schematic diagram;

[0059] Figure 6 A flow chart of a three-dimensional model rendering method applied to a rendering server provided by an embodiment of the present application;

[0060] Figure 7 A structural schematic diagram of a three-dimensional model rendering display device provided by an embodiment of the present application;

[0061] Figure 8 An architecture schematic diagram of a rendering server end provided by an embodiment of the present application;

[0062] Figure 9 A connection block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0063] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and embodiments, so that how the present application applies technical means to solve technical problems and achieves corresponding technical effects can be fully understood and implemented. The embodiments of the present application and each feature in the embodiments can be combined with each other without conflict, and the formed technical solutions are all within the protection scope of the present application.

[0064] Embodiment one

[0065] The present embodiment provides a three-dimensional model distributed cloud rendering method, which is applied to a client.

[0066] Figure 1 A flowchart illustrating a 3D model rendering method applied to a client, as provided in this application embodiment, is shown below. Figure 1 As shown, the method in this embodiment includes:

[0067] Step 110: Control the model splitting server to split the target model into multiple sub-models and send each sub-model to the rendering server corresponding to each sub-model.

[0068] Optionally, the target model may include a BIM 3D data model, and this step can be skipped if the target model has been split into multiple sub-models.

[0069] Optionally, the specific splitting process can be completed separately from the client in advance, and the resulting multiple sub-models can be sent to multiple rendering servers, with each rendering server rendering one sub-model. There is a one-to-one correspondence between the sub-models and the rendering servers.

[0070] Step 120: Control each rendering server to perform clock synchronization operation and determine the maximum network transmission delay between each rendering server.

[0071] Optionally, the client can be used as a reference clock to synchronize with the rendering server, and the network latency between the client and the rendering server can be obtained during the time synchronization.

[0072] In some embodiments, clock synchronization instructions are sent to multiple rendering servers based on a clock synchronization protocol, so that the multiple rendering servers perform clock synchronization operations and determine network transmission latency respectively.

[0073] Optionally, clock synchronization between the client and the rendering server can be achieved based on a clock synchronization protocol, which can be the IEEE 1588V2 protocol.

[0074] It should be noted that clock synchronization can be a process that is continuously executed at preset time intervals, because once the synchronization process is stopped, the clocks of the client and the server will gradually become out of sync.

[0075] In some embodiments, the maximum network transmission delay between the multiple rendering servers is determined based on the network transmission delay sent by the multiple rendering servers.

[0076] During clock synchronization with the rendering server, the network transmission latency with each server is simultaneously acquired, and the maximum network transmission latency DelayMax is determined based on the network transmission latency of each rendering server.

[0077] Step 130, in response to the preset browsing operation determination instruction time stamp, and according to the operation parameters of the preset browsing operation, the instruction time stamp and the maximum network transmission delay, generate rendering instructions.

[0078] Optionally, the preset browsing operation can include operations based on user operations on a keyboard, mouse or gestures on a touch screen.

[0079] Further, the operation parameters are obtained according to the corresponding user operations.

[0080] In some embodiments, the rendering instructions include one or more of the following information:

[0081] Instruction ID, operation parameters, rendering parameters, encoding parameters, instruction time stamp and maximum network transmission delay.

[0082] Optionally, the rendering instruction content includes: instruction ID, operation parameters, rendering parameters, encoding parameters, instruction time stamp, and the maximum network transmission delay between the client and the multiple rendering servers (DelayMax).

[0083] Optionally, before the rendering instructions are sent, the current time stamp is obtained, and the rendering instructions are generated according to the current time stamp, the operation parameters and the maximum network transmission delay (it can be understood that the current time stamp in the rendering instruction content is the instruction time stamp), and the rendering instructions are sent to the multiple rendering servers at the same time, so that the instruction time stamps in the rendering instructions sent to the multiple rendering servers are the same. Further, some contents of the rendering instructions are explained:

[0084] 1) Instruction ID: the instruction ID can uniquely identify an instruction, i.e. the serial number of the rendering instruction;

[0085] 2) Operation parameters: such as the actions of panning, rotating and scaling, and the speed, direction, starting point and ending point of the actions;

[0086] 3) Rendering parameters: such as camera view, clipping plane and pixel resolution;

[0087] 4) Encoding parameters: such as encoding method, code rate and frame rate;

[0088] 5) Instruction time stamp: the time when the client sends the rendering instruction (the operation instruction sent to the multiple rendering servers uses the same time stamp);

[0089] 6) the maximum network transmission delay DelayMax between the client and the plurality of rendering servers, i.e. the maximum network transmission delay between the client and the plurality of rendering servers. It should be noted that the client sends rendering instructions to each rendering server, and the rendering instructions obtained by each rendering server are consistent, and the parameters used for rendering each sub-model are consistent.

[0090] Step 140, the rendering instructions are sent to the rendering servers simultaneously, so that the rendering servers respectively render the sub-models and generate video streams corresponding to the sub-models and feedback the video streams.

[0091] Further, the plurality of rendering servers respectively render the sub-models in response to the rendering instructions and generate video streams corresponding to the sub-models and feedback the video streams to the current client.

[0092] Step 150, each video stream is decoded to generate a video image frame set corresponding to each video stream.

[0093] Optionally, the video stream sent by the rendering server is decoded, and image information of each frame in each video stream is recovered to generate a video image frame set corresponding to each video stream.

[0094] Step 160, clock synchronization is performed on all video image frame sets, and a target image frame is generated according to a plurality of video image frames at the same time in all video image frame sets.

[0095] Optionally, for all video image frames in each video image frame set, a plurality of video image frames at the same time are obtained from each video image frame set, and then a target image frame corresponding to the same time is generated according to the plurality of video image frames at the same time.

[0096] Optionally, the rendering and encoding of the sub-models by the rendering servers are synchronized, so that each decoded frame image needs to be synchronized after the client receives the multiple video streams.

[0097] In some embodiments, the target image frame is generated according to the plurality of video image frames at the same time in all video image frame sets, including:

[0098] A pixel point visible distance value set of the plurality of video image frames is obtained respectively;

[0099] A minimum visible distance data value set is determined from the plurality of pixel point visible distance value sets;

[0100] A new pixel point set is determined according to the minimum visible distance data value set and the plurality of video image frames.

[0101] generating the one target image frame according to the new pixel point set.

[0102] Optionally, when generating one new target image frame according to multiple video image frames at the same time in all video image frame sets, the occlusion relationship can be obtained based on the ranging of image pixels from different servers, and all video image frames at the same time in multiple video image frame sets can be integrated into one new target image frame according to the occlusion relationship.

[0103] Optionally, the client receives video streams from each server, and needs to complete the following operations:

[0104] 1) Extracting information data embedded by the server in each video stream, such as rendering instruction serial number (instruction ID), model number after splitting, distance from each frame / pixel point of the video stream to the content of the camera (distance from the model pixel point to the observation point Ln xy , wherein n represents the model number after splitting, and xy represents the pixel position);

[0105] 2) Decoding each video stream to restore each frame and each pixel point;

[0106] 3) Synchronizing and merging the image frames of each video stream, considering that the occlusion relationship of the model will change after the angle rotation of multiple small models (refer to Figure 4 ), so the Ln xy value of each pixel of each image frame needs to be compared, and only the pixel point with the smallest Ln xy value (the point closest to the observer) is taken when the image frames are merged.

[0107] Those skilled in the art can understand that after the merging processing of the video frames of multiple video image frames at the same time in each video image frame set, one new target image frame will be generated, and after the merging processing of the video frames of multiple video image frames at the same time in another video image frame set, another new target image frame will be generated. For example, there are 5 video image frame sets (each video image frame set contains 100 frames of images), after synchronization operation; then the first frame of image is taken from each of the 5 video image frame sets, and then the 5 first frames of image are merged to process one new image (the first target image frame); then the second frame of image is taken from each of the 5 video image frame sets, and then the 5 second frames of image are merged to process one new image (the second target image frame); and so on. When all the 100 frames of image in each video image frame set are merged, a set containing 100 newly generated target image frames will be generated.

[0108] Step 170, display all newly generated target image frames.

[0109] Display all newly generated target image frames on the client, and then display the three-dimensional model through a series of two-dimensional image frames.

[0110] The three-dimensional model distributed cloud rendering method provided in the embodiment separates the video rendering process and balances the load of a group of rendering servers. Specifically, the model splitting server is controlled by the client to split the target model into multiple sub-models and send each sub-model to the corresponding rendering server; the clock synchronization operation is performed on each rendering server, and the maximum network transmission delay between the rendering servers is determined; the instruction timestamp is determined in response to the preset browsing operation of the target model, and the rendering instruction is generated according to the operation parameters of the preset browsing operation, the instruction timestamp, and the maximum network transmission delay; the rendering instruction is sent to the rendering servers at the same time, so that the rendering servers render the sub-models respectively and generate video streams corresponding to the sub-models and feedback the video streams; the decoding operation is performed on each video stream to generate a video image frame set corresponding to each video stream; the clock synchronization operation is performed on all video image frame sets, and a target image frame is generated according to multiple video image frames at the same time in all video image frame sets; and finally, all newly generated target image frames are displayed on the client. For the three-dimensional data model distributed cloud rendering method, multiple servers can share the rendering, whether it is a small or large-scale model, and the real-time and smooth rendering capability is achieved, with the advantages of small delay, high rendering efficiency, and the like, and the user experience of three-dimensional data model rendering is improved.

[0111] Embodiment Two

[0112] The embodiment provides a three-dimensional model distributed cloud rendering method, which is applied to a model splitting server. Figure 2 A flowchart of a three-dimensional model rendering method applied to a model splitting server provided in the embodiment is shown in FIG. 1. Figure 2 The method provided in the embodiment includes the following steps.

[0113] Step 210, in response to the splitting instruction sent by the client, the target model is split into multiple sub-models according to a preset splitting strategy.

[0114] Optionally, the client sends a splitting instruction to control the model splitting server to split the target model into multiple sub-models.

[0115] Optionally, the splitting process can include the following parts: splitting of the model, loading of the server-side model, publishing, etc. The splitting operation can be performed in response to a splitting instruction sent by the client, so as to split the target model into a plurality of sub-models based on the coordinate system of the target model according to a preset splitting strategy.

[0116] In some embodiments, the preset splitting strategy includes:

[0117] The target model is split into a plurality of sub-models based on the coordinate system of the target model according to a preset splitting strategy, and the relative coordinate positions between the plurality of sub-models after splitting remain unchanged.

[0118] Optionally, how to split the target model can be selected according to actual needs, but the plurality of sub-models after splitting are still based on the coordinate system of the large model, and the relative positions between the plurality of sub-models remain unchanged.

[0119] The specific splitting principle can refer to Figures 3-5 , wherein, Figure 3 is a model splitting schematic diagram provided by an embodiment of the present application, Figure 4 is a model rotation and occlusion relationship schematic diagram provided by an embodiment of the present application, Figure 5 is another model rotation and occlusion relationship schematic diagram.

[0120] In step 220, each sub-model is sent to a rendering server corresponding to the sub-model, so that each rendering server renders the corresponding sub-model and records the model number of the corresponding sub-model.

[0121] Optionally, the small models after splitting are rendered by a plurality of rendering servers respectively, and one rendering server undertakes the rendering of one small model.

[0122] The three-dimensional model distributed cloud rendering method provided by the embodiment includes: splitting a target model into a plurality of sub-models according to a preset splitting strategy in response to a splitting instruction sent by a client, and sending each sub-model to a rendering server corresponding to the sub-model, so that each rendering server renders the corresponding sub-model and records the model number of the corresponding sub-model. A super large model is split into a plurality of sub-models, and each sub-model is rendered by a plurality of servers respectively, so that the user can browse the three-dimensional model in real time and smoothly on the client.

[0123] Embodiment three

[0124] The embodiment provides a three-dimensional model distributed cloud rendering method, which is applied to a rendering server, Figure 6 A flowchart of a three-dimensional model rendering method applied to a rendering server is provided by the embodiment of the present application, as shown in Figure 6 The method of the embodiment includes:

[0125] In step 610, in response to the clock synchronization instruction sent by the client, the clock synchronization operation is performed with the client, the network transmission delay is obtained, and the network transmission delay is sent to the client to enable the client to determine the maximum network transmission delay.

[0126] Optionally, the client sends a clock synchronization instruction to control each rendering server to perform a clock synchronization operation with the client respectively, and each rendering server feeds back the respective network transmission delay to the client. The client can also obtain the network transmission delay according to the response data of each rendering server.

[0127] Optionally, the rendering server implements the clock synchronization with the client based on a clock synchronization protocol (such as IEEE 1588V2 protocol).

[0128] The clock synchronization accuracy is not less than ±200μs. Through the IEEE 1588V2 protocol, the client can obtain the delay of each rendering server, and take the maximum delay value from them, which is recorded as DelayMax.

[0129] In step 620, in response to the rendering instruction sent by the client, the current sub-model is rendered to generate a rendering result, the rendering result is video encoded to generate a video stream, and the video stream is sent to the client.

[0130] Optionally, the rendering server receives the rendering instruction of the client, parses and executes the related rendering instruction requirements, and finally encodes the rendering result (a series of two-dimensional images) to generate a video stream.

[0131] In some embodiments, the rendering operation on the current sub-model in response to the rendering instruction sent by the client comprises:

[0132] In response to the rendering instruction sent by the client, the current sub-model is rendered in the case that the current time reaches a preset time, and the preset time comprises a time after delaying the maximum network transmission delay on the basis of the instruction timestamp.

[0133] Optionally, after each rendering server receives the rendering instruction, based on the local clock, the independent rendering model is performed after the instruction timestamp+DelayMax, which ensures the synchronization of rendering of each server, and the instruction timestamp can be obtained from the rendering instruction.

[0134] In some embodiments, in a pixel processing stage in the rendering process, a visible distance value of each pixel point corresponding to the content to the observation point is determined for each video image frame, and a set of pixel visible distance values of each video image frame is generated according to the visible distance values of all pixel points corresponding to the content to the observation point in each video image frame.

[0135] Optionally, the rendering of the model includes Application, Geometry Processing, Rasterization, and Pixel Processing, and the synchronization of the rendering process between the servers is achieved through a synchronization mechanism.

[0136] Further, in the pixel processing stage in the model rendering process, the distance of each pixel point corresponding to the content to the camera (or observation point) is calculated and denoted as Ln xy , where n represents the number of the split model, and xy represents the pixel position.

[0137] Further, in the camera ranging operation, the distance from the camera observation point to the model needs to be measured for each pixel point, Figure 5 Another model rotation occlusion relationship diagram is shown in FIG. 4, where the positions of the camera to the yellow ball and the red ball are different. Figure 5 The positions of the pixel points rendered by different rendering servers are one-to-one corresponding.

[0138] Further, the server encodes the rendering result to form a real-time streaming media. Since the clocks of the rendering servers are synchronized, the starting time points of the rendering are consistent, and the received instruction contents of the client are also consistent, the image frames of the streaming media formed by the server model rendering are also synchronized.

[0139] In some embodiments, after the video stream is generated by encoding the rendering result, and before the video stream is sent to the client, the method further includes:

[0140] Embedding rendering information in the video stream, the rendering information including one or more of the instruction ID of the rendering instruction and the set of pixel visible distance values.

[0141] Optionally, after the rendering servers render the respective sub-models to form streaming media, some information data (rendering information) needs to be embedded in the streaming media data for integration at the client, including the serial number of the rendering instruction (instruction ID), the model number of the split sub-model corresponding to the video stream, Ln xy , and the like.

[0142] Optionally, the video stream is encapsulated in the MPEG-2 standard TS (Transport stream) format, the PID field of the TS packet PacketHead is a unique identification mark in the TS stream, and the content of the Packet Data is determined by the PID. When the PID is 0, the Packet Data is a PAT table (Program Association Table, PAT for short), which is used to indicate the PID number of the PMT table (Program Map Table, PMT for short) corresponding to the video content sequence number; when the PID is the PID number of the PMT, the Packet Data is the PMT table of a certain content sequence number, which is used to indicate the PID number corresponding to the video, audio and data TS related to the content. A line can be added to the PMT table to indicate the instruction ID of the rendering instruction, the model number of the sub-model, and the distance value of each pixel of each frame.

[0143] Further, in order to reduce the data amount of the data sent by the server to the client, only the relevant rendering information data is embedded in the I frame video image, and the P frame and the B frame are not embedded.

[0144] In addition, in order to compress the data bytes of Ln xy , the pixels of the content object that do not exist in the field of view of the observer can be empty; and Ln xy of the first frame of the I frame is an absolute value, and Ln xy uploaded on the subsequent I frame can be a relative value between the first frame Ln xy . xy

[0145] Optionally, the network interface and protocol stack can be used for instruction transmission, video stream media transmission, and clock synchronization signal transmission between the rendering server and the client.

[0146] The three-dimensional model distributed cloud rendering method provided by the embodiment comprises the following steps: in response to a clock synchronization instruction sent by a client, performing clock synchronization operation with the client and obtaining network transmission delay, and sending the network transmission delay to the client to enable the client to determine the maximum network transmission delay; in response to a rendering instruction sent by the client, performing rendering operation on a current sub-model to generate a rendering result, performing video encoding on the rendering result to generate a video stream, and sending the video stream to the client. By rendering respective sub-models by multiple rendering servers, the user can browse the three-dimensional model in real time and smoothly on the client.

[0147] Embodiment four

[0148] ​The embodiment of the application provides a three-dimensional model rendering display device, which can be used for executing the three-dimensional model distributed cloud rendering method of the application applied to a client. For details not disclosed in the device embodiment, please refer to the method embodiment of the application. The device provided by the embodiment comprises:

[0149] A splitting module is configured to control a model splitting server to split a target model into a plurality of sub-models and send each sub-model to a corresponding rendering server;

[0150] A determining module is configured to control each rendering server to perform a clock synchronization operation and determine a maximum network transmission delay between the rendering servers;

[0151] An instruction generating module is configured to determine an instruction timestamp in response to a preset browsing operation of the target model, and generate a rendering instruction according to an operation parameter of the preset browsing operation, the instruction timestamp and the maximum network transmission delay;

[0152] An instruction sending module is configured to send the rendering instruction to the rendering servers simultaneously, so that the rendering servers render the respective sub-models and generate video streams corresponding to the respective sub-models and feedback the video streams;

[0153] A decoding module is configured to perform a decoding operation on each video stream to generate a video image frame set corresponding to each video stream;

[0154] A merging module is configured to perform a clock synchronization operation on all video image frame sets, and generate a target image frame according to a plurality of video image frames at the same time in all video image frame sets;

[0155] A display module is configured to display all newly generated target image frames.

[0156] In some embodiments, the rendering instruction comprises one or more of the following information:

[0157] An instruction ID, an operation parameter, a rendering parameter, an encoding parameter, an instruction timestamp and a maximum network transmission delay.

[0158] In some embodiments, the merging module comprises an acquisition unit, a first determining unit, a second determining unit and a synthesizing unit, wherein:

[0159] The acquisition unit is configured to acquire a pixel point visible distance value set of each of the plurality of video image frames;

[0160] The first determining unit is configured to determine a minimum visible distance data value set from the plurality of pixel point visible distance value sets;

[0161] A second determining unit is configured to determine a new pixel point set according to the minimum visible distance data set and the plurality of video image frames.

[0162] A synthesizing unit is configured to generate the target image frame according to the new pixel point set.

[0163] Optionally, the three-dimensional model rendering display device in the embodiments of the present application can refer to Figure 7 , Figure 7 a structural schematic diagram of a three-dimensional model rendering display device provided by the embodiments of the present application.

[0164] Those skilled in the art can understand that Figure 7 the structures shown in the above embodiments do not constitute a limitation on the device of the embodiments of the present application, and can include more or fewer modules / units than the drawings, or combine certain modules / units, or different arrangements of modules / units.

[0165] It should be noted that the above-mentioned various modules / units can be functional modules or program modules, and can be implemented by software or hardware. For the modules / units implemented by hardware, the above-mentioned various modules / units can be located in the same processor; or the above-mentioned various modules / units can also be located in different processors in any combination.

[0166] The device provided in the embodiment comprises: a splitting module, configured to control a model splitting server to split a target model into a plurality of sub-models and respectively send each sub-model to a rendering server corresponding to the sub-model; a determination module, configured to control each rendering server to respectively perform a clock synchronization operation and determine a maximum network transmission delay between the rendering servers; an instruction generation module, configured to determine an instruction timestamp in response to a preset browsing operation on the target model, and generate a rendering instruction according to an operation parameter of the preset browsing operation, the instruction timestamp and the maximum network transmission delay; an instruction sending module, configured to simultaneously send the rendering instruction to the rendering servers, so that the rendering servers respectively render the sub-models and generate video streams corresponding to the sub-models and feed back the video streams; a decoding module, configured to respectively perform a decoding operation on each video stream to generate a video image frame set corresponding to each video stream; a merging module, configured to perform a clock synchronization operation on all video image frame sets, and generate a target image frame according to a plurality of video image frames at the same time in all video image frame sets; and a display module, configured to display all newly generated target image frames. For the distributed cloud rendering method of the three-dimensional data model, rendering can be shared by multiple servers, whether the model is small or large, the model has real-time and smooth rendering capability, has the advantages of small delay, high rendering efficiency and the like, and improves the user experience of rendering the three-dimensional data model.

[0167] Embodiment five

[0168] The embodiment provides a three-dimensional model splitting system. The system embodiment can be used to execute the three-dimensional model rendering method applied to the model splitting system of the application. For details not disclosed in the system embodiment, refer to the method embodiments of the application. The system provided in the embodiment comprises:

[0169] The splitting module is configured to split the target model into a plurality of sub-models according to a preset splitting strategy in response to a splitting instruction sent by the client.

[0170] The sending module is configured to respectively send each sub-model to a rendering server corresponding to the sub-model, so that the rendering server renders the corresponding sub-model and records a model number of the corresponding sub-model.

[0171] In some embodiments, the preset splitting strategy comprises:

[0172] The target model is split into a plurality of sub-models according to a preset splitting strategy based on a coordinate system of the target model, and the relative coordinate positions between the plurality of sub-models after splitting remain unchanged.

[0173] The system provided in the embodiment comprises: a splitting module configured to split a target model into a plurality of sub-models according to a preset splitting strategy in response to a splitting instruction sent by a client; and a sending module configured to send each sub-model to a rendering server corresponding to the sub-model, so that the rendering server renders the corresponding sub-model and records a model number of the corresponding sub-model. A super-large model is split into a plurality of sub-models, and each sub-model is rendered by a plurality of servers, so that the user can browse the three-dimensional model in real time and smoothly on the client.

[0174] Embodiment six

[0175] The embodiment provides a three-dimensional model rendering system. The system embodiment can be used to execute the three-dimensional model rendering method embodiment of the model splitting system applied in the application. For details not disclosed in the system embodiment, refer to the method embodiment of the application. The system provided in the embodiment comprises:

[0176] one or more rendering servers; wherein the one or more rendering servers each comprise:

[0177] a synchronization module configured to perform a clock synchronization operation with the client and obtain a network transmission delay in response to a clock synchronization instruction sent by the client, and send the network transmission delay to the client to enable the client to determine a maximum network transmission delay;

[0178] a rendering module configured to perform a rendering operation on a current sub-model to generate a rendering result in response to a rendering instruction sent by the client, perform video encoding on the rendering result to generate a video stream, and send the video stream to the client.

[0179] In some embodiments, the rendering module is configured to perform a rendering operation on the current sub-model in a case where a current time reaches a preset time in response to a rendering instruction sent by the client, the preset time comprising a time after delaying the maximum network transmission delay on the basis of an instruction timestamp.

[0180] In some embodiments, in a pixel processing stage in the rendering process, a visible distance value of content corresponding to each pixel point to an observation point is determined for each video image frame, and a pixel visible distance value set of each video image frame is generated according to the visible distance values of content corresponding to all pixel points to the observation point in the each video image frame.

[0181] In some embodiments, further comprising an embedding module, configured to embed rendering information in the video stream after the rendering result is video encoded to generate the video stream, and before the video stream is sent to the client, the rendering information comprising one or more of the instruction ID of the rendering instruction, and one or more of the set of pixel point visibility distance values.

[0182] Optionally, the rendering server in the embodiments of the present application can refer to Figure 8 , Figure 8 A schematic diagram of an architecture of a rendering server end provided by the embodiments of the present application.

[0183] Those skilled in the art can understand that the structure shown in the above Figure 8 does not constitute a limitation on the device of the embodiments of the present application, and can include more or less modules / cells than the diagram, or combine certain modules / cells, or different arrangement of modules / cells.

[0184] The system provided by the embodiments includes one or more rendering servers, wherein each of the one or more rendering servers includes a synchronization module configured to perform clock synchronization operation with the client and obtain network transmission delay in response to a clock synchronization instruction sent by the client, and send the network transmission delay to the client to enable the client to determine the maximum network transmission delay; and a rendering module configured to perform rendering operation on a current sub-model to generate a rendering result in response to a rendering instruction sent by the client, and video encode the rendering result to generate a video stream and send the video stream to the client. By rendering respective sub-models by multiple rendering servers, the effect of enabling the user to browse the three-dimensional model in real time and smoothly at the client can be achieved.

[0185] Embodiment Seven

[0186] The embodiments also provide a computer readable storage medium, which stores a computer program, and the computer program can implement the method steps in the foregoing method embodiments when executed by a processor. The embodiments will not be repeated here.

[0187] The computer-readable storage medium can also individually include a computer program, a data file, a data structure, etc., or a combination thereof. The computer-readable storage medium or the computer program can be specifically designed and understood by those skilled in the computer software field, or can be known and available to those skilled in the computer software field. Examples of the computer-readable storage medium include: magnetic media, such as a hard disk, a floppy disk, and a magnetic tape; optical media, such as a CD ROM disk and a DVD; a magneto-optical medium, such as an optical disk; and a hardware device specifically configured to store and execute a computer program, such as a read-only memory (ROM), a random access memory (RAM), a flash memory; or a server, an app application store, etc. Examples of the computer program include machine code (e.g., code generated by a compiler) and a file containing high-level code that can be executed by a computer by using an interpreter. The described hardware device can be configured to function as one or more software modules to perform the above-described operations and methods, and vice versa. In addition, the computer-readable storage medium can be distributed in a networked computer system, and the program code or computer program can be stored and executed in a distributed manner.

[0188] Embodiment Eight

[0189] Figure 9 A connection block diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 9. The electronic device 900 can include one or more processors 901, a memory 902, a multimedia component 903, an input / output (I / O) interface 904, and a communication component 905. Figure 9

[0190] The one or more processors 901 are configured to perform all or part of the steps in the above-described method embodiments. The memory 902 is configured to store various types of data, which can include, for example, instructions of any application program or method in the electronic device, and application-related data.

[0191] ​The one or more processors 901 can be implemented with an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor or other electronic devices implementing the present application.

[0192] The memory 902 can be implemented with any type of volatile or nonvolatile storage devices or a combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic storage, a flash memory, a magnetic disk, or a compact disk.

[0193] The multimedia component 903 can include a screen, which can be a touch screen, and audio components for outputting and / or inputting audio signals. For example, the audio components can include a microphone for receiving external audio signals. The received audio signals can be further stored in the memory or transmitted through the communication component. The audio components also include at least one speaker for outputting audio signals.

[0194] The I / O interface 904 provides an interface between the one or more processors 901 and other interface modules, which can be a keyboard, a mouse, a button, and the like. The buttons can be virtual buttons or physical buttons.

[0195] The communication component 905 is configured to perform wired or wireless communication between the electronic device 900 and other devices. The wired communication includes communication through a network port, a serial port, etc., and the wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, 5G, or a combination of one or more of them. Therefore, the corresponding communication component 905 can include a Wi-Fi module, a Bluetooth module, and an NFC module.

[0196] In summary, the three-dimensional model distributed cloud rendering method, the three-dimensional model rendering display device, the three-dimensional model splitting system, the three-dimensional model rendering system, the storage medium, and the electronic device provided by the present application are provided. The three-dimensional model distributed cloud rendering method provided by the embodiment separates the video rendering process and balances the load of each rendering server. The three-dimensional model distributed cloud rendering method applied to the client includes: splitting a target model into multiple sub-models by a model splitting server controlled by the client and sending each sub-model to a corresponding rendering server; controlling each rendering server to perform clock synchronization operations and determine the maximum network transmission delay between the rendering servers; determining an instruction time stamp in response to a preset browsing operation of the target model, and generating a rendering instruction according to the operation parameters of the preset browsing operation, the instruction time stamp, and the maximum network transmission delay; sending the rendering instruction to each rendering server simultaneously to enable each rendering server to render its own sub-model and generate a corresponding video stream and feedback the video stream; decoding each video stream to generate a corresponding video image frame set; performing clock synchronization operations on all video image frame sets, and generating a target image frame according to multiple video image frames at the same time in all video image frame sets; and finally displaying all newly generated target image frames on the client. For the three-dimensional data model distributed cloud rendering method, multiple servers can share the rendering. Whether it is a small or large-scale model, it has real-time and smooth rendering capability, has the advantages of small delay, high rendering efficiency, and the like, and improves the user experience of three-dimensional data model rendering.

[0197] It should also be understood that the methods or systems disclosed in the embodiments provided in this application can also be implemented in other ways. The method or system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functions, and operations of possible implementations of methods and apparatus according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, computer program segment, or part of a computer program, which includes one or more computer programs for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings, and may actually be executed substantially in parallel. They may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer programs.

[0198] In this application, the terms "comprise", "contain", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, the element defined by the phrase "comprises a" does not exclude the presence of additional identical elements in the process, method, device, or apparatus that includes the element; if there is a description of "first", "second", etc., it is only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of indicated technical features; in the description of the present application, unless otherwise specified, the term "a plurality of" or "a plurality" means at least two; if there is a description of a server, it should be noted that the server can be a stand-alone physical server or terminal, or a server cluster composed of multiple physical servers, or a cloud server capable of providing cloud server, cloud database, cloud storage and CDN and other basic cloud computing services; if there is a description of a smart terminal or a mobile device in the present application, it should be noted that the smart terminal or mobile device can be a mobile phone, a tablet computer, a smart watch, a netbook, a wearable electronic device, a personal digital assistant (PDA), an augmented reality technology device (AR), a virtual reality device (VR), a smart television, a smart sound, a personal computer (PC), etc., but is not limited thereto, and the specific form of the smart terminal or mobile device is not specially limited in the present application.

[0199] Finally, it should be noted that in the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "one example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0200] Although the embodiments of the present application have been shown and described above, it is understood that all the above-described embodiments are exemplary only, the contents described are merely adopted for the purpose of facilitating the understanding of the present application, and are not intended to limit the present application. Any person skilled in the art to which the present application belongs can make any modification and change in the form and details without departing from the spirit and scope of the present application, but the protection scope of the present application shall be subject to the scope defined by the appended claims.

Claims

1. A distributed cloud rendering method for 3D models, characterized in that, The method is applied to a client, and the method comprises: controlling a model splitting server to split a target model into multiple sub-models and send each sub-model to a corresponding rendering server; controlling each rendering server to perform clock synchronization and determine the maximum network transmission delay between the rendering servers; determining an instruction timestamp in response to a preset browsing operation on the target model, and generating a rendering instruction according to the operation parameters of the preset browsing operation, the instruction timestamp, and the maximum network transmission delay; sending the rendering instruction to each rendering server to cause each rendering server to render the corresponding sub-model and generate a video stream corresponding to the sub-model and feed back the video stream; decoding each video stream to generate a set of video image frames corresponding to each video stream; performing clock synchronization on all sets of video image frames, and generating a target image frame from multiple video image frames at the same time according to all sets of video image frames; displaying all newly generated target image frames.

2. The method of claim 1, wherein, The rendering instruction comprises one or more of the following information: instruction ID, operation parameters, rendering parameters, encoding parameters, instruction timestamp, and maximum network transmission delay.

3. The method of claim 1, wherein, The generation of a target image frame from multiple video image frames at the same time according to all sets of video image frames comprises: obtaining a set of pixel point visual distance values of the multiple video image frames; determining a set of minimum visual distance data values from the multiple sets of pixel point visual distance values; determining a new set of pixel points according to the set of minimum visual distance data values and the multiple video image frames; generating the target image frame according to the new set of pixel points.

4. A three-dimensional model distributed cloud rendering method, characterized in that, The method is applied to a rendering server, and the method comprises: performing clock synchronization with a client in response to a clock synchronization instruction sent by the client, obtaining a network transmission delay, and sending the network transmission delay to the client to enable the client to determine the maximum network transmission delay; performing rendering on a current sub-model in response to a rendering instruction sent by the client to generate a rendering result, video encoding the rendering result to generate a video stream, and sending the video stream to the client; wherein the generation of the rendering instruction comprises determining an instruction timestamp in the client in response to a preset browsing operation on a target model, and generating the rendering instruction according to the operation parameters of the preset browsing operation, the instruction timestamp, and the maximum network transmission delay; the current sub-model is obtained by splitting the target model by a model splitting server.

5. The method of claim 4, wherein, The rendering on the current sub-model in response to the rendering instruction sent by the client comprises: performing rendering on the current sub-model in response to the rendering instruction sent by the client, under the condition that the current time reaches a preset time, which comprises a time delayed by the maximum network transmission delay from the instruction timestamp.

6. The method of claim 4, wherein, in a pixel processing stage in the rendering process, determining a visible distance value of each pixel point corresponding content to the viewpoint for each video image frame, and generating a pixel visible distance value set of each video image frame according to the visible distance values of all pixel points corresponding content to the viewpoint in each video image frame.

7. The method of claim 6, wherein, after video encoding the rendering result to generate a video stream, and before sending the video stream to the client, further comprising: embedding rendering information in the video stream, the rendering information including one or more of the instruction ID of the rendering instruction, and one or more of the pixel visible distance value set.

8. A three-dimensional model rendering display apparatus, characterized by comprising: comprising: a splitting module configured to control a model splitting server to split a target model into a plurality of sub-models and send each sub-model to a corresponding rendering server; a determining module configured to control each rendering server to perform a clock synchronization operation and determine a maximum network transmission delay between the rendering servers; an instruction generating module configured to determine an instruction timestamp in response to a preset browsing operation on the target model, and generate a rendering instruction according to operation parameters of the preset browsing operation, the instruction timestamp, and the maximum network transmission delay; an instruction sending module configured to send the rendering instruction to each rendering server simultaneously, so that each rendering server renders its own sub-model and generates a video stream corresponding to the sub-model and feeds back the video stream; a decoding module configured to decode each video stream to generate a set of video image frames corresponding to the video stream; a merging module configured to perform a clock synchronization operation on all sets of video image frames, and generate a target image frame from a plurality of video image frames at the same time from all sets of video image frames; a display module configured to display all newly generated target image frames.

9. A three-dimensional model rendering system, comprising: comprising: a plurality of rendering servers; wherein each rendering server comprises: a synchronization module configured to perform a clock synchronization operation with a client and obtain a network transmission delay in response to a clock synchronization instruction sent by the client, and send the network transmission delay to the client to enable the client to determine a maximum network transmission delay; a rendering module configured to render a current sub-model to generate a rendering result in response to a rendering instruction sent by the client, and video encode the rendering result to generate a video stream and send the video stream to the client; wherein the rendering instruction is generated by determining an instruction timestamp in response to a preset browsing operation on a target model at the client, and generating the rendering instruction according to operation parameters of the preset browsing operation, the instruction timestamp, and the maximum network transmission delay; the current sub-model is a sub-model sent by the client, and the sub-model is obtained by splitting the target model by a model splitting server.

10. A computer-readable storage medium, characterized in that, The computer program stored in the computer readable storage medium, when executed by one or more processors, implements the method of any one of claims 1-3 or 4-7.

11. An electronic device, comprising: A device comprising a memory and one or more processors, the memory having stored thereon a computer program, the memory and the one or more processors being communicatively coupled, and the computer program, when executed by the one or more processors, performing the method of any one of claims 1-3 or 4-7.

Images