Optimization method of real-time rendering picture

Abstract

The invention relates to the technical field of three-dimensional rendering, in particular to a real-time image rendering optimization method, which comprises the following steps of: when a scheduling center receives a rendering request sent by a user side, detecting whether rendering equipment is connected with the user side or not; if not, the scheduling center schedules any rendering device to be connected with the user side based on an exclusive scheduling algorithm; the dispatching center detects the user picture frame rate of the rendering equipment in the rendering process in real time, and judges whether a picture rendered by the current rendering equipment is healthy or not based on a picture health threshold value; when the user side is not healthy, the dispatching center detects whether rendering equipment which is not connected with the user side exists in all rendering equipment connected with the dispatching center or not; and when the rendering equipment exists, the scheduling center switches another rendering equipment to be connected with the user side based on an exclusive scheduling algorithm. The method has the beneficial effects that the stable output of the stream rendering picture can be ensured by switching the rendering equipment in the stream rendering process, and the influence of picture lagging or service collapse on the user experience is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional rendering, and in particular to an optimization method for real-time rendering pictures. Background Art

[0002] Streaming rendering is an important milestone in the development of the Internet. It is a more advanced form of the Internet and a real-time network. It can achieve real-time transmission of high-definition video across the entire network, which is currently impossible with the Internet. It will push many Internet applications towards high-definition video and high-definition face-to-face, and will ultimately achieve a world without distance, where the distance between people around the world is just the distance of a screen. Therefore, the use of streaming rendering is becoming more and more widespread.

[0003] During the streaming rendering process, the stability of the rendered image is crucial to the user experience. However, today's common streaming rendering services only provide rendering services to users through a single rendering device. When a single rendering device is used for rendering, once an abnormality occurs during the rendering process, that is, the image freezes or the service crashes, it affects the stable output of the streaming rendering image and the user experience. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an optimization method for real-time rendering of pictures, which solves the technical problem that when a single rendering device provides services, once the picture freezes or the service crashes during operation, it directly affects the stability of the stream rendering picture and thus affects the user experience.

[0006] (II) Technical solution

[0007] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0008] In a first aspect, an embodiment of the present invention provides a method for optimizing a real-time rendering picture, the method being implemented based on a pre-set real-time rendering system, the system comprising a dispatching center, and at least one rendering device connected to the dispatching center, the method comprising:

[0009] S11, when the dispatch center receives a rendering request sent by a user terminal, it detects whether there is any rendering device connected to the user terminal to provide rendering services for the user terminal;

[0010] If it does not exist, the scheduling center schedules any rendering device to connect to the user terminal to provide rendering services to the user terminal based on a preset exclusive scheduling algorithm;

[0011] S12, the dispatch center detects in real time the user picture frame rate of the rendering device connected to the user terminal during the rendering process, and determines whether the picture rendered by the current rendering device is healthy based on a preset picture health threshold; the user picture frame rate is the frame rate of the picture rendered by the rendering device;

[0012] If it is unhealthy, the dispatch center detects whether there is a rendering device that is not connected to the user terminal to provide rendering services among all the rendering devices connected to it;

[0013] S13. If yes, the scheduling center switches another rendering device to connect with the user terminal to provide rendering service for the user terminal based on a preset exclusive scheduling algorithm, and repeats steps S12 and S13.

[0014] Optionally, the picture health threshold includes a frame rate threshold and a frame rate abnormality times threshold;

[0015] Then, the S12 includes:

[0016] S12-1, the dispatch center detects the user screen frame rate of the rendering device connected to the user terminal in real time during the rendering process, and determines whether the user screen frame rate of the current rendering device during the rendering process is abnormal based on a preset frame rate threshold, and records the number of abnormal user screen frame rates;

[0017] During this period, when the user screen frame rate detected by the dispatch center is less than the frame rate threshold, the number of frame rate anomalies increases by one;

[0018] When the user screen frame rate detected by the dispatch center is greater than or equal to the frame rate threshold, the number of frame rate anomalies decreases by one;

[0019] S12-2, the dispatch center detects the number of abnormal frame rates of the user screen in real time, and determines whether the screen rendered by the current rendering device is healthy based on the frame rate abnormality number threshold;

[0020] When the number of abnormal frame rates of the user screen is greater than or equal to the frame rate abnormal number threshold, it is determined that the screen rendered by the current rendering device is unhealthy.

[0021] Optionally, the S12 further includes:

[0022] While the current rendering device provides rendering services to the user end, the dispatch center records the user screen frame rate of the rendering device in the rendering process in real time;

[0023] The S13 further includes:

[0024] When another rendering device is switched to connect with the client to provide rendering service for the client, the dispatch center records the service duration of the rendering device before switching to provide rendering service for the client, and stores the service duration as historical service duration in a preset historical service duration database.

[0025] Optionally, the method further comprises:

[0026] When the dispatch center dispatches any rendering device to connect to the user terminal to provide rendering services for the user terminal, the total running times preset by the rendering device are updated, that is, the total running times of the rendering device are increased by one;

[0027] When the dispatch center determines that the image rendered by any rendering device is unhealthy, the preset crash count of the rendering device is updated, that is, the crash count of the rendering device is increased by one;

[0028] Then, after S13, the step further includes:

[0029] S14. When the dispatch center determines that the images rendered by all the connected rendering devices are unhealthy, the frame rate stability of each rendering device is obtained based on the frame rates of all user images when each rendering device provides rendering services for the user terminal, the total number of operations corresponding to each rendering device, the number of crashes corresponding to each rendering device, and all historical service durations corresponding to each rendering device in the historical service duration database, as well as a preset formula 1; the formula 1 is:

[0030]

[0031] Wherein, S is the frame rate stability corresponding to the rendering device, AFR is the average frame rate when the rendering device provides rendering services to the user terminal, SD is the frame rate standard deviation when the rendering device provides rendering services to the user terminal, the average frame rate and the frame rate standard deviation are both obtained based on the frame rates of all user screens when the rendering device provides rendering services to the user terminal, ε is a preset minimum constant, HAD is the historical average service duration corresponding to the rendering device, the historical average service duration is obtained based on all historical service durations corresponding to the rendering device, MAX_DUR is the maximum running duration preset by the rendering device, k is a preset weight coefficient, C is the number of crashes of the rendering device, and T is the total number of runs of the rendering device;

[0032] Based on the frame rate stability corresponding to each rendering device, the dispatch center switches to the rendering device with the highest frame rate stability to connect with the user terminal and provide rendering services to the user terminal.

[0033] Optionally, the method further comprises:

[0034] The scheduling center schedules the connected rendering devices to restart in sequence based on the pre-set restart period;

[0035] During this period, if the dispatch center detects that any rendering device has been connected to the user terminal to provide rendering services to the user terminal, it will detect whether other rendering devices that are not connected to the user terminal have been restarted;

[0036] If so, the scheduling center switches another rendering device that has been restarted to connect to the user terminal to provide rendering services for the user terminal based on a preset exclusive scheduling algorithm, and schedules the rendering device before the switching to restart;

[0037] If not, the dispatch center schedules other rendering devices that are not connected to the user end to restart until it is detected that other rendering devices that are not connected to the user end have completed the restart.

[0038] Optionally, the method further comprises:

[0039] At preset intervals, the dispatch center refreshes the total number of runs and crashes corresponding to the rendering device to zero.

[0040] Optionally, the S13 further includes:

[0041] If it does not exist, the current rendering device is protected and monitored based on a preset daemon; the daemon is used to restart the rendering service when the rendering device crashes during the rendering service provided by the user terminal.

[0042] Optionally, S12 further includes:

[0043] The dispatch center obtains the device status of the current rendering device in real time, and adjusts the image health threshold in real time based on a preset dynamic threshold adjustment strategy and a preset image health threshold; the image health threshold includes a frame rate threshold;

[0044] The device status includes: GPU temperature, GPU usage, video memory usage, CPU usage, memory usage and disk usage;

[0045] The dynamic threshold adjustment strategy includes:

[0046] When the GPU temperature is greater than 80 degrees and less than 100 degrees, the first frame rate threshold increases by 1 for every 5 degrees increase; when the GPU temperature is greater than 100 degrees, the first frame rate threshold increases by 10 for every 5 degrees increase;

[0047] When the GPU usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase;

[0048] When the video memory usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase;

[0049] When the CPU usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase;

[0050] When the memory usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase;

[0051] When the disk usage is greater than 80%, the disk usage enters an alarm state. For every 5% increase, the first frame rate threshold increases by 0.5.

[0052] Optionally, each of the rendering devices comprises a camera lens;

[0053] Then the S12 further includes:

[0054] The dispatch center detects in real time the lens parameter information of the camera lens of the rendering device connected to the user terminal during the rendering process; the lens parameter information includes: lens position and lens rotation parameters;

[0055] The S13 further includes:

[0056] When another rendering device is switched to connect with the client to provide rendering service for the client, the dispatch center sends the latest lens parameter information of the rendering device before switching to the rendering device after switching to synchronize the lens parameter information of the rendering device after switching.

[0057] Optionally, S13 further includes: when switching another rendering device to connect to the user terminal to provide rendering service for the user terminal, the dispatch center uses the most recent cut-off frequency of the rendering device before the switching as the picture provided to the user terminal until the switching is completed, and hides the cut-off frequency.

[0058] (III) Beneficial effects

[0059] The beneficial effects of the present invention are as follows: a method for optimizing real-time rendering images of the present invention, since it deploys multiple rendering devices to provide streaming rendering services, can ensure stable output of streaming rendering images by switching rendering devices during the streaming rendering process, thereby reducing the impact of screen freezes or service crashes on user experience compared to the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flow chart of a method for optimizing a real-time rendering image provided by Embodiment 1 of the present invention;

[0061] Figure 2 A flowchart of a method for optimizing a real-time rendering image provided by Embodiment 2 of the present invention;

[0062] Figure 3A shot synchronization flow chart of a method for optimizing real-time rendering of images provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0063] In order to better explain the present invention and facilitate understanding, some words are explained below.

[0064] Crash Ratio (CR): The ratio of the number of crashes to the total number of runs within a period of time.

[0065] Historical Average Duration (HAD): indicates the historical average running time of the video stream. A longer value usually indicates a more stable system.

[0066] Maximum Possible Duration (MAX_DUR): This refers to the maximum operating duration in this article, which indicates the maximum duration or design duration that the system can run.

[0067] Exclusive scheduling algorithm: All current service information is collected through the client program deployed on each device. Initially, all services are in the scheduling state. Once a service is selected for scheduling, it will be marked as scheduled. The next time there is a scheduling demand, the service with the scheduling mark will be bypassed. Once the next service is scheduled, the previous service will automatically remove the scheduled mark.

[0068] The present invention will be described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0069] An optimization method for real-time rendering of images proposed in an embodiment of the present invention adopts the deployment of multiple rendering devices to provide streaming rendering services. Compared with the prior art, it can ensure the stable output of streaming rendering images by switching rendering devices during the streaming rendering process, thereby reducing the impact of image freezes or service crashes on user experience.

[0070] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0071] Example 1

[0072] An embodiment of the present invention provides a method for optimizing a real-time rendering picture. The method is implemented based on a real-time rendering system that provides streaming rendering services to users, wherein the system includes a dispatching center and at least one rendering device that is communicatively connected to the dispatching center. Generally, the dispatching center has at least two rendering devices under its jurisdiction to achieve real-time switching during the rendering process.

[0073] The method is as Figure 1 As shown, including:

[0074] S11, when the dispatch center receives a rendering request sent by a user terminal, it detects whether there is any rendering device connected to the user terminal to provide rendering services for the user terminal;

[0075] When it does not exist (that is, all rendering devices under the scheduling center do not provide rendering services to the user terminal), the scheduling center schedules any rendering device to connect to the user terminal to provide rendering services to the user terminal based on a preset exclusive scheduling algorithm.

[0076] S12, the dispatch center detects in real time the user picture frame rate of the rendering device connected to the user terminal during the rendering process, and determines whether the picture rendered by the current rendering device is healthy based on a preset picture health threshold; the user picture frame rate is the frame rate of the picture rendered by the rendering device;

[0077] When it is unhealthy, the dispatch center detects whether there is a rendering device among all the rendering devices connected to it that is not connected to the user end to provide rendering services. Specifically, it includes:

[0078] S12-1, the dispatch center detects the user screen frame rate of the rendering device connected to the user terminal in real time during the rendering process, and determines whether the user screen frame rate of the current rendering device during the rendering process is abnormal based on a preset frame rate threshold, and records the number of abnormal user screen frame rates;

[0079] During this period, when the user screen frame rate detected by the dispatch center is less than the frame rate threshold, the number of frame rate anomalies increases by one;

[0080] When the user screen frame rate detected by the dispatch center is greater than or equal to the frame rate threshold, the number of frame rate anomalies decreases by one;

[0081] Among them, the frame rate threshold is a dynamic threshold, which is adjusted in real time based on the dynamic threshold adjustment strategy, that is, the dispatch center obtains the device status of the current rendering device in real time, and adjusts the picture health threshold in real time based on the pre-set dynamic threshold adjustment strategy and the pre-set picture health threshold; the picture health threshold includes the frame rate threshold; the device status includes: GPU temperature, GPU usage, video memory usage, CPU usage, memory occupancy and disk occupancy; the dynamic threshold adjustment strategy includes:

[0082] When the GPU temperature is greater than 80 degrees and less than 100 degrees, the first frame rate threshold increases by 1 for every 5 degrees increase; when the GPU temperature is greater than 100 degrees, the first frame rate threshold increases by 10 for every 5 degrees increase;

[0083] When the GPU usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase;

[0084] When the video memory usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase;

[0085] When the CPU usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase;

[0086] When the memory usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase;

[0087] When the disk usage is greater than 80%, the disk usage enters an alarm state. For every 5% increase, the first frame rate threshold increases by 0.5.

[0088] S12-2, the dispatch center detects the number of abnormal frame rates of the user screen in real time, and determines whether the screen rendered by the current rendering device is healthy based on the frame rate abnormality threshold; the frame rate abnormality threshold is artificially set in advance based on actual conditions.

[0089] When the number of abnormal frame rates of the user screen is greater than or equal to the frame rate abnormal number threshold, it is determined that the screen rendered by the current rendering device is unhealthy.

[0090] While the current rendering device is providing rendering services to the user end, the dispatch center records the user screen frame rate of the rendering device in the rendering process in real time;

[0091] When the dispatch center dispatches any rendering device to connect to the user terminal to provide rendering services for the user terminal, the total running times preset by the rendering device are updated, that is, the total running times of the rendering device are increased by one;

[0092] When the dispatch center determines that the image rendered by any rendering device is unhealthy, the preset crash count of the rendering device is updated, that is, the crash count of the rendering device is increased by one; the total number of runs and crashes needs to be reset. Generally, the total number of runs and crashes are refreshed at preset time intervals. The preset time is a manually set non-service time, which can usually be reset at a fixed time every day. At this time, all rendering devices are not connected to the user end to provide rendering services.

[0093] S13. If yes, the scheduling center switches another rendering device to connect with the user terminal to provide rendering service for the user terminal based on a preset exclusive scheduling algorithm, and repeats steps S12 and S13.

[0094] If it does not exist, the current rendering device is protected and monitored based on a preset daemon; the daemon is used to restart the rendering service when the rendering device crashes during the rendering service provided by the user terminal, and jump to S12.

[0095] When another rendering device is switched to connect with the user terminal to provide rendering services for the user terminal, the dispatch center records the service duration of the rendering device before switching to provide rendering services for the user terminal, and stores the service duration as historical service duration in a preset historical service duration database.

[0096] S14. When the dispatch center determines that the images rendered by all the connected rendering devices are unhealthy, the frame rate stability of each rendering device is obtained based on the frame rates of all user images when each rendering device provides rendering services for the user terminal, the total number of operations corresponding to each rendering device, the number of crashes corresponding to each rendering device, and all historical service durations corresponding to each rendering device in the historical service duration database, as well as a preset formula 1; the formula 1 is:

[0097]

[0098] Wherein, S is the frame rate stability corresponding to the rendering device, AFR is the average frame rate when the rendering device provides rendering services to the user terminal, SD is the frame rate standard deviation when the rendering device provides rendering services to the user terminal, the average frame rate and the frame rate standard deviation are both obtained based on the frame rates of all user screens when the rendering device provides rendering services to the user terminal, ε is a preset minimum constant, HAD is the historical average service duration corresponding to the rendering device, the historical average service duration is obtained based on all historical service durations corresponding to the rendering device, MAX_DUR is the maximum running duration preset by the rendering device, k is a preset weight coefficient, C is the number of crashes of the rendering device, and T is the total number of runs of the rendering device;

[0099] Based on the frame rate stability of each rendering device, the dispatch center switches to the rendering device with the highest frame rate stability to connect to the user terminal and provide rendering services to the user terminal. That is, the rendering device with the best current rendering effect is selected by scoring to provide rendering services to the user terminal.

[0100] Due to the particularity of streaming rendering services, if the unified service runs continuously for 24 hours a day, the stability and risks may be greatly improved; the daily time periods can be divided into high-frequency usage periods and low-frequency usage periods. For example, 6:00-24:00 is the high-frequency usage period, and 0:00-24:00 is the low-frequency usage period. It is generally believed that all rendering devices do not provide rendering services during the low-frequency usage period. During the low-frequency period, one machine will be automatically selected to shut down the service every hour, and it will automatically restart after one hour. Then, in the next hour, other machines will be selected in turn to shut down automatically until all machines have completed the restart (if the number of machines is large, the segment time can be adjusted to half an hour). That is, based on the pre-set restart period, the scheduling center will schedule the connected rendering devices to restart or restart the rendering services of the rendering devices in sequence;

[0101] During this period, if the dispatch center detects that any rendering device has been connected to the user terminal to provide rendering services to the user terminal, it will detect whether other rendering devices that are not connected to the user terminal have been restarted;

[0102] If so, the scheduling center switches another rendering device that has been restarted to connect to the user terminal to provide rendering services for the user terminal based on a preset exclusive scheduling algorithm, and schedules the rendering device before the switching to restart;

[0103] If not, the dispatch center schedules other rendering devices that are not connected to the user end to restart until it is detected that other rendering devices that are not connected to the user end have completed the restart.

[0104] When the rendering device is switched, the lens of the rendering device before and after the switch needs to be synchronized, so lens parameter information is needed, that is, the dispatch center detects the lens parameter information of the camera lens of the rendering device connected to the user terminal in real time during the rendering process; the lens parameter information includes: lens position and lens rotation parameters. When another rendering device is switched to connect to the user terminal to provide rendering services for the user terminal, the dispatch center sends the latest lens parameter information of the rendering device before the switch to the rendering device after the switch to synchronize the lens parameter information of the rendering device after the switch.

[0105] When another rendering device is switched to connect with the client to provide rendering services for the client, the dispatch center uses the most recent cut-off frequency of the rendering device before the switch as the picture provided to the client until the switch is completed and hides the cut-off frequency.

[0106] This real-time example provides an optimization method for real-time rendering images. During the streaming rendering process, the rendering device is switched in real time based on the user frame rate of the current image to ensure the stability of the streaming rendering image. When all rendering devices are unable to provide high-quality rendering services, the rendering status of each rendering device is scored, and the most stable rendering device under the current conditions is selected to provide rendering services to the user end, thereby improving the user experience.

[0107] Example 2

[0108] This embodiment provides a method for optimizing real-time rendering images, and its flowchart is as follows: Figure 2 As shown, including:

[0109] The method is implemented based on the real-time rendering system described in Example 1. Usually, the system includes two modes, namely, an automatic switching mode and a manual switching mode, which can be selected manually.

[0110] When in automatic switching mode and receiving a rendering request from the client, the dispatch center will detect whether the rendering service has started. If not, it will select any rendering device to connect to the client for rendering service. During the service, the dispatch center will detect in real time whether the rendering service provided by the rendering device to the client is disconnected. Once disconnected, it will immediately switch to the service of another rendering device through the exclusive scheduling algorithm.

[0111] In the process of providing streaming rendering services to the user end, if the streaming rendering service is disconnected, it is certain that the real-time rendering screen cannot be provided. However, the health of the service does not mean that the screen seen by the user is stable. It is possible that although the service is healthy, the continuously output screen is very stuck and the frame rate is also very low, which has seriously affected the normal use. At this time, the real-time rendering screen is also considered unstable. Therefore, the dispatch center needs to detect the current user screen frame rate in real time when the rendering device provides rendering services to the user end. And the pre-set screen health threshold is used to determine whether the screen rendered by the current rendering device is healthy.

[0112] The image health threshold includes the frame rate threshold and the frame rate abnormality threshold. The frame rate threshold is the lower limit of the unhealthy frame rate value.

[0113] The dispatch center uses polling to detect the user's screen frame rate of the current screen in real time. When the frame rate is less than the frame rate threshold, it is recorded, that is, the number of frame rate anomalies increases by one. When the frame rate is greater than or equal to the frame rate threshold, it is also recorded, that is, the number of frame rate anomalies decreases by one. This is done to eliminate the impact of a single anomaly. When the rendering screen is running, there may indeed be a moment of freeze, which does not mean that the user really wants to switch services.

[0114] Determine whether the number of frame rate anomalies exceeds the frame rate anomaly threshold. If so, the image is considered to be continuously unhealthy and needs to be switched to another rendering device.

[0115] If there are some extreme cases, such as the services of two rendering devices are judged to be unhealthy, it is still necessary to select a relatively more stable one, so it is necessary to judge the frame rate stability of each rendering device.

[0116] To determine the frame rate stability of each Xu Ran device, it is necessary to obtain the average frame rate and frame rate standard deviation during rendering service. The average frame rate is based on the user screen frame rate obtained in real time during the rendering process, which is obtained through the following formula:

[0117]

[0118] Where N is the total duration of the time window / the amount of user screen frame rate collected, and Fi is the user screen frame rate;

[0119] The frame rate standard deviation is based on the user screen frame rate obtained in real time during the rendering process and the average power, that is:

[0120]

[0121] Based on the preset formula 1, the frame rate stability corresponding to each rendering device is obtained; the formula 1 is:

[0122]

[0123] Wherein, S is the frame rate stability corresponding to the rendering device, AFR is the average frame rate when the rendering device provides rendering services to the user terminal, SD is the frame rate standard deviation when the rendering device provides rendering services to the user terminal, and the average frame rate and the frame rate standard deviation are both obtained based on the frame rates of all user screens when the rendering device provides rendering services to the user terminal; ε is a preset minimum constant (usually set to a positive number, such as 0.001, to avoid the denominator being 0); HAD is the historical average service duration corresponding to the rendering device, and the historical average service duration is obtained based on all historical service durations corresponding to the rendering device; MAX_DUR is the maximum running duration preset for the rendering device; k is a preset weight coefficient, which is used to adjust the impact of the crash ratio on the stability score and can be set according to requirements; C is the number of crashes of the rendering device; T is the total number of runs of the rendering device; This part is used to evaluate the frame rate stability of the video. The higher the average frame rate and the smaller its standard deviation (fluctuation), the higher the score, which means the video is smoother. The part is used to evaluate the duration of the video stream. The closer the ratio is to 1, the more stable the system is, the longer the usage time is, and the higher the score is. The -k×CR part represents the negative impact of each unit crash ratio on stability. The higher the crash ratio, the lower the score is, because it indicates the instability of the system.

[0124] For the sake of convenience, let's take an example and get a set of collected frame rate data: [25, 30, 28, 27, 26], and calculate the frame rate stability of the rendering device; that is,

[0125] AFR=(25+30+28+27+26) / 5=27.2FPS;

[0126]

[0127] Count the number of crashes 2 and the total number of runs 10, set k to 5, then -k×CR=-5×2 / 10=-1;

[0128] The historical average service time is 5 days, and the designed maximum service time is 7 days.

[0129]

[0130] but,

[0131] The frame rate threshold can generally be changed dynamically according to the device status of the rendering device, that is, an initial default value of the frame rate threshold is set; the frame rate threshold is adjusted in real time according to the real-time acquisition of the current device status of the rendering device and the pre-set dynamic threshold adjustment strategy.

[0132] That is, when the GPU temperature is greater than 80 degrees, it starts to reach the alarm state, with 5 degrees as one level, and the threshold value increases by 1 for each increase; when it reaches 100 degrees, the threshold value increases by 10 for each increase (at this time, it is considered that there is a risk of hardware damage and needs to be switched immediately);

[0133] GPU usage rate: When the usage rate exceeds 80%, the alarm state begins to be reached. 5% is the first level, and the threshold is increased by 1 for each additional level.

[0134] Video memory usage: When the usage exceeds 80%, the alarm state begins to be reached. 5% is the first level, and the threshold is increased by 1 for each additional level.

[0135] CPU usage: When the CPU usage exceeds 80%, an alarm is triggered, with the threshold value +0.5;

[0136] Memory usage: When the disk usage exceeds 80%, an alarm is triggered, with the threshold value +0.5;

[0137] Disk usage: When the disk usage exceeds 80%, an alarm is triggered, with the threshold value + 0.5.

[0138] For example, the default value of the frame rate threshold is set to 20FPS;

[0139] At a certain moment, the monitoring data is: GPU temperature 88 degrees, alarm, threshold increased by 2, to 22; GPU utilization rate 90%, alarm, threshold increased by 2 (85%~90%), to 24; video memory utilization rate 75%, no change; CPU utilization rate 70%, no change; memory occupancy rate 85%, alarm, threshold increased by 0.5, to 24.5; disk occupancy rate 60%, no change; the final calculated threshold is 24.5 frames.

[0140] The abnormal frame rate threshold is divided into three levels, namely 3, 5 or 7, which can be set based on actual needs.

[0141] The above is an automatic switching mode, which will automatically judge based on the frame rate. However, whether the picture is stable is also subject to human subjectivity. It is possible that the frame rate collected by the service is just stuck near the threshold, sometimes stuck and sometimes smooth, resulting in the cumulative count of unhealthy frame rates always failing to reach the set threshold. However, at this time, it is necessary to manually judge whether it is necessary to switch, and a manual forced switching method is also provided.

[0142] Due to the particularity of the streaming rendering service, if the same service runs continuously for 24 hours a day, the risk of stability may increase significantly; currently, the daily time periods are divided into high-frequency usage periods and low-frequency usage periods, for example: 6:00-24:00 is the high-frequency usage period, and the night from 24:00 to 6:00 is the low-frequency usage period; in the low-frequency period, one machine will be automatically selected to shut down the service every hour, and it will automatically restart after one hour, and then in the next hour, other machines will be selected in turn to shut down automatically until all machines have completed the restart (if there are a large number of machines, the segment time can be adjusted to half an hour).

[0143] When there is only one rendering device, you need to rely on the daemon that comes with the service itself; the daemon will monitor the process of the UE program, and when the process crashes, the daemon will restart the service.

[0144] When multiple rendering devices are switched, in order to ensure the consistency of the displayed content (consistency of lens parameters), the camera lens information needs to be synchronized. The lens synchronization process diagram is as follows: Figure 3 shown.

[0145] The service currently being rendered by the actual scheduler saves the lens parameter information of the camera lens every 1 second. Once a service switch occurs, the newly scheduled service will read the current lens parameter information to avoid a large difference in the画面 between two different services. In theory, the last frame of the previous video stream service is the lens of the first frame of the new service after the switch.

[0146] The lens parameter information includes the lens position and the lens rotation parameters. The lens position is the X, Y, and Z coordinates of the lens, and the lens rendering parameters are the axis rotation parameters of X, Z, and Z; Roll represents the X-axis (generally defaulting to zero without modification), Pitch represents the Y-axis (-90 < Y < 90), and Yaw represents the Z-axis (-180 to 180).

[0147] There will be a brief black screen phenomenon between disconnecting the previous connection and the completion of the next connection and the display of the画面. During this period, to prevent the black screen, the screenshot of the previous moment is used as the background for loading. When the new connection switch is completed, the screenshot is hidden. Therefore, it is necessary to set the frequency of taking screenshots of the stream-rendered画面 every 1 second.

[0148] An optimization method for real-time rendering of the画面 provided by an embodiment of the present invention performs real-time switching of the rendering device based on the user's画面 frame rate of the current画面 during the stream rendering process to ensure the stability of the stream-rendered画面. When all rendering devices are unable to provide high-quality rendering services, the rendering situation of each rendering device is scored, and the most stable rendering device in the current situation is selected to provide rendering services for the user side, improving the user experience.

[0149] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality" means two or more, unless otherwise specifically defined.

[0150] In the present invention, unless otherwise clearly defined and limited, the terms "installation", "connection", "connection", "fixation", etc. should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or integrated; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal communication of two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0151] In the present invention, unless otherwise clearly specified and limited, when a first feature is “on” or “below” a second feature, it may be that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Moreover, when a first feature is “above”, “above” or “above” a second feature, it may be that the first feature is directly above or obliquely above the second feature, or it may simply mean that the first feature is higher in level than the second feature. When a first feature is “below”, “below” or “below” a second feature, it may be that the first feature is directly below or obliquely below the second feature, or it may simply mean that the first feature is lower in level than the second feature.

[0152] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0153] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for optimizing a real-time rendering image, characterized in that: The method is implemented based on a pre-set real-time rendering system, the system includes a dispatch center, and at least one rendering device connected to the dispatch center, and the method includes: S11, when the dispatch center receives a rendering request sent by a user terminal, it detects whether there is any rendering device connected to the user terminal to provide rendering services for the user terminal; If it does not exist, the scheduling center schedules any rendering device to connect to the user terminal to provide rendering services for the user terminal based on a preset exclusive scheduling algorithm; S12, the dispatch center detects in real time the user picture frame rate of the rendering device connected to the user terminal during the rendering process, and determines whether the picture rendered by the current rendering device is healthy based on a preset picture health threshold; the user picture frame rate is the frame rate of the picture rendered by the rendering device; If it is unhealthy, the dispatch center detects whether there is a rendering device that is not connected to the user terminal to provide rendering services among all the rendering devices connected to it; S13. If yes, the scheduling center switches another rendering device to connect with the user terminal to provide rendering service for the user terminal based on a preset exclusive scheduling algorithm, and repeats steps S12 and S13.

2. The method for optimizing real-time rendering images according to claim 1, characterized in that: The picture health threshold includes a frame rate threshold and a frame rate abnormality times threshold; Then, the S12 includes: S12-1, the dispatch center detects the user screen frame rate of the rendering device connected to the user terminal in real time during the rendering process, and determines whether the user screen frame rate of the current rendering device during the rendering process is abnormal based on a preset frame rate threshold, and records the number of abnormal user screen frame rates; During this period, when the user screen frame rate detected by the dispatch center is less than the frame rate threshold, the number of frame rate anomalies increases by one; When the user screen frame rate detected by the dispatch center is greater than or equal to the frame rate threshold, the number of frame rate anomalies decreases by one; S12-2, the dispatch center detects the number of abnormal frame rates of the user screen in real time, and determines whether the screen rendered by the current rendering device is healthy based on the frame rate abnormality number threshold; When the number of abnormal frame rates of the user screen is greater than or equal to the frame rate abnormal number threshold, it is determined that the screen rendered by the current rendering device is unhealthy.

3. The method for optimizing real-time rendering images according to claim 1, characterized in that: The S12 further includes: While the current rendering device provides rendering services to the user end, the dispatch center records the user screen frame rate of the rendering device in the rendering process in real time; The S13 further includes: When another rendering device is switched to connect with the client to provide rendering service for the client, the dispatch center records the service duration of the rendering device before switching to provide rendering service for the client, and stores the service duration as historical service duration in a preset historical service duration database.

4. The method for optimizing real-time rendering images according to claim 3, characterized in that: The method further comprises: When the dispatch center dispatches any rendering device to connect to the user terminal to provide rendering services for the user terminal, the total running times preset by the rendering device are updated, that is, the total running times of the rendering device are increased by one; When the dispatch center determines that the image rendered by any rendering device is unhealthy, the preset crash count of the rendering device is updated, that is, the crash count of the rendering device is increased by one; Then, after S13, the step further includes: S14. When the dispatch center determines that the images rendered by all the connected rendering devices are unhealthy, the frame rate stability of each rendering device is obtained based on the frame rates of all user images when each rendering device provides rendering services for the user terminal, the total number of operations corresponding to each rendering device, the number of crashes corresponding to each rendering device, and all historical service durations corresponding to each rendering device in the historical service duration database, as well as a preset formula 1; the formula 1 is: Wherein, S is the frame rate stability corresponding to the rendering device, AFR is the average frame rate when the rendering device provides rendering services to the user terminal, SD is the frame rate standard deviation when the rendering device provides rendering services to the user terminal, the average frame rate and the frame rate standard deviation are both obtained based on the frame rates of all user screens when the rendering device provides rendering services to the user terminal, ε is a preset minimum constant, HAD is the historical average service duration corresponding to the rendering device, the historical average service duration is obtained based on all historical service durations corresponding to the rendering device, MAX_DUR is the maximum running duration preset by the rendering device, k is a preset weight coefficient, C is the number of crashes of the rendering device, and T is the total number of runs of the rendering device; Based on the frame rate stability corresponding to each rendering device, the dispatch center switches to the rendering device with the highest frame rate stability to connect with the user terminal and provide rendering services to the user terminal.

5. The method for optimizing real-time rendering images according to claim 1, characterized in that: The method further comprises: The scheduling center schedules the connected rendering devices to restart in sequence based on the pre-set restart period; During this period, if the dispatch center detects that any rendering device has been connected to the user terminal to provide rendering services to the user terminal, it will detect whether other rendering devices that are not connected to the user terminal have been restarted; If so, the scheduling center switches another rendering device that has been restarted to connect to the user terminal to provide rendering services for the user terminal based on a preset exclusive scheduling algorithm, and schedules the rendering device before the switching to restart; If not, the dispatch center schedules other rendering devices that are not connected to the user end to restart until it is detected that other rendering devices that are not connected to the user end have completed the restart.

6. The method for optimizing real-time rendering images according to claim 4, characterized in that: The method further comprises: At preset intervals, the dispatch center refreshes the total number of runs and crashes corresponding to the rendering device to zero.

7. The method for optimizing real-time rendering images according to claim 1, characterized in that: The S13 further includes: If it does not exist, the current rendering device is protected and monitored based on a preset daemon; the daemon is used to restart the rendering service when the rendering device crashes during the rendering service provided by the user terminal.

8. The method for optimizing real-time rendering images according to claim 1, characterized in that: The S12 also includes: The dispatch center obtains the device status of the current rendering device in real time, and adjusts the image health threshold in real time based on a preset dynamic threshold adjustment strategy and a preset image health threshold; the image health threshold includes a frame rate threshold; The device status includes: GPU temperature, GPU usage, video memory usage, CPU usage, memory usage and disk usage; The dynamic threshold adjustment strategy includes: When the GPU temperature is greater than 80 degrees and less than 100 degrees, the first frame rate threshold increases by 1 for every 5 degrees increase; when the GPU temperature is greater than 100 degrees, the first frame rate threshold increases by 10 for every 5 degrees increase; When the GPU usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase; When the video memory usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase; When the CPU usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase; When the memory usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase; When the disk usage is greater than 80%, the disk usage enters an alarm state. For every 5% increase, the first frame rate threshold increases by 0.

5.

9. The method for optimizing real-time rendering images according to claim 1, characterized in that: Each of the rendering devices includes a camera lens; Then the S12 further includes: The dispatch center detects in real time the lens parameter information of the camera lens of the rendering device connected to the user terminal during the rendering process; the lens parameter information includes: lens position and lens rotation parameters; The S13 further includes: When another rendering device is switched to connect with the client to provide rendering service for the client, the dispatch center sends the latest lens parameter information of the rendering device before switching to the rendering device after switching to synchronize the lens parameter information of the rendering device after switching.

10. The method for optimizing real-time rendering images according to claim 9, characterized in that: The S13 also includes: when another rendering device is switched to connect with the user terminal to provide rendering service for the user terminal, the dispatch center uses the most recent cut-off frequency of the rendering device before the switch as the picture provided to the user terminal until the switch is completed, and hides the cut-off frequency.

Images