Frame loss prediction method and electronic equipment

Abstract

The invention provides a frame loss prediction method and electronic equipment. The method comprises the steps that a game application issues a rendering instruction of an Nth frame; performing feature matching on the rendering instruction based on the identification strategy; if the first feature is matched, frame loss prediction is carried out on the Nth frame based on the moment when the Nth frame is matched with the first feature, the moment when the historical frame is matched with the first feature and the moment when the historical frame is matched with the end of the historical frame, and the historical frame is at least a part of frames before the Nth frame. According to the invention, targeted prediction can be carried out on the frame loss based on the frame dimension, and the accuracy of frame loss prediction is improved.

Description

[0001] This application is a divisional application of the invention patent application with application number 202310878100.3 submitted to the China Patent Office, application date July 18, 2023, and invention name “Frame loss prediction method and electronic device”. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a frame loss prediction method and electronic device. Background Art

[0003] Game rendering is a process of visualization and presentation. Game rendering can make the game screen more beautiful and more realistic. In addition, in order to pursue real-time interactivity, the game screen is rendered in real time. At present, in order to provide players with high-quality game screens and make players have a stronger sense of immersion and better user experience in game operations, game rendering is becoming more and more complex, and the performance overhead of electronic devices when performing game rendering is also increasing.

[0004] Since the game screen is rendered in real time, that is, each frame needs to be rendered in real time during the game, and since different electronic devices are affected by factors such as chips and systems, there are performance differences between different electronic devices. In addition, complex game rendering increases the performance overhead of electronic devices. Therefore, frame loss is inevitable during the game.

[0005] Under normal circumstances, electronic device systems track and predict game loads to perform macro-dimensional system resource scheduling to improve the adverse effects of frame loss. Since frame loss is a frame-dimensional problem, and macro-dimensional game load tracking and prediction cannot make targeted predictions of frame loss, this method cannot accurately predict frame loss. Summary of the invention

[0006] The embodiments of the present application provide a frame loss prediction method, an electronic device, a computer-readable storage medium, a computer program product, and a chip system, which can not only perform targeted prediction of frame loss based on the frame dimension, but also improve the accuracy of frame loss prediction.

[0007] In a first aspect, an embodiment of the present application provides a frame loss prediction method, comprising: intercepting a rendering instruction issued by an application based on the Nth frame; in the case of each time a rendering instruction is intercepted, performing a feature match on at least one of the rendering instructions that has been intercepted through at least one recognition strategy; wherein the at least one recognition strategy corresponds to at least one feature one-to-one; in the case of matching a first feature, predicting a frame loss for the Nth frame according to the moment when the first feature is matched, the moment when the first feature is matched in a historical frame, and the moment when the historical frame ends; wherein the at least one feature includes the first feature, and before this matching, the first feature has not been matched in the Nth frame, and the historical frame is at least a part of the frames before the Nth frame; any one of the at least one feature appears and only appears once in different frames of the application, and the first duration of the candidate feature corresponding to multiple frames of the application including the Nth frame conforms to a normal distribution; the candidate feature is any one of the at least one feature, and the first duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame of the multiple frames and the moment when the any frame ends.

[0008] As can be seen from the above, by intercepting the rendering instructions issued by the application based on the Nth frame, and in the case of each interception of a rendering instruction, at least one of the intercepted rendering instructions is matched with features through at least one recognition strategy, and in the case of matching the first feature, according to the time of matching the first feature, the time of matching the first feature in the historical frame and the time of matching the end of the historical frame, the Nth frame is predicted to be lost, which provides a method for targeted prediction of frame loss based on the frame dimension. In addition, since the candidate feature is any one of the at least one feature, the first duration of the candidate feature corresponding to the multiple frames of the application including the Nth frame conforms to the normal distribution, and the first duration of the candidate feature is the duration between the time when the candidate feature is matched in any frame of the multiple frames to the time when the arbitrary frame is matched. Therefore, during the running of the application, when the first duration of the candidate feature corresponding to one frame of the application is known, the first duration of the candidate feature corresponding to another frame of the application can be inferred based on the first duration of the candidate feature corresponding to the frame, and the accuracy of the inferred first duration of the candidate feature corresponding to another frame of the application is high. In this way, in the process of predicting the frame loss of the Nth frame according to the moment of matching the first feature, the moment of matching the first feature in the historical frame, and the moment of matching the end of the historical frame, the first duration of the first feature corresponding to the historical frame can be determined according to the moment of matching the first feature in the historical frame and the moment of matching the end of the historical frame. In combination with the above, it can be known that the first duration corresponding to the first feature corresponding to the historical frame can be used as the first duration of the first feature corresponding to the estimated Nth frame, and the accuracy of the first duration of the estimated Nth frame is high. Based on this, the rendering end time of the Nth frame can be determined according to the moment of matching the first feature and the first duration of the first feature corresponding to the historical frame, and the accuracy of the determined rendering end time of the Nth frame is high. In this way, the frame loss prediction of the Nth frame can be performed according to the rendering end time of the Nth frame, thereby improving the accuracy of the frame loss prediction. In addition, since any one of the at least one feature appears in different frames of the application and only appears once, and at least one feature includes the first feature, it can ensure that the first feature is matched in the Nth frame, thereby ensuring the reliability of the frame loss prediction method provided by the present application.

[0009] In a possible implementation, the frame loss prediction for the Nth frame based on the moment when the first feature is matched, the moment when the first feature is matched in the historical frame, and the moment when the historical frame is matched to the end includes: determining the first duration of the first feature corresponding to the historical frame based on the moment when the first feature is matched in the historical frame and the moment when the historical frame is matched to the end; determining the rendering end moment of the Nth frame based on the moment when the first feature is matched and the first duration of the first feature corresponding to the historical frame; and predicting frame loss for the Nth frame based on the rendering end moment and the moment when synthesis of the Nth frame starts.

[0010] In a possible implementation, the frame loss prediction for the Nth frame based on the rendering end time and the start time of synthesizing the Nth frame includes: determining the timeout ratio according to the start time of synthesizing the Nth frame, the rendering end time, and the first duration of the first feature corresponding to the historical frame; and predicting the frame loss for the Nth frame based on the relative size of the timeout ratio and a preset ratio.

[0011] In a possible implementation, when the first feature is matched, predicting frame loss for the Nth frame based on the moment when the first feature is matched, the moment when the first feature is matched in the historical frame, and the moment when the historical frame ends, includes: when the first feature is matched and the current state of the Nth frame is that frame loss is not predicted, predicting frame loss for the Nth frame based on the moment when the first feature is matched, the moment when the first feature is matched in the historical frame, and the moment when the historical frame ends.

[0012] In a possible implementation, the number of the at least one feature is multiple; the method also includes: obtaining a first latest response time each time a rendering instruction is intercepted; when the first latest response time is obtained, performing frame loss prediction on the Nth frame according to the time when the rendering instruction is intercepted and the first latest response time obtained; wherein the first latest response time obtained refers to the time when the latest execution of the rendering instruction corresponding to the second feature in the Nth frame is allowed to start; the first latest response time obtained is updated according to the time when the second feature is matched in the historical frame, the time when the historical frame ends, the time when the synthesis of the Nth frame starts, and the first time; the at least one feature includes the second feature, and the second feature is the first feature that was matched in the Nth frame most recently before the moment when the rendering instruction was intercepted; when the second feature is not the last one matched in the Nth frame, the first time is the time when the second feature is matched in the Nth frame. the moment when the first feature is matched in the historical frame; when the second feature is the last one to be matched in the Nth frame, the first moment is the moment when the matching to the historical frame ends; for any two features of the at least one feature, the order of the moments when the any two features are matched in any frame of the application is fixed; the second duration of the candidate feature corresponding to multiple frames of the application including the Nth frame conforms to the normal distribution; when the candidate feature is not the last one to be matched in any frame of the multiple frames, the second duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame and the moment when the next adjacent feature is matched in any frame; when the candidate feature is the last one to be matched in any frame of the multiple frames, the second duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame and the moment when the matching to the any frame ends.

[0013] As can be seen from the above, by intercepting the rendering instructions issued by the application based on the Nth frame, and obtaining the first latest response time each time a rendering instruction is intercepted, and when the first latest response time is obtained, the Nth frame is predicted to be lost according to the moment of intercepting the rendering instruction and the first latest response time obtained, a method for predicting frame loss based on the frame dimension is provided. In addition, since the candidate feature is any one of the at least one feature, the first duration and the second duration of the candidate feature corresponding to the multiple frames of the application including the Nth frame are both in accordance with the normal distribution, and the first duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame and the moment when the arbitrary frame is matched. In the case where the candidate feature is not the last one to be matched in any frame among the multiple frames, the second duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame and the moment when the next adjacent feature is matched in the arbitrary frame. In the case where the candidate feature is the last one to be matched in any frame among the multiple frames, the second duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame and the moment when the arbitrary frame is matched. Therefore, during the running of the application, when the first duration and the second duration of the candidate feature corresponding to one frame of the application are known, the first duration and the second duration of the candidate feature corresponding to another frame of the application can be inferred based on the first duration and the second duration of the candidate feature corresponding to the frame, and the accuracy of the inferred first duration and the second duration of the candidate feature corresponding to another frame of the application is high. In this way, the first latest response moment obtained is updated based on the moment when the second feature is matched in the historical frame, the moment when the historical frame ends, the moment when the Nth frame is started to be synthesized, and the first moment. In other words, in the process of updating the first latest response moment obtained, the first duration of the second feature corresponding to the historical frame can be determined based on the moment when the second feature is matched in the historical frame and the moment when the historical frame ends, the second duration of the second feature corresponding to the historical frame can be determined based on the moment when the second feature is matched in the historical frame and the first moment, and then the first latest response moment obtained is updated based on the moment when the Nth frame starts to be synthesized, the first duration and the second duration of the second feature corresponding to the historical frame. In combination with the above, it can be seen that the first duration and the second duration of the second feature corresponding to the historical frame can be respectively used as the first duration and the second duration of the second feature corresponding to the estimated N-th frame, and the accuracy of the first duration and the second duration of the second feature corresponding to the estimated N-th frame is high, thereby improving the accuracy of the updated first latest response moment, thereby making the accuracy of the obtained first latest response moment high, thereby improving the accuracy of frame loss prediction.In addition, since any one of the at least one feature appears and only appears once in different frames of the application, for any two of the at least one feature, the order in which the two features are matched in any frame of the application is fixed, and since at least one feature includes the second feature, the reliability of updating the first latest response moment obtained is ensured, thereby ensuring the reliability of frame loss prediction.

[0014] In a possible implementation, obtaining the first latest response time each time one of the rendering instructions is intercepted includes: obtaining the first latest response time each time one of the rendering instructions is intercepted and the current state of the Nth frame is that no frame loss is predicted.

[0015] In a possible implementation, the initial value of the first latest response time is empty; the method also includes: in the case of matching the first feature, updating the first latest response time according to the time when the first feature is matched in the historical frame, the time when the historical frame ends, the time when the synthesis of the Nth frame starts, and the second time; wherein, in the case that the first feature is not the last one matched in the historical frame, the second time is the time when the third feature is matched in the historical frame; the at least one feature includes the third feature, and the third feature is a feature matched after the first feature and adjacent to the first feature in the feature matching of the historical frame; in the case that the first feature is the last one matched in the historical frame, the second time is the time when the historical frame ends; the updated first latest response time refers to the time when the rendering instruction corresponding to the first feature in the Nth frame is allowed to start executing at the latest.

[0016] As can be seen from the above, in the case of matching the first feature, the first latest response time is updated according to the time when the first feature is matched in the historical frame, the time when the historical frame ends, the time when the synthesis of the Nth frame begins, and the second time, providing a way to update the first latest response time. In addition, since the candidate feature is any one of the at least one feature, the first duration and the second duration of the candidate feature corresponding to the multiple frames of the application including the Nth frame are both in accordance with the normal distribution, and the first duration of the candidate feature is the duration between the time when the candidate feature is matched in any frame and the time when the arbitrary frame ends. In the case where the candidate feature is not the last one to be matched in any frame among the multiple frames, the second duration of the candidate feature is the duration between the time when the candidate feature is matched in any frame and the time when the next adjacent feature is matched in the arbitrary frame. In the case where the candidate feature is the last one to be matched in any frame among the multiple frames, the second duration of the candidate feature is the duration between the time when the candidate feature is matched in any frame and the time when the arbitrary frame ends. Therefore, during the running of the application, when the first duration and second duration of the candidate feature corresponding to one frame of the application are known, the first duration and second duration of the candidate feature corresponding to another frame of the application can be inferred based on the first duration and second duration of the candidate feature corresponding to the frame, and the accuracy of the inferred first duration and second duration of the candidate feature corresponding to the other frame of the application is high. In this way, in the process of updating the first latest response time based on the moment when the first feature is matched in the historical frame, the moment when the historical frame ends, the moment when the Nth frame starts to be synthesized, and the second moment, the first duration of the first feature corresponding to the historical frame can be determined based on the moment when the first feature is matched in the historical frame and the moment when the historical frame ends, the second duration of the first feature corresponding to the historical frame can be determined based on the moment when the first feature is matched in the historical frame and the second moment, and then the first latest response time is updated based on the moment when the Nth frame starts to be synthesized, the first duration and second duration of the first feature corresponding to the historical frame. Furthermore, as can be seen from the above, the first duration and the second duration of the first feature corresponding to the historical frame can be used as the first duration and the second duration of the first feature corresponding to the estimated Nth frame, respectively, and the accuracy of the first duration and the second duration of the first feature corresponding to the estimated Nth frame is high, thereby improving the accuracy of updating the first latest response time. In addition, since any one of the at least one feature appears and only appears once in different frames of the application, for any two of the at least one feature, the order of the moments when any two features are matched in any frame of the application is fixed, and since at least one feature includes the first feature, it is ensured that the first feature can be matched in the Nth frame, and it is ensured that the order of appearance of the first feature and other features of the at least one feature in each frame is fixed, thereby ensuring the reliability of updating the first latest response time.

[0017] In a possible implementation, in the case where the first feature is matched, updating the first latest response time according to the moment when the first feature is matched in the historical frame, the moment when the historical frame ends, the moment when synthesis of the Nth frame starts, and the second moment includes: in the case where it is predicted that there is no frame loss in the Nth frame according to the moment when the first feature is matched, the moment when the first feature is matched in the historical frame, and the moment when the historical frame ends, updating the first latest response time according to the moment when the first feature is matched in the historical frame, the moment when the historical frame ends, the moment when synthesis of the Nth frame starts, and the second moment.

[0018] In a possible implementation, the method further includes: when matching the first feature, obtaining the Vsync signal time and the number of frame accumulations; determining the time to start synthesizing the Nth frame based on the obtained Vsync signal time, the obtained number of frame accumulations and the Vsync signal period.

[0019] In a possible implementation, the method further includes: monitoring the Vsync signal time and the frame accumulation number; and updating the Vsync signal time and / or the frame accumulation number when the Vsync signal time and / or the frame accumulation number change.

[0020] In a possible implementation, the initial state of the Nth frame is that frame loss is not predicted; the method further includes: when frame loss is predicted in the Nth frame, configuring the current state of the Nth frame as predicted frame loss.

[0021] In a possible implementation manner, the method further includes: when the first feature is matched, recording a time when the first feature is matched as a time when the first feature is matched in the Nth frame.

[0022] In a possible implementation, the method further includes: intercepting and matching the end-of-frame rendering instruction of the Nth frame, and, when the end-of-frame rendering instruction of the Nth frame is matched, recording the moment when the end-of-frame rendering instruction of the Nth frame is matched as the moment when the Nth frame ends.

[0023] In a possible implementation, when there are multiple at least one feature, for any two of the at least one feature, the moments when the two features are matched in any frame of the application are different.

[0024] Since any two of at least one feature are matched at different times in any frame of the application, that is, in the matching process of any frame of the application, the situation where multiple first features are matched at the same time is avoided, and thus the situation where frame loss prediction is performed for the Nth frame based on multiple first features when multiple first features are matched at the same time is avoided. This ensures the accuracy of the prediction and reduces the number of frame loss predictions.

[0025] In a second aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory; wherein the processor and the memory are coupled, and the memory is used to store a computer program, and when the computer program is executed by the processor, the electronic device executes any one of the methods described in the first aspect.

[0026] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes any one of the methods described in the first aspect.

[0027] In a fourth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program code. When the computer program code is executed by a processor, the processor executes any one of the methods described in the first aspect.

[0028] In a fifth aspect, an embodiment of the present application provides a chip system, which includes a processor and a communication interface; the processor is used to call and run a computer program stored in the storage medium from a storage medium to execute any method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The schematic diagram of the frame display process is shown in FIG. Figure 1 ;

[0030] Figure 2 The schematic diagram of the frame display process is shown in FIG. Figure 2 ;

[0031] Figure 3 A schematic diagram of a frame display process based on game load tracking and prediction is shown;

[0032] Figure 4 Shows the frame display process based on the boost mechanism Figure 1 ;

[0033] Figure 5 Shows the frame display process based on the boost mechanism Figure 2 ;

[0034] Figure 6 A schematic diagram showing that the first duration of the candidate feature conforms to the normal distribution;

[0035] Figure 7 A schematic diagram showing three features matched during the rendering process of a frame of a game application is shown;

[0036] Figure 8 A schematic diagram showing a process of determining at least one feature and at least one identification strategy of a game application;

[0037] Fig. 9 The rendering process of a frame in game A is shown;

[0038] Fig.10 A schematic diagram of screening at least one rendering operation according to an embodiment of the present application;

[0039] Fig.11 A schematic diagram of a rendering operation 1 shown in an embodiment of the present application;

[0040] Fig.12 A schematic diagram of a rendering operation 2 shown in an embodiment of the present application;

[0041] Fig.13 A schematic diagram of rendering operation 3 shown in an embodiment of the present application;

[0042] Fig.14 A schematic diagram of an instruction called before executing a copy operation according to an embodiment of the present application;

[0043] Fig.15 A schematic diagram of the instruction glInvalidateFramebuffer shown in an embodiment of the present application;

[0044] Fig.16 A schematic diagram of the status information of the instruction glInvalidateFramebuffe shown in an embodiment of the present application;

[0045] Fig.17 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0046] Fig.18 A schematic diagram of a layered structure of an electronic device provided in an embodiment of the present application;

[0047] Fig.19 A schematic diagram of the matching process based on the Nth frame of game A is shown;

[0048] Fig. 20 This is a schematic diagram of feature matching based on the Nth frame shown in an embodiment of the present application;

[0049] Fig.21A schematic diagram of the interaction between software modules provided in the embodiment of the present application;

[0050] Fig. 22 Schematic diagram of frame loss prediction provided in the embodiment of the present application Figure 1 ;

[0051] Fig.23 Schematic diagram of frame loss prediction provided in the embodiment of the present application Figure 2 ;

[0052] Fig.24 Schematic diagram of frame loss prediction provided in the embodiment of the present application Figure 3 ;

[0053] Fig.25 A flowchart of a frame loss prediction method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the present application, unless otherwise specified, "at least one" means one or more, and "a plurality of" means two or more than two. In the description of the present application, unless otherwise specified, "first", "second", "third", etc. are used to distinguish different objects, and are not limitations on order or precedence.

[0055] Game rendering is a process of visualization and presentation. Game rendering can make the game screen more beautiful and more realistic. In addition, in order to pursue real-time interactivity, the game screen is rendered in real time.

[0056] At present, in order to provide players with high-quality game images so that players can have a stronger sense of immersion and a better user experience in game operations, game rendering is becoming more and more complex, and the performance overhead of electronic devices when performing game rendering is also increasing accordingly.

[0057] Since the game screen is rendered in real time, that is, each frame needs to be rendered in real time during the game, and different electronic devices are affected by factors such as chips and systems, resulting in performance differences between different electronic devices, and complex game rendering increases the performance overhead of electronic devices. Therefore, frame loss is inevitable during the game.

[0058] Frame loss in games generally refers to the phenomenon of screen lag and freeze caused by the failure to refresh the game screen in time. Frame loss will directly affect the player's experience and immersion, and also affect the player's evaluation of the game.

[0059] Next, frame loss will be specifically explained based on the game's frame display process.

[0060] The frame display process may at least include: frame rendering, frame synthesis and frame display. Among them, frame rendering is completed by the game application, frame synthesis is completed by the synthesizer (ie, SurfaceFlinger), and frame display is completed by the display.

[0061] In order to ensure the smoothness of the display and avoid display freezes and other phenomena, electronic devices generally display based on the Vsync (vertical synchronization) signal to synchronize frame rendering, frame synthesis and frame display. Specifically, the Vsync signal is used to control the start of frame rendering, frame synthesis and frame display. In other words, game applications, synthesizers and displays are triggered when the Vsync signal arrives. The Vsync signal is a periodic signal, and the Vsync signal period can be set according to the screen refresh rate of the display. For example, when the screen refresh rate is 60Hz, the Vsync signal period can be 16.6ms, that is, the electronic device generates a control signal every 16.6ms to trigger the Vsync signal periodically.

[0062] It should be noted that in other feasible implementations, the game application may not strictly follow the Vsync signal to trigger frame rendering, that is, after the game application completes the rendering of one frame, it can render the next frame without the Vsync signal triggering it.

[0063] The display process of a frame can be described as follows:

[0064] When a VSYNC signal arrives, the game application renders the frame, and after rendering, stores the rendered frame of the frame (i.e., the rendered image) in storage queue 1. When the next VSYNC signal arrives, the synthesizer obtains the rendered frame of the frame from storage queue 1, synthesizes the display frame of the frame (i.e., the synthesized image) based on the rendered frame of the frame, and stores the display frame of the frame in storage queue 2. When the next VSYNC signal arrives, the display obtains the display frame of the frame from storage queue 2 and displays it.

[0065] It should be noted that in the display process of a frame, the frame rendering of the frame must be completed before the trigger signal of the frame synthesis (VSYNC signal) of the frame arrives, so that the frame can be synthesized when the trigger signal of the frame synthesis arrives, and the frame can be displayed when the trigger signal of the frame display (VSYNC signal) arrives, so as to ensure the smoothness of the display. If the frame rendering of the frame is not completed before the trigger signal of the frame synthesis of the frame arrives, the frame cannot be synthesized when the trigger signal of the frame synthesis of the frame arrives, and then the frame cannot be displayed when the trigger signal of the frame display of the frame arrives, but the previously displayed content is retained, so that frame loss occurs.

[0066] For example, Figure 1 The schematic diagram of the frame display process is shown in FIG. Figure 1 .Depend on Figure 1 It can be seen that the frame display process can be as follows:

[0067] When VSYNC0 arrives, the game application renders the Nth frame, the compositor synthesizes the N-1th frame, and the display displays the N-2th frame.

[0068] When VSYNC1 arrives, the game application renders the N+1th frame, the compositor synthesizes the Nth frame, and the monitor displays the N-1th frame.

[0069] When VSYNC2 arrives, the game application renders the N+2 frame, the compositor synthesizes the N+1 frame, and the monitor displays the N frame.

[0070] When VSYNC3 arrives, the game application renders the N+3 frame, the compositor synthesizes the N+2 frame, and the monitor displays the N+1 frame.

[0071] Obviously, by Figure 1 It can be seen that for each frame, frame rendering is completed before the trigger signal of frame synthesis arrives. For each frame, synthesis is completed within the corresponding VSYNC signal cycle. Therefore, for each frame, the corresponding frame is displayed when the corresponding VSYNC signal arrives, ensuring the smoothness of the display.

[0072] For example, Figure 2 The schematic diagram of the frame display process is shown in FIG. Figure 2 .Depend on Figure 2 It can be seen that the rendering of the Nth frame was not completed before the arrival of VSYNC1, resulting in the inability of the synthesizer to synthesize the Nth frame when VSYNC1 arrives, which in turn causes the display to be unable to display the Nth frame when VSYNC2 arrives, but continues to display the N-1th frame, resulting in frame loss.

[0073] In order to alleviate the occurrence of frame loss, under normal circumstances, the system of the electronic device tracks and predicts the game load to perform macro-dimensional system resource scheduling, thereby improving the adverse effects of frame loss. However, since frame loss is a frame-dimensional problem, and the tracking and prediction of the game load in the macro dimension cannot make targeted predictions of frame loss, this method cannot accurately predict frame loss, and thus cannot effectively improve the adverse effects of frame loss.

[0074] For example, Figure 3 FIG. 4 shows a schematic diagram of the frame display process based on game load tracking and prediction. Figure 3 As shown in , the response point of game load tracking and prediction is the end of each frame. Figure 3It can be seen that although the game load is tracked and predicted at the end of the Nth frame, and the system resources are scheduled in time to render the N+1th frame in advance (that is, the N+1th frame is rendered before VSYNC1 arrives), the rendering of the N+1th frame is still not completed before the arrival of VSYNC2, resulting in the synthesizer being unable to synthesize the N+1th frame when VSYNC2 arrives, and then when VSYNC3 arrives, the display cannot display the N+1th frame, but continues to display the Nth frame, resulting in frame loss. Obviously, this method cannot accurately predict frame loss, and thus cannot effectively improve the impact of frame loss.

[0075] In the related art, a Boost mechanism can be used to predict frame loss. The Boost mechanism is: a threshold can be set according to the theoretical time consumption of a single frame. During the frame rendering process, when the current time consumption of the frame exceeds the threshold (that is, when frame loss is predicted), the system resources are quickly scheduled to speed up the frame rendering and avoid frame loss. In this way, there are high requirements for the accuracy of threshold setting. Once the threshold setting is inaccurate, the problem of inaccurate prediction will arise, resulting in the failure to quickly schedule the system resources when the system resources should be quickly scheduled, resulting in frame loss, or the rapid scheduling of system resources when the system resources should not be quickly scheduled, resulting in waste of system resources, etc.

[0076] For example, Figure 4 Shows the frame display process based on the boost mechanism Figure 1 .like Figure 4 As shown, the threshold is less than the theoretical time consumption of a single frame. During the rendering of the N+1th frame, when the current time consumption of the N+1th frame exceeds the threshold, a response is triggered to quickly schedule system resources to speed up the rendering of the N+1th frame.

[0077] From the results, we can see that the actual rendering time of the N+1 frame is less than the theoretical rendering time of a single frame, and the N+1 frame can also be displayed correctly. The fast scheduling of system resources does shorten the rendering time of the N+1 frame.

[0078] However, when the threshold is less than the theoretical time consumption of a single frame, there is a situation where the rendering of the N+1th frame can be completed and the N+1th frame can be correctly displayed before the arrival of VSYNC2 without intervention. Therefore, compared with the case of no intervention, the above-mentioned boost mechanism does not make a more beneficial effect on the result (i.e., the correct display of the N+1th frame), but instead causes a waste of system resources because of the rapid scheduling of system resources. In other words, the frame loss prediction is inaccurate, resulting in the scheduling of system resources when the system resources should not be quickly scheduled, resulting in a waste of system resources.

[0079] For example, Figure 5 Shows the frame display process based on the boost mechanism Figure 2.like Figure 5 As shown, the threshold is set to be greater than the theoretical time consumption of a single frame. During the rendering of the N+1th frame, when the current time consumption of the N+1th frame exceeds the threshold, a response is triggered to quickly schedule system resources to speed up the rendering of the N+1th frame.

[0080] From the results, the actual time consumption of the N+1th frame is greater than the theoretical time consumption of a single frame, and the rendering of the N+1th frame is not completed before VSYNC2 arrives. As a result, the N+1th frame cannot be synthesized when VSYNC2 arrives, which in turn causes the N+1th frame to not be displayed correctly, that is, when VSYNC3 arrives, the N+1th frame is not displayed, but the Nth frame continues to be displayed.

[0081] Obviously, since the threshold is greater than the theoretical time consumption of a single frame, the system resources are scheduled too late (that is, the frame loss prediction is inaccurate, and the system resources are not scheduled in time when they should be scheduled). As a result, even if the system resources are scheduled, the rendering of the N+1 frame cannot be completed before the arrival of VSYNC2, which causes the N+1 frame to not be displayed correctly and frame loss occurs.

[0082] In order to solve the above technical problems, an embodiment of the present application provides a frame loss prediction method. The frame loss prediction method can be applied to an electronic device. The electronic device can be installed with an application that displays a video to a user by rendering frames in real time. The video can be understood as a continuous playback of multiple frames. The application that displays a video to a user by rendering frames in real time includes but is not limited to a game application.

[0083] During the process of the above-mentioned application being run in the electronic device, the electronic device can perform frame loss prediction during the process of the above-mentioned application executing frame rendering by executing the frame loss prediction method provided in the embodiment of the present application.

[0084] Before introducing the embodiments of the present application, first, some terms or concepts involved in the embodiments of the present application are explained by taking game applications as an example. It should be understood that the present application does not specifically limit the naming of the following terms. The following terms may have other names. The renamed terms still meet the following related term explanations.

[0085] The game application has at least one feature and at least one identification strategy, wherein the at least one feature corresponds to the at least one identification strategy. The number of features of different game applications may be the same or different.

[0086] The at least one feature may be understood as at least one node that appears during the rendering process of any frame of the game application. Each of the at least one identification strategy is a strategy that matches the corresponding feature during the rendering process of any frame of the game application.

[0087] At least one feature has at least two of the following properties:

[0088] Feature 1: Any one of the at least one feature appears in different frames of the game application and only appears once. In other words, the candidate feature is any one of the at least one feature, and the candidate feature appears in each frame of the game application and only appears once in each frame.

[0089] Feature 2: The first duration of candidate features corresponding to multiple frames of game applications conforms to the normal distribution.

[0090] The first duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame of the multiple frames and the moment when the matching of the any frame ends.

[0091] It should be noted that any of the above two frames refer to the same frame. For example, Figure 6 The schematic diagram shows that the first duration of the candidate feature conforms to the normal distribution. Figure 6 In , the horizontal axis is the first duration of the candidate feature, and the vertical axis is the number of frames. Figure 6 It can be seen that the number of frames corresponding to different values ​​of the first duration of the candidate feature. For multiple frames of the game application, the values ​​of the first duration of the candidate feature are mainly concentrated around 4.9ms.

[0092] Since the first duration of the candidate features conforms to the normal distribution, during the running of the game application, if the first duration of the candidate features corresponding to a frame of the game application is known, the first duration of the candidate features corresponding to another frame of the game application can be inferred based on the first duration of the candidate features corresponding to the frame, and the inferred first duration of the candidate features corresponding to another frame of the game application has a high accuracy rate.

[0093] On this basis, optionally, at least one feature may also have the following characteristic 3, or may also have the following characteristic 4 and characteristic 5, or may also have the following characteristics 3 to 5, wherein:

[0094] Characteristic 3: When the number of at least one feature is multiple, for any two features of the at least one feature, the time when the two features are matched in any frame of the game application is different. In other words, the time when the two features are matched in the same frame is different.

[0095] Feature 4: When there are multiple features in at least one feature, for any two features in the at least one feature, the order of the time when the two features are matched in any frame of the game application is fixed. In other words, the order of the time when the at least one feature is matched in any frame is fixed.

[0096] Feature 5: The second duration of the candidate features corresponding to multiple frames of the game application conforms to the normal distribution.

[0097] Among them, when the candidate feature is not the last one matched in any frame of the multiple frames of the game application, the second duration of the candidate feature is the duration between the moment the candidate feature is matched in any frame and the moment the next adjacent feature is matched in the any frame.

[0098] When the candidate feature is the last one to be matched in any frame of the multiple frames of the game application, the second duration of the candidate feature is the duration between the time when the candidate feature is matched in any frame and the time when the matching of the any frame ends. It should be noted that any frame in the above two places refers to the same frame.

[0099] Since the second duration of the candidate features all conform to the normal distribution, during the running of the game application, if the second duration of the candidate features corresponding to a frame of the game application is known, the second duration of the candidate features corresponding to another frame of the game application can be inferred based on the second duration of the candidate features corresponding to the frame, and the inferred second duration of the candidate features corresponding to another frame of the game application has a high accuracy rate.

[0100] The rendering instruction corresponding to the candidate feature in any frame of the multiple frames of the game application includes the rendering instruction intercepted in the first time period and issued based on the any frame.

[0101] In the case that the candidate feature is not the last one matched in the arbitrary frame, the first time period may refer to the time period between the moment when the candidate feature is matched in the arbitrary frame and the moment when the next adjacent feature is matched in the arbitrary frame.

[0102] In the case where the candidate feature is the last one to be matched in the arbitrary frame, the first time period may refer to the time period between the moment when the candidate feature is matched in the arbitrary frame and the moment when the matching of the arbitrary frame ends.

[0103] Exemplarily, taking a game application including three features as an example, the rendering instructions corresponding to each of the three features and the first duration and the second duration of each feature are explained. Figure 7 A schematic diagram showing three features matched during the rendering process of one frame of a game application.

[0104] The game application renders the frame by issuing a rendering instruction stream for the frame. T0 is the moment when the first rendering instruction issued by the game application based on the frame is intercepted. T4 is the moment when the rendering of the frame ends (i.e., the moment when the frame ends is matched).

[0105] T1 is the moment when feature 1 is matched in the frame, T2 is the moment when feature 2 is matched in the frame, and T3 is the moment when feature 3 is matched in the frame. The rendering instructions corresponding to feature 1 in the frame include rendering instructions issued by the game application intercepted in the time period of T1 to T2 based on the frame. The rendering instructions corresponding to feature 2 in the frame include rendering instructions issued by the game application intercepted in the time period of T2 to T3 based on the frame. The rendering instructions corresponding to feature 3 in the frame include rendering instructions issued by the game application intercepted in the time period of T3 to T4 based on the frame.

[0106] The first duration of feature 1 corresponding to the frame is the duration between T1 and T4, the first duration of feature 2 corresponding to the frame is the duration between T2 and T4, and the first duration of feature 3 corresponding to the frame is the duration between T3 and T4.

[0107] The second duration of feature 1 corresponding to the frame is the duration between T1 and T2, the second duration of feature 2 corresponding to the frame is the duration between T2 and T3, and the second duration of feature 3 corresponding to the frame is the duration between T3 and T4.

[0108] It should be noted that, the “matching… in… frame” mentioned above can be understood as “matching… in the rendering process of… frame”.

[0109] Next, the process of determining at least one feature and at least one identification strategy of a game application is described.

[0110] Figure 8 A schematic diagram of a process for determining at least one feature and at least one identification strategy of a game application is shown. Figure 8 As shown, it may include:

[0111] S801, capturing multiple sample rendering instruction streams from the rendering instruction stream sent by the game application, wherein the multiple sample rendering instruction streams correspond one-to-one to multiple frames of the game application, in other words, each sample rendering instruction stream in the multiple sample rendering instruction streams is a rendering instruction stream sent by the game application based on the corresponding frame.

[0112] In order to make the multiple sample rendering instruction streams diverse and to improve the accuracy of subsequent feature determination and recognition strategies, in other feasible embodiments, the rendering instruction streams of frames corresponding to different game scenes can be captured from the rendering instruction stream issued by the game application, and the rendering instruction streams of frames corresponding to different game scenes can be used as multiple sample rendering instruction streams, that is, multiple sample rendering instruction streams correspond to multiple different game scenes.

[0113] In order to improve the consistency of multiple sample rendering instruction streams and facilitate subsequent analysis, multiple sample rendering instruction streams can be captured under the same settings (for example, 60 frame rate, extreme image quality, and ultra-high resolution).

[0114] Exemplarily, this step is described by taking Game B as an example, capturing the rendering instruction streams of frames corresponding to different game scenes from the rendering instruction stream issued by Game B to obtain X rendering instruction streams (i.e., X sample rendering instruction streams), and the X rendering instruction streams are respectively: RDC-1, RDC-2, ..., RDC-X, where X is an integer.

[0115] S802, filter out at least one rendering operation from a plurality of sample rendering instruction streams, wherein a first rendering operation has corresponding instruction segments in different sample rendering instruction streams, and the first rendering operation is any one of the at least one filtered rendering operation.

[0116] Exemplarily, for each sample rendering instruction stream, each rendering instruction in the sample rendering instruction stream can be analyzed to divide the sample rendering instruction stream into multiple instruction segments, wherein each instruction segment in the multiple instruction segments corresponds to a rendering operation. By comparing the rendering operations corresponding to each instruction segment in different sample rendering instruction streams in the multiple sample rendering instruction streams, the rendering operations existing in each sample rendering instruction stream are screened out to obtain at least one rendering operation.

[0117] Each rendering operation in the at least one rendering operation screened out appears in each sample rendering instruction stream, and each rendering operation has a corresponding instruction segment in different sample rendering instruction streams.

[0118] Rendering operations may include, but are not limited to, drawing dynamic clouds, environmental disturbances, terrain, G-buffer (geometry buffer), linear depth map, Occlusion Culling, shadows, SSAO (Screen-Space Ambient Occlusion), SSR (Screen Space Reflection), translucent effects, motion blur, first binding of a frame buffer object with a specified resolution, second use of a texture object with a tag of specified content, first rendering of a shadow map, completion of rendering of specific content, etc.

[0119] Taking the game application Game A as an example, the rendering process of one frame in Game A is described. Fig. 9 The rendering process of a frame in game A is shown. Fig. 9In the rendering, the rendering of a frame goes through the following rendering operations: drawing dynamic clouds, environmental disturbances, terrain, G-buffer, linear depth map, Occlusion Culling, shadows, SSAO, SSR, translucent effects, CharacterAmbientSensors (character environment perception simulation), lightingPass (lighting simulation), MV (MotionVector, motion vector), TAA (Temporal Anti-Aliasing, time anti-aliasing), motion blur, Bloom (floodlight), Uber, UI (User Interface, user interface).

[0120] Exemplarily, the exemplary description is continued with the example of the above-mentioned game B. Fig.10 A schematic diagram of filtering at least one rendering operation according to an embodiment of the present application.

[0121] like Fig.10 As shown, each of the X rendering instruction streams (RDC-1 to RDC-X) includes M rendering instructions. Specifically, RDC-1 includes rendering instructions i11, i12, ..., i1M. RDC-2 includes rendering instructions i21, i22, ..., i2M. RDC-3 includes rendering instructions i31, i32, ..., i3M. RDC-X includes rendering instructions ix1, ix2, ..., ixM. It should be noted that the number of rendering instructions included in different rendering instruction streams may be the same or different, that is, for different rendering instruction streams, the value of M may be the same or different, and the embodiments of the present application do not make any special limitation on this.

[0122] based on Fig.10 , analyze each rendering instruction in RDC-1 to divide RDC-1 into multiple instruction segments. Each of the multiple instruction segments corresponds to a rendering operation. Similarly, based on each rendering instruction stream in the remaining rendering instruction streams (RDC-2~RDC-X), the above analysis is performed to divide each remaining rendering instruction stream into multiple instruction segments. It should be noted that the number of multiple instruction segments obtained based on different rendering instruction streams can be the same or different, and this application does not make any special limitation on this.

[0123] Compare the rendering operations corresponding to each instruction segment in different rendering instruction streams in the X rendering instruction streams (RDC-1 to RDC-X), and select the rendering operations that exist in each of the X rendering instruction streams. Fig.10 It can be seen that the selected rendering operations are: rendering operation 1, rendering operation 2, and rendering operation 3. Each rendering operation has a corresponding instruction segment in a different rendering instruction stream. Fig.10As shown, the instruction segment corresponding to the rendering operation in a rendering instruction stream is the instruction segment framed by the rendering operation in the rendering instruction stream.

[0124] For example, Fig.11 As shown, rendering operation 1 is to render all non-UI content offline into the frame buffer (framebuffer1). Fig.12 As shown in FIG. 2 , rendering operation 2 is to copy the image to the default frame buffer (framebuffer0). Fig.13 As shown, rendering operation 3 is to directly draw the UI part in the default frame buffer (framebuffer0).

[0125] S803, analyzing the instruction segment corresponding to each rendering operation in at least one rendering operation to obtain a recognition strategy corresponding to at least one rendering operation. According to each rendering operation in at least one rendering operation, a corresponding feature is configured for the corresponding recognition strategy to obtain at least one feature corresponding to at least one recognition strategy. Verify whether at least one feature meets the characteristics described above in the rendering process of multiple frames of the game application through at least one recognition strategy. For the features that meet the characteristics described above, retain the feature and the recognition strategy corresponding to the feature. For the features that do not meet the characteristics described above, it is necessary to re-analyze the instruction segment corresponding to the rendering operation corresponding to the feature to redetermine the recognition strategy, and verify it according to the recognition strategy determined by the reproduction until the verification passes. Or for the features that do not meet the characteristics described above, eliminate the feature and the recognition strategy corresponding to the feature from at least one feature and at least one recognition strategy, and re-screen the rendering operation, and determine the recognition strategy based on the re-screened rendering operation, and verify it until it passes the verification.

[0126] Each of the at least one recognition strategy is a strategy for matching corresponding features during the rendering process of any frame of the game application.

[0127] In a feasible implementation, the first rendering operation is any one of the at least one rendering operation selected. Below, the process of determining the identification strategy corresponding to the first rendering operation is described by taking the first rendering operation as an example. The specific process can be described as follows:

[0128] Analyze the instruction segment corresponding to the first rendering operation (the instruction segment corresponding to the first rendering operation in different sample rendering instruction streams) to determine the specific rendering instruction that needs to be executed when implementing the first rendering operation. The specific rendering instruction refers to a rendering instruction that must be executed and is easy to match when implementing the first rendering operation. Next, the identification strategy corresponding to the first rendering operation can be determined by analyzing the specific rendering instruction and the status information of the specific rendering instruction.

[0129] The status information of a specific rendering instruction includes, but is not limited to, parameter information and rendering pipeline information when the specific rendering instruction appears.

[0130] It should be noted that the above description on determining the identification strategy is only exemplary and is not used to limit the method of determining the identification strategy in this application. Since the first rendering operation is any one of the at least one rendering operation, the identification strategy corresponding to the other rendering operations can be determined with reference to the above description. In this way, at least one identification strategy is obtained.

[0131] Exemplarily, the exemplary description is continued with the example of the above-mentioned game B.

[0132] The method of determining the identification strategy corresponding to the rendering operation 2 can be described as follows:

[0133] Rendering operation 2 is to copy it to the default frame buffer. Fig.14 As shown, before executing the copy operation, the instruction glInvalidateFramebuffer (i.e., a specific rendering instruction) must be called. Since the instruction glInvalidateFramebuffer only appears three times during the rendering process of a frame (relative to other rendering instructions, the fewer times it appears, the better the matching efficiency). Therefore, the instruction glInvalidateFramebuffer is determined as a specific rendering instruction.

[0134] The process of determining the recognition strategy based on the instruction glInvalidateFramebuffer can be as follows:

[0135] First, obtain the current state information of the instruction glInvalidateFramebuffer. Then, compare the current state information of the instruction glInvalidateFramebuffer with the state information of the other two times the instruction glInvalidateFramebuffer appears. Finally, the difference part of the comparison, the instruction glInvalidateFramebuffer itself, and the current state information of the instruction glInvalidateFramebuffer can be combined as the identification strategy corresponding to rendering operation 2. Fig.15 and Fig.16 The identification strategy corresponding to the rendering operation 2 is shown, wherein the identification strategy corresponding to the rendering operation 2 includes the instruction glInvalidateFramebuffer itself (such as Fig.15 As shown), the status information of the instruction glInvalidateFramebuffe at this time (such as Fig.16 shown) and the difference parts of the comparison (not shown in the figure).

[0136] It should be noted that the method of determining the identification strategy corresponding to the rendering operation 1 and the identification strategy corresponding to the rendering operation 3 can refer to the above description, which will not be repeated here.

[0137] Below, taking the first rendering operation as an example, the process of configuring a corresponding feature for the recognition strategy corresponding to the first rendering operation according to the first rendering operation is described. The specific process can be described as follows:

[0138] The name of the first rendering operation may be configured as a feature corresponding to the identification strategy corresponding to the first rendering operation. The first rendering operation may also be renamed to configure the renamed name as a feature corresponding to the identification strategy corresponding to the first rendering operation. For example, the name of the first rendering operation may be renamed as Feature 1 to configure Feature 1 as a feature corresponding to the identification strategy corresponding to the first rendering operation.

[0139] It should be noted that the process of configuring a corresponding feature for the recognition strategy corresponding to other rendering operations according to other rendering operations can refer to the above description.

[0140] It should be noted that verifying whether at least one feature satisfies the above characteristics may refer to verifying whether at least one feature satisfies characteristics 1 and 2. Optionally, verifying whether at least one feature satisfies the above characteristics may also refer to verifying whether at least one feature satisfies characteristics 1, 2, and 3. Optionally, verifying whether at least one feature satisfies the above characteristics may also refer to verifying whether at least one feature satisfies characteristics 1, 2, 4, and 5. Optionally, verifying whether at least one feature satisfies the above characteristics may also refer to verifying whether at least one feature satisfies characteristics 1 to 5.

[0141] The process of verifying whether at least one feature meets the characteristics described above during the rendering process of multiple frames of the game application by using at least one recognition strategy may be as follows:

[0142] During the running of the game application, at least one feature is matched during the rendering process of each frame of the game application through at least one recognition strategy, and the end-of-frame rendering instruction of each frame is matched. According to the time when each feature of the at least one feature is matched in each frame, the number of times each feature is matched, and the time when the end-of-frame rendering instruction of each frame is matched (i.e., the time when each frame ends), it is verified whether the at least one feature meets the characteristics described above.

[0143] It should be noted that the above verification needs to verify each frame, but since the game will continue to display the screen as long as the game is not stopped during the running process, and since it is difficult to exhaustively verify all frames, a time period can be limited during the verification process, for example, verification is performed based on each frame of the game application within a preset time period. The preset time period can be, for example, 1 hour, 2 hours, 5 hours, etc., and the embodiments of the present application do not specifically limit this.

[0144] Exemplarily, the exemplary description is continued with the example of the above-mentioned game B.

[0145] By analyzing the instruction segments corresponding to rendering operation 1, rendering operation 2, and rendering operation 3 in turn, the identification strategies obtained are: Identify-1, Identify-2, and Identify-3. Rendering operation 1, rendering operation 2, and rendering operation 3 are renamed as Feature 1, Feature 2, and Feature 3, respectively.

[0146] If feature 1, feature 2, and feature 3 are verified to meet the above five characteristics through identification strategies Identify-1, Identify-2, and Identify-3, the final features and identification strategies of game B are shown in Table 1:

[0147]

[0148] Table 1

[0149] Among them, the corresponding relationship between feature 1, feature 2, feature 3 and Identify-1, Identify-2, Identify-3 is shown in Table 1.

[0150] It should be noted that the above description on determining the characteristics and identification strategies of game applications is only exemplary and is not intended to limit the method of determining the characteristics and identification strategies in this application.

[0151] The electronic devices to which the frame loss prediction method provided in the embodiments of the present application is applied may be mobile phones, tablet computers, wearable electronic devices, enhanced display devices, virtual display devices, laptop computers, super mobile personal computers, netbooks, personal digital assistants, game consoles, etc. The embodiments of the present application do not specifically limit the specific types of electronic devices.

[0152] Hereinafter, the structure of the electronic device is described by taking a mobile phone as an example. Fig.17 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0153] like Fig.17As shown, the electronic device 1700 may include a processor 1710, an external memory interface 1720, an internal memory 1721, a universal serial bus (USB) interface 1730, a charging management module 1740, a power management module 1741, a battery 1742, an antenna 1, an antenna 2, a mobile communication module 1750, a wireless communication module 1760, an audio module 1770, a speaker 1770A, a receiver 1770B, a microphone 1770C, an earphone interface 1770D, a sensor module 1780, a button 1790, a motor 1791, an indicator 1792, a camera 1793, a display screen 1794, and a subscriber identification module (SIM) card interface 1795, etc. Among them, the sensor module 1780 may include a pressure sensor 1780A, a gyroscope sensor 1780B, an air pressure sensor 1780C, a magnetic sensor 1780D, an acceleration sensor 1780E, a distance sensor 1780F, a proximity light sensor 1780G, a fingerprint sensor 1780H, a temperature sensor 1780J, a touch sensor 1780K, an ambient light sensor 1780L, a bone conduction sensor 1780M, etc.

[0154] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 1700. In other embodiments of the present application, the electronic device 1700 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0155] The processor 1710 may include one or more processing units, for example, the processor 1710 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0156] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0157] A memory may also be provided in the processor 1710 for storing instructions and data. In some embodiments, the memory in the processor 1710 is a cache memory. The memory may store instructions or data that the processor 1710 has just used or cyclically used. If the processor 710 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, and the waiting time of the processor 1710 is reduced, thereby improving the efficiency of the system. In an embodiment of the present application, the processor 1710 may also execute a computer program stored in the internal memory 1721, so that the electronic device 7100 executes the frame loss prediction method provided in the embodiment of the present application.

[0158] In some embodiments, the processor 1710 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0159] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 1710 may include multiple groups of I2C buses. The processor 1710 may be coupled to the touch sensor 1780K, the charger, the flash, the camera 1793, etc. through different I2C bus interfaces. For example: the processor 1710 may be coupled to the touch sensor 1780K through the I2C interface, so that the processor 1710 communicates with the touch sensor 1780K through the I2C bus interface, thereby realizing the touch function of the electronic device 1700.

[0160] The I2S interface can be used for audio communication. In some embodiments, the processor 1710 can include multiple groups of I2S buses. The processor 1710 can be coupled to the audio module 1770 via the I2S bus to achieve communication between the processor 1710 and the audio module 1770. In some embodiments, the audio module 1770 can transmit an audio signal to the wireless communication module 1760 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0161] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 1770 and the wireless communication module 1760 can be coupled via a PCM bus interface. In some embodiments, the audio module 1770 can also transmit audio signals to the wireless communication module 1760 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0162] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 1710 and the wireless communication module 1760. For example, the processor 1710 communicates with the Bluetooth module in the wireless communication module 1760 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 1770 can transmit an audio signal to the wireless communication module 1760 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0163] The MIPI interface can be used to connect the processor 1710 with peripheral devices such as the display screen 1794 and the camera 1793. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 1710 and the camera 1793 communicate via the CSI interface to implement the shooting function of the electronic device 1700. The processor 1710 and the display screen 1794 communicate via the DSI interface to implement the display function of the electronic device 1700.

[0164] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 1710 with the camera 1793, the display 1794, the wireless communication module 1760, the audio module 1770, the sensor module 1780, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0165] The USB interface 1730 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 1730 can be used to connect a charger to charge the electronic device 1700, and can also be used to transfer data between the electronic device 1700 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0166] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 1700. In other embodiments of the present application, the electronic device 1700 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0167] The charging management module 1740 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 1740 may receive charging input from a wired charger through the USB interface 1730. In some wireless charging embodiments, the charging management module 1740 may receive wireless charging input through a wireless charging coil of the electronic device 1700. While the charging management module 1740 is charging the battery 1742, it may also power the electronic device through the power management module 1741.

[0168] The power management module 1741 is used to connect the battery 1742, the charging management module 1740 and the processor 1710. The power management module 1741 receives input from the battery 1742 and / or the charging management module 1740, and supplies power to the processor 1710, the internal memory 1721, the display screen 1794, the camera 1793, and the wireless communication module 1760. The power management module 1741 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 1741 can also be set in the processor 1710. In other embodiments, the power management module 1741 and the charging management module 1740 can also be set in the same device.

[0169] The wireless communication function of the electronic device 1700 can be implemented through antenna 1, antenna 2, mobile communication module 1750, wireless communication module 1760, a modem processor, and a baseband processor.

[0170] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 1700 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0171] The mobile communication module 1750 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 1700. The mobile communication module 1750 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 1750 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 1750 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 1750 can be set in the processor 1710. In some embodiments, at least some of the functional modules of the mobile communication module 1750 can be set in the same device as at least some of the modules of the processor 1710.

[0172] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 1770A, a receiver 1770B, etc.), or displays an image or video through a display screen 1794. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 1710 and be set in the same device as the mobile communication module 1750 or other functional modules.

[0173] The wireless communication module 1760 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 1700. The wireless communication module 1760 can be one or more devices integrating at least one communication processing module. The wireless communication module 1760 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 1710. The wireless communication module 1760 can also receive the signal to be sent from the processor 1710, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0174] In some embodiments, the antenna 1 of the electronic device 1700 is coupled to the mobile communication module 1750, and the antenna 2 is coupled to the wireless communication module 1760, so that the electronic device 1700 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0175] The electronic device 1700 implements the display function through a GPU, a display screen 1794, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 1794 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1710 may include one or more GPUs that execute program instructions to generate or change display information.

[0176] The display screen 1794 is used to display images, videos, etc. The display screen 1794 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 1700 may include 1 or N display screens 1794, where N is a positive integer greater than 1.

[0177] The electronic device 1700 can realize the shooting function through ISP, camera 1793, video codec, GPU, display screen 1794 and application processor.

[0178] ISP is used to process the data fed back by camera 1793. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 793.

[0179] Camera 1793 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 1700 may include 1 or N cameras 1793, where N is a positive integer greater than 1.

[0180] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 1700 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0181] Video codecs are used to compress or decompress digital videos. The electronic device 1700 may support one or more video codecs. Thus, the electronic device 1700 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0182] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 1700 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0183] The external memory interface 1720 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1700. The external memory card communicates with the processor 1710 through the external memory interface 1720 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0184] The internal memory 1721 can be used to store computer executable program codes, which include instructions. The internal memory 1721 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 1700 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 1721 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 1710 executes various functional applications and data processing of the electronic device 1700 by running instructions stored in the internal memory 1721 and / or instructions stored in a memory provided in the processor.

[0185] The electronic device 1700 can implement audio functions such as music playing and recording through the audio module 1770, the speaker 1770A, the receiver 1770B, the microphone 1770C, the headphone jack 1770D, and the application processor.

[0186] The audio module 1770 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 1770 can also be used to encode and decode audio signals. In some embodiments, the audio module 1770 can be arranged in the processor 1710, or some functional modules of the audio module 1770 can be arranged in the processor 1710.

[0187] The speaker 1770A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 1700 can listen to music or listen to a hands-free call through the speaker 1770A.

[0188] The receiver 1770B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 1700 receives a call or voice message, the voice can be received by placing the receiver 1770B close to the human ear.

[0189] Microphone 1770C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting their mouth close to the microphone 1770C to input the sound signal into the microphone 1770C. The electronic device 1700 can be provided with at least one microphone 1770C. In other embodiments, the electronic device 1700 can be provided with two microphones 1770C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 1700 can also be provided with three, four or more microphones 1770C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0190] The earphone jack 1770D is used to connect a wired earphone and can be a USB interface 1730 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0191] The pressure sensor 1780A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 1780A can be set on the display screen 1794. There are many types of pressure sensors 1780A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 1780A, the capacitance between the electrodes changes. The electronic device 1700 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 1794, the electronic device 1700 detects the touch operation intensity according to the pressure sensor 1780A. The electronic device 1700 can also calculate the touch position based on the detection signal of the pressure sensor 1780A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to a first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0192] The gyro sensor 1780B can be used to determine the motion posture of the electronic device 1700. In some embodiments, the angular velocity of the electronic device 1700 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 1780B. The gyro sensor 1780B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 1780B detects the angle of the electronic device 1700 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 1700 through reverse movement to achieve anti-shake. The gyro sensor 1780B can also be used for navigation and somatosensory game scenes.

[0193] The air pressure sensor 1780C is used to measure air pressure. In some embodiments, the electronic device 1700 calculates the altitude through the air pressure value measured by the air pressure sensor 1780C to assist in positioning and navigation.

[0194] The magnetic sensor 1780D includes a Hall sensor. The electronic device 1700 can use the magnetic sensor 1780D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 1700 is a flip phone, the electronic device 1700 can detect the opening and closing of the flip cover according to the magnetic sensor 1780D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0195] The acceleration sensor 1780E can detect the magnitude of the acceleration of the electronic device 1700 in all directions (generally three axes). When the electronic device 1700 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0196] The distance sensor 1780F is used to measure the distance. The electronic device 1700 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 1700 can use the distance sensor 1780F to measure the distance to achieve fast focusing.

[0197] The proximity light sensor 1780G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 1700 emits infrared light outward through the light emitting diode. The electronic device 1700 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 1700. When insufficient reflected light is detected, the electronic device 1700 can determine that there is no object near the electronic device 1700. The electronic device 1700 can use the proximity light sensor 1780G to detect that the user holds the electronic device 1700 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 1780G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0198] The ambient light sensor 1780L is used to sense the brightness of the ambient light. The electronic device 1700 can adaptively adjust the brightness of the display screen 1794 according to the perceived brightness of the ambient light. The ambient light sensor 1780L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 1780L can also cooperate with the proximity light sensor 1780G to detect whether the electronic device 1700 is in a pocket to prevent accidental touches.

[0199] The fingerprint sensor 1780H is used to collect fingerprints. The electronic device 1700 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0200] The temperature sensor 1780J is used to detect temperature. In some embodiments, the electronic device 1700 uses the temperature detected by the temperature sensor 1780J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 1780J exceeds a threshold, the electronic device 1700 reduces the performance of the processor located near the temperature sensor 1780J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 1700 heats the battery 1742 to avoid abnormal shutdown of the electronic device 1700 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 1700 performs a boost on the output voltage of the battery 1742 to avoid abnormal shutdown caused by low temperature.

[0201] The touch sensor 1780K is also called a "touch device". The touch sensor 1780K can be set on the display screen 1794, and the touch sensor 1780K and the display screen 1794 form a touch screen, also called a "touch screen". The touch sensor 1780K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 1794. In other embodiments, the touch sensor 1780K can also be set on the surface of the electronic device 1700, which is different from the position of the display screen 1794.

[0202] The bone conduction sensor 1780M can obtain a vibration signal. In some embodiments, the bone conduction sensor 1780M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 1780M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 1680M can also be set in an earphone and combined into a bone conduction earphone. The audio module 1770 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 1780M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 1780M to realize a heart rate detection function.

[0203] The key 1790 includes a power key, a volume key, etc. The key 1790 may be a mechanical key or a touch key. The electronic device 1700 may receive key input and generate key signal input related to user settings and function control of the electronic device 1700.

[0204] Motor 1791 can generate vibration prompts. Motor 1791 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 1794, motor 1791 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0205] Indicator 1792 can be an indicator light, which can be used to indicate the charging status, power changes, and can also be used to indicate messages, missed calls, notifications, etc.

[0206] The SIM card interface 1795 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 1700 by inserting it into the SIM card interface 1795 or pulling it out from the SIM card interface 1795. The electronic device 1700 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 1795 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 1795 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 1795 can also be compatible with different types of SIM cards. The SIM card interface 1795 can also be compatible with external memory cards. The electronic device 1700 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 1700 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 1700 and cannot be separated from the electronic device 1700.

[0207] The hierarchical structure of the electronic device may include several layers, each of which has a clear role and division of labor. The layers communicate with each other through software interfaces. Below, the hierarchical structure of the electronic device provided in the embodiment of the present application is described by taking a game application as an example.

[0208] Fig.18 This is a schematic diagram of the hierarchical structure of an electronic device provided in an embodiment of the present application. Fig.18 As shown, the hierarchical structure of the electronic device may include three layers, from top to bottom: an application (APP) layer, an application framework layer (abbreviated as FWK) and a system resource layer.

[0209] The application layer can include a series of application packages. Fig.18 As shown, the application package may include a game application.

[0210] It should be noted that the application package may also include other applications, such as calendar, calls, maps, navigation, WLAN, Bluetooth, music, video, short messages, browsers, WeChat, Alipay, Taobao, and other applications that display videos to users through real-time rendering frames.

[0211] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0212] like Fig.18 As shown, the application framework layer may include: a system resource scheduler, a compositor (SurfaceFlinger), and a frame loss prediction service, etc.

[0213] The system resource scheduler can be used to schedule various system resources in the system resource layer.

[0214] The synthesizer is used to trigger frame synthesis when the Vsync signal arrives.

[0215] The frame loss prediction service can interact with the game application, the system resource scheduler, and the synthesizer. The frame loss prediction service can predict the frame loss of the Nth frame by intercepting the rendering instruction issued by the game application based on the Nth frame and combining at least one identification strategy and at least one feature of the game application, while the game application is running on the electronic device and in the process of the game application issuing a rendering instruction based on the Nth frame (i.e., the game application is rendering the Nth frame).

[0216] Specifically, the frame loss prediction service may include: an interception module, a cache module, an identification module, a statistics module and a prediction module. Among them:

[0217] The interception module can sequentially intercept rendering instructions issued by the game application based on the Nth frame. The Nth frame is one of the multiple frames displayed by the game application to the user. N is an integer greater than 0. It should be noted that the intercepted rendering instructions include the end-of-frame rendering instructions of the Nth frame.

[0218] It should be noted that the purpose of the game application issuing rendering instructions based on the Nth frame is to render the Nth frame through the image processor in response to the rendering instructions. Therefore, the interception module cannot affect the original issuance path of the rendering instructions in the process of intercepting the rendering instructions issued by the game application based on the Nth frame. That is, it can be understood that the rendering instructions issued by the game application based on the Nth frame are divided into two paths, one of which is transmitted to the image processor according to the original path, so that the image processor renders the Nth frame in response to the rendering instructions, and the other path is intercepted by the interception module and transmitted to the corresponding module. Only in this way can the frame loss prediction of the Nth frame be performed during the rendering process of the Nth frame.

[0219] The cache module can trigger a cache operation every time the interception module intercepts a rendering instruction.

[0220] A cache operation may refer to updating the intercepted rendering instruction into the cache queue. Optionally, a cache operation may also refer to updating the intercepted rendering instruction and the state information of the intercepted rendering instruction into the cache queue.

[0221] The state information of the rendering instruction may include parameter information when the rendering instruction appears and the rendering pipeline information of the rendering instruction.

[0222] The rendering pipeline information of the rendering instruction includes, but is not limited to, information such as program, texture, uniform, vertex, and rasterizer state extracted from the rendering instruction.

[0223] It should be noted that the rendering instruction issued based on the Nth frame will carry the identification information of the Nth frame. In addition, if the value of N is 1 (i.e., the first frame), updating to the cache queue may refer to adding the intercepted rendering instruction (or the intercepted rendering instruction and the status information of the intercepted rendering instruction, etc.) to the cache queue. If the value of N is greater than 1, updating to the cache queue may refer to modifying the parameter information of the same rendering instruction of the N-1th frame stored in the cache queue (or the parameter information of the intercepted rendering instruction and the status information of the intercepted rendering instruction, etc.) through the parameter information of the intercepted rendering instruction (or the parameter information of the intercepted rendering instruction and the status information of the intercepted rendering instruction, etc.).

[0224] The identification module can be used to trigger a matching operation each time the information in the cache queue is updated. A matching operation may refer to a feature matching of the information of the Nth frame currently cached in the cache queue (based on the rendering instructions / rendering instructions intercepted in the Nth frame and the status line information of the rendering instructions, etc.) through at least one identification strategy to determine whether the first feature can be matched. In the case of matching the first feature, the moment when the first feature is matched is determined as the moment when the first feature is matched in the Nth frame. At least one feature includes a first feature, and before this matching, the first feature has not been matched in the Nth frame.

[0225] In the above matching method, since the rendering instructions are intercepted one by one, and each time the interception module intercepts a rendering instruction, the cache module triggers a cache operation (i.e., updates once). Since the condition for triggering the matching operation is the cache queue update, multiple matching operations will be triggered during the entire rendering process of the Nth frame. Since the data of each matching operation includes the information updated by the cache module this time and the information updated before this time. Therefore, using the above matching method, the same feature will be repeatedly matched in different matching operations based on the Nth frame. However, since any one of the at least one feature appears in different frames of the game application and only appears once, therefore, in each matching operation based on the Nth frame, only the feature that is matched this time and not matched before this matching meets the requirements of this application. Based on this, in the matching process of the recognition module, the purpose of the matching operation is defined to determine whether the first feature is matched, wherein the first feature was not matched in the Nth frame before this matching.

[0226] For example, game A has three features and three recognition strategies. The three features are feature A, feature B, and feature C, and the three recognition strategies are recognition strategy A, recognition strategy B, and recognition strategy C.

[0227] Fig.19 A schematic diagram of the matching process based on the Nth frame of game A is shown. Fig.19 At least five matching operations based on the Nth frame are shown. Each time a matching operation is performed, the rendering instructions in the cache queue are updated. Each time a matching operation is performed, the rendering instructions currently stored in the cache queue are traversed in turn through the three identification strategies.

[0228] Depend on Fig.19 It can be seen that in the first and second matching operations, no features were matched. In the third matching operation, feature A was matched based on rendering instructions 1 to 3. In the fourth matching operation, feature A was matched based on rendering instructions 1 to 4. In the fifth matching operation, feature A and feature B were matched based on rendering instructions 1 to 5.

[0229] Obviously, since the feature was not matched in the first and second matching operations, the first feature was not matched in the first and second matching operations. Since feature A was matched in the third matching operation, and since feature A was not matched in the first and second matching operations, feature A is determined as the first feature, that is, the first feature matched in the third matching operation is feature A. Since feature A was matched in the fourth matching operation, but since feature A was also matched in the third matching operation, feature A is not the first feature, so the first feature was not matched in the fourth matching operation. Since feature A and feature B were matched in the fifth matching operation, but since feature A was also matched in the third and fourth matching operations, feature A is not the first feature, and since feature B was not matched in the first to fourth matching operations, feature B is determined as the first feature, that is, the first feature matched in the fifth matching operation is feature B.

[0230] For example, in order to facilitate the determination of the first feature. Before the start of the Nth frame, the state of each feature of the at least one feature of the game application is marked as unmatched. In this way, during the entire matching process of the Nth frame, when a feature is matched through at least one recognition strategy, if the state of the matched feature is unmatched, the matched feature is determined as the first feature, and the state of the matched feature is marked from unmatched to matched. If the state of the matched feature is matched, it means that the matched feature is not the first feature.

[0231] The identification module is also used to match the end-of-frame rendering instruction of the Nth frame from the cache queue.

[0232] The statistics module can be used to record the moment when the first feature is matched as the moment when the first feature is matched in the Nth frame, and record the moment when the frame end rendering instruction of the Nth frame is matched as the moment when the Nth frame is matched to the end.

[0233] The statistics module can also be used to count and record the time when each first feature is matched in the historical frame and the time when the historical frame ends matching. The historical frame is at least a part of the frame before the Nth frame.

[0234] Exemplarily, the statistics module can complete statistics and records in the following manner.

[0235] When the recognition module matches the first feature, the statistics module records the time when the first feature is matched as the time when the first feature is matched in the Nth frame. When the recognition module matches the end-of-frame rendering instruction of the Nth frame, the statistics module records the time when the end-of-frame rendering instruction of the Nth frame is matched as the time when the Nth frame is matched to the end.

[0236] It should be noted that for each frame of the game application, the execution process of the statistics module is as shown above. Based on this, during the matching process of the historical frames (i.e., at least a portion of the frames from the 1st frame to the N-1th frame), the statistics module counts and records the time when each first feature is matched in the historical frames and the time when the matching of the historical frames ends.

[0237] The prediction module is used to perform frame loss prediction on the Nth frame. Specifically, the prediction module may use at least one of the following two frame loss prediction methods to perform frame loss prediction on the Nth frame.

[0238] The first frame loss prediction method can be shown as follows:

[0239] When the recognition module matches the first feature, the prediction module predicts frame loss for the Nth frame according to the time when the first feature is matched, the time when the first feature is matched in the historical frame, and the time when the historical frame ends.

[0240] In combination with the above, it can be seen that during the operation of the game application, when the first duration of the candidate feature corresponding to a frame of the game application is known, the first duration of the candidate feature corresponding to another frame of the game application can be inferred based on the first duration of the candidate feature corresponding to the frame, and the accuracy of the inferred first duration of the candidate feature corresponding to the other frame of the game application is high. Among them, the candidate feature is any one of the at least one feature. In this way, in the first prediction method, when the first feature is matched, the first duration of the first feature corresponding to the historical frame can be determined based on the moment when the first feature is matched in the historical frame and the moment when the historical frame is matched to the end, and then the rendering end time of the Nth frame is determined based on the moment when the first feature is matched and the first duration of the first feature corresponding to the historical frame, and the frame loss prediction of the Nth frame is performed based on the rendering end time of the Nth frame. In combination with the above, it can be seen that since the first duration of the first feature corresponding to the historical frame can be used as the first duration of the first feature corresponding to the inferred Nth frame, and the accuracy of the inferred first duration of the first feature corresponding to the Nth frame is high, the accuracy of determining the moment when the rendering of the Nth frame ends is improved, thereby improving the accuracy of the frame loss prediction. In addition, since any one of the at least one feature appears and only appears once in different frames of the application, and at least one feature includes the first feature, it is possible to ensure that the first feature is matched in the Nth frame, thereby ensuring the reliability of the first frame loss prediction method.

[0241] Exemplarily, the prediction module may perform frame loss prediction on the Nth frame in the following manner.

[0242] First, according to the time when the first feature is matched in the historical frame and the time when the historical frame ends, the first duration of the first feature corresponding to the historical frame is determined.

[0243] The first duration of the first feature corresponding to the historical frame is the duration between the moment when the first feature is matched in the historical frame and the moment when the historical frame ends.

[0244] It should be noted that, in the case of multiple historical frames, the first duration of the first feature can be calculated based on each historical frame to obtain multiple first durations of the first feature. The average of the multiple first durations of the first feature is determined as the first duration of the first feature corresponding to the historical frame determined in this step.

[0245] Then, the rendering end time of the Nth frame is determined according to the time when the first feature is matched and the first duration of the first feature corresponding to the historical frame.

[0246] The specific calculation formula can be shown as follows:

[0247] T2=CurrentTime1+FeatureEndTime

[0248] Among them, T2 is the rendering end time of the Nth frame, CurrentTime1 is the time when the first feature is matched, and FeatureEndTime is the first duration of the first feature corresponding to the historical frame.

[0249] Finally, according to the rendering end time of the Nth frame and the synthesis start time of the Nth frame, the frame loss prediction is performed on the Nth frame.

[0250] The process of determining the time to start synthesizing the Nth frame can be described as follows:

[0251] When the recognition module matches the first feature, the Vsync signal time and the number of frame accumulations are obtained, and the time to start synthesizing the Nth frame is determined according to the obtained Vsync signal time, the obtained number of frame accumulations and the Vsync signal period.

[0252] The specific calculation formula can be shown as follows:

[0253] T1=Vsync+(BufferNum+1)*VsyncPeriod

[0254] Wherein, T1 is the time when the Nth frame starts to be synthesized, Vsync is the obtained Vsync signal time, BufferNum is the obtained frame accumulation number, and VsyncPeriod is the Vsync signal period.

[0255] According to the rendering end time of the Nth frame and the synthesis start time of the Nth frame, specific implementation methods of predicting the frame loss of the Nth frame include but are not limited to the following two:

[0256] Method 1: Compare the rendering end time of the Nth frame with the time when the Nth frame starts to be synthesized. If the rendering end time of the Nth frame is less than the time when the Nth frame starts to be synthesized, it is determined that there is no frame loss in the Nth frame. If the rendering end time of the Nth frame is not less than the time when the Nth frame starts to be synthesized, it is determined that there is frame loss in the Nth frame.

[0257] Method 2: Determine the timeout ratio according to the time when the Nth frame is synthesized, the time when the Nth frame is rendered, and the first duration of the first feature corresponding to the historical frame. Perform frame loss prediction for the Nth frame according to the relative size of the timeout ratio and the preset ratio.

[0258] The formula for calculating the overtime ratio can be shown as follows:

[0259]

[0260] Among them, Delta is the timeout ratio, T1 is the time when the synthesis of the Nth frame starts, T2 is the time when the rendering of the Nth frame ends, and FeatureEndTime is the first duration of the first feature corresponding to the historical frame.

[0261] According to the relative size of the timeout ratio and the preset ratio, the method of predicting the frame loss of the Nth frame can be as follows:

[0262] The timeout ratio is compared with the preset ratio. If the timeout ratio is greater than the preset ratio, it is determined that there is frame loss in the Nth frame. If the timeout ratio is not greater than the preset ratio, it is determined that there is no frame loss in the Nth frame.

[0263] The preset ratio can be determined according to the maximum value of the allowed end time of rendering the Nth frame later than the start time of synthesizing the Nth frame. It should be noted that the maximum value of the allowed end time of rendering the Nth frame later than the start time of synthesizing the Nth frame can be obtained through a large number of experimental verifications.

[0264] By determining the timeout ratio and predicting the frame loss of the Nth frame based on the timeout ratio and the preset ratio, the influence of the error at the end of rendering of the Nth frame can be eliminated, further improving the accuracy of the frame loss prediction.

[0265] It should be noted that when N is equal to 1 (i.e., the first frame), since there is no data corresponding to the historical frame, no frame loss prediction is performed for frame 1. Since there is no frame loss in frame 1 under normal circumstances, even if no frame loss prediction is performed for frame 1, it will not affect the overall effect of the above frame loss prediction.

[0266] It should be noted that, except for the prediction module, other modules still need to perform corresponding processing on the first frame.

[0267] The second frame loss prediction method can be shown as follows:

[0268] Since the second frame loss prediction method involves the first latest response time, the first latest response time is described before describing the second frame loss prediction method. It should be noted that the premise of using the second frame loss prediction method is that at least one feature satisfies at least feature one, feature two, feature four and feature five.

[0269] Before starting to render the Nth frame, the first latest response time is set to be null, that is, the initial value of the first latest response time is null.

[0270] The timing and method of updating the first latest response time may be as follows:

[0271] When the recognition module matches the first feature, the prediction module can update the first latest response time according to the time when the first feature is matched in the historical frame, the time when the historical frame ends, the time when the Nth frame synthesis starts, and the second time.

[0272] The updated first latest response time refers to the time at which the rendering instruction corresponding to the first feature in the Nth frame is allowed to start executing at the latest.

[0273] If the first feature is not the last one matched in the historical frame, the second moment is the moment when the third feature is matched in the historical frame. The at least one feature includes a third feature, and the third feature is a feature matched after the first feature and adjacent to the first feature in the feature matching of the historical frame. If the first feature is the last one matched in the historical frame, the second moment is the moment when the matching of the historical frame ends.

[0274] Specifically, the process of updating the first latest response time may be as follows:

[0275] First, the difference between the time when the historical frame is matched to the end and the time when the first feature is matched in the historical frame is determined as the first duration of the first feature corresponding to the historical frame.

[0276] It should be noted that, when there are multiple historical frames, multiple first durations of the first feature can be determined based on the multiple historical frames. Then, the average of the multiple first durations of the first feature is determined as the first duration of the first feature corresponding to the historical frame determined in this step.

[0277] Then, the difference between the second moment and the moment when the first feature is matched in the historical frame is determined as the second duration of the first feature corresponding to the historical frame.

[0278] It should be noted that, when there are multiple historical frames, multiple second durations of the first feature can be determined based on the multiple historical frames, and the average of the multiple second durations of the first feature is determined as the second duration of the first feature corresponding to the historical frame determined in this step. The second moment has been described above and will not be repeated here.

[0279] Finally, the first latest response time is updated according to the time when the synthesis of the Nth frame starts, the first duration of the first feature corresponding to the historical frame, and the second duration of the first feature corresponding to the historical frame.

[0280] Specifically, the calculation formula of the first latest response time can be as follows:

[0281] DynamicTime=T1-(FeatureEndTime-FeatureTime)

[0282] Among them, DynamicTime is the first latest response time, T1 is the time to start synthesizing the Nth frame, FeatureEndTime is the first duration of the first feature corresponding to the historical frame, and FeatureTime is the second duration of the first feature corresponding to the historical frame.

[0283] As can be seen from the above, the update time of the first latest response time is when the first feature is matched. That is, when the Nth frame is rendered, the first latest response time is empty, and then, every time a first feature is matched in the Nth frame, the first latest response time is updated based on the relevant data of the first feature.

[0284] In combination with the above, it can be known that during the operation of the game application, when the first duration and the second duration of the candidate feature corresponding to a frame of the game application are known, the first duration and the second duration of the candidate feature corresponding to another frame of the game application can be inferred based on the first duration and the second duration of the candidate feature corresponding to the frame, and the accuracy of the inferred first duration and the second duration of the candidate feature corresponding to another frame of the game application is high. Among them, the candidate feature is any one of the at least one feature. In this way, in the process of updating the first latest response time, the first duration of the first feature corresponding to the historical frame can be determined based on the time when the first feature is matched in the historical frame and the time when the historical frame ends, and then the second duration of the first feature corresponding to the historical frame can be determined based on the time when the first feature is matched in the historical frame and the second time, and finally the first latest response time is updated based on the time when the Nth frame is synthesized and the first duration and the second duration of the first feature corresponding to the historical frame. In combination with the above, it can be seen that since the first duration and the second duration of the first feature corresponding to the historical frame can be used as the first duration and the second duration of the first feature corresponding to the estimated Nth frame, respectively, and the accuracy of the first duration and the second duration of the first feature corresponding to the estimated Nth frame is high, the accuracy of updating the first latest response time is improved. In addition, since any one of the at least one feature appears and only appears once in different frames of the application, at least one feature includes the first feature, and the order of the moments when any two of the at least one feature are matched in any frame of the application is fixed, it is ensured that the first feature can be matched in the Nth frame, and it is also ensured that the order of appearance of the first feature and other features of the at least one feature in each frame is fixed, thereby ensuring the reliability of updating the first latest response time.

[0285] Below, the process of updating the first latest response time is described by taking the process of performing feature matching in the Nth frame of the game application as an example. The game application includes three features, namely, feature 1, feature 2, and feature 3. The game application includes three recognition strategies, namely, recognition strategy 1, recognition strategy 2, and recognition strategy 3.

[0286] like Fig. 20 As shown, during the rendering process of the Nth frame, the first features matched in the Nth frame by the three recognition strategies are: Feature 1, Feature 2 and Feature 3. The moment when the first rendering instruction (i.e., the first rendering instruction) issued by the game application based on the Nth frame is intercepted is T0, the moment when Feature 1 is matched in the Nth frame is T1, the moment when Feature 2 is matched in the Nth frame is T2, the moment when Feature 3 is matched in the Nth frame is T3, the moment when the end-of-frame rendering instruction of the Nth frame is matched in the Nth frame (i.e., the moment when the end of the Nth frame is matched) is T4, and the moment when the synthesis of the Nth frame begins is T5.

[0287] exist Fig. 20 In the example, the duration between the starting position of the virtual frame corresponding to feature 1 and the ending position of the virtual frame corresponding to feature 3 is the first duration of feature 1 corresponding to the historical frame. The duration between the starting position of the virtual frame corresponding to feature 2 and the ending position of the virtual frame corresponding to feature 3 is the first duration of feature 2 corresponding to the historical frame. The duration between the starting position of the virtual frame corresponding to feature 3 and the ending position of the virtual frame corresponding to feature 3 is the first duration of feature 3 corresponding to the historical frame.

[0288] The duration between the starting position of the virtual frame corresponding to feature 1 and the starting position of the virtual frame corresponding to feature 2 is the second duration of feature 1 corresponding to the historical frame. The duration between the starting position of the virtual frame corresponding to feature 2 and the starting position of the virtual frame corresponding to feature 3 is the second duration of feature 2 corresponding to the historical frame. The duration between the starting position of the virtual frame corresponding to feature 3 and the ending position of the virtual frame corresponding to feature 3 is the second duration of feature 3 corresponding to the historical frame.

[0289] Combined with the above description, it can be known that when feature 1 is matched in the Nth frame, the difference between the first duration of feature 1 corresponding to the historical frame and the second duration of feature 1 corresponding to the historical frame is the duration between the starting position of the virtual frame where feature 2 is located and the ending position of the virtual frame where feature 3 is located. The difference between the time T5 when the synthesis of the Nth frame starts and the above difference is T6 (i.e., the value of the first latest response time after update). At this time, the updated first latest response time T6 refers to the time when the rendering instruction corresponding to feature 1 in the Nth frame (i.e., the rendering instruction issued by the game application intercepted in the time period from T1 to T2 based on the Nth frame) is allowed to start executing at the latest.

[0290] Similarly, when feature 2 is matched in the Nth frame, the updated first latest response time is T7. At this time, the updated first latest response time T7 refers to the time at which the rendering instruction corresponding to feature 2 in the Nth frame (i.e., the rendering instruction issued by the game application intercepted in the time period from T2 to T3 based on the Nth frame) is allowed to start executing at the latest.

[0291] Similarly, when feature 3 is matched in the Nth frame, the updated first latest response time is T5. At this time, the updated first latest response time T5 refers to the time at which the rendering instruction corresponding to feature 3 in the Nth frame (i.e., the rendering instruction issued by the game application intercepted in the time period from T3 to T4 based on the Nth frame) is allowed to start executing at the latest.

[0292] Based on this, since the initial value of the first latest response time is empty, and since the first latest response time is updated once each time a first feature is matched in the Nth frame, and the updated first latest response time refers to the time at which the rendering instruction corresponding to the first feature in the Nth frame is allowed to start executing at the latest, the second frame loss prediction method can be described as follows:

[0293] Every time the interception module intercepts a rendering instruction, the prediction module obtains the first latest response time, and when the first latest response time is obtained, the prediction module performs frame loss prediction for the Nth frame based on the time when the rendering instruction is intercepted and the obtained first latest response time.

[0294] If the first latest response time is not obtained, it means that the value of the first latest response time is empty. In other words, before the rendering instruction is intercepted, the first feature has not been matched in the Nth frame. The first latest response time has not been updated, and the first latest response time is still empty. Based on this, since the first latest response time is not obtained, the frame loss prediction is not performed for the Nth frame.

[0295] If the first latest response time is obtained, it means that the first latest response time has been updated at least once. In other words, before the rendering instruction is intercepted, at least one first feature is matched in the Nth frame, and the first latest response time is updated at least once based on the relevant data of the at least one matched first feature. Based on this, since the first latest response time is obtained, the frame loss prediction is performed on the Nth frame based on the time when the rendering instruction is intercepted and the obtained first latest response time.

[0296] It should be noted that the first latest response time obtained is updated when the first feature is matched most recently in the Nth frame before the rendering instruction is intercepted. For ease of description and distinction, the first feature matched most recently in the Nth frame before the rendering instruction is intercepted is used as the second feature, wherein at least one feature includes the second feature.

[0297] Based on this, the obtained first latest response time refers to the time at which the rendering instruction corresponding to the second feature in the Nth frame is allowed to start executing at the latest.

[0298] The first latest response time obtained is updated according to the time when the second feature is matched in the historical frame, the time when the historical frame ends, the time when the synthesis of the Nth frame starts, and the first time.

[0299] Among them, when the second feature is not the last one matched in the Nth frame, the first moment is the moment when the first feature is matched in the historical frame. The first feature here is the first feature that was matched in the Nth frame most recently after the rendering instruction was intercepted (combined with the execution order of each module in the above text, the first feature here is the first feature recognized by the recognition module). When the second feature is the last one matched in the Nth frame, the first moment is the moment when the historical frame is matched to the end. The specific implementation process can be found above and will not be repeated here.

[0300] It can be seen from the above that, in the case of each interception of a rendering instruction, the first latest response moment is obtained, and in the case of obtaining the first latest response moment, the frame loss prediction of the Nth frame is performed according to the moment of intercepting the rendering instruction and the first latest response moment obtained, which provides a method for making targeted predictions of frame loss based on the frame dimension. In addition, in combination with the above, it can be seen that during the operation of the game application, when the first duration and the second duration of the candidate feature corresponding to a frame of the game application are known, the first duration and the second duration of the candidate feature corresponding to another frame of the game application can be respectively inferred according to the first duration and the second duration of the candidate feature corresponding to the frame, and the accuracy of the inferred first duration and the second duration of the candidate feature corresponding to another frame of the game application is high. Among them, the candidate feature is any one of the at least one feature. In this way, in the second frame loss prediction method, since the obtained first latest response moment is updated according to the moment of matching the second feature in the historical frame, the moment of matching the end of the historical frame, the moment of starting to synthesize the Nth frame, and the first moment. In other words, in the process of updating the first latest response moment, the first duration of the second feature corresponding to the historical frame can be determined according to the moment when the second feature is matched in the historical frame and the moment when the historical frame ends, and the second duration of the second feature corresponding to the historical frame can be determined according to the moment and the first moment when the second feature is matched in the historical frame, and then the first latest response moment is updated according to the moment when the Nth frame is synthesized, the first duration and the second duration of the second feature corresponding to the historical frame. In combination with the above, it can be known that the first duration and the second duration of the second feature corresponding to the historical frame can be used as the first duration and the second duration of the second feature corresponding to the estimated Nth frame, respectively, and the accuracy of the estimated first duration and the second duration of the second feature corresponding to the Nth frame is high, thereby improving the accuracy of the updated first latest response moment, thereby making the accuracy of the first latest response moment high, and further improving the accuracy of the frame loss prediction. In addition, since any one of the at least one feature appears and only appears once in different frames of the application, for any two of the at least one feature, the order in which the two features are matched in any frame of the application is fixed, and since at least one feature includes the second feature, the reliability of updating the first latest response moment obtained is ensured, thereby ensuring the reliability of frame loss prediction.

[0301] For example, Fig. 20 As shown in FIG. 1 , if the time when the rendering instruction is intercepted is within the time period from T0 to T1, since the first feature has not been matched in the Nth frame before the time when the rendering instruction is intercepted, the value of the first latest response time is empty. In this way, when the rendering instruction is intercepted, since the first latest response time cannot be obtained, no frame loss prediction is performed on the Nth frame.

[0302] If the moment when the rendering instruction is intercepted is within the time period from T1 to T2, then since feature 1 (i.e., the second feature) is matched in the Nth frame before the moment when the rendering instruction is intercepted, the first latest response moment obtained is updated when feature 1 is matched in the Nth frame. In this way, when the rendering instruction is intercepted, since the first latest response moment can be obtained, the frame loss prediction of the Nth frame is performed based on the moment when the rendering instruction is intercepted and the first latest response moment obtained, wherein the first latest response moment obtained refers to the moment when the rendering instruction corresponding to feature 1 in the Nth frame is allowed to start executing at the latest.

[0303] Similarly, if the moment when the rendering instruction is intercepted is within the time period from T2 to T3, the first latest response moment obtained is updated when feature 2 (i.e., the second feature) is matched in the Nth frame. The first latest response moment obtained refers to the moment when the rendering instruction corresponding to feature 2 in the Nth frame is allowed to start executing at the latest. In this way, when the rendering instruction is intercepted, the frame loss prediction of the Nth frame is performed based on the moment when the rendering instruction is intercepted and the first latest response moment obtained.

[0304] Similarly, if the moment when the rendering instruction is intercepted is within the time period from T3 to T4, the first latest response moment obtained is updated when feature 3 (i.e., the second feature) is matched in the Nth frame. The first latest response moment obtained refers to the moment when the rendering instruction corresponding to feature 3 in the Nth frame is allowed to start executing at the latest. In this way, when the rendering instruction is intercepted, the frame loss prediction of the Nth frame is performed based on the moment when the rendering instruction is intercepted and the first latest response moment obtained.

[0305] Exemplarily, according to the time when the rendering instruction is intercepted and the first latest response time obtained, the method of predicting the frame loss of the Nth frame includes but is not limited to the following two methods:

[0306] Method 1: Compare the first latest response moment obtained with the moment when the rendering instruction is intercepted. If the moment when the rendering instruction is intercepted is greater than the first latest response moment obtained, it is predicted that there is frame loss in the Nth frame. If the moment when the rendering instruction is intercepted is not greater than the first latest response moment obtained, it is predicted that there is no frame loss in the Nth frame.

[0307] Method 2: First, the second latest response time can be determined by the following formula.

[0308] AllowTime=(PresetCoe*DynamicTime1)

[0309] Wherein, AllowTime is the second latest response time, PresetCoe is the preset response coefficient, and DynamicTime1 is the first latest response time obtained.

[0310] Then, the second latest response time is compared with the time when the rendering instruction is intercepted. If the time when the rendering instruction is intercepted is greater than the second latest response time, it is predicted that there is frame loss in the Nth frame. If the time when the rendering instruction is intercepted is not greater than the second latest response time, it is predicted that there is no frame loss in the Nth frame.

[0311] Based on this, optionally, the frame loss prediction service can also set up an acquisition module. The acquisition module can be used to monitor the Vsync signal time and the number of frame accumulations (i.e., the number of frame accumulations in storage queue 1), and when the Vsync signal time and / or the number of frame accumulations change, update the Vsync signal time and / or the number of frame accumulations to provide data basis for determining the moment to start synthesizing the Nth frame. It should be noted that when the game application starts running, the acquisition module starts working, and when the recognition module matches the first feature, the prediction module obtains the Vsync signal time and the number of frame accumulations from the acquisition module to determine the moment to start synthesizing the Nth frame.

[0312] From the above, it can be seen that since the Vsync signal is used to synchronously control frame synthesis and frame display, and since the Vsync signal time and the number of frame accumulations are monitored, and when the Vsync signal time and / or the number of frame accumulations are monitored to change, the Vsync signal time and / or the number of frame accumulations are updated, therefore, in the process of determining the moment to start synthesizing the Nth frame, the obtained Vsync signal time and the number of frame accumulations are the most recently updated. Therefore, the moment to start synthesizing the Nth frame determined by the obtained Vsync signal time and the number of frame accumulations and the Vsync signal period is highly accurate, thereby further improving the accuracy of frame loss prediction.

[0313] Optionally, the initial state of the Nth frame is that frame loss is not predicted. Based on this, when the prediction module predicts that frame loss exists in the Nth frame, the current state of the Nth frame is configured as predicted frame loss.

[0314] Exemplarily, a frame loss flag may be set for the Nth frame to indicate the current state of the Nth frame through the frame loss flag. The initial value of the frame loss flag is false, and “false” indicates that no frame loss is predicted for the Nth frame. When the prediction module predicts that frame loss exists for the Nth frame, the frame loss flag is modified to true, and “true” indicates that frame loss is predicted for the Nth frame.

[0315] On this basis, optionally, in order to reduce the workload of the prediction module and save resources, in the first frame loss prediction method, the prediction module can perform frame loss prediction on the Nth frame in the following manner.

[0316] The prediction module can predict frame loss for the Nth frame according to the time when the first feature is matched, the time when the first feature is matched in the historical frame, and the time when the historical frame ends when the first feature is matched, when the recognition module matches the first feature and the current state of the Nth frame is that frame loss is not predicted. The specific implementation process can be found above.

[0317] Optionally, similarly, in order to reduce the workload of the prediction module and save resources, in the second frame loss prediction method, the prediction module may specifically perform frame loss prediction on the Nth frame in the following manner.

[0318] The prediction module can obtain the first latest response time when the interception module intercepts a rendering instruction and the current state of the Nth frame is that no frame loss is predicted, and, when the first latest response time is obtained, perform frame loss prediction for the Nth frame according to the time when the rendering instruction is intercepted and the obtained first latest response time. The specific implementation process can be found above and will not be repeated here.

[0319] Optionally, in order to further reduce the workload of the prediction module and save resources, the prediction module may specifically update the first latest response time in the following manner.

[0320] The prediction module can update the first latest response time according to the time when the first feature is matched, the time when the first feature is matched in the historical frame, and the time when the historical frame ends, when it is predicted that there is no frame loss in the Nth frame, according to the time when the first feature is matched in the historical frame, the time when the historical frame ends, the time when the Nth frame starts to be synthesized, and the second time. The specific implementation process can be found above and will not be repeated here.

[0321] Optionally, the frame loss prediction service may also include an offline determination module ( Fig.18 (not shown), the offline determination module may be used to determine at least one feature and at least one identification strategy of the game application.

[0322] The offline determination module determines at least one feature and at least one identification strategy of the game application at the following times:

[0323] First, after the game application is installed on the electronic device, the offline determination module determines at least one feature and at least one identification strategy of the game application.

[0324] Second, after the game application is updated, the offline determination module determines at least one feature and at least one identification strategy of the game application.

[0325] Thirdly, after the system or chip of the electronic device is updated, the offline determination module determines at least one feature and at least one identification strategy of the game application.

[0326] It should be noted that in some other possible implementations, the characteristics and identification strategies of the game application can also be determined by the background server of the game application. Based on this, when the game application is installed on the electronic device, the background server can synchronize the characteristics and identification strategies of the game application to the electronic device. In addition, after the game application is updated, the background server can redetermine the characteristics and identification strategies of the game application, and synchronize the redetermined characteristics and identification strategies to the electronic device.

[0327] It should be noted that the method for determining the characteristics and identification strategies of game applications can refer to the relevant instructions above.

[0328] Optionally, the frame loss prediction service can also influence the decision of the system resource scheduler according to the frame loss prediction result, so that the system resource scheduler can make reasonable calls to system resources to avoid frame loss. For example, when it is predicted that there is frame loss in the Nth frame, the prediction module sends frame loss indication information to the system resource scheduler, so that the system resource scheduler responds to the frame loss indication information and schedules system resources to speed up the rendering of the Nth frame, thereby avoiding the occurrence of frame loss.

[0329] It should be noted that the application framework layer may also include a window manager, a content provider, a view system, a phone manager, a notification manager, etc. The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to the application. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text notification icon may include a view for displaying text and a view for displaying pictures. The phone manager is used to provide communication functions for electronic devices. For example, the management of call status (including connected, hung up, etc.). The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify the completion of downloads, message reminders, etc. The notification manager can also be a notification that appears in the top status bar of the system in the form of an icon or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0330] like Fig.18As shown, the system resource layer may include: central processing unit (CPU), graphics processing unit (GPU), double rate synchronous dynamic random access memory (DDR), cache, I / O input / output, SurfaceView, temperature control, screen resolution, screen refresh rate, touch, volume, brightness, network, etc. Among them, the CPU can be used to control each module in the application framework layer to realize their respective functions, and the GPU can be used to perform corresponding rendering processing according to the API called by the instructions processed by each module in the application framework layer. I / O input / output is divided into two parts: IO device and IO interface. SurfaceView is the inherited class of View, which has a built-in Surface specifically for drawing, and can draw to the screen in a thread other than the main thread.

[0331] It should be noted that, for each other frame of the game application, the processing of each other frame by each module can refer to the processing of the Nth frame by each module.

[0332] It should be noted that the above description of the hierarchical structure of the electronic device is only exemplary and is not intended to limit the hierarchical structure of the electronic device in this application.

[0333] Below Fig.18 Taking the hierarchical structure of the electronic device shown as an example, the interaction between the software modules involved in the frame loss prediction is exemplarily described. Fig.21 The following is a schematic diagram of the interaction between software modules provided in the embodiment of the present application. Fig.21 Said may include:

[0334] 2101. The game application issues rendering instructions based on the Nth frame.

[0335] In the embodiment of the present application, the number of rendering instructions issued based on the Nth frame is multiple.

[0336] 2102. The interception module intercepts the rendering instruction issued based on the Nth frame.

[0337] In the embodiment of the present application, the interception module intercepts rendering instructions one by one.

[0338] In a possible implementation, the interception module may indiscriminately intercept rendering instructions issued by the game application based on the Nth frame.

[0339] In another possible implementation, the interception module may only intercept rendering instructions corresponding to at least one identification strategy of the game application. As can be seen from the above, the identification strategy is obtained based on the analysis of specific rendering instructions, so the specific rendering instructions can be used as rendering instructions corresponding to the identification strategy. In this way, the interception module only intercepts rendering instructions corresponding to at least one identification strategy during the process of intercepting rendering instructions, which can reduce the number of intercepted rendering instructions, reduce the demand for cache space, and also reduce the number of matches. It should be noted that the interception module also intercepts the end-of-frame rendering instruction of the Nth frame.

[0340] 2103. Whenever the interception module intercepts a rendering instruction, it transmits the rendering instruction to the cache module.

[0341] Optionally, 2104. the interception module may also transmit the rendering instruction to the prediction module each time a rendering instruction is intercepted.

[0342] It should be noted that there is no order of execution between 2103 and 2104.

[0343] Optionally, 2105. the prediction module receives and responds to the rendering instruction, obtains the first latest response time, and, when the first latest response time is obtained, performs frame loss prediction for the Nth frame according to the time when the rendering instruction is intercepted and the obtained first latest response time. The specific implementation process can be referred to above.

[0344] It should be noted that in another possible implementation method, the implementation process of 2105 can be specifically as follows: the prediction module receives and responds to the rendering instruction, and when the current state of the Nth frame is that frame loss is not predicted, the first latest response moment is obtained, and when the first latest response moment is obtained, the frame loss of the Nth frame is predicted based on the moment when the rendering instruction is intercepted and the first latest response moment obtained.

[0345] 2106. After receiving each rendering instruction, the cache module updates the rendering instruction to the cache queue. Optionally, after receiving each rendering instruction, the cache module may also obtain the status information of the rendering instruction, and update the rendering instruction and the status information of the rendering instruction to the cache queue. The specific implementation is detailed above.

[0346] 2107. After each update of the information in the cache queue (i.e., cached rendering instructions and / or status information of rendering instructions, etc.), the cache module sends update indication information to the identification module to indicate that the information in the cache queue has been updated through the update indication information.

[0347] 2108. The identification module receives and responds to the update indication information, and performs a feature match on the information currently cached in the cache queue (rendering instructions and / or state information of rendering instructions, etc.) through at least one identification strategy of the game application. The specific implementation method can be referred to above.

[0348] 2109. When the recognition module matches the first feature, the recognition module sends matching indication information to the statistical module and the prediction module. The matching indication information is used to indicate the matching of the first feature and the time when the first feature is matched.

[0349] It should be noted that the recognition module also needs to match the end-of-frame rendering instruction of the Nth frame, and when matching the end-of-frame rendering instruction of the Nth frame, transmit the time when the end-of-frame rendering instruction of the Nth frame is matched to the statistics module.

[0350] 2110. The statistics module receives and responds to the matching indication information, and records the time when the first feature is matched as the time when the first feature is matched in the Nth frame. It should be noted that the statistics module also needs to record the time when the frame end rendering instruction of the Nth frame is matched as the time when the Nth frame is matched. The statistics module also records the time when each first feature is matched in the historical frame and the time when the historical frame is matched.

[0351] 2111. The prediction module receives and responds to the matching indication information, and performs frame loss prediction for the Nth frame according to the time when the first feature is matched (obtained from the matching indication information), the time when the first feature is matched in the historical frame, and the time when the historical frame ends. The specific implementation process can be referred to above. The time when the first feature is matched in the historical frame and the time when the historical frame ends can be obtained from the statistical module.

[0352] It should be noted that in another possible implementation, the process of 2111 can also be specifically as follows: the prediction module receives and responds to the matching indication information, and when the current state of the Nth frame is that no frame loss is predicted, the frame loss prediction for the Nth frame is performed based on the moment when the first feature is matched, the moment when the first feature is matched in the historical frame, and the moment when the historical frame ends.

[0353] It should be noted that there is no difference in the execution order between 2110 and 2111.

[0354] Optionally, 2112. the prediction module may also configure the current state of the Nth frame as predicted frame loss when predicting that frame loss exists in the Nth frame. It should be noted that the initial state of the Nth frame is that no frame loss is predicted.

[0355] It should be noted that when it is predicted that frame loss occurs in the Nth frame during the prediction process of 2111 and 2105, 2112 is executed.

[0356] Optionally, 2113. the prediction module receives and responds to the matching indication information, updates the first latest response time, and provides data support for executing 2105. The specific implementation process is as mentioned above.

[0357] It should be noted that there is no order of execution between 2113 and 2111. In another possible implementation, the specific execution process of 2113 may be: when it is predicted in 2111 that there is no frame loss in the Nth frame, the first latest response time is updated.

[0358] Optionally, 2114. The acquisition module can monitor the Vsync signal time and the number of frame accumulations, and update the Vsync signal time and / or the number of frame accumulations when the Vsync signal time and / or the number of frame accumulations change. It should be noted that there is no limitation on the order of 2114 and the above steps. 2114 starts working after the game application is started. 2114 can provide data support for the prediction module to determine the time to start synthesizing the Nth frame.

[0359] It should be noted that when the Nth frame is rendered, the frame loss prediction process for the Nth frame is started. That is, during the rendering of the Nth frame, the frame loss prediction process is executed to predict the Nth frame. When it is predicted that the Nth frame has a frame loss, the system resource scheduler is used to quickly schedule system resources to speed up the rendering of the Nth frame, so as to complete the rendering of the Nth frame before the moment of starting to synthesize the Nth frame arrives, thereby avoiding frame loss.

[0360] Below, we take the game application as an example and combine Figure 22 to Figure 24 The frame loss prediction is described as follows. The game application includes three features, namely, feature 1, feature 2, and feature 3. The game application includes three recognition strategies, namely, recognition strategy 1, recognition strategy 2, and recognition strategy 3.

[0361] It should be noted that Figure 22 to Figure 24 The solid-line frame in indicates information of frame loss prediction for the Nth frame during the rendering process of the Nth frame of the game application, and the dotted-line frame indicates information of historical frames of the game application.

[0362] like Fig. 22 As shown, during the rendering process of the Nth frame, the first features matched from the Nth frame by the three recognition strategies are: feature 1, feature 2, and feature 3. The three arrows respectively indicate the moments when feature 1, feature 2, and feature 3 are matched in the Nth frame.

[0363] The duration between the starting position of the virtual frame where feature 1 is located and the ending position of the virtual frame where feature 3 is located indicates the first duration of feature 1 corresponding to the historical frame. The duration between the starting position of the virtual frame where feature 2 is located and the ending position of the virtual frame where feature 3 is located indicates the first duration of feature 2 corresponding to the historical frame. The duration between the starting position of the virtual frame where feature 3 is located and the ending position of the virtual frame where feature 3 is located indicates the first duration of feature 3 corresponding to the historical frame.

[0364] VSYNC1 is the time to start synthesizing the Nth frame. Fig. 22 It can be seen that when feature 1 is matched, the rendering end time of the Nth frame (i.e., the time corresponding to the end position of the virtual frame where feature 3 is located) determined according to the first duration of feature 1 corresponding to the time when feature 1 is matched in the Nth frame and the historical frame is before the VSYNC1 signal. Therefore, based on feature 1, it is predicted that there is no frame loss in the Nth frame. Similarly, when feature 2 is matched, it is predicted that there is no frame loss in the Nth frame based on feature 2. Similarly, when feature 3 is matched, it is predicted that there is no frame loss in the Nth frame based on feature 3.

[0365] like Fig.23 It is shown that during the rendering process of the Nth frame, the first feature matched from the Nth frame by the three recognition strategies is feature 1. The arrow indicates the moment when feature 1 is matched in the Nth frame. The duration between the starting position of the virtual frame where feature 1 is located and the ending position of the virtual frame where feature 3 is located indicates the first duration of feature 1 corresponding to the historical frame. VSYNC1 is the moment when the synthesis of the Nth frame starts.

[0366] Depend on Fig.23 It can be seen that when feature 1 is matched, the rendering end time of the Nth frame (i.e., the time corresponding to the end position of the virtual frame where feature 3 is located) determined according to the time when feature 1 is matched in the Nth frame and the first duration of feature 1 corresponding to the historical frame is after the VSYNC1 signal. Therefore, it is predicted that there is frame loss in the Nth frame based on feature 1. At this time, the frame rendering of the Nth frame can be accelerated by quickly scheduling system resources to avoid frame loss.

[0367] like Fig.24 It shows that during the rendering process of the Nth frame, the first feature matched from the Nth frame by the three recognition strategies is feature 1, and the first feature matched next is feature 2. The two arrows on the upper side respectively indicate the moments when feature 1 and feature 2 are matched in the Nth frame.

[0368] The duration between the starting position of the virtual frame where feature 1 is located and the ending position of the virtual frame where feature 3 is located indicates the first duration of feature 1 corresponding to the historical frame. The duration between the starting position of the virtual frame where feature 2 is located and the ending position of the virtual frame where feature 3 is located indicates the first duration of feature 2 corresponding to the historical frame. The duration between the starting position of the virtual frame where feature 2 is located and the starting position of the virtual frame where feature 3 is located indicates the second duration of feature 2 corresponding to the historical frame. VSYNC1 is the time when the synthesis of the Nth frame starts.

[0369] Depend on Fig.24 It can be seen that when feature 1 is matched, the rendering end time of the Nth frame (i.e., the time corresponding to the end position of the virtual frame where feature 3 is located) determined based on the time when feature 1 is matched in the Nth frame and the first duration of feature 1 corresponding to the historical frame is before the VSYNC1 signal. Therefore, it is predicted that there is no frame loss in the Nth frame based on feature 1. Similarly, when feature 2 is matched, it is predicted that there is no frame loss in the Nth frame based on feature 2.

[0370] When feature 2 is matched, the first latest response time is updated. Fig.24 In FIG. 1 , the lower arrow indicates the updated first latest response time. After the first latest response time is updated according to feature 2, when a rendering instruction is intercepted, due to the time when the rendering instruction is intercepted (i.e. Fig.24 The time corresponding to the position indicated by the black triangle in the figure is after the first latest response time after the update, so it is predicted based on the rendering instruction that the Nth frame has a frame loss. At this time, the rendering of the Nth frame can be accelerated by quickly scheduling system resources to avoid the occurrence of frame loss.

[0371] Based on the above description, Fig.25 A flowchart of a frame loss prediction method provided in an embodiment of the present application is shown in FIG. Fig.25 As shown, the method may include:

[0372] 2501. Intercept the rendering instruction issued by the application based on the Nth frame. The application refers to an application that displays a video to a user by rendering frames in real time, such as a game application. The specific implementation process here is detailed in 2102 above and the relevant instructions in the interception module. The application has at least one identification strategy and at least one feature. Among them, at least one identification strategy corresponds to at least one feature.

[0373] 2502. Every time a rendering instruction is intercepted, a feature match is performed on at least one intercepted rendering instruction through at least one recognition strategy.

[0374] Exemplarily, the specific execution process of 2502 may be as follows:

[0375] Each time a rendering instruction is intercepted, the intercepted rendering instruction (or the rendering instruction and the status information of the rendering instruction, etc.) is cached in a cache queue, and each time the information in the cache queue is updated, a feature match is performed on the information currently stored in the cache queue (at least one rendering instruction that has been intercepted / at least one rendering instruction that has been intercepted and the status information of at least one rendering instruction, etc.) through at least one recognition strategy.

[0376] For the detailed implementation process of this part, please refer to the relevant instructions in 2106 to 2109, the identification module and the cache module.

[0377] 2503. When the first feature is matched, a frame loss prediction is performed on the Nth frame according to the time when the first feature is matched, the time when the first feature is matched in the historical frame, and the time when the historical frame ends.

[0378] Among them, at least one feature includes a first feature. Before this match, the first feature was not matched in the Nth frame, and the historical frame is at least a part of the frames before the Nth frame. Any one of the at least one feature appears in different frames of the application and only appears once, and the first duration of the candidate feature corresponding to multiple frames of the application including the Nth frame conforms to the normal distribution. The candidate feature is any one of the at least one feature, and the first duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame of the multiple frames and the moment when the matching of the any frame ends.

[0379] For the specific implementation process of 2503, please refer to the relevant instructions in 2111 and the first frame loss prediction method in the prediction module.

[0380] As can be seen from the above, by intercepting the rendering instructions issued by the application based on the Nth frame, and in the case of each interception of a rendering instruction, at least one of the intercepted rendering instructions is matched with features through at least one recognition strategy, and in the case of matching the first feature, according to the time of matching the first feature, the time of matching the first feature in the historical frame and the time of matching the end of the historical frame, the Nth frame is predicted to be lost, which provides a method for targeted prediction of frame loss based on the frame dimension. In addition, since the candidate feature is any one of the at least one feature, the first duration of the candidate feature corresponding to the multiple frames of the application including the Nth frame conforms to the normal distribution, and the first duration of the candidate feature is the duration between the time when the candidate feature is matched in any frame of the multiple frames to the time when the arbitrary frame is matched. Therefore, during the running of the application, when the first duration of the candidate feature corresponding to one frame of the application is known, the first duration of the candidate feature corresponding to another frame of the application can be inferred based on the first duration of the candidate feature corresponding to the frame, and the accuracy of the inferred first duration of the candidate feature corresponding to another frame of the application is high. In this way, in the process of predicting the frame loss of the Nth frame according to the moment of matching the first feature, the moment of matching the first feature in the historical frame, and the moment of matching the end of the historical frame, the first duration of the first feature corresponding to the historical frame can be determined according to the moment of matching the first feature in the historical frame and the moment of matching the end of the historical frame. In combination with the above, it can be known that the first duration corresponding to the first feature corresponding to the historical frame can be used as the first duration of the first feature corresponding to the estimated Nth frame, and the accuracy of the first duration of the estimated Nth frame is high. Based on this, the rendering end time of the Nth frame can be determined according to the moment of matching the first feature and the first duration of the first feature corresponding to the historical frame, and the accuracy of the determined rendering end time of the Nth frame is high. In this way, the frame loss prediction of the Nth frame can be performed according to the rendering end time of the Nth frame, thereby improving the accuracy of the frame loss prediction. In addition, since any one of the at least one feature appears in different frames of the application and only appears once, and at least one feature includes the first feature, it can ensure that the first feature is matched in the Nth frame, thereby ensuring the reliability of the frame loss prediction method provided by the present application.

[0381] In a possible implementation process, according to the time when the first feature is matched, the time when the first feature is matched in the historical frame, and the time when the historical frame ends, the process of predicting the frame loss for the Nth frame can be as follows:

[0382] First, according to the time when the first feature is matched in the historical frame and the time when the historical frame is matched to the end, the first duration of the first feature corresponding to the historical frame is determined. Then, according to the time when the first feature is matched and the first duration of the first feature corresponding to the historical frame, the rendering end time of the Nth frame is determined. Finally, according to the rendering end time of the Nth frame and the time when the synthesis of the Nth frame starts, the frame loss prediction is performed on the Nth frame.

[0383] Exemplarily, the method for determining the time to start synthesizing the Nth frame is: when the first feature is matched, the Vsync signal time and the number of frame accumulations are obtained. The time to start synthesizing the Nth frame is determined based on the obtained Vsync signal time, the obtained number of frame accumulations and the Vsync signal period.

[0384] Exemplarily, according to the rendering end time of the Nth frame and the synthesis start time of the Nth frame, the method of predicting the frame loss of the Nth frame includes but is not limited to the following two methods:

[0385] The first method is to compare the time when the synthesis of the Nth frame starts with the time when the rendering of the Nth frame ends, and determine whether there is a frame loss in the Nth frame according to the comparison result.

[0386] The second method is to determine the timeout ratio according to the time when the Nth frame is synthesized, the time when the Nth frame is rendered, and the first duration of the first feature corresponding to the historical frame, and predict the frame loss of the Nth frame according to the relative size of the timeout ratio and the preset ratio.

[0387] In order to reduce the workload and save resources, in a possible implementation, when the first feature is matched, according to the time when the first feature is matched, the time when the first feature is matched in the historical frame, and the time when the historical frame ends, the specific implementation process of performing frame loss prediction on the Nth frame can be as follows:

[0388] When the first feature is matched and the current state of the Nth frame is that no frame loss is predicted, frame loss is predicted for the Nth frame according to the time when the first feature is matched, the time when the first feature is matched in the historical frame, and the time when the historical frame ends.

[0389] Optionally, for any two features among at least one feature, the order of the moments when any two features are matched in any frame of the application is fixed. The second duration of the candidate features corresponding to multiple frames of the application including the Nth frame conforms to the normal distribution. In the case where the candidate feature is not the last one matched in any frame among the multiple frames, the second duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame and the moment when the next adjacent feature is matched in any frame. In the case where the candidate feature is the last one matched in any frame among the multiple frames, the second duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame and the moment when the matching ends in any frame.

[0390] Based on this, the frame loss prediction method may also include:

[0391] When a rendering instruction is intercepted, the first latest response time is obtained. When the first latest response time is obtained, a frame loss prediction is performed on the Nth frame according to the time when the rendering instruction is intercepted and the obtained first latest response time.

[0392] Among them, the first latest response moment obtained refers to the moment that allows the latest start of execution of the rendering instruction corresponding to the second feature in the Nth frame. The first latest response moment obtained is updated according to the moment when the second feature is matched in the historical frame, the moment when the historical frame is matched to the end, the moment when the synthesis of the Nth frame starts, and the first moment. At least one feature includes a second feature, and the second feature is the first feature that was most recently matched in the Nth frame before the rendering instruction was intercepted. In the case where the second feature is not the last one matched in the Nth frame, the first moment is the moment when the first feature is matched in the historical frame. In the case where the second feature is the last one matched in the Nth frame, the first moment is the moment when the historical frame is matched to the end. The specific implementation process can be referred to above.

[0393] As can be seen from the above, by intercepting the rendering instructions issued by the application based on the Nth frame, and obtaining the first latest response time each time a rendering instruction is intercepted, and when the first latest response time is obtained, the Nth frame is predicted to be lost according to the moment of intercepting the rendering instruction and the first latest response time obtained, a method for predicting frame loss based on the frame dimension is provided. In addition, since the candidate feature is any one of the at least one feature, the first duration and the second duration of the candidate feature corresponding to the multiple frames of the application including the Nth frame are both in accordance with the normal distribution, and the first duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame and the moment when the arbitrary frame is matched. In the case where the candidate feature is not the last one to be matched in any frame among the multiple frames, the second duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame and the moment when the next adjacent feature is matched in the arbitrary frame. In the case where the candidate feature is the last one to be matched in any frame among the multiple frames, the second duration of the candidate feature is the duration between the moment when the candidate feature is matched in any frame and the moment when the arbitrary frame is matched. Therefore, during the running of the application, when the first duration and the second duration of the candidate feature corresponding to one frame of the application are known, the first duration and the second duration of the candidate feature corresponding to another frame of the application can be inferred based on the first duration and the second duration of the candidate feature corresponding to the frame, and the accuracy of the inferred first duration and the second duration of the candidate feature corresponding to another frame of the application is high. In this way, the first latest response moment obtained is updated based on the moment when the second feature is matched in the historical frame, the moment when the historical frame ends, the moment when the Nth frame is started to be synthesized, and the first moment. In other words, in the process of updating the first latest response moment obtained, the first duration of the second feature corresponding to the historical frame can be determined based on the moment when the second feature is matched in the historical frame and the moment when the historical frame ends, the second duration of the second feature corresponding to the historical frame can be determined based on the moment when the second feature is matched in the historical frame and the first moment, and then the first latest response moment obtained is updated based on the moment when the Nth frame starts to be synthesized, the first duration and the second duration of the second feature corresponding to the historical frame. In combination with the above, it can be seen that the first duration and the second duration of the second feature corresponding to the historical frame can be respectively used as the first duration and the second duration of the second feature corresponding to the estimated N-th frame, and the accuracy of the first duration and the second duration of the second feature corresponding to the estimated N-th frame is high, thereby improving the accuracy of the updated first latest response moment, thereby making the accuracy of the obtained first latest response moment high, thereby improving the accuracy of frame loss prediction.In addition, since any one of the at least one feature appears and only appears once in different frames of the application, for any two of the at least one feature, the order in which the two features are matched in any frame of the application is fixed, and since at least one feature includes the second feature, the reliability of updating the first latest response moment obtained is ensured, thereby ensuring the reliability of frame loss prediction.

[0394] In order to reduce the workload and save resources, in a possible implementation, when a rendering instruction is intercepted, the first latest response time is obtained. When the first latest response time is obtained, the specific implementation process of performing frame loss prediction on the Nth frame according to the time when the rendering instruction is intercepted and the obtained first latest response time can be as follows:

[0395] When a rendering instruction is intercepted and the current state of the Nth frame is that frame loss is not predicted, the first latest response time is obtained. When the first latest response time is obtained, frame loss prediction is performed on the Nth frame according to the time when the rendering instruction is intercepted and the obtained first latest response time.

[0396] Based on this, the method further includes:

[0397] When the first feature is matched, the first latest response time is updated according to the time when the first feature is matched in the historical frame, the time when the historical frame ends, the time when the synthesis of the Nth frame starts, and the second time.

[0398] Among them, the initial value of the first latest response time is empty. When the first feature is not the last one matched in the historical frame, the second time is the time when the third feature is matched in the historical frame. At least one feature includes a third feature, and the third feature is a feature that is matched after the first feature and adjacent to the first feature in the feature matching of the historical frame. When the first feature is the last one matched in the historical frame, the second time is the time when the matching of the historical frame ends. The updated first latest response time refers to the time that allows the latest start of execution of the rendering instruction corresponding to the first feature in the Nth frame.

[0399] As can be seen from the above, in the case of matching the first feature, the first latest response time is updated according to the time when the first feature is matched in the historical frame, the time when the historical frame ends, the time when the synthesis of the Nth frame begins, and the second time, providing a way to update the first latest response time. In addition, since the candidate feature is any one of the at least one feature, the first duration and the second duration of the candidate feature corresponding to the multiple frames of the application including the Nth frame are both in accordance with the normal distribution, and the first duration of the candidate feature is the duration between the time when the candidate feature is matched in any frame and the time when the arbitrary frame ends. In the case where the candidate feature is not the last one to be matched in any frame among the multiple frames, the second duration of the candidate feature is the duration between the time when the candidate feature is matched in any frame and the time when the next adjacent feature is matched in the arbitrary frame. In the case where the candidate feature is the last one to be matched in any frame among the multiple frames, the second duration of the candidate feature is the duration between the time when the candidate feature is matched in any frame and the time when the arbitrary frame ends. Therefore, during the running of the application, when the first duration and second duration of the candidate feature corresponding to one frame of the application are known, the first duration and second duration of the candidate feature corresponding to another frame of the application can be inferred based on the first duration and second duration of the candidate feature corresponding to the frame, and the accuracy of the inferred first duration and second duration of the candidate feature corresponding to the other frame of the application is high. In this way, in the process of updating the first latest response time based on the moment when the first feature is matched in the historical frame, the moment when the historical frame ends, the moment when the Nth frame starts to be synthesized, and the second moment, the first duration of the first feature corresponding to the historical frame can be determined based on the moment when the first feature is matched in the historical frame and the moment when the historical frame ends, the second duration of the first feature corresponding to the historical frame can be determined based on the moment when the first feature is matched in the historical frame and the second moment, and then the first latest response time is updated based on the moment when the Nth frame starts to be synthesized, the first duration and second duration of the first feature corresponding to the historical frame. Furthermore, as can be seen from the above, the first duration and the second duration of the first feature corresponding to the historical frame can be used as the first duration and the second duration of the first feature corresponding to the estimated Nth frame, respectively, and the accuracy of the first duration and the second duration of the first feature corresponding to the estimated Nth frame is high, thereby improving the accuracy of updating the first latest response time. In addition, since any one of the at least one feature appears and only appears once in different frames of the application, for any two of the at least one feature, the order of the moments when any two features are matched in any frame of the application is fixed, and since at least one feature includes the first feature, it is ensured that the first feature can be matched in the Nth frame, and it is ensured that the order of appearance of the first feature and other features of the at least one feature in each frame is fixed, thereby ensuring the reliability of updating the first latest response time.

[0400] In order to reduce the workload and save resources, in a possible implementation, when the first feature is matched, according to the time when the first feature is matched in the historical frame, the time when the historical frame ends, the time when the Nth frame is synthesized, and the second time, the specific implementation process of updating the first latest response time can be as follows:

[0401] When it is predicted that there is no frame loss in the Nth frame based on the moment when the first feature is matched, the moment when the first feature is matched in the historical frame and the moment when the historical frame ends, the first latest response moment is updated based on the moment when the first feature is matched in the historical frame, the moment when the historical frame ends, the moment when synthesis of the Nth frame starts and the second moment.

[0402] For the specific implementation of the above steps, please refer to the relevant description in the second frame loss prediction method in the previous text.

[0403] Optionally, the frame loss prediction method may further include: monitoring the Vsync signal time and the number of frame accumulations, and updating the Vsync signal time and / or the number of frame accumulations when the Vsync signal time and / or the number of frame accumulations change.

[0404] As can be seen from the above, by monitoring the Vsync signal time and the number of frame accumulations, the Vsync signal time and / or the number of frame accumulations can be updated when changes are detected in the monitored Vsync signal time and / or the number of frame accumulations. In this way, in the process of determining the time to start synthesizing the Nth frame, the obtained Vsync signal time and the number of frame accumulations are the latest updated, so the time to start synthesizing the Nth frame determined by the obtained Vsync signal time and the number of frame accumulations and the Vsync signal period is highly accurate.

[0405] Optionally, the frame loss prediction method may further include: the initial state of the Nth frame is that frame loss is not predicted. Based on this, the method may further include: when frame loss is predicted for the Nth frame, configuring the current state of the Nth frame as that frame loss is predicted.

[0406] Optionally, the frame loss prediction method may further include: when the first feature is matched, recording the time when the first feature is matched as the time when the first feature is matched in the Nth frame.

[0407] Optionally, the frame loss prediction method may further include: intercepting and matching the end-of-frame rendering instruction of the Nth frame, and, when the end-of-frame rendering instruction of the Nth frame is matched, recording the moment when the end-of-frame rendering instruction of the Nth frame is matched as the moment when the Nth frame ends.

[0408] Optionally, when there are multiple at least one feature, for any two of the at least one feature, the moments when the two features are matched in any frame of the application are different.

[0409] Since any two of at least one feature are matched at different times in any frame of the application, that is, in the matching process of any frame of the application, the situation where multiple first features are matched at the same time is avoided, and thus the situation where frame loss prediction is performed for the Nth frame based on multiple first features when multiple first features are matched at the same time is avoided. This ensures the accuracy of the prediction and reduces the number of frame loss predictions.

[0410] It should be noted that the relevant instructions for the above steps have been explained in detail above and will not be repeated here.

[0411] The present application also provides a chip system, which includes a processor and a communication interface; the processor is used to call and run a computer program stored in a storage medium from a storage medium to execute the method described in any method embodiment of the present application.

[0412] The present application also provides a computer program product, which, when executed by a processor, implements the method described in any method embodiment of the present application.

[0413] The computer program product can be stored in a memory, and is finally converted into an executable target file that can be executed by a processor after preprocessing, compiling, assembling and linking.

[0414] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the method described in any method embodiment of the present application is implemented. The computer program can be a high-level language program or an executable target program.

[0415] The computer-readable storage medium may be a volatile memory or a nonvolatile memory, or may include both a volatile memory and a nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM).

[0416] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0417] In several embodiments provided in the present application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0418] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0419] In addition, the terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0420] In short, the above is only a preferred embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application should be included in the protection scope of this application.

Claims

1. A frame loss prediction method, characterized in that: The method is applied to an electronic device, the electronic device includes a game application, and the method includes: The game application issues a rendering instruction for the Nth frame; Based on the recognition strategy, feature matching is performed on the rendering instruction; If the first feature is matched, a frame loss prediction is performed on the Nth frame based on the moment when the Nth frame matches the first feature, the moment when the historical frame matches the first feature, and the moment when the matching of the historical frame ends, wherein the historical frame is at least a portion of the frames before the Nth frame.

2. The method according to claim 1, characterized in that The game application includes at least one feature and at least one identification strategy, the at least one feature corresponds to the at least one identification strategy in a one-to-one manner, and the at least one feature includes the first feature.

3. The method according to claim 2, characterized in that The game application includes a candidate feature, where the candidate feature is any one of the at least one feature, and the candidate feature appears in different frames of the game application and only appears once.

4. The method according to claim 3, characterized in that The first duration of the candidate features corresponding to different frames of the game application conforms to a normal distribution, wherein the first duration of the candidate features is the duration between the moment when any frame among the different frames matches the candidate feature and the moment when the matching of any frame ends.

5. The method according to claim 1, characterized in that The performing frame loss prediction on the Nth frame based on the time when the Nth frame matches the first feature, the time when the historical frame matches the first feature, and the time when the historical frame ends matching, includes: Determining a first duration of the first feature corresponding to the historical frame based on a time when the historical frame matches the first feature and a time when the historical frame ends matching; Determining a rendering end time of the Nth frame based on a time when the Nth frame matches the first feature and a first duration of the first feature corresponding to the historical frame; Based on the rendering end time of the Nth frame and the synthesis start time of the Nth frame, frame loss prediction is performed on the Nth frame.

6. The method according to claim 5, characterized in that The performing frame loss prediction on the Nth frame based on the rendering end time of the Nth frame and the start time of synthesizing the Nth frame includes: Determine the timeout ratio based on the end time of rendering the Nth frame, the time when synthesis of the Nth frame starts, and the first duration of the first feature corresponding to the historical frame; Based on the relative size of the timeout ratio and the preset ratio, a frame loss prediction is performed on the Nth frame.

7. The method according to any one of claims 1 to 6, characterized in that If the first feature is matched, based on the time when the Nth frame matches the first feature, the time when the historical frame matches the first feature, and the time when the historical frame ends matching, performing frame loss prediction on the Nth frame, includes: If the first feature is matched and the current state of the Nth frame is that frame loss is not predicted, frame loss is predicted for the Nth frame based on the moment when the Nth frame matches the first feature, the moment when the historical frame matches the first feature, and the moment when the matching of the historical frame ends.

8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: If the first feature is matched, the Vsync signal time and frame accumulation quantity are obtained; Based on the Vsync signal time, the frame accumulation number and the Vsync signal period, a time to start synthesizing the Nth frame is determined.

9. The method according to claim 8, characterized in that The method further comprises: Monitor the Vsync signal time and the frame accumulation quantity; If the Vsync signal time and / or the frame accumulation number changes, the Vsync signal time and / or the frame accumulation number is updated.

10. The method according to any one of claims 1 to 6, characterized in that: The initial state of the Nth frame is that no frame loss is predicted; If it is predicted that the Nth frame has frame loss, the current state of the Nth frame is configured as predicted frame loss.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The processor is coupled to the memory, and the memory is used to store a computer program. When the computer program is executed by the processor, the electronic device executes the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 10.

Images