Load detection method, device, apparatus, medium, program product and chip

Abstract

The present disclosure relates to a load detection method, device, equipment, medium, program product and chip. The load detection method comprises: in the processing process of a current image frame, in response to the first processor calling the application programming interface of the second processor, obtaining the interface call information of the current image frame; based on the interface call information of the current image frame and a plurality of historical image frames, determining the load information of the current image frame. The interface call information of the current image frame and the plurality of historical image frames is used as the basis for determining the load information of the current image frame, the change of the application programming interface of the second processor called by the first processor reflects the load change of the second processor, the accuracy of the load information is ensured, the interface call information of the current image frame can be obtained before the second processor starts working, the supply of the second processor can be adjusted in time to ensure the display effect of the current image frame, and the overall performance of the second processor is improved.

Description

Technical Field

[0001] This disclosure relates to the field of load detection, specifically to a load detection method, apparatus, equipment, medium, program product, and chip. Background Technology

[0002] In recent years, with the rapid development of image processing and display technologies and the continuous iteration of related devices, the requirements for display frame rate stability and the performance requirements of related devices have been increasing. During image processing, it is necessary to detect changes in the load of the Graphics Processing Unit (GPU) in order to adjust the GPU's frequency supply accordingly, thereby avoiding frame drops due to insufficient supply or power waste due to excessive supply. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a load detection method, apparatus, device, medium, program product, and chip.

[0004] According to a first aspect of the present disclosure, a load detection method is provided, the load detection method comprising: During the processing of the current image frame, in response to the first processor calling the application programming interface of the second processor, the interface call information of the current image frame is obtained; Based on the interface call information of the current image frame and the interface call information of multiple historical image frames, the load information of the current image frame is determined. The interface call information is used to characterize the call status of the first processor to at least one of the application programming interfaces of the second processor, and the load information is used to characterize the load change of the second processor during the processing of the current image frame.

[0005] In this embodiment, during the processing of the current image frame, when the first processor calls the application programming interface (API) of the second processor, the interface call information of the current image frame is obtained. Based on this API call information and the API call information of multiple historical image frames, the load information of the current image frame is determined. This enables the detection of load changes in the second processor, facilitating adjustments to the power supply to the second processor based on the detection results to ensure frame rate stability. Using the API call information of the current image frame and multiple historical image frames as the basis for determining the load information of the current image frame, and utilizing changes in the first processor's API calls to the second processor to reflect load changes in the second processor, ensures the accuracy of the load information. Furthermore, the API call information of the current image frame can be obtained before the second processor starts working, enabling timely determination of load information and adjustment of the second processor's power supply to ensure the display effect of the current image frame, thus improving the overall performance of the second processor.

[0006] In some embodiments of this disclosure, the processing of each image frame includes a first processing process corresponding to a first processor and a second processing process corresponding to a second processor, wherein the start time of the first processing process is prior to the start time of the second processing process.

[0007] In this embodiment, the first processing procedure corresponding to the first processor and the second processing procedure corresponding to the second processor are used as the processing procedures for each image frame. Different processing functions and corresponding processing effects can be achieved through the first and second processing procedures respectively, which is beneficial to improving the processing efficiency and quality of the current image frame. The start time of the first processing procedure is configured to be before the start time of the second processing procedure, so that interface call information can be obtained after the first processing procedure starts and before the second processing procedure starts, and that load information can be determined after the first processing procedure starts and before the second processing procedure starts, further ensuring the accuracy and timeliness of the load information.

[0008] In some embodiments of this disclosure, the first processor is configured to invoke the application programming interface of the second processor during the first processing.

[0009] In this embodiment, the first processor is configured to call the application programming interface of the second processor during the first processing, so that the interface call information can be obtained after the first processing starts and before the second processing starts, and the load information can be determined after the first processing starts and before the second processing starts, thereby further ensuring the accuracy and timeliness of the load information.

[0010] In some embodiments of this disclosure, determining the load information of the current image frame includes: Before the second processing begins, the load information of the current image frame is determined.

[0011] In this embodiment, the load information of the current image frame is determined before the second processing begins, so that the load information can be determined after the first processing and before the second processing begins. This allows the supply of the second processor to be adjusted in advance based on the load information, which helps to reduce the delay in responding to sudden changes in the load of the second processor and thus improves the stability of the frame rate.

[0012] In some embodiments of this disclosure, the load detection method further includes: In response to the end of the processing of the current image frame, the interface call information of the current image frame is used as the interface call information of a historical image frame; The determination of the load information of the current image frame based on the interface call information of the current image frame and the interface call information of multiple historical image frames includes: In response to the number of historical image frames reaching a preset threshold, the load information of the current image frame is determined based on the interface call information of the current image frame and the interface call information of each of the historical image frames.

[0013] In this embodiment, at the end of the processing of the current image frame, the interface call information of the current image frame is used as the interface call information of a historical image frame. This realizes the updating and supplementation of the interface call information of historical image frames, ensuring the timeliness of the interface call information of historical image frames. When the number of historical image frames reaches a preset threshold, the load information of the current image frame is determined based on the interface call information of the current image frame and the interface call information of each historical image frame. This avoids fluctuations and errors caused by an insufficient number of historical image frames, further improving the accuracy of the load information.

[0014] In some embodiments of this disclosure, the load change includes non-abrupt, sudden increase, or sudden decrease.

[0015] In this embodiment, non-mutation, sudden increase, or sudden decrease are used as the load change characteristics represented by the load information. This not only enables the determination of whether a sudden change has occurred in the load, but also determines the manner in which the load has changed. This facilitates maintaining, increasing, or decreasing the supply of the second processor based on the detection results of non-mutation, sudden increase, or sudden decrease, thereby ensuring frame rate stability and avoiding power waste, which is beneficial for further improving the overall performance of the second processor.

[0016] In some embodiments of this disclosure, the first processor includes a central processing unit, and the second processor includes an image processor.

[0017] In this embodiment, the central processing unit and the image processor are respectively used as the first processor and the second processor. The image frame processing can be performed by the central processing unit and the image processor to realize image data calculation, drawing and display. The power supply of the image processor can be adjusted in a timely manner according to the load information to ensure the display effect of the current image frame and improve the overall performance of the image processor.

[0018] In some embodiments of this disclosure, the interface call information includes at least one of the following: the number of interface calls, the drawing area, and the amount of data.

[0019] In this embodiment, at least one of the number of interface calls, the drawing area, and the amount of data is used as the interface call information. By leveraging the strong correlation between the number of interface calls, the drawing area, and the amount of data and the load of the second processor, the interface call information can accurately reflect the load of the second processor, providing a basis for determining the load information and ensuring the accuracy of the load information.

[0020] In some embodiments of this disclosure, the application programming interface includes at least one of a binding interface, a shader selection interface, an indexed drawing interface, a vertex indexing interface, a framebuffer interface, a viewport interface, a texture update interface, a buffer initialization interface, an instance drawing interface, a vertex drawing interface, a texture creation interface, and a shader calculation interface.

[0021] In this embodiment, the application programming interface includes at least one of the following: binding interface, shader selection interface, indexed drawing interface, vertex indexing interface, frame buffer interface, viewport interface, texture update interface, buffer initialization interface, instance drawing interface, vertex drawing interface, texture creation interface, and shader calculation interface. The interface call information is used to characterize the call status, so that the interface call information can accurately reflect the load status of the second processor, providing a basis for determining the load information and ensuring the accuracy of the load information.

[0022] In some embodiments of this disclosure, determining the load information of the current image frame based on the interface call information of the current image frame and the interface call information of multiple historical image frames includes: Based on the interface call information of the current image frame, a first call information feature vector is generated; Based on the interface call information of each of the historical image frames, a second call information feature vector is generated; The first call information feature vector and the second call information feature vector are input into the target classification model to obtain the load information of the current image frame.

[0023] In this embodiment, a first call information feature vector is generated based on the interface call information of the current image frame, and a second call information feature vector is generated based on the interface call information of each historical image frame. The first and second call information feature vectors are then input into the target classification model to obtain the load information of the current image frame. The load information is determined by using the target classification model, which facilitates the improvement of the accuracy of the load information through the structural design and training process of the target classification model.

[0024] In some embodiments of this disclosure, the target classification model includes a first encoder, a second encoder, and a first classifier, wherein the first encoder and the second encoder have the same weight parameters, and the same weight parameters are used to construct a Siamese network; The step of inputting the first call information feature vector and the second call information feature vector into the target classification model to obtain the load information of the current image frame includes: The first call information feature vector and the second call information feature vector are respectively input to the first encoder and the second encoder to obtain the first load feature corresponding to the first call information feature vector and the second load feature corresponding to the second call information feature vector. The first load feature and the second load feature are input into the first classifier to obtain the load information of the current image frame.

[0025] In this embodiment, a target classification model is composed of a first encoder, a second encoder, and a first classifier. It can obtain the corresponding first load feature and second load feature by inputting the first call information feature vector and the second call information feature vector into the first encoder and the second encoder, respectively. The first load feature and the second load feature are then input into the first classifier to obtain the load information of the current image frame. This allows the target classification model to output the load information based on the input first call information feature vector and the second call information feature vector, which facilitates the improvement of the accuracy of the load information through the structural design and training process of the target classification model.

[0026] In some embodiments of this disclosure, both the first encoder and the second encoder include a fully connected layer, a linear correction unit, and a plurality of residual blocks connected in sequence.

[0027] In this embodiment, a first encoder and a second encoder are constructed by a fully connected layer, a linear correction unit, and multiple residual blocks connected in sequence. This enables the first encoder to output corresponding first load features and second load features based on the input first call information feature vector and second call information feature vector. The characteristics of the fully connected layer, the linear correction unit, and the residual blocks are used to ensure the rationality of the structure of the first encoder and the second encoder, thereby further improving the accuracy of the load information.

[0028] In some embodiments of this disclosure, the training process of the target classification model includes: Based on the interface call information of multiple sample image frames, a first sample feature vector and a second sample feature vector are generated. The processing of one sample image frame corresponding to the first sample feature vector is after the processing of multiple sample image frames corresponding to the second sample feature vector. The first sample feature vector and the second sample feature vector are respectively input into the third encoder and the fourth encoder of the initial classification model to obtain the third load feature corresponding to the first sample feature vector and the fourth load feature corresponding to the second sample feature vector. The loss function is determined based on the feature distance between the third load feature and the fourth load feature; Based on the loss function, the third encoder and the fourth encoder are subjected to first parameter optimization processing; The third load feature and the fourth load feature are input into the second classifier of the initial classification model to obtain the load information of the sample image frame; Based on the load information of the sample image frame, the second classifier is subjected to second parameter optimization processing to obtain the target classification model. The third encoder and the fourth encoder after the first parameter optimization processing are respectively constituted as the first encoder and the second encoder. The second classifier after the second parameter optimization processing is constituted as the first classifier.

[0029] In this embodiment, the target classification model is trained through the above training process, enabling the first encoder and second encoder of the target classification model to output accurate load features, and the first classifier of the target classification model to output accurate load information, thus ensuring the functional realization of the target classification model and further improving the accuracy of the load information.

[0030] In some embodiments of this disclosure, the training process of the target classification model further includes: Determine the classification labels corresponding to the first sample feature vector and the second sample feature vector, wherein the classification labels are used to characterize whether the first sample feature vector and the second sample feature vector belong to the same sample category; The step of determining the loss function based on the feature distance between the third load feature and the fourth load feature includes: Based on the feature distance between the third load feature and the fourth load feature and the classification label, the loss function is determined such that the loss function is configured to be positively correlated with the feature distances corresponding to the first sample feature vector and the second sample feature vector belonging to the same sample category, and negatively correlated with the feature distances corresponding to the first sample feature vector and the second sample feature vector belonging to different sample categories.

[0031] In this embodiment, by determining the classification labels corresponding to the first and second sample feature vectors, and based on the feature distance between the third and fourth load features and the classification labels, a loss function is determined. This loss function is configured to be positively correlated with the feature distances corresponding to the first and second sample feature vectors belonging to the same sample category, and negatively correlated with the feature distances corresponding to the first and second sample feature vectors belonging to different sample categories. This realizes the application of contrastive learning, which can improve the generalization ability, feature representation, flexibility, stability, and applicability of the target classification model by clustering similar samples and separating dissimilar samples.

[0032] According to a second aspect of the present disclosure, a load detection device is provided, the load detection device comprising: The acquisition module is used to acquire interface call information of the current image frame in response to the first processor calling the application programming interface of the second processor during the processing of the current image frame. The determining module is used to determine the load information of the current image frame based on the interface call information of the current image frame and the interface call information of multiple historical image frames. The interface call information is used to characterize the call status of the first processor to at least one of the application programming interfaces of the second processor, and the load information is used to characterize the load change of the second processor during the processing of the current image frame.

[0033] In some embodiments of this disclosure, the processing of each image frame includes a first processing process corresponding to a first processor and a second processing process corresponding to a second processor, wherein the start time of the first processing process is prior to the start time of the second processing process.

[0034] In some embodiments of this disclosure, the first processor is configured to invoke the application programming interface of the second processor during the first processing.

[0035] In some embodiments of this disclosure, the determining module is further configured to: determine the load information of the current image frame before the second processing begins.

[0036] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising: processor; Memory used to store processor-executable instructions; The processor is configured to perform the load detection method as described in the first aspect.

[0037] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the load detection method as described in the first aspect.

[0038] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the load detection method as described in the first aspect.

[0039] According to a sixth aspect of the present disclosure, a chip is provided, the chip including a processor and an interface, the processor being configured to read instructions to execute the load detection method as described in the first aspect.

[0040] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: using the interface call information of the current image frame and multiple historical image frames as the basis for determining the load information of the current image frame, and using the changes in the call status of the application programming interface of the first processor to the second processor to reflect the load changes of the second processor, the accuracy of the load information is guaranteed. Furthermore, the interface call information of the current image frame can be obtained before the second processor starts working, which can timely determine the load information and adjust the supply of the second processor to ensure the display effect of the current image frame, thereby improving the overall performance of the second processor.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0043] Figure 1 This is a flowchart illustrating a load detection method according to an exemplary embodiment.

[0044] Figure 2 This is a flowchart illustrating, according to an exemplary embodiment, the determination of the load information of the current image frame based on the interface call information of the current image frame and the interface call information of multiple historical image frames.

[0045] Figure 3 This is a schematic diagram of a target classification model according to an exemplary embodiment.

[0046] Figure 4 This is a flowchart illustrating, according to an exemplary embodiment, how a first call information feature vector and a second call information feature vector are input into a target classification model to obtain the load information of the current image frame.

[0047] Figure 5 This is a flowchart illustrating the training process of a target classification model according to an exemplary embodiment.

[0048] Figure 6 This is a block diagram of a load detection device according to an exemplary embodiment.

[0049] Figure 7 This is a block diagram of an electronic device according to an exemplary embodiment.

[0050] In the picture: 10 - Acquisition module; 20 - Determination module; 400 - Electronic device; 402 - Processing component; 404 - Memory; 406 - Power supply component; 408 - Multimedia component; 410 - Audio component; 412 - Input / output interface; 414 - Sensor component; 416 - Communication component; 420 - Processor. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0052] In recent years, with the rapid development of image processing and display technologies and the continuous iteration of related devices, the requirements for the stability of display frame rates and the performance requirements of related devices have become increasingly stringent. During image processing, it is necessary to determine the changes in graphics processing load through specific methods in order to adjust the frequency supply of the image processor accordingly. This avoids frame drops due to insufficient frequency supply during load surges or power waste due to excessive frequency supply during load drops.

[0053] In related technologies, the load of an image processor (IPC) in the next frame can be predicted a priori based on historical data, or the actual load of the IPC can be monitored in real time using a posterior approach. However, predicting the IPC load a priori, which only uses historical data prior to the next frame and does not consider the actual processing of the next frame, has low accuracy in predicting discontinuous load changes caused by complex scene transitions or random user input. Monitoring the actual load of the IPC in real time using a posterior approach requires continuous and high-frequency monitoring and judgment, resulting in high operational overhead and a delay in load determination, making it difficult to respond promptly to sudden load changes.

[0054] Based on this, this disclosure provides a load detection method. During the processing of the current image frame, when the first processor calls the application programming interface (API) of the second processor, the method obtains the API call information of the current image frame and determines the load information of the current image frame based on the API call information of the current image frame and the API call information of multiple historical image frames. This enables the detection of load changes in the second processor, facilitating adjustments to the supply to the second processor based on the detection results to ensure frame rate stability. By using the API call information of the current image frame and multiple historical image frames as the basis for determining the load information of the current image frame, and by utilizing changes in the API calls of the first processor to the second processor to reflect changes in the load of the second processor, the accuracy of the load information is ensured. Furthermore, the API call information of the current image frame can be obtained before the second processor starts working, enabling timely determination of load information and adjustment of the supply to the second processor to ensure the display effect of the current image frame, thereby improving the overall performance of the second processor.

[0055] In one exemplary embodiment, a load detection method is provided, applied to a display device, which may include, for example, a mobile phone, tablet computer, or wristband, a display device with image display capabilities. (Reference) Figure 1 As shown, the load detection method includes: S100. During the processing of the current image frame, in response to the first processor calling the application programming interface of the second processor, the interface call information of the current image frame is obtained.

[0056] In step S100, the display device is configured with a first processor and a second processor for performing image processing processes such as image data parsing, image data calculation, image drawing, and image display. The first processor may include, for example, a central processing unit (CPU) to perform image data parsing and image data calculation processes, and the second processor may include, for example, an image processor to perform image drawing and image rendering processes.

[0057] The current image frame is the image frame whose corresponding processing is currently being executed, and will be the next image frame to be displayed. During the processing of the current image frame, when it is detected that the first processor is calling the application programming interface (API) of the second processor, it means that the first processor is issuing an image processing task, such as image drawing, to the second processor by calling the API of the second processor. The API call information of the current image frame is then obtained to reflect the calls made by the first processor to at least one API of the second processor during the processing of the current image frame. For example, the API call information may include the number of times the central processing unit calls multiple APIs of the image processor and the parameter information of the called functions.

[0058] S200. Based on the interface call information of the current image frame and the interface call information of multiple historical image frames, determine the load information of the current image frame. The interface call information is used to characterize the call status of the first processor to at least one application programming interface of the second processor, and the load information is used to characterize the load change of the second processor during the processing of the current image frame.

[0059] In step S200, the display device can also acquire interface call information of multiple historical image frames, so as to reflect the call status of the first processor to at least one application programming interface of the second processor during the processing of each historical image frame through the interface call information of multiple historical image frames. The interface call information of multiple historical image frames can be acquired, for example, during the processing of each historical image frame.

[0060] Since the image processing task of the second processor is issued by the first processor calling the application programming interface of the second processor, the load of the second processor when executing the corresponding processing procedure is strongly correlated with the calling status of its application programming interface. This allows the interface call information of the current image frame and the interface call information of the historical image frames to reflect the load of the second processor during the processing of the current image frame and the processing of the historical image frames, respectively.

[0061] Therefore, the load information of the current image frame can be determined based on the interface call information of the current image frame and the interface call information of multiple historical image frames. This load information characterizes the change in the load of the second processor during the processing of the current image frame compared to the load during the processing of historical image frames. For example, feature vectors corresponding to the interface call information of the current and historical image frames can be generated, and these feature vectors can be input into the target classification model to output the corresponding load information. The load information may include, for example, whether the load of the image processor undergoes a sudden change, and if such a change occurs, whether it is a sudden increase or a sudden decrease.

[0062] It should be noted that, compared to related technologies that predict image processor load based solely on historical data using a priori methods, the aforementioned load detection method uses the interface call information of the current image frame as the basis for determining load information. This integrates the determination of load information with the processing of the current image frame, and since the first processor has already called the application programming interface of the second processor, it achieves a "semi-predictive" effect compared to the "pure prediction" of related technologies, thus ensuring the accuracy of load information. Compared to related technologies that use a posteriori method to monitor the actual load of the image processor in real time, the aforementioned load detection method obtains interface call information when the first processor calls the application programming interface of the second processor. This allows for the determination of the load information of the current image frame before the second processor actually starts working, facilitating timely adjustments to the second processor's power supply based on the load information, thereby reducing the delay in responding to sudden changes in the second processor's load.

[0063] In this embodiment, during the processing of the current image frame, when the first processor calls the application programming interface (API) of the second processor, the interface call information of the current image frame is obtained. Based on this API call information and the API call information of multiple historical image frames, the load information of the current image frame is determined. This enables the detection of load changes in the second processor, facilitating adjustments to the power supply to the second processor based on the detection results to ensure frame rate stability. Using the API call information of the current image frame and multiple historical image frames as the basis for determining the load information of the current image frame, and utilizing changes in the first processor's API calls to the second processor to reflect load changes in the second processor, ensures the accuracy of the load information. Furthermore, the API call information of the current image frame can be obtained before the second processor starts working, enabling timely determination of load information and adjustment of the second processor's power supply to ensure the display effect of the current image frame, thus improving the overall performance of the second processor.

[0064] In some embodiments, the processing of each image frame includes a first processing procedure corresponding to a first processor and a second processing procedure corresponding to a second processor, wherein the start time of the first processing procedure is prior to the start time of the second processing procedure.

[0065] The processing of each current image frame and historical image frame includes a first processing step executed by the first processor and a second processing step executed by the second processor, to achieve different processing functions and corresponding processing effects. The start time of the first processing step is before the start time of the second processing step; that is, the first processing step is executed by the first processor first, and then the second processing step is executed by the second processor. In some cases, the first processor can call the application programming interface of the second processor through the first processing step, which starts earlier, to issue image processing tasks to the second processor, thereby controlling the second processor to execute the second processing step, which starts later. This ensures that the interface call information can be obtained after the first processing step starts and before the second processing step starts, and that the load information can be determined after the first processing step and before the second processing step starts.

[0066] For example, the first processing procedure corresponding to the first processor may include image data parsing, image data calculation, resource allocation and task scheduling performed by the central processing unit, and the second processing procedure corresponding to the second processor may include vertex processing, rasterization, pixel rendering, image compositing and output performed by the image processor. The start time of the second processing procedure corresponding to the image processor may be located between the start time and the end time of the first processing procedure, or may be located after the end time of the first processing procedure.

[0067] In this embodiment, the first processing procedure corresponding to the first processor and the second processing procedure corresponding to the second processor are used as the processing procedures for each image frame. Different processing functions and corresponding processing effects can be achieved through the first and second processing procedures respectively, which is beneficial to improving the processing efficiency and quality of the current image frame. The start time of the first processing procedure is configured to be before the start time of the second processing procedure, so that interface call information can be obtained after the first processing procedure starts and before the second processing procedure starts, and that load information can be determined after the first processing procedure starts and before the second processing procedure starts, further ensuring the accuracy and timeliness of the load information.

[0068] In some embodiments, the first processor is configured to invoke the application programming interface of the second processor during a first processing step.

[0069] The first processor is configured to call the application programming interface (API) of the second processor during a corresponding first processing step. This ensures that the first processor can call the API of the second processor through the first processing step, which starts earlier, to issue image processing tasks to the second processor. This, in turn, controls the second processor to execute the second processing step, which starts later. The API call information can be obtained after the first processing step begins and before the second processing step begins, and the load information can be determined after the first processing step and before the second processing step begins. For example, the central processing unit is configured to call the API of the image processor during the resource allocation and task invocation phase of the first processing step.

[0070] In this embodiment, the first processor is configured to call the application programming interface of the second processor during the first processing, so that the interface call information can be obtained after the first processing starts and before the second processing starts, and the load information can be determined after the first processing starts and before the second processing starts, thereby further ensuring the accuracy and timeliness of the load information.

[0071] In some embodiments, determining the load information of the current image frame includes: determining the load information of the current image frame before the start of the second processing procedure.

[0072] Before the second processing step corresponding to the second processor begins, the display device determines the load information of the current image frame based on the interface call information of the current image frame and the interface call information of multiple historical image frames. This allows the load information to be determined after the first processing step and before the second processing step begins. For example, after obtaining the interface call information of the current image frame, the load information of the current image frame is determined immediately, so that the determination of the load information is completed before the image rendering and other processing steps performed by the image processor. In this case, the power supply of the second processor can be adjusted in advance based on the determined load information before the second processing step begins, thereby eliminating or suppressing frame rate fluctuations caused by sudden load changes after the second processing step begins.

[0073] In this embodiment, the load information of the current image frame is determined before the second processing begins, so that the load information can be determined after the first processing and before the second processing begins. This allows the supply of the second processor to be adjusted in advance based on the load information, which helps to reduce the delay in responding to sudden changes in the load of the second processor and thus improves the stability of the frame rate.

[0074] In some embodiments, the load detection method further includes: in response to the end of the processing of the current image frame, treating the interface call information of the current image frame as the interface call information of a historical image frame.

[0075] During the processing of each current image frame, the interface call information of the current image frame is used as the interface call information of a historical image frame. This allows for the updating and supplementation of the interface call information of historical image frames, providing a basis for determining the load information of the next current image frame and ensuring the timeliness of the interface call information of historical image frames, thereby further improving the accuracy of the load information.

[0076] Based on the interface call information of the current image frame and the interface call information of multiple historical image frames, the load information of the current image frame is determined, including: in response to the number of historical image frames reaching a preset threshold, the load information of the current image frame is determined based on the interface call information of the current image frame and the interface call information of each historical image frame.

[0077] It is understandable that if the current image frame is the first image frame, or if the number of historical image frames is small, the interface call information of the historical image frames is limited, which can easily lead to large fluctuations or errors. This makes it difficult to accurately reflect the historical load of the second processor using the interface call information of the historical image frames, thus compromising the accuracy of the load information. Therefore, when the number of historical image frames reaches a preset threshold, the load information of the current image frame can be determined based on the interface call information of the current image frame and the interface call information of each historical image frame. This ensures the accuracy of the load information through sufficient interface call information from historical image frames. For example, the preset threshold could be 20.

[0078] In this embodiment, at the end of the processing of the current image frame, the interface call information of the current image frame is used as the interface call information of a historical image frame. This realizes the updating and supplementation of the interface call information of historical image frames, ensuring the timeliness of the interface call information of historical image frames. When the number of historical image frames reaches a preset threshold, the load information of the current image frame is determined based on the interface call information of the current image frame and the interface call information of each historical image frame. This avoids fluctuations and errors caused by an insufficient number of historical image frames, further improving the accuracy of the load information.

[0079] In some embodiments, load changes include non-abrupt, sudden increase, or sudden decrease.

[0080] The load information, representing the load change of the second processor during the processing of the current image frame, can include non-abrupt, sudden increase, and sudden decrease. A non-abrupt change indicates that the load of the second processor during the processing of the current image frame does not change significantly compared to the processing of historical image frames, and the current power supply of the second processor can be maintained. A sudden increase indicates that the load of the second processor during the processing of the current image frame has a significant increase compared to the processing of historical image frames, requiring an increase in the power supply of the second processor to ensure frame rate stability. A sudden decrease indicates that the load of the second processor during the processing of the current image frame has a significant decrease compared to the processing of historical image frames, requiring a decrease in the power supply of the second processor to avoid wasting power.

[0081] In this embodiment, non-mutation, sudden increase, or sudden decrease are used as the load change characteristics represented by the load information. This not only enables the determination of whether a sudden change has occurred in the load, but also determines the manner in which the load has changed. This facilitates maintaining, increasing, or decreasing the supply of the second processor based on the detection results of non-mutation, sudden increase, or sudden decrease, thereby ensuring frame rate stability and avoiding power waste, which is beneficial for further improving the overall performance of the second processor.

[0082] In some embodiments, the first processor includes a central processing unit, and the second processor includes an image processor.

[0083] The central processing unit (CPU) and the image processor (IPC) can be used as the first and second processors, respectively. The CPU performs image data parsing and calculation, while the IPC performs image drawing and rendering. When the CPU calls the IPC's application programming interface (API), it obtains the API call information for the current image frame. Based on this information and the API call information of multiple historical image frames, the CPU's load information is determined to ensure that the IPC's load changes appropriately during the processing of the current image frame.

[0084] In this embodiment, the central processing unit and the image processor are respectively used as the first processor and the second processor. The image frame processing can be performed by the central processing unit and the image processor to realize image data calculation, drawing and display. The power supply of the image processor can be adjusted in a timely manner according to the load information to ensure the display effect of the current image frame and improve the overall performance of the image processor.

[0085] In some embodiments, the interface call information includes at least one of the following: the number of interface calls, the drawing area, and the amount of data.

[0086] The interface call information may include, for example, at least one of the following: the number of interface calls, the drawing area, and the amount of data when the first processor calls the application programming interface of the second processor. Since the number of interface calls, the drawing area, and the amount of data are all strongly correlated with the load of the second processor when it executes the corresponding processing procedure, the interface call information can accurately reflect the load of the second processor in the processing of the current image frame and the processing of historical image frames, and thus can serve as a basis for determining the load information.

[0087] In this embodiment, at least one of the number of interface calls, the drawing area, and the amount of data is used as the interface call information. By leveraging the strong correlation between the number of interface calls, the drawing area, and the amount of data and the load of the second processor, the interface call information can accurately reflect the load of the second processor, providing a basis for determining the load information and ensuring the accuracy of the load information.

[0088] In some embodiments, the application programming interface includes at least one of a binding interface, a shader selection interface, an indexed drawing interface, a vertex indexing interface, a framebuffer interface, a viewport interface, a texture update interface, a buffer initialization interface, an instance drawing interface, a vertex drawing interface, a texture creation interface, and a shader computation interface.

[0089] Application programming interfaces (APIs) can include at least one of the following: binding interface, shader selection interface, indexed drawing interface, vertex indexing interface, framebuffer interface, viewport interface, texture update interface, buffer initialization interface, instance drawing interface, vertex drawing interface, texture creation interface, and shader calculation interface, as shown in the above embodiments. This enables the first processor to call the above interfaces of the second processor and characterize the calling status of the above interfaces through interface call information.

[0090] The binding interface, glBindTexture, is used to bind a specified texture object to the current context, enabling subsequent drawing operations to use the texture. The interface call information corresponding to the binding interface can be the number of times the binding interface is called.

[0091] The shader selection interface, glUseProgram, is used to select the shader program to be used, thereby defining the rendering method of the image. The interface call information corresponding to the shader selection interface can be the number of times the shader selection interface is called.

[0092] The indexed drawing interface, gles_draw_draw_elements, is used to draw primitives using an array of indices. It can retrieve vertex data from the vertex buffer based on the provided indices and draw it in a specified manner. The interface call information corresponding to the indexed drawing interface can be the number of times the indexed drawing interface is called.

[0093] The vertex index interface, gles2_draw_draw_elements_base_vertex, is used to draw primitives using an index array and allows specifying a base vertex index, which can further improve memory utilization and drawing flexibility. The interface call information corresponding to the vertex index interface can be the number of times the vertex index interface is called.

[0094] The framebuffer interface, glBindFramebuffer, is used to bind framebuffer objects for generating shadow maps and post-processing effects. The interface call information corresponding to the framebuffer interface is the number of times the framebuffer interface is called.

[0095] The viewport interface, or glViewport, is used to set the size and position of the viewport to define the display area of ​​the image on the screen. The interface call information corresponding to the viewport interface can be the number of times the viewport interface is called and the drawing area.

[0096] The texture update interface, glTexSubImage2D, is used to update texture data. It can replace or add new texture data to an already created texture object. The interface call information corresponding to the texture update interface can be the number of times the texture update interface is called and the drawing area.

[0097] The buffer initialization interface, glBufferData, is used to create and initialize the data storage corresponding to the buffer for subsequent drawing operations. The interface call information corresponding to the buffer initialization interface can be the number of times the buffer initialization interface is called and the amount of data.

[0098] The instance drawing interface, gles2_draw_draw_elements_instanced, is used to draw multiple identical geometric shapes in a single drawing process, thereby improving the efficiency of drawing a large number of similar elements. The interface call information corresponding to the instance drawing interface can be the number of times the instance drawing interface is called.

[0099] The vertex drawing interface, gles_draw_draw_arrays, is used to draw using vertex arrays. It connects vertices sequentially according to the specified starting vertex and number of vertices to form primitives. The interface call information corresponding to the vertex drawing interface can be the number of times the vertex drawing interface is called.

[0100] The texture creation interface, glTexImage2D, is used to create and initialize two-dimensional texture images. The interface call information corresponding to the texture creation interface can be the number of times the texture creation interface is called and the drawing area.

[0101] The shader compute interface, glDispatchCompute, is used to start compute shaders to perform general computational tasks such as physics simulations, which are then used for subsequent drawing operations. The interface call information corresponding to the shader compute interface can be the number of times the shader compute interface is called.

[0102] In this embodiment, the application programming interface includes at least one of the following: binding interface, shader selection interface, indexed drawing interface, vertex indexing interface, frame buffer interface, viewport interface, texture update interface, buffer initialization interface, instance drawing interface, vertex drawing interface, texture creation interface, and shader calculation interface. The interface call information is used to characterize the call status, so that the interface call information can accurately reflect the load status of the second processor, providing a basis for determining the load information and ensuring the accuracy of the load information.

[0103] In some embodiments, reference Figure 2 As shown, based on the interface call information of the current image frame and the interface call information of multiple historical image frames, the load information of the current image frame is determined, including: S210. Generate a first call information feature vector based on the interface call information of the current image frame.

[0104] In step S210, a first call information feature vector is generated based on the interface call information of the current image frame, and used as input to the subsequent target classification model. For example, if the interface call information of the current image frame includes interface call information of multiple application programming interfaces API_info_1, API_info_2, ..., API_info_N, where N is a positive integer greater than 2, then a first call information feature vector A = (API_info_1, API_info_2, ..., API_info_N) can be generated.

[0105] S220. Based on the interface call information of each historical image frame, generate a second call information feature vector.

[0106] In step S220, a second call information feature vector is generated based on the interface call information of each historical image frame, and used as input to the subsequent target classification model. For example, if the interface call information of each historical image frame includes the interface call information of multiple application programming interfaces (APIs) API_info_1, API_info_2, ..., API_info_N, then the average values ​​avg(API_info_1), avg(API_info_2), ..., avg(API_info_N) of the interface call information of each API for multiple historical image frames can be calculated, and the second call information feature vector B = ((API_info_1), avg(API_info_2), ..., avg(API_info_N)) is generated.

[0107] S230. Input the first call information feature vector and the second call information feature vector into the target classification model to obtain the load information of the current image frame.

[0108] In step S230, after obtaining the first and second call information feature vectors, these vectors are input into the pre-trained target classification model. This allows the target classification model to output the load information of the current image frame based on the interface call information included in the first and second call information feature vectors, using its algorithm. For example, the target classification model can be a model based on Siamese Network and contrastive learning.

[0109] In this embodiment, a first call information feature vector is generated based on the interface call information of the current image frame, and a second call information feature vector is generated based on the interface call information of each historical image frame. The first and second call information feature vectors are then input into the target classification model to obtain the load information of the current image frame. The load information is determined by using the target classification model, which facilitates the improvement of the accuracy of the load information through the structural design and training process of the target classification model.

[0110] In some embodiments, the target classification model includes a first encoder, a second encoder, and a first classifier, wherein the first encoder and the second encoder have the same weight parameters, and the same weight parameters are used to construct a Siamese network.

[0111] refer to Figure 3As shown, the target classification model consists of a first encoder, a second encoder, and a first classifier. The first encoder and the second encoder have the same weight parameters. The weight parameters can control the connection strength of the encoder, optimize the encoder performance, reflect the importance of features, and prevent overfitting. By using the same weight parameters, the construction of the Siamese network can be achieved, ensuring consistent feature extraction for inputs with the same structure.

[0112] refer to Figure 4 As shown, the first and second call information feature vectors are input into the target classification model to obtain the load information of the current image frame, including: S231. Input the first call information feature vector and the second call information feature vector into the first encoder and the second encoder respectively to obtain the first load feature corresponding to the first call information feature vector and the second load feature corresponding to the second call information feature vector.

[0113] In step S231, the first call information feature vector is input to the first encoder to output a first load feature corresponding to the first call information feature vector, and the second call information feature vector is input to the second encoder to output a second load feature corresponding to the second call information feature vector. Since the first and second call information feature vectors contain interface call information characterizing the interface call status, feature extraction by the first and second encoders ensures that the obtained first and second load features accurately reflect the current and historical load status of the second processor.

[0114] S232. Input the first load feature and the second load feature into the first classifier to obtain the load information of the current image frame.

[0115] In step S232, the first load feature and the second load feature are input to the first classifier. The classification judgment function of the first classifier can determine whether the corresponding load change is non-abrupt, abruptly increased, or abruptly decreased based on the current load and historical load of the second processor represented by the first load feature and the second load feature, so as to determine the load information of the current image frame.

[0116] In this embodiment, a target classification model is composed of a first encoder, a second encoder, and a first classifier. It can obtain the corresponding first load feature and second load feature by inputting the first call information feature vector and the second call information feature vector into the first encoder and the second encoder, respectively. The first load feature and the second load feature are then input into the first classifier to obtain the load information of the current image frame. This allows the target classification model to output the load information based on the input first call information feature vector and the second call information feature vector, which facilitates the improvement of the accuracy of the load information through the structural design and training process of the target classification model.

[0117] In some embodiments, both the first encoder and the second encoder include a fully connected layer, a linear correction unit, and a plurality of residual blocks connected in sequence.

[0118] like Figure 3 As shown, both the first and second encoders consist of sequentially connected fully connected (FC) layers, rectified linear units (ReLU), and multiple residual blocks. The fully connected layers integrate and transform the data, mapping the input features to new feature representations for tasks such as classification or regression. The rectified linear unit (ReLU) is a commonly used activation function that introduces non-linearity, enabling the model to learn and represent more complex feature relationships. The residual blocks address the vanishing gradient problem by introducing residual connections, thereby capturing richer features and allowing the model to converge to a better solution.

[0119] In this embodiment, a first encoder and a second encoder are constructed by a fully connected layer, a linear correction unit, and multiple residual blocks connected in sequence. This enables the first encoder to output corresponding first load features and second load features based on the input first call information feature vector and second call information feature vector. The characteristics of the fully connected layer, the linear correction unit, and the residual blocks are used to ensure the rationality of the structure of the first encoder and the second encoder, thereby further improving the accuracy of the load information.

[0120] In some embodiments, reference Figure 5 As shown, the training process of the target classification model includes: S310. Based on the interface call information of multiple sample image frames, generate a first sample feature vector and a second sample feature vector. The processing of one sample image frame corresponding to the first sample feature vector is after the processing of multiple sample image frames corresponding to the second sample feature vector.

[0121] In step S310, during the processing of multiple sample image frames, interface call information for each sample image frame is collected, and a first sample feature vector and a second sample feature vector are generated based on the interface call information of each sample image frame, to serve as subsequent input samples. The first sample feature vector corresponds to one image frame, and the second sample feature vector corresponds to multiple sample image frames. The processing of one sample image frame corresponding to the first sample feature vector occurs after the processing of multiple sample image frames corresponding to the second sample feature vector, so that the first sample feature vector and the second sample feature vector can respectively represent the current load and historical load of the second processor at a certain historical moment.

[0122] S320. Input the first sample feature vector and the second sample feature vector into the third encoder and the fourth encoder of the initial classification model, respectively, to obtain the third load feature corresponding to the first sample feature vector and the fourth load feature corresponding to the second sample feature vector.

[0123] In step S320, the initial classification model includes a third encoder and a fourth encoder. The first sample feature vector and the second sample feature vector are input to the third encoder and the fourth encoder respectively to obtain the third load feature and the fourth load feature, so as to characterize the current load and historical load of the second processor in the processing of sample image frames.

[0124] S330. Determine the loss function based on the feature distance between the third load feature and the fourth load feature.

[0125] In step S330, the feature distance between the third load feature and the fourth load feature is calculated to characterize the similarity between them. The corresponding loss function is then determined based on the feature distance, serving as the basis for subsequent first parameter optimization. For example, the feature distance between the third load feature and the fourth load feature could be, for instance, Euclidean distance. European distance It can be determined by the following formula (1): (1) in, This is the third load characteristic. This is the fourth load characteristic.

[0126] S340. Based on the loss function, perform first parameter optimization processing on the third encoder and the fourth encoder.

[0127] In step S340, a first parameter optimization process can be performed on the third and fourth encoders according to the loss function, so that the third and fourth encoders can output more accurate load features, thereby achieving the training of the third and fourth encoders. It should be noted that the first parameter optimization process for the third and fourth encoders can be an iterative process by minimizing the loss function, allowing the parameters of the third and fourth encoders to be optimized multiple times, so that the third and fourth encoders can output more accurate load features after each execution of the first parameter optimization process.

[0128] S350. Input the third load feature and the fourth load feature into the second classifier of the initial classification model to obtain the load information of the sample image frame.

[0129] In step S350, the initial classification model further includes a second classifier. The third load feature and the fourth load feature are input into the second classifier of the initial classification model to obtain the load information of the sample image frame, so as to characterize the load change of the second processor in the processing of the sample image frame.

[0130] S360. Based on the load information of the sample image frame, the second classifier is optimized by the second parameter to obtain the target classification model. The third encoder and the fourth encoder after the first parameter optimization are respectively formed by the first encoder and the second encoder. The second classifier after the second parameter optimization is formed by the first classifier.

[0131] In step S360, a second parameter optimization process can be performed on the second classifier based on the load information of the sample image frames, so that the second classifier can output more accurate load information, thereby achieving the training of the second classifier. For example, the second classifier can be trained using cross-entropy loss to further constrain feature representation and improve the performance of the second classifier.

[0132] After the third encoder, fourth encoder and second classifier of the initial classification model have been trained, the third encoder and fourth encoder can be used as the first encoder and second encoder of the target classification model, respectively, and the second classifier can be used as the first classifier of the target classification model. This completes the training process of the target classification model, ensuring that the first encoder and second encoder of the target classification model can output accurate load features, and that the first classifier of the target classification model can output accurate load information.

[0133] In this embodiment, the target classification model is trained through the above training process, enabling the first encoder and second encoder of the target classification model to output accurate load features, and the first classifier of the target classification model to output accurate load information, thus ensuring the functional realization of the target classification model and further improving the accuracy of the load information.

[0134] In some embodiments, the training process of the target classification model further includes: determining the classification labels corresponding to the first sample feature vector and the second sample feature vector, wherein the classification labels are used to characterize whether the first sample feature vector and the second sample feature vector belong to the same sample category.

[0135] After generating the first sample feature vector and the second sample feature vector, it is necessary to determine the classification labels corresponding to the first sample feature vector and the second sample feature vector, so as to characterize whether the first sample feature vector and the second sample feature vector belong to the same category through the classification labels, thereby providing a basis for subsequent comparative learning applications.

[0136] For example, the actual detection values ​​of the active counters of the second processor during the processing of the sample image frames corresponding to the first and second sample feature vectors can be collected to characterize the actual running cycle of the second processor. Then, using algorithms such as mean, moving average, or autoregression, the predicted value of the active counter corresponding to the first sample feature vector is determined based on the actual detection value of the active counter corresponding to the second sample feature vector.

[0137] If the difference between the actual detected value and the predicted value of the active count corresponding to the feature vector of the first sample is small, then the classification label is determined. A value of 0 indicates that the first and second sample feature vectors belong to the same sample category, representing a non-abrupt change in load. If the actual detected value of the active count corresponding to the first sample feature vector is greater than the predicted value, and the difference is large, then the classification label is determined. A value of 1 indicates that the first and second sample feature vectors belong to different sample categories, representing a sudden increase in workload. If the actual detected value of the activity count corresponding to the first sample feature vector is less than the predicted value, and the difference is large, then the classification label is determined. The value is -1, indicating that the first sample feature vector and the second sample feature vector belong to different sample categories, representing a sudden drop in load change.

[0138] The loss function is determined based on the feature distance between the third and fourth load features, including: determining the loss function based on the feature distance between the third and fourth load features and the classification label, such that the loss function is configured to be positively correlated with the feature distances corresponding to the first and second sample feature vectors belonging to the same sample category, and negatively correlated with the feature distances corresponding to the first and second sample feature vectors belonging to different sample categories.

[0139] When determining the loss function based on the feature distance between the third and fourth load features, the classification label and the feature distance can be used together as the basis for determining the loss function. This allows the loss function to be positively correlated with the feature distances between the first and second sample feature vectors belonging to the same sample category, and negatively correlated with the feature distances between the first and second sample feature vectors belonging to different sample categories, thus enabling the application of contrastive learning.

[0140] For example, it can be based on the feature distance between the third load feature and the fourth load feature. and tags The loss function is determined by formula (2) as shown below. : (2) in, These are preset hyperparameters that can be set according to your needs.

[0141] In this case, during the subsequent training of the third and fourth encoders by performing the first parameter optimization processing through the loss function, the feature distance between the third and fourth load features corresponding to the first and second sample feature vectors belonging to the same sample category is as small as possible, while the feature distance between the third and fourth load features corresponding to the first and second sample feature vectors belonging to different sample categories is larger, thus achieving the clustering of samples of the same type and the separation of samples of different types.

[0142] In this embodiment, by determining the classification labels corresponding to the first and second sample feature vectors, and based on the feature distance between the third and fourth load features and the classification labels, a loss function is determined. This loss function is configured to be positively correlated with the feature distances corresponding to the first and second sample feature vectors belonging to the same sample category, and negatively correlated with the feature distances corresponding to the first and second sample feature vectors belonging to different sample categories. This realizes the application of contrastive learning, which can improve the generalization ability, feature representation, flexibility, stability, and applicability of the target classification model by clustering similar samples and separating dissimilar samples.

[0143] In one exemplary embodiment, a load detection device is provided, applied to a display device, with reference to... Figure 6 As shown, the load detection device includes an acquisition module 10 and a determination module 20. The acquisition module 10 is used to acquire interface call information of the current image frame in response to the first processor calling the application programming interface of the second processor during the processing of the current image frame. The determination module 20 is used to determine the load information of the current image frame based on the interface call information of the current image frame and the interface call information of multiple historical image frames. The interface call information is used to characterize the calling status of at least one application programming interface of the second processor by the first processor, and the load information is used to characterize the load change of the second processor during the processing of the current image frame.

[0144] In this embodiment, during the processing of the current image frame, when the first processor calls the application programming interface (API) of the second processor, the acquisition module 10 acquires the API call information of the current image frame, and the determination module 20 determines the load information of the current image frame based on the API call information of the current image frame and the API call information of multiple historical image frames. This enables the detection of load changes in the second processor, facilitating adjustments to the supply to the second processor based on the detection results to ensure frame rate stability. Using the API call information of the current image frame and multiple historical image frames as the basis for determining the load information of the current image frame, and utilizing the changes in the first processor's API calls to the second processor to reflect the load changes of the second processor, ensures the accuracy of the load information. Furthermore, the API call information of the current image frame can be acquired before the second processor starts working, enabling timely determination of the load information and adjustment of the second processor's supply to ensure the display effect of the current image frame, thus improving the overall performance of the second processor.

[0145] In one embodiment, the processing of each image frame includes a first processing process corresponding to a first processor and a second processing process corresponding to a second processor, wherein the start time of the first processing process is earlier than the start time of the second processing process.

[0146] In one embodiment, the first processor is configured to invoke the application programming interface of the second processor during a first processing step.

[0147] In one embodiment, the determining module 20 is further configured to: determine the load information of the current image frame before the second processing begins.

[0148] In one embodiment, the acquisition module 10 is further configured to: in response to the end of the processing of the current image frame, use the interface call information of the current image frame as the interface call information of a historical image frame; the determination module 20 is further configured to: in response to the number of historical image frames reaching a preset number threshold, determine the load information of the current image frame based on the interface call information of the current image frame and the interface call information of each historical image frame.

[0149] In one embodiment, the load change includes non-abrupt, sudden increase, or sudden decrease.

[0150] In one embodiment, the first processor includes a central processing unit, and the second processor includes an image processor.

[0151] In one embodiment, the interface call information includes at least one of the following: the number of interface calls, the drawing area, and the amount of data.

[0152] In one embodiment, the application programming interface includes at least one of a binding interface, a shader selection interface, an indexed drawing interface, a vertex indexing interface, a framebuffer interface, a viewport interface, a texture update interface, a buffer initialization interface, an instance drawing interface, a vertex drawing interface, a texture creation interface, and a shader computation interface.

[0153] In one embodiment, the determining module 20 is further configured to: generate a first call information feature vector based on the interface call information of the current image frame; generate a second call information feature vector based on the interface call information of each historical image frame; and input the first call information feature vector and the second call information feature vector into the target classification model to obtain the load information of the current image frame.

[0154] In one embodiment, the target classification model includes a first encoder, a second encoder, and a first classifier, wherein the first encoder and the second encoder have the same weight parameters; the determination module 20 is further configured to: input the first call information feature vector and the second call information feature vector to the first encoder and the second encoder respectively to obtain the first load feature corresponding to the first call information feature vector and the second load feature corresponding to the second call information feature vector; input the first load feature and the second load feature to the first classifier to obtain the load information of the current image frame.

[0155] In one embodiment, both the first encoder and the second encoder include a fully connected layer, a linear correction unit, and a plurality of residual blocks connected in sequence.

[0156] In one embodiment, the load detection device further includes a training module, which is configured to: generate a first sample feature vector and a second sample feature vector based on interface call information of multiple sample image frames, wherein the processing of one sample image frame corresponding to the first sample feature vector is after the processing of multiple sample image frames corresponding to the second sample feature vector; input the first sample feature vector and the second sample feature vector to the third encoder and the fourth encoder of the initial classification model, respectively, to obtain the third load feature corresponding to the first sample feature vector and the fourth load feature corresponding to the second sample feature vector; determine a loss function based on the feature distance between the third load feature and the fourth load feature; perform a first parameter optimization process on the third encoder and the fourth encoder based on the loss function; input the third load feature and the fourth load feature to the second classifier of the initial classification model to obtain the load information of the sample image frames; and perform a second parameter optimization process on the second classifier based on the load information of the sample image frames to obtain the target classification model, wherein the third encoder and the fourth encoder after performing the first parameter optimization process constitute the first encoder and the second encoder, respectively, and the second classifier after performing the second parameter optimization process constitutes the first classifier.

[0157] In one embodiment, the training module is further configured to: determine the classification labels corresponding to the first sample feature vector and the second sample feature vector, wherein the classification labels are used to characterize whether the first sample feature vector and the second sample feature vector belong to the same sample category; and determine a loss function based on the feature distance between the third load feature and the fourth load feature and the classification labels, such that the loss function is configured to be positively correlated with the feature distances corresponding to the first sample feature vector and the second sample feature vector belonging to the same sample category, and negatively correlated with the feature distances corresponding to the first sample feature vector and the second sample feature vector belonging to different sample categories.

[0158] In one exemplary embodiment, an electronic device is provided, which may include, for example, a display device with image display function such as a mobile phone, tablet computer, or wristband, and the electronic device is capable of performing the load detection method described above.

[0159] refer to Figure 7 As shown, the electronic device 400 may include one or more of the following components: processing component 402, memory 404, power supply component 406, multimedia component 408, audio component 410, input / output (I / O) interface 412, sensor component 414, and communication component 416.

[0160] Processing component 402 typically controls the overall operation of electronic device 400, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 402 may include one or more processors 420 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 402 may include one or more modules to facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402.

[0161] Memory 404 is configured to store various types of data to support the operation of electronic device 400. Examples of such data include instructions for any application or method operating on electronic device 400, contact data, phonebook data, messages, pictures, videos, etc. Memory 404 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0162] Power supply component 406 provides power to various components of electronic device 400. Power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 400.

[0163] Multimedia component 408 includes a screen that provides an output interface between electronic device 400 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 408 includes a front-facing camera module and / or a rear-facing camera module. When electronic device 400 is in an operating mode, such as shooting mode or video mode, the front-facing camera module and / or rear-facing camera module may receive external multimedia data. Each front-facing camera module and rear-facing camera module may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0164] Audio component 410 is configured to output and / or input audio signals. For example, audio component 410 includes a microphone (MIC) configured to receive external audio signals when electronic device 400 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 404 or transmitted via communication component 416. In some embodiments, audio component 410 also includes a speaker for outputting audio signals.

[0165] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0166] Sensor assembly 414 includes one or more sensors for providing state assessments of various aspects of electronic device 400. For example, sensor assembly 414 may receive the on / off state of electronic device 400, the relative positioning of components such as the display and keypad of electronic device 400, changes in position of electronic device 400 or a component of electronic device 400, the presence or absence of user contact with electronic device 400, orientation or acceleration / deceleration of electronic device 400, and temperature changes of electronic device 400. Sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 414 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0167] Communication component 416 is configured to facilitate wired or wireless communication between electronic device 400 and other terminals. Electronic device 400 can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 416 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 416 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0168] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing terminals (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods shown in the above embodiments or combinations thereof.

[0169] In one exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, which can be executed by a processor 420 of an electronic device 400 to perform the methods shown in the above embodiments or combinations thereof. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage terminal, etc. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is able to perform the methods shown in the above embodiments or combinations thereof.

[0170] In one exemplary embodiment, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the method described above.

[0171] In one exemplary embodiment, a chip is provided, including a processor and an interface, the processor being configured to read instructions to execute the method described above.

[0172] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0173] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0174] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0175] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0176] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A load detection method, characterized in that, The load detection method includes: During the processing of the current image frame, in response to the first processor calling the application programming interface of the second processor, the interface call information of the current image frame is obtained; Based on the interface call information of the current image frame and the interface call information of multiple historical image frames, the load information of the current image frame is determined. The interface call information is used to characterize the call status of the first processor to at least one of the application programming interfaces of the second processor, and the load information is used to characterize the load change of the second processor during the processing of the current image frame.

2. The load detection method according to claim 1, characterized in that, The processing of each image frame includes a first processing step corresponding to the first processor and a second processing step corresponding to the second processor. The start time of the first processing step is earlier than the start time of the second processing step.

3. The load detection method according to claim 2, characterized in that, The first processor is configured to invoke the application programming interface of the second processor during the first processing.

4. The load detection method according to claim 2, characterized in that, Determining the load information of the current image frame includes: Before the second processing begins, the load information of the current image frame is determined.

5. The load detection method according to claim 1, characterized in that, The load detection method further includes: In response to the end of the processing of the current image frame, the interface call information of the current image frame is used as the interface call information of a historical image frame; The determination of the load information of the current image frame based on the interface call information of the current image frame and the interface call information of multiple historical image frames includes: In response to the number of historical image frames reaching a preset threshold, the load information of the current image frame is determined based on the interface call information of the current image frame and the interface call information of each of the historical image frames.

6. The load detection method according to claim 1, characterized in that, The load changes include non-abrupt, sudden increase, or sudden decrease.

7. The load detection method according to any one of claims 1 to 6, characterized in that, The first processor includes a central processing unit, and the second processor includes an image processor.

8. The load detection method according to any one of claims 1 to 6, characterized in that, The interface call information includes at least one of the following: number of interface calls, drawing area, and data volume.

9. The load detection method according to any one of claims 1 to 6, characterized in that, The application programming interface includes at least one of the following: binding interface, shader selection interface, indexed drawing interface, vertex indexing interface, framebuffer interface, viewport interface, texture update interface, buffer initialization interface, instance drawing interface, vertex drawing interface, texture creation interface, and shader calculation interface.

10. The load detection method according to any one of claims 1 to 6, characterized in that, The determination of the load information of the current image frame based on the interface call information of the current image frame and the interface call information of multiple historical image frames includes: Based on the interface call information of the current image frame, a first call information feature vector is generated; Based on the interface call information of each of the historical image frames, a second call information feature vector is generated; The first call information feature vector and the second call information feature vector are input into the target classification model to obtain the load information of the current image frame.

11. The load detection method according to claim 10, characterized in that, The target classification model includes a first encoder, a second encoder, and a first classifier. The first encoder and the second encoder have the same weight parameters, which are used to construct a Siamese network. The step of inputting the first call information feature vector and the second call information feature vector into the target classification model to obtain the load information of the current image frame includes: The first call information feature vector and the second call information feature vector are respectively input to the first encoder and the second encoder to obtain the first load feature corresponding to the first call information feature vector and the second load feature corresponding to the second call information feature vector. The first load feature and the second load feature are input into the first classifier to obtain the load information of the current image frame.

12. The load detection method according to claim 11, characterized in that, Both the first encoder and the second encoder include a fully connected layer, a linear correction unit, and multiple residual blocks connected in sequence.

13. The load detection method according to claim 11, characterized in that, The training process of the target classification model includes: Based on the interface call information of multiple sample image frames, a first sample feature vector and a second sample feature vector are generated. The processing of one sample image frame corresponding to the first sample feature vector is after the processing of multiple sample image frames corresponding to the second sample feature vector. The first sample feature vector and the second sample feature vector are respectively input into the third encoder and the fourth encoder of the initial classification model to obtain the third load feature corresponding to the first sample feature vector and the fourth load feature corresponding to the second sample feature vector. The loss function is determined based on the feature distance between the third load feature and the fourth load feature; Based on the loss function, the third encoder and the fourth encoder are subjected to first parameter optimization processing; The third load feature and the fourth load feature are input into the second classifier of the initial classification model to obtain the load information of the sample image frame; Based on the load information of the sample image frame, the second classifier is subjected to second parameter optimization processing to obtain the target classification model. The third encoder and the fourth encoder after the first parameter optimization processing are respectively constituted as the first encoder and the second encoder. The second classifier after the second parameter optimization processing is constituted as the first classifier.

14. The load detection method according to claim 13, characterized in that, The training process of the target classification model also includes: Determine the classification labels corresponding to the first sample feature vector and the second sample feature vector, wherein the classification labels are used to characterize whether the first sample feature vector and the second sample feature vector belong to the same sample category; The step of determining the loss function based on the feature distance between the third load feature and the fourth load feature includes: Based on the feature distance between the third load feature and the fourth load feature and the classification label, the loss function is determined such that the loss function is configured to be positively correlated with the feature distances corresponding to the first sample feature vector and the second sample feature vector belonging to the same sample category, and negatively correlated with the feature distances corresponding to the first sample feature vector and the second sample feature vector belonging to different sample categories.

15. A load detection device, characterized in that, The load detection device includes: The acquisition module is used to acquire interface call information of the current image frame in response to the first processor calling the application programming interface of the second processor during the processing of the current image frame. The determining module is used to determine the load information of the current image frame based on the interface call information of the current image frame and the interface call information of multiple historical image frames. The interface call information is used to characterize the call status of the first processor to at least one of the application programming interfaces of the second processor, and the load information is used to characterize the load change of the second processor during the processing of the current image frame.

16. The load detection device according to claim 15, characterized in that, The processing of each image frame includes a first processing step corresponding to the first processor and a second processing step corresponding to the second processor. The start time of the first processing step is earlier than the start time of the second processing step.

17. The load detection device according to claim 16, characterized in that, The first processor is configured to invoke the application programming interface of the second processor during the first processing.

18. The load detection device according to claim 16, characterized in that, The determining module is also used for: Before the second processing begins, the load information of the current image frame is determined.

19. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; The processor is configured to perform the load detection method as described in any one of claims 1 to 14.

20. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the load detection method as described in any one of claims 1 to 14.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the load detection method as described in any one of claims 1 to 14.

22. A chip, characterized in that, The chip includes a processor and an interface, the processor being configured to read instructions to execute the load detection method as described in any one of claims 1 to 14.

Images