Method and apparatus for scheduling codec under multiple instances, storage medium, terminal
By acquiring the real-time performance requirements of the codecs, prioritizing the scheduling of real-time codecs and alternately scheduling non-real-time codecs, the problem of poor system performance when multiple codecs work simultaneously is solved, thus improving the user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SPREADTRUM SEMICON (NANJING) CO LTD
- Filing Date
- 2022-11-24
- Publication Date
- 2026-07-21
Smart Images
Figure CN116126487B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of codec technology, and in particular to a method and apparatus for scheduling codecs under multiple instances, a storage medium, and a terminal. Background Technology
[0002] Currently, the market for devices related to codecs, such as mobile phones, mainly focuses on the performance of codecs operating in a single instance, where a single instance refers to watching only one video or using only the recording function at any given time. However, with hardware upgrades, increasingly complex application scenarios, higher user demands, and the emergence of various benchmarking software, the need for multiple codecs to work simultaneously is becoming increasingly urgent.
[0003] When multiple codecs work simultaneously, most current codec hardware uses time-division multiplexing to meet the needs of codecs under multiple instances. However, this method results in poor system performance and affects user experience. Summary of the Invention
[0004] The technical problem solved by the embodiments of the present invention is that when multiple codecs work simultaneously, the system performance is poor, which affects the user experience.
[0005] To address the aforementioned technical problems, this invention provides a method for scheduling codecs in a multi-instance environment, comprising: when multiple instances are running concurrently, obtaining real-time performance requirement parameters of the codecs corresponding to the multiple instances respectively, wherein the real-time performance requirement parameters are used to determine whether a codec is a real-time codec; scheduling the work of each codec based on the current actual performance satisfaction of the real-time codec and the real-time performance requirement parameters of each codec, wherein the scheduling priority of the real-time codec is higher than that of the non-real-time codec.
[0006] Optionally, scheduling the work of each codec based on the current actual performance of the real-time codec and the real-time performance requirement parameters of each codec includes: determining the codecs corresponding to multiple instances, including real-time codecs and non-real-time codecs, based on the real-time performance requirement parameters of each codec; determining whether the current actual performance meets the performance requirements of the real-time codec before encoding and decoding the first frame of data or after the previous frame of data encoding and decoding is completed; and scheduling the work of the non-real-time codec based on the remaining performance if the current actual performance meets the performance requirements of the real-time codec.
[0007] Optionally, the number of non-real-time codecs is multiple, and the step of scheduling the operation of non-real-time codecs according to the remaining performance based on the current actual performance meets the performance requirements of the real-time codec includes: scheduling multiple non-real-time codecs to work alternately according to the remaining performance based on the current actual performance meets the performance requirements of the real-time codec.
[0008] Optionally, if the current actual performance meets the performance requirements of the real-time codec, scheduling multiple non-real-time codecs to work alternately based on the remaining performance includes: if the current actual performance meets the performance requirements of the real-time codec, deleting the identifier of the current real-time codec from the real-time codec sequence, the real-time codec sequence being used to indicate the working real-time encoder; waiting for the real-time codec sequence to be empty and the number of working non-real-time codecs to meet a specified number, then scheduling the waiting non-real-time codecs to work.
[0009] Optionally, the scheduling method for the codecs under multiple instances further includes: if the currently working non-real-time codec finishes encoding and decoding a frame of data, it notifies the waiting non-real-time codecs to start working.
[0010] Optionally, the scheduling method for the codec under multiple instances further includes: if the current actual performance does not meet the performance requirements of the real-time codec, pausing the operation of the non-real-time codec so that the non-real-time codec is in a waiting state.
[0011] Optionally, the following method can be used to determine whether the current actual performance meets the performance requirements of the real-time codec: determine whether the actual time required to encode and decode one frame is greater than the theoretical time; if the actual time is greater than the theoretical time, then determine that the actual performance does not meet the performance requirements of the real-time codec; if the actual time is not greater than the theoretical time, then determine that the actual performance meets the performance requirements of the real-time codec.
[0012] Optionally, the method for scheduling codecs under multiple instances further includes: when multiple instances are running concurrently, determining the operating frequency of the codec corresponding to each instance based on the frame size and frame rate required by each instance; determining the final operating frequency based on the operating frequency of the codec corresponding to each instance; and using the final operating frequency to control the operation of the codec, wherein the final operating frequency satisfies the frequency requirements of all working codecs.
[0013] Optionally, obtaining the real-time performance requirement parameters of the codecs corresponding to the multiple instances includes: obtaining the real-time performance requirement parameters of the codecs corresponding to the multiple instances through a pre-configured calling interface of the hardware abstraction layer.
[0014] To address the aforementioned issues, this invention also provides a scheduling device for codecs in a multi-instance environment, comprising: an acquisition unit, configured to acquire real-time performance requirement parameters of codecs corresponding to multiple concurrent instances, wherein the real-time performance requirement parameters are used to determine whether a codec is a real-time codec; and a scheduling unit, configured to schedule the work of each codec based on the current actual performance satisfaction of the real-time codec and the real-time performance requirement parameters of each codec, wherein the scheduling priority of the real-time codec is higher than that of the non-real-time codec.
[0015] To address the aforementioned issues, embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, performs the steps of any of the above-described multi-instance codec scheduling methods.
[0016] To address the aforementioned issues, this invention also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes the steps of any of the above-described multi-instance codec scheduling methods.
[0017] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects:
[0018] In this embodiment of the invention, when multiple instances run concurrently, the real-time performance requirement parameters of the codecs corresponding to each instance are obtained. These parameters are used to determine whether a codec is a real-time codec. Based on the current performance requirements of the real-time codecs and the real-time performance requirement parameters of each codec, the work of each codec is scheduled. Since real-time codecs have a higher scheduling priority than non-real-time codecs, scheduling multiple codecs based on the current performance requirements of the real-time codecs and the real-time performance requirement parameters of each codec allows for priority scheduling of real-time codecs. This enables the scheduling of non-real-time codecs while meeting their performance requirements, thereby fully utilizing system performance and improving user experience. Attached Figure Description
[0019] Figure 1 This is a flowchart of a multi-instance codec scheduling method according to an embodiment of the present invention;
[0020] Figure 2 It is a flowchart for obtaining real-time performance requirement parameters;
[0021] Figure 3 This is a flowchart of a multi-instance codec scheduling method in a typical application scenario;
[0022] Figure 4 This is a schematic diagram of the structure of a codec scheduling device under multiple instances in an embodiment of the present invention. Detailed Implementation
[0023] As mentioned above, most current codec hardware uses time-sharing multiplexing to meet the needs of multiple instances. However, this approach, which treats all codecs equally, cannot guarantee performance when the system is running multiple codecs, has poor resource allocation, and affects user experience.
[0024] To address the aforementioned issues, in this embodiment of the invention, when multiple instances run concurrently, real-time performance requirement parameters for the codecs corresponding to each instance are obtained. These parameters determine whether a codec is a real-time codec. Based on the current performance requirements of the real-time codecs and the real-time performance requirement parameters for each codec, the work of each codec is scheduled. Since real-time codecs have a higher scheduling priority than non-real-time codecs, scheduling multiple codecs based on the current performance requirements of the real-time codecs and the real-time performance requirement parameters for each codec allows for priority scheduling of real-time codecs. This enables the scheduling of non-real-time codecs while still meeting their performance requirements, thereby fully utilizing system performance and improving user experience.
[0025] To make the above-mentioned objectives, features and beneficial effects of the embodiments of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0026] This invention provides a method for scheduling codecs under multiple instances. This method can be executed by a terminal, a chip or chip module within the terminal that has multi-instance codec scheduling functionality, a chip or chip module within the terminal that has data processing capabilities, or a baseband chip within the terminal. The terminal can be various suitable devices such as mobile phones, digital TVs, smartwatches, dashcams, and tablet computers.
[0027] Reference Figure 1 The present invention provides a flowchart of a codec scheduling method under multiple instances, which may specifically include the following steps:
[0028] Step 11: When multiple instances are running concurrently, obtain the real-time performance requirement parameters of the codecs corresponding to the multiple instances respectively. The real-time performance requirement parameters are used to determine whether the codec is a real-time codec.
[0029] Step 12: Based on the current performance of the real-time codec and the real-time performance requirements of each codec, schedule the work of each codec, wherein the scheduling priority of the real-time codec is higher than that of the non-real-time codec.
[0030] In practice, multiple instances of encoding / decoding related operations will run concurrently within the same time period. That is, multiple encoding / decoding related operations will be used at a certain moment or within a certain time period. These multiple encoding / decoding related operations may or may not be triggered simultaneously.
[0031] For example, recording the screen playback while a video is playing will result in two concurrent instances. Specifically, one instance is created in response to a video playback operation. If the user triggers the recording of the screen playback during video playback, another instance is created in response to that operation. In this case, two concurrent instances exist: one for playing the video and one for recording the screen playback. A codec is needed to decode the instance playing the video, and another codec is needed to encode the instance recording the screen playback.
[0032] For example, playing a video while making a video call will run two concurrent instances.
[0033] For example, during a video call, using two or more different playback platforms to play the video will result in three or more concurrent instances.
[0034] For example, some devices may lack the performance to support new encoding formats, requiring conversion from unsupported to supported formats for playback. This encoding conversion needs to be performed in the background. Simultaneously, if other instances require encoding / decoding, multiple instances will run concurrently. Here, encoding / decoding refers to either encoding or decoding.
[0035] It is understandable that there are other scenarios where multiple instances may run concurrently, which will not be listed here.
[0036] The real-time performance requirement parameters obtained in step 11 are used to indicate the codec's performance requirements on the system. These parameters indicate whether the codec is a real-time codec. A real-time codec is one that the system must meet the operating rate requirements. Non-real-time codecs require the system to make every effort to meet the requirements, but do not impose strict requirements; that is, the system may not guarantee that it will meet the requirements.
[0037] Real-time performance requirements can be codec priority, codec operating rate, or codec decoding rate. It should be noted that different manufacturers or different codec models may have different names or specific configuration parameters for real-time performance requirements.
[0038] For example, if the codec priority configured in the real-time performance requirement parameters is higher than the set priority, or if a specific operation rate or a specific decoding rate is configured, or if the real-time performance requirement parameters are configured to specify a first identifier, then it is determined to be a real-time codec.
[0039] For example, if the codec's real-time performance requirement parameter does not have a codec priority configured, or the configured codec priority is lower than the set priority, or the operation rate is the default value, or the decoding rate is the default value, or the real-time performance requirement parameter is configured to specify a second identifier, then it is determined to be a non-real-time codec.
[0040] The first specified identifier can be 0, and the second specified identifier can be 1. It is understood that the first specified identifier can be 1, and the second specified identifier can be 0. Alternatively, other characters can be used as the first or second specified identifier.
[0041] In some non-limiting embodiments, the real-time performance requirements of codecs corresponding to multiple instances can be obtained through the pre-configured calling interface of the Hardware Abstraction Layer (HAL).
[0042] For example, if the real-time performance requirement parameter is Priority, a Priority call interface can be pre-configured for calling Priority.
[0043] For example, if the real-time performance requirement parameter is OperatingRate, the OperatingRate call interface can be pre-configured for calling OperatingRate.
[0044] A codec typically comprises two parts: an encoder and a decoder. The encoder has an encoding component that performs compression (encoding) functions. The decoder has a decoding component that performs decompression (decoding) functions. Understandably, some codecs may have both an encoder and a decoder, while others may only have an encoder, or only a decoder.
[0045] Taking the real-time performance requirement parameters Priority and OperatingRate as an example, Priority and OperatingRate are configured by the upper-layer application. A public parameter class can be encapsulated to store or manage the parameters of the encoding and decoding components (encoding and decoding components). Both encoding and decoding can define the Priority and OperatingRate of the codec, which can be stored as parent classes of component parameters in the public parameter class. The encoding or decoding component inherits from the parent class of component parameters, meaning it can obtain Priority and OperatingRate from the parent class. Specifically, the interface getPriority_l() and getOperatingRate_l() are defined in the member functions of the public parameter class. The encoding and decoding components can obtain the Priority of the corresponding Codec by calling the getPriority_l() interface and obtain the OperatingRate of the Codec by calling the getOperatingRate_l() interface.
[0046] Reference Figure 2 This paper presents a flowchart for obtaining real-time performance requirement parameters. The following section uses Priority and OperatingRate as examples to illustrate the specific process for obtaining these parameters. The process may include the following steps:
[0047] Step 21: Obtain the Priority and OperatingRate parameters set by the upper-layer application.
[0048] Step 22: Configure codec parameters.
[0049] Codec configure parameters can include Priority and Operating Rate.
[0050] Step 23: Pass through the playback framework and interact with parameters.
[0051] Step 24: Place Priority and OperatingRate in the parent class of the component parameters.
[0052] Step 25: The encoding or decoding component inherits from the parent class of the component parameter.
[0053] Step 26: The encoding or decoding component obtains Priority and OperatingRate from the parent class of the component parameters.
[0054] In practice, when multiple instances are running concurrently, the codecs corresponding to the multiple instances, determined based on the real-time performance requirement parameters, can include three cases: only real-time codecs; only non-real-time codecs; and both real-time and non-real-time codecs.
[0055] In the specific implementation of step 12, the codecs corresponding to multiple instances are determined according to the real-time performance requirement parameters of each codec, including real-time codecs and non-real-time codecs. Before the encoding and decoding of the first frame of data or after the encoding and decoding of the previous frame of data is completed, it is determined whether the current actual performance meets the performance requirements of the real-time codec; if the current actual performance meets the performance requirements of the real-time codec, the work of the non-real-time codec is scheduled according to the remaining performance.
[0056] Furthermore, if the current actual performance meets the performance requirements of the real-time codec, multiple non-real-time codecs are scheduled to work alternately based on the remaining performance. For example, if there are two non-real-time codecs, denoted as non-real-time Codec1 and non-real-time Codec2 respectively, and the current actual performance meets the performance requirements of the real-time codec, then non-real-time Codec1 and non-real-time Codec2 are scheduled alternately.
[0057] In some non-limiting embodiments, if the current actual performance meets the performance requirements of the real-time codec, the identifier of the current real-time codec is removed from the real-time codec sequence, which is used to indicate the working real-time encoder; if the number of non-real-time codecs waiting to satisfy the condition that the real-time codec sequence is empty and the number of working non-real-time codecs meets a specified number, then the waiting non-real-time codecs are scheduled to work.
[0058] Furthermore, the requirement that the number of non-real-time codecs waiting to satisfy the condition that the real-time codec sequence is empty and working satisfies a specified number includes: the number of non-real-time codecs waiting to satisfy the condition that the real-time codec sequence is empty and working is zero, that is, the real-time codec sequence is empty and no non-real-time codecs are working.
[0059] Once the currently operating non-real-time codec has finished encoding and decoding a frame of data, it notifies the waiting non-real-time codecs to begin operation. The waiting non-real-time codecs then pause their operation to avoid consuming resources (such as CPU resources).
[0060] In practice, if the current performance does not meet the performance requirements of the real-time codec, the operation of the non-real-time codec is paused, causing it to wait. The waiting non-real-time codec will temporarily stop encoding and decoding.
[0061] In some non-limiting embodiments, the following method is used to determine whether the current actual performance meets the performance requirements of the real-time codec: determine whether the actual time required to encode and decode one frame is greater than the theoretical time; if the actual time is greater than the theoretical time, then determine that the actual performance does not meet the performance requirements of the real-time codec; if the actual time is not greater than the theoretical time, then determine that the actual performance meets the performance requirements of the real-time codec.
[0062] In other non-limiting embodiments, taking decoding as an example, it is determined whether the current actual number of decoded frames meets the frame rate requirement of the real-time codec. If the current actual number of decoded frames meets the frame rate requirement of the real-time codec, then the actual performance is determined to meet the performance requirements of the real-time codec. Conversely, if the current actual number of decoded frames does not meet the frame rate requirement of the real-time codec, then the actual performance is determined to not meet the performance requirements of the real-time codec.
[0063] For example, if a real-time codec requires a frame rate of 30 frames per second, at the 2nd second, it is determined whether 60 frames have been decoded. If less than 60 frames have been decoded, the actual performance is deemed insufficient to meet the real-time codec's requirements. The real-time codec is then scheduled, while the non-real-time codec is paused, placing it in a waiting state. If 60 frames have been decoded, the actual performance is deemed sufficient to meet the real-time codec's requirements, and the non-real-time codec can be scheduled to work.
[0064] In practical implementation, after creating an instance, the Priority and OperatingRate of the codec corresponding to the current instance can be obtained. Codec scheduling is then performed based on these two parameters. A resource scheduling module can be used for codec scheduling. For example, the codec component can call a resource scheduling function to determine whether the currently working codec is a real-time codec or a non-real-time codec based on Priority and OperatingRate. For instance, if the OperatingRate value is less than or equal to 0, it is determined to be a non-real-time codec. If it is a real-time codec, it is determined whether the actual time required to encode and decode one frame is greater than the theoretical time. If the actual time is greater than the theoretical time, the system cannot meet the performance requirements of the current real-time codec, and the current codec is added to the real-time codec sequence. The process() function is executed, called once for each frame of data processed. This function will continuously loop until the encoding and decoding task ends. The resource scheduling module is set in the HAL layer and is used to schedule the codec before encoding and decoding each frame of data. Each frame of data can be either image data or audio data.
[0065] Conversely, if the system can meet the performance requirements of the current real-time codec, then the current codec is removed from the real-time codec sequence. In other words, if the system performance is sufficient, the non-real-time codec is scheduled to start working.
[0066] After a non-real-time codec finishes encoding and decoding a frame (i.e., after the process() function is executed), the non-real-time codec that has completed encoding and decoding a frame is removed from the non-real-time codec sequence, and the waiting non-real-time codecs are notified. This is used to implement the alternating encoding and decoding of non-real-time codecs. The waiting non-real-time codecs that receive the notification can start working when the conditions are met (such as the real-time codec sequence is empty and the number of working non-real-time codecs meets the specified number).
[0067] In practical implementation, when multiple instances are running concurrently, the operating frequency of the codec for each instance is determined based on the required frame size and frame rate of each instance; the final operating frequency is determined based on the operating frequency of the codec for each instance; the operation of the codec is controlled by the final operating frequency, wherein the final operating frequency satisfies the frequency requirements of all working codecs.
[0068] Taking video encoding and decoding as an example, the frame size can be the size of the image, such as the number of pixels in the image.
[0069] In some embodiments, a threshold can be preset, with different thresholds corresponding to different frequency levels. For each instance, the product of frame size (frame_size) and frame rate (frame_rate) is calculated. Based on the relationship between the product and the threshold, the frequency level is determined, and then the operating frequency of the codec corresponding to the instance is determined based on the frequency level.
[0070] For example, four frequency levels are defined: Level 1, Level 2, Level 3, and Level 4. Different frequency levels operate at different frequencies.
[0071] Level 4: frame_rate*frame_size≥2560*1440*30;
[0072] Level 3: frame_rate*frame_size≥1920*1080*30;
[0073] Level 2: frame_rate*frame_size≥1280*720*30;
[0074] Level 1: else, where else represents other cases, referring to cases where Level 2, Level 3, and Level 4 are not satisfied.
[0075] In other embodiments, the operating frequency of the codec can be determined based on the codec priority and operation rate of the instance. Specifically, while meeting the minimum operating frequency requirement for a single instance, the mapping relationship between codec priority, operation rate, and operating frequency can be pre-configured and stored. Subsequently, the corresponding operating frequency can be found from the mapping relationship based on the codec priority and operation rate.
[0076] In other embodiments, for each instance, the frequency level of the codec corresponding to the instance can be determined based on the product of frame size and frame rate, and the relationship between the product and a threshold. Then, the operating frequency and frequency level of the codec corresponding to the instance are determined based on the codec priority and operating rate, thus determining the final operating frequency. Since different terminals have different chip performance, determining the frequency level first and then comprehensively considering the codec priority and operating rate to determine the final operating frequency allows for the determination of an operating frequency compatible with the terminal, helping to fully utilize the system's codec performance.
[0077] For example, four concurrent instances are designated Instance 1, Instance 2, Instance 3, and Instance 4. Instance 1 plays a video in Advanced Video Coding (AVC) format, at 1080p resolution and 30 frames per second (fps). Instance 2 records a video in AVC format, at 2K resolution and 30 fps. Instance 3 transcodes the video, converting the resolution from 1080p to 720p. Instance 4 transcodes the video, converting the resolution from 2K to 1080p.
[0078] Instance 1 has a real-time performance requirement of Priority of 0 and Operating Rate of 30. Instance 2 has a real-time performance requirement of Priority of 0 and Operating Rate of 30. Instance 3 has a real-time performance requirement of Priority of 1. Instance 4 has no real-time performance requirement parameters set, i.e., neither Priority nor Operating Rate is set.
[0079] Based on the real-time performance requirements parameters of each instance, the codec corresponding to instance 1 is a real-time codec, the codec corresponding to instance 2 is a real-time codec, the codec corresponding to instance 3 is a non-real-time codec, and the codec corresponding to instance 4 is a non-real-time codec.
[0080] The frequency levels can be divided into four levels as described above, or they can be identified as low frequency (Low FRQ), mid frequency (Mid FRQ), and high frequency (Highest FRQ). The operating frequency of the low frequency band meets the first frequency range, the operating frequency of the mid frequency band meets the second frequency range, and the operating frequency of the high frequency band meets the third frequency range. The operating frequencies of the low frequency band, mid frequency band, and high frequency band increase sequentially.
[0081] For example, Priority values of 1, 2, and 3 correspond to Highest FRQ, Mid FRQ, and Low FRQ, respectively. Instance 1 corresponds to a real-time codec, which only needs to run on Low FRQ to meet the minimum operating frequency requirement for a single instance. Instance 2 also corresponds to a real-time codec, similarly requiring only Low FRQ. Instance 3 corresponds to a non-real-time codec, running on Highest FRQ according to the set priority parameters. Instance 4 corresponds to a non-real-time codec, running on Mid FRQ.
[0082] If the codecs corresponding to instance 1, instance 2, instance 3 and instance 4 are running simultaneously, the final working frequency setting should be based on the highest FRQ required by the codec corresponding to instance 3, according to the maximum demand.
[0083] If the real-time codecs of instance 1 and instance 2 and the non-real-time codec of instance 4 (Priority set to 2) run simultaneously, according to the highest frequency requirement of a single codec, the non-real-time codec of instance 4 needs to run in Mid FRQ. However, since Mid FRQ cannot meet the rate requirements of the real-time codecs of instance 1 and instance 2, the final working frequency needs to be set to Highest FRQ.
[0084] In some embodiments, when a new instance is detected, the final operating frequency is re-determined based on the multiple concurrent instances following the new instance.
[0085] In some embodiments, hardware encoding and decoding can be employed. The encoding / decoding component implements its functions by calling a hardware encoding / decoding library. That is, the encoding / decoding component calls the driver side, ultimately invoking the system-on-a-chip (SoC) for encoding and decoding. Therefore, selecting an appropriate hardware codec operating frequency plays a crucial role in maximizing the system's encoding and decoding performance. The final operating frequency can be selected based on parameters such as frame size, frame rate, Priority, and OperatingRate. The encoding / decoding component obtains the Priority and OperatingRate parameters and passes them to the driver side for encoding and decoding via the DecInit() and EncInit() functions. The DecInit() function is used for decoding, and the EncInit() function is used for encoding.
[0086] As shown above, when multiple instances run concurrently, the real-time performance requirement parameters of the codecs corresponding to each instance are obtained. These parameters determine whether a codec is a real-time codec. Based on the current performance requirements of the real-time codecs and their respective real-time performance requirement parameters, the work of each codec is scheduled. Since real-time codecs have a higher scheduling priority than non-real-time codecs, scheduling multiple codecs based on the current performance requirements of the real-time codecs and their respective real-time performance requirement parameters allows for prioritizing the scheduling of real-time codecs. This enables the scheduling of non-real-time codecs while still meeting their performance requirements, thereby fully utilizing system performance and improving user experience.
[0087] Reference Figure 3 This paper presents a flowchart of a multi-instance codec scheduling method in a typical application scenario. For ease of understanding, this embodiment uses three concurrent instances, namely instance 1, instance 2, and instance 3, as examples for illustration. The scheduling method may specifically include the following steps.
[0088] Step 301, Instance 1 corresponds to real-time Codec1.
[0089] Step 302, Instance 2 corresponds to non-real-time Codec1.
[0090] Step 303, Instance 3 corresponds to non-real-time Codec2.
[0091] In some non-limiting embodiments, based on the concurrent instances of operations related to encoding and decoding, the concurrent instances of operations related to encoding and decoding are typically detected, that is, the corresponding instances are created.
[0092] Depending on the creation time of Instance 1, Instance 2, and Instance 3, steps 301, 302, and 303 can be executed simultaneously or not simultaneously. Instance 1, Instance 2, and Instance 3 must exist simultaneously for at least a certain period of time.
[0093] After creating the corresponding instances, it can be determined whether the codec is a real-time codec based on the real-time performance requirement parameters Priority and / or OperatingRate for each instance. Specifically, the codec corresponding to instance 1 is real-time Codec1, the codec corresponding to instance 2 is non-real-time Codec1, and the codec corresponding to instance 3 is non-real-time Codec2.
[0094] Step 304: Determine whether the encoding / decoding process has ended.
[0095] If the result is negative, meaning encoding / decoding has not ended, proceed to step 305. If the result is positive, meaning encoding / decoding has ended, then the process terminates. Taking video as an example, the end of encoding / decoding can include the following situations: For instance, if video playback is detected as canceled, such as closing the playing video, then encoding / decoding ends. Another example is when the video playback resolution is switched; in this case, the encoding / decoding of the instance corresponding to the previous video resolution ends, and a new instance corresponding to the switched resolution is created. Yet another example is when video playback finishes; in this case, the encoding / decoding of that instance ends.
[0096] Step 305: Call the resource scheduling function.
[0097] Step 306: Determine whether the actual time required to encode and decode one frame is greater than the theoretical time.
[0098] If the judgment result is yes, proceed to step 307; if the judgment result is no, proceed to step 308.
[0099] In practice, theoretical time refers to the encoding and decoding time required for an instance of a real-time codec, which can be calculated based on the frame rate required by the instance.
[0100] In practical implementation, a real-time codec queue and a non-real-time codec queue can be configured. The real-time codec queue is used to indicate the operation of real-time codecs, and the non-real-time codec queue is used to indicate the operation of non-real-time codecs. The real-time codec queue contains the identifiers of the scheduled real-time codecs. The non-real-time codec queue contains the identifiers of the non-real-time codecs. Each codec identifier corresponds one-to-one with a codec and is used to identify the codec.
[0101] Step 307: Real-time codec performance cannot be guaranteed, limiting the operation of non-real-time codecs.
[0102] In practical implementation, when the real-time codec performance cannot be guaranteed, if there is already a real-time codec identifier in the real-time codec queue, the real-time codec identifier will remain in the queue. If there is no real-time codec identifier in the queue, the real-time codec identifier will be added to the queue.
[0103] When there are multiple real-time codecs, if the performance of all real-time codecs is not satisfactory, then all real-time codec identifiers are in the real-time codec queue.
[0104] In some non-limiting embodiments, if the performance of some real-time codecs among multiple real-time codecs is not satisfactory, the identifiers of the real-time codecs that do not meet the performance requirements can be added to the real-time codec queue.
[0105] Furthermore, it's possible to notify the non-real-time codec that it's currently in a restricted working state. Restricting the non-real-time codec's operation can be achieved by pausing the process function of the non-real-time codec instance, such as the `process()` function. Since the `process()` function in the codec component is called once for each frame of data processed during encoding / decoding, and this function continuously loops until the encoding / decoding task is completed, pausing the execution of the `process()` function pauses the codec's encoding / decoding operations.
[0106] Step 308: If the real-time codec performance is satisfied, schedule the non-real-time codec to work.
[0107] In practice, when the performance of the real-time codec is satisfactory, the real-time codec identifier can be removed from the real-time codec queue, and a non-real-time codec identifier can be added to the non-real-time codec queue to schedule the work of the non-real-time codec.
[0108] Furthermore, it can also notify non-real-time codecs to work.
[0109] Step 309: Process each frame of data.
[0110] In practice, the process() function can be executed. Once the process() function is completed, the encoding and decoding of the current frame of data is finished, and the encoding and decoding of the next frame of data can begin.
[0111] After step 309 is executed, continue with step 304.
[0112] After step 308 is completed, you can jump to step 312 and / or step 317.
[0113] In practice, whether step 312, step 317, or both are executed after step 308 depends on the non-real-time codec's operational configuration. For example, if the configuration allows non-real-time codecs to work alternately, then non-real-time codec1 will be scheduled to work in the previous frame, and non-real-time codec2 will be scheduled to work in the next frame, and so on, until the encoding and decoding of the instance is complete. Alternatively, if the configuration allows non-real-time codecs to work simultaneously, then non-real-time codec1 and non-real-time codec2 will be scheduled to work simultaneously.
[0114] Taking the alternating scheduling of non-real-time codecs as an example, when scheduling non-real-time codec1, step 312 is executed.
[0115] Step 312: Wait for the real-time codec sequence to be empty and for no non-real-time codecs to be working.
[0116] Step 313: Process each frame of data.
[0117] Process each frame of data, which means executing the process() function corresponding to the non-real-time codec.
[0118] Step 314: Remove executed non-real-time codecs from the non-real-time codec sequence.
[0119] That is, after the process() function finishes executing, the executed non-real-time codec function is removed from the non-real-time codec sequence.
[0120] After step 314 is completed, steps 310 and 311 are executed. Step 310 can be the same as steps 304 and 315; that is, steps 310, 304, and 315 are shared. Similarly, step 311 can be the same as steps 305 and 316; that is, steps 311, 305, and 316 are shared. This saves system computational overhead. Therefore, regardless of whether step 305, 311, or 316 is executed, the process proceeds to step 306.
[0121] After step 314 is completed, the encoding and decoding of the current frame can also be completed through non-real-time Codec2.
[0122] When scheduling non-real-time Codec2 to work, execute step 317.
[0123] Step 317: Wait for the real-time codec sequence to be empty and for no non-real-time codecs to be working.
[0124] Step 318: Process each frame of data.
[0125] Process each frame of data, which means executing the process() function corresponding to the non-real-time codec.
[0126] Step 319: Remove executed non-real-time codecs from the non-real-time codec sequence.
[0127] That is, after the process() function finishes executing, the executed non-real-time codec function is removed from the non-real-time codec sequence.
[0128] After step 319 is completed, the encoding and decoding of the current frame can also be completed through non-real-time Codec1.
[0129] It should be noted that step 312 above is an example of the suitability of alternately scheduling non-real-time codecs when the performance of the real-time codec is satisfied. When the scheduling conditions for the non-real-time codec are configured differently, the specific content of step 312 can be adjusted to the corresponding configuration conditions, which will not be listed here.
[0130] This invention also provides a scheduling device for codecs in a multi-instance environment, referring to... Figure 4 The provided diagram illustrates the structure of a multi-instance codec scheduling device according to an embodiment of the present invention. The multi-instance codec scheduling device 40 includes:
[0131] The acquisition unit 41 is used to acquire the real-time performance requirement parameters of the codecs corresponding to the multiple instances when multiple instances are running concurrently. The real-time performance requirement parameters are used to determine whether the codec is a real-time codec.
[0132] The scheduling unit 42 is used to schedule the work of each codec according to the current actual performance satisfaction of the real-time codec and the real-time performance requirement parameters of each codec, wherein the scheduling priority of the real-time codec is higher than that of the non-real-time codec.
[0133] In specific implementation, the working principle and workflow of the multi-instance codec scheduling device 40 can be found in the description of the multi-instance codec scheduling method provided in any of the above embodiments, and will not be repeated here.
[0134] In specific implementation, the aforementioned multi-instance codec scheduling device 40 may correspond to a chip in the terminal that has multi-instance codec scheduling function, such as a SOC (System-On-a-Chip), baseband chip, etc.; or correspond to a chip module in the terminal that includes multi-instance codec scheduling function; or correspond to a chip module with data processing function chip; or correspond to the terminal.
[0135] This invention also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, performs the steps of the multi-instance codec scheduling method provided in any of the above embodiments of this invention.
[0136] The computer-readable storage medium may include non-volatile or non-transitory memory, and may also include optical discs, hard disk drives, solid-state drives, etc.
[0137] Specifically, in this embodiment of the invention, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0138] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DR RAM).
[0139] This invention also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes the steps of the codec scheduling method under multiple instances provided in any of the above embodiments.
[0140] The memory and the processor are coupled, and the memory can be located inside or outside the terminal. The memory and the processor can be connected via a communication bus.
[0141] Terminals can include, but are not limited to, mobile phones, computers, tablets, digital TVs, smartwatches, dashcams, and other terminal devices, as well as servers, cloud platforms, etc.
[0142] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.
[0143] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0144] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or in a combination of hardware and software functional units. For example, for various devices or products applied to or integrated into a chip, each module / unit can be implemented using hardware such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware such as circuits; for various devices or products applied to or integrated into a chip module, each module / unit can be implemented using hardware such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.
[0145] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.
[0146] In the embodiments of this application, "multiple" refers to two or more.
[0147] The descriptions of "first," "second," "third," etc., appearing in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They have no order and do not indicate any special limitation on the number of devices in the embodiments of this application, nor do they constitute any limitation on the embodiments of this application.
[0148] It should be noted that the sequence number of each step in this embodiment does not represent a limitation on the execution order of each step.
[0149] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.
Claims
1. A scheduling method for a codec in a multi-instance environment, characterized in that, include: When multiple instances are running concurrently, the real-time performance requirement parameters of the codecs corresponding to the multiple instances are obtained. The real-time performance requirement parameters are used to determine whether the codec is a real-time codec. Based on the current performance requirements of the real-time codecs and the real-time performance requirements of each codec, the work of each codec is scheduled, wherein the scheduling priority of the real-time codecs is higher than that of the non-real-time codecs. The process of scheduling the work of each codec based on the current performance satisfaction of the real-time codec and the real-time performance requirements of each codec includes: Based on the real-time performance requirements parameters of each codec, the codecs corresponding to multiple instances are determined, including real-time codecs and non-real-time codecs. Before encoding and decoding the first frame of data or after encoding and decoding the previous frame of data is completed, determine whether the current actual performance meets the performance requirements of the real-time codec. If the current actual performance meets the performance requirements of the real-time codec, the work of the non-real-time codec is scheduled according to the remaining performance. The method for determining whether the current actual performance meets the performance requirements of the real-time codec is as follows: determine whether the actual time required to encode and decode one frame is greater than the theoretical time; if the actual time is greater than the theoretical time, then the actual performance does not meet the performance requirements of the real-time codec; if the actual time is not greater than the theoretical time, then the actual performance meets the performance requirements of the real-time codec.
2. The scheduling method for codecs under multiple instances as described in claim 1, characterized in that, The number of non-real-time codecs is multiple. The step of scheduling the work of the non-real-time codecs based on the remaining performance, if the current actual performance meets the performance requirements of the real-time codec, includes: If the current actual performance meets the performance requirements of the real-time codec, multiple non-real-time codecs are scheduled to work alternately based on the remaining performance.
3. The scheduling method for codecs under multiple instances as described in claim 2, characterized in that, If the current actual performance meets the performance requirements of the real-time codec, multiple non-real-time codecs are scheduled to work alternately based on the remaining performance, including: If the current actual performance meets the performance requirements of the real-time codec, the identifier of the current real-time codec is removed from the real-time codec sequence, which is used to indicate the working real-time encoder. If the number of non-real-time codecs waiting for the real-time codec sequence to be empty and in operation meets a specified number, then the waiting non-real-time codecs will be scheduled to start working.
4. The scheduling method for codecs under multiple instances as described in claim 3, characterized in that, Also includes: If the currently working non-real-time codec finishes encoding and decoding a frame of data, it notifies the waiting non-real-time codec to start working.
5. The scheduling method for codecs under multiple instances as described in claim 1, characterized in that, Also includes: If the current actual performance does not meet the performance requirements of the real-time codec, the operation of the non-real-time codec is paused so that the non-real-time codec is in a waiting state.
6. The scheduling method for codecs under multiple instances as described in claim 1, characterized in that, Also includes: When multiple instances run concurrently, the operating frequency of the codec for each instance is determined based on the frame size and frame rate required by each instance. The final operating frequency is determined based on the operating frequency of the codec corresponding to each instance. The operation of the codec is controlled by the final operating frequency, wherein the final operating frequency satisfies the frequency requirements of all operating codecs.
7. The scheduling method for codecs under multiple instances as described in claim 1, characterized in that, The step of obtaining the real-time performance requirement parameters of the codecs corresponding to multiple instances includes: By using the pre-configured calling interface of the hardware abstraction layer, the real-time performance requirements parameters of the codecs corresponding to multiple instances can be obtained.
8. A scheduling device for a codec in a multi-instance environment, characterized in that, include: The acquisition unit is used to acquire the real-time performance requirement parameters of the codecs corresponding to the multiple concurrent instances, wherein the real-time performance requirement parameters are used to determine whether the codec is a real-time codec. The scheduling unit is used to schedule the work of each codec based on the current actual performance satisfaction of the real-time codec and the real-time performance requirement parameters of each codec, wherein the scheduling priority of the real-time codec is higher than that of the non-real-time codec. The scheduling unit is configured to determine, based on the real-time performance requirement parameters of each codec, the codecs corresponding to multiple instances, including real-time codecs and non-real-time codecs; before encoding and decoding the first frame of data or after the previous frame of data encoding and decoding is completed, determine whether the current actual performance meets the performance requirements of the real-time codec; if the current actual performance meets the performance requirements of the real-time codec, schedule the work of the non-real-time codecs according to the remaining performance; wherein, the determination of whether the current actual performance meets the performance requirements of the real-time codec is made in the following manner: determine whether the actual time required to encode and decode one frame is greater than the theoretical time; if the actual time is greater than the theoretical time, determine that the actual performance does not meet the performance requirements of the real-time codec; if the actual time is not greater than the theoretical time, determine that the actual performance meets the performance requirements of the real-time codec.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a processor, performs the steps of the scheduling method for the codec under multiple instances as described in any one of claims 1 to 7.
10. A terminal comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the scheduling method for the codec under multiple instances as described in any one of claims 1 to 7.