Method for scheduling of coding resources and electronic device

Abstract

The application provides a coding resource scheduling method and an electronic device, and relates to the technical field of coding. When a video service application requests coding processing, the remaining available resource amount of a hardware codec is queried through a macro block capability query interface and a coding management service module, and then it is judged whether the current remaining hardware coding resource meets the coding requirement of the application. According to the judgment result, the coding strategy is adjusted: the resolution or code rate is reduced, the coding capability of the application is reduced, and it is ensured that the video service can continue to be executed; or the coding mode is changed, and then software coding is applied, and it is ensured that the video service can continue to be executed; or if the service party cannot reduce the resolution or code rate, and cannot use software coding, the video service is normally terminated, and the user is prompted, and it is ensured that the application does not crash.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of coding and decoding technology, in particular to a coding and decoding resource scheduling method and an electronic device. BACKGROUND

[0002] With the improvement of the hardware performance of the electronic device, the electronic device can support video services such as video recording, video playing, video editing, video content analysis, video content extraction, and video screen projection.

[0003] In order to reduce the occupation of storage space and facilitate transmission over the network, the video is usually compressed, that is, video encoding. When the electronic device plays the video, the video obtained by the electronic device is usually an encoded video file, so the electronic device decodes the already encoded video file. Then, the video can be normally played after being decoded. At present, the video can be encoded or decoded by the hardware codec in the electronic device.

[0004] However, when the electronic device simultaneously processes multiple video services, the multiple video services all need to occupy hardware codec resources for encoding or decoding, which may result in insufficient hardware codec resources, leading to failure of hardware codec and thus video service, affecting the user experience. SUMMARY

[0005] In view of this, the present application provides a coding and decoding resource scheduling method and an electronic device, which can solve the problem of service failure or program abnormal exit due to insufficient hardware codec resources when the electronic device simultaneously processes multiple services.

[0006] In a first aspect, the present application provides a coding and decoding resource scheduling method applied to an electronic device, wherein the electronic device is provided with a hardware codec. The scheduling method can include: receiving a coding and decoding request of a first service; obtaining a remaining available resource amount of the hardware codec; if the coding and decoding resource amount requested by the first service is greater than the remaining available resource amount of the hardware codec, obtaining a target coding and decoding strategy, the target coding and decoding strategy being used to indicate to reduce the coding and decoding demand of the first service or to perform software coding and decoding on the first service; and then performing coding and decoding processing on the first service according to the target coding strategy.

[0007] When the first service request is applied for codec processing, the remaining available resource amount of the hardware codec is queried based on the macro block capability query interface provided in the application, and then it is determined whether the current remaining hardware codec resource meets the application codec demand. According to the determination result, the codec strategy is adjusted: the resolution or code rate is reduced, the application codec capability is reduced, and it is ensured that the first service can continue to be executed; or the codec mode is changed, and then software codec is applied, and it is ensured that the first service can continue to be executed; or if the service party cannot reduce the resolution or code rate, and cannot use software codec, the first service is normally terminated, and the user is prompted, and it is ensured that the application does not crash.

[0008] In some possible implementation ways, the target codec strategy can include: adjusting the codec parameter of the first service, so that the codec resource amount requested by the first service is less than or equal to the remaining available resource amount of the hardware codec; and performing codec processing on the first service by the hardware codec according to the adjusted codec parameter of the first service. The codec resource amount requested by the first service is determined according to the codec parameter of the first service.

[0009] The codec parameter of the first service includes video resolution, code rate and / or frame rate. For example, the above-mentioned adjusting the codec parameter of the first service includes: reducing at least one of the video resolution, code rate and frame rate.

[0010] In actual implementation, the video display performance (for example, definition or smoothness) corresponding to the adjusted codec parameter of the first service is lower than the video display performance corresponding to the codec parameter requested by the first service. It can be understood that the application scheme can reduce the codec demand or quality to ensure that the service can normally perform codec processing, thereby ensuring that the service can normally run.

[0011] In some possible implementation ways, the target codec strategy can include: performing codec processing on the first service by the software codec according to the codec parameter requested by the first service. It can be understood that the application scheme can perform soft codec by CPU to ensure that the service can normally perform codec processing, thereby ensuring that the service can normally run.

[0012] In some possible implementation ways, the codec resource of the hardware codec is a macro block resource, and the remaining available resource amount of the hardware codec is a remaining macro block number.

[0013] In some possible implementation manners, the adjusting the codec parameter of the first service comprises: adjusting the codec parameter of the first service according to the remaining available resource amount of the hardware codec, so that the codec resource amount requested by the first service is less than or equal to the remaining available resource amount of the hardware codec. The codec resource amount requested by the first service is determined according to the codec parameter of the first service.

[0014] In some possible implementation manners, the method further can comprise: storing hardware codec resource available information, the hardware codec resource available information being used to indicate the remaining available resource amount of the hardware codec, and the hardware codec resource available information being updated when a codec resource is occupied and being updated when an occupied codec resource is released.

[0015] The obtaining the remaining available resource amount of the hardware codec comprises: obtaining the remaining available resource amount of the hardware codec according to the hardware codec resource available information.

[0016] In some possible implementation manners, the method further can comprise: storing hardware codec resource occupation information, the hardware codec resource occupation information being used to indicate the occupied resource amount of the hardware codec, and the hardware codec resource occupation information being updated when a codec resource is occupied and being updated when an occupied codec resource is released.

[0017] The obtaining the remaining available resource amount of the hardware codec comprises: determining the remaining available resource amount of the hardware codec according to the hardware codec resource occupation information and hardware codec total resource information. The hardware codec total resource information is used to indicate the maximum resource amount supported by the hardware codec.

[0018] In some possible implementation manners, the electronic device is further provided with a codec management service module. The codec management service module can be used to store the hardware codec resource available information and / or the hardware codec resource occupation information.

[0019] In some possible implementation manners, the determining the remaining available resource amount of the hardware codec comprises: the codec management service module obtaining the remaining available resource amount of the hardware codec according to the stored hardware codec resource available information; or the codec management service module determining the remaining available resource amount of the hardware codec according to the hardware codec total resource information and the stored hardware codec resource occupation information.

[0020] In some possible implementation manners, the method further can comprise: the codec management service module obtaining hardware codec total resource information corresponding to the hardware codec.

[0021] In some possible implementation manners, the electronic device is installed with a target application, and the first service is a service initiated by the target application. In this case, the adjusting the codec parameter of the first service comprises: adjusting the codec parameter of the first service by the target application, or adjusting the codec parameter of the first service by the codec management service module.

[0022] In some possible implementation manners, before the adjusting the codec parameter of the first service by the target application, the method further comprises: sending, by the codec management service module, a first message to the target application, the first message being used to indicate the remaining available resource amount of the hardware codec and the codec resource amount requested by the first service being greater than the remaining available resource amount of the hardware codec.

[0023] The adjusting the codec parameter of the first service by the target application comprises: adjusting, by the target application, the codec parameter of the first service after receiving the first message.

[0024] In some possible implementation manners, the method can further comprise: receiving, by the target application, a start operation of the first service by a user; and in response to the start operation, sending, by the target application, a codec request of the first service to the codec management service module, wherein the codec request comprises the codec parameter requested by the first service.

[0025] In some possible implementation manners, after the sending, by the target application, of the codec request of the first service to the codec management service module, the method can further comprise: receiving, by the codec management service module, the codec request of the first service; and calculating, by the codec management service module, the codec resource amount requested by the first service according to the codec parameter requested by the first service.

[0026] In some possible implementation manners, the method can further comprise: when the first service completes the codec, releasing, by the hardware codec, the hardware codec resource occupied by the first service; then sending, by the hardware codec, a second message to the codec management service module, the second message being used to indicate that the hardware codec resource occupied by the first service has been released; and then updating, by the codec management service module, the stored hardware codec resource available information or the hardware codec resource occupation information according to the second message.

[0027] In some possible implementation manners, after the determining the remaining available resource amount of the hardware codec, the method can further comprise:

[0028] If the codec resource amount of the first service request is less than or equal to the available resource amount of the hardware codec, the hardware codec is used to perform the codec processing on the first service according to the codec parameter of the first service request.

[0029] In some possible implementation manners, after the codec processing on the first service, the method further includes: obtaining, by the target application, the decoded data or the encoded data obtained by performing the codec processing on the first service; and displaying, by the target application, the decoded data or storing the encoded data.

[0030] In some possible implementation manners, the method further includes: if neither the codec hardware nor the codec software supports performing the codec processing on the first service, normally terminating the first service; and displaying, on an interface of the target application, a prompt information for prompting that the first service is terminated due to insufficient codec resources.

[0031] In some possible implementation manners, the first service can be a video playing, video recording or screen projection service.

[0032] In a second aspect, the present application provides a codec resource scheduling apparatus, which includes units for performing the method in the first aspect. The apparatus can correspond to the method described in the first aspect, and the related description of the units in the apparatus can refer to the description of the first aspect. For brevity, the description is not repeated here.

[0033] The method described in the first aspect can be implemented by hardware, or the corresponding software can be executed by hardware. The hardware or software includes one or more modules or units corresponding to the above functions. For example, processing modules or units, display modules or units, and the like.

[0034] In a third aspect, the present application provides an electronic device, which includes a processor and a memory. The memory is used to store computer programs or instructions, and the processor is used to execute the computer programs or instructions stored in the memory, so that the method in the first aspect is executed.

[0035] For example, the processor is used to execute the computer programs or instructions stored in the memory, so that the apparatus executes the method in the first aspect.

[0036] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program (also referred to as instructions or codes) for implementing the method in the first aspect.

[0037] For example, when the computer program is executed by a computer, the computer can execute the method in the first aspect.

[0038] In a fifth aspect, the present application provides a chip comprising a processor. The processor is configured to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation manner thereof.

[0039] Optionally, the chip further comprises a memory, and the memory is connected to the processor through a circuit or a wire.

[0040] In a sixth aspect, the present application provides a chip system comprising a processor. The processor is configured to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation manner thereof.

[0041] Optionally, the chip system further comprises a memory, and the memory is connected to the processor through a circuit or a wire.

[0042] In a seventh aspect, the present application provides a computer program product comprising a computer program (also referred to as instructions or codes), which, when executed by a computer, causes the computer to implement the method in the first aspect.

[0043] It can be understood that the beneficial effects of the above-mentioned second aspect to seventh aspect can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A schematic diagram of a video playing process provided by an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a system architecture to which a coding resource scheduling method provided by an embodiment of the present application is applied;

[0046] Figure 3 A schematic diagram of interaction between modules in a system architecture to which a coding resource scheduling method provided by an embodiment of the present application is applied;

[0047] Figure 4 A schematic diagram of an interface when a coding resource scheduling method provided by an embodiment of the present application is applied to a video playing scenario;

[0048] Figure 5 A schematic diagram of an interface when a coding resource scheduling method provided by an embodiment of the present application is applied to a video playing scenario;

[0049] Figure 6 A schematic diagram of an interface when a coding resource scheduling method provided by an embodiment of the present application is applied to a video playing scenario;

[0050] Figure 7 A flowchart of a coding resource scheduling method provided by an embodiment of the present application;

[0051] Figure 8 One of the scene schematic diagrams to which the coding resource scheduling method provided by the embodiments of the present application is applied;

[0052] Figure 9 One of the scene schematic diagrams to which the coding resource scheduling method provided by the embodiments of the present application is applied;

[0053] Figure 10 One of the scene schematic diagrams to which the coding resource scheduling method provided by the embodiments of the present application is applied;

[0054] Figure 11 One of the scene schematic diagrams to which the coding resource scheduling method provided by the embodiments of the present application is applied;

[0055] Figure 12 One of the scene schematic diagrams to which the coding resource scheduling method provided by the embodiments of the present application is applied;

[0056] Figure 13 The structural schematic diagram of a coding resource scheduling device provided by the embodiments of the present application;

[0057] Figure 14 The structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0059] The term “and / or” in the present document is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol “ / ” in the present document represents the relationship of or, for example, A / B represents A or B.

[0060] The terms “first” and “second” and the like in the specification and claims of the present document are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first message and the second message are used to distinguish different messages, and are not used to describe a specific order of the messages.

[0061] In the embodiments of the present application, the word "exemplary" or "for example" is used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "for example" is intended to present concepts in a concrete manner.

[0062] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0063] In order to facilitate understanding of the embodiments of the present application, some terms of the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0064] Software coding: also known as software decoding, refers to encoding or decoding media data by software (or program) using devices such as central processing units (CPU). The program used by software coding can be called a software codec. For example, a software codec such as conventional x264, x265, etc. can be used to encode or decode media data, which has the advantage of flexibility but the disadvantage of slow speed.

[0065] Hardware coding: also known as hard coding, refers to encoding or decoding media data using hardware such as graphics processing units (GPU), dedicated digital signal processors (DSP), field programmable gate arrays (FPGA), application specific integrated circuits (ASIC) chips, etc.

[0066] The hardware (such as coding chip) and driver program used by hardware coding can be collectively referred to as a hardware codec. For example, a hardware codec can use an application program interface (API) provided by a hardware vendor to encode or decode media data, which has the advantage of fast speed but the disadvantage of lack of flexibility.

[0067] It can be understood that the embodiments of the present application collectively refer to software coding and software decoding responsible by CPU as software coding, and collectively refer to hardware coding and hardware decoding not responsible by CPU as hardware coding.

[0068] Hardware codec resource: also referred to as a hardware codec resource, can refer to the maximum number of macroblocks that a hardware codec can support (or referred to as a macroblock resource or a macroblock capability). Generally, a codec chip will identify the maximum macroblock capability supported when it is shipped. Alternatively, the hardware codec resource can be a macroblock resource, but can also be other possible computing units for hardware codec, which can be determined according to actual use requirements, and the embodiments of the present application are not limited. For ease of illustration, the hardware codec resource is taken as a macroblock resource in the following example.

[0069] It should be noted that in the process of encoding and decoding, the number of remaining macroblocks can reflect the current encoding and decoding capability of the hardware codec. The more the number of remaining macroblocks, the better the encoding and decoding capability of the hardware codec, and correspondingly, the better the display performance (such as clarity or smoothness) of the video data obtained by encoding and decoding.

[0070] It should be noted that the media data that needs to be encoded and decoded can be video data, or audio data, or other possible media data, which can be determined according to actual use requirements, and the embodiments of the present application are not limited. For ease of illustration, the encoding and decoding of video data is taken as an example in the following.

[0071] It should also be noted that the present application is applicable to encoding media data, and is also applicable to decoding media data. For ease of illustration, the decoding of video data is taken as an example in the following.

[0072] The decoding scenario involved in the present application will be described below with the example of playing a video by an electronic device. As shown in Figure 1 When the user triggers the electronic device to play the video, the electronic device first obtains the video file and puts the video file into the memory buffer. These video files are encapsulated format data, including audio stream and video stream. The encapsulated format can be AVI, MOV or MP4 format, and the video file can be understood as a container that contains various encoded data. After obtaining the video file, the electronic device performs encapsulated format processing on the video file, separates the video stream and the audio stream, and obtains audio compression data and video compression data. Further, the electronic device decodes the audio compression data to obtain audio sampling data, and decodes the video compression data to obtain video pixel data. Then the electronic device synchronously processes the audio sampling data and the video pixel data to obtain the video data to be played. Finally, based on the video data to be played, the video picture is displayed on the display and the video sound is output through the loudspeaker.

[0073] The decoding scenario involved in the present application can be Figure 1The decoding stage shown in the middle dashed box, such as audio decoding and video decoding, is described below by taking video decoding as an example. In the video decoding stage, the video application usually directly requests the hardware codec (such as a codec chip) to decode the video compression data. If the hardware codec resource is insufficient, the video decoding cannot be completed, which will cause the video playback to fail or the program to abnormally exit.

[0074] In some common scenarios, when the electronic device simultaneously processes multiple codec tasks, the hardware codec resource may be occupied, resulting in insufficient hardware codec resources, and thus some codec tasks may fail. That is, when the hardware codec macro block resource is insufficient, various video service scenarios will fail, which is a serious problem. The following exemplary five exemplary scenarios are described.

[0075] Scenario 1: When the user triggers the video application on the electronic device to play a video, the video application first performs codec configuration, and after the configuration is successful, the hardware codec can be preferentially performed. If other applications occupy the hardware codec macro block resource at this time, the video playback may not be able to obtain sufficient hardware codec macro block resources for decoding, resulting in failure of the video playback, or even abnormal exit of the video application.

[0076] Scenario 2: When the electronic device is playing a video, if the user triggers the electronic device to perform screen projection, the hardware codec macro block resource may be insufficient when the screen projection is performed due to the video playback service occupying the hardware codec macro block resource, resulting in failure of the screen projection, or even abnormal exit of the screen projection application.

[0077] Scenario 3: When the electronic device is performing screen projection, if the user triggers the electronic device to perform video playback, the video playback may not be able to obtain sufficient hardware codec macro block resources for decoding due to the screen projection service occupying the hardware codec macro block resource, resulting in failure of the video playback, or even abnormal exit of the video application.

[0078] Scenario 4: When the electronic device is playing a video through a video application, the video is decoded when played. If the hardware codec macro block resource is occupied by the background when parsing a video thumbnail (such as a cover), the hardware codec macro block resource may be insufficient when the video is played, resulting in failure of the video playback, or even abnormal exit of the video application.

[0079] Scenario 5: When a user edits a video through an electronic device, if multiple video editing is applied at the same time, multiple hardware codec is instantiated, and the multiple hardware codec occupies the hardware codec macro block resource at the same time (macro block overflow occurs), which may cause insufficient hardware codec macro block resource during video editing, resulting in video editing failure or even program abnormal exit.

[0080] Since the macro block resource or capability is directly managed by the hardware codec (for example, a codec chip), and the current Android system framework does not uniformly manage the macro block resource or capability of the hardware codec, when multiple hardware codecs are started by an application, multiple hardware codec resources are concurrently applied, and the phenomenon that the macro block resource requested by the application is greater than the hardware codec macro block resource (that is, macro block overflow) may easily occur, thereby causing hardware codec failure and various problems.

[0081] In view of this, the embodiment of the present application provides a codec resource scheduling method. When an application video service requests codec processing, the remaining available resource amount of the hardware codec is queried through the codec management service module provided in the present application based on the macro block capability query interface provided in the present application, and then it is judged whether the current remaining hardware codec resource meets the application codec requirement. According to the judgment result, the codec strategy is adjusted: the resolution or code rate is reduced, the applied codec capability is reduced, and it is ensured that the video service can continue to be executed; or the codec mode is changed, and then software codec is applied, and it is ensured that the video service can continue to be executed; or if the service party cannot reduce the resolution or code rate, and cannot use software codec, the video service is normally terminated, and the user is prompted, and it is ensured that the application does not crash.

[0082] The electronic device in the embodiment of the present application can be an electronic device with an operating system. The operating system can be an Android operating system, can be an ios operating system, and can also be other possible operating systems, which are not limited in the embodiment of the present application. The software environment to which the codec resource scheduling method provided in the embodiment of the present application is applied is introduced below taking the Android operating system as an example.

[0083] Figure 2 A possible architecture schematic diagram of the Android operating system provided in the embodiment of the present application is shown in FIG. 1. Figure 2 As shown in FIG. 1, the architecture of the Android operating system includes five layers, which are: an application (application, APP) layer, an APP framework layer, a system runtime library layer, a hardware abstraction layer (hardware abstract layer, HAL), and a kernel layer (which can be a Linux kernel layer).

[0084] The APP layer includes various application programs (including system application programs and third-party application programs) in the Android operating system. For example, in the scheme of the present application, the application program layer can include a camera application, a video application, a screen projection application, and a video editing application, and the like; of course, other application programs that can require scheduling of codec resources for coding and decoding can also be included, and the specific application programs can be determined according to actual use requirements, and the embodiments of the present application are not limited in this regard.

[0085] In some embodiments, when the electronic device records a video through the camera application, in order to reduce the occupation of the video on the storage space, the electronic device compresses the generated video data, that is, performs video coding.

[0086] In some embodiments, when the electronic device performs video projection through the screen projection application, in order to transmit the video data to be projected to the electronic device supporting the projection through the network, the electronic device encodes the video data to be projected.

[0087] In some embodiments, when the electronic device plays a video through the video application, the electronic device first acquires the video data. Since the acquired video data is compressed and encoded video data, the electronic device decodes the already compressed and encoded video data. Then, the video data can be normally played after being decoded.

[0088] In some embodiments, when the electronic device edits a video through the video editing application, the electronic device decodes and encodes the video data.

[0089] In some embodiments, for video content analysis and video content extraction, the electronic device decodes the video data. For example, when an I frame or a frame of a video is acquired as a cover, or when key content in the video data is analyzed, the video data needs to be decoded.

[0090] The APP framework layer is the framework of the application program. Developers can develop some application programs based on the APP framework layer under the condition of complying with the development principles of the framework of the application program.

[0091] For example, in the scheme of the present application, the APP framework layer can include a video recording module MediaRecorder, a media player MediaPlayer, a media codec MediaCodec, and a media codec manager HnMediaCodecManager; of course, other application program frameworks that can be related to coding and decoding can also be included, and the specific application program frameworks can be determined according to actual use requirements, and the embodiments of the present application are not limited in this regard.

[0092] The system runtime library layer includes a library (also referred to as a system library) and an Android operating system runtime environment.

[0093] The Android operating system runtime environment is used to provide a software environment for the Android operating system, and is responsible for scheduling and management of the Android system. The Android operating system runtime environment includes a core library and a virtual machine. The core library includes two parts: one part is a function function that needs to be called by the java language, and the other part is the core library of the Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer into binary files. The virtual machine is used to perform functions such as management of object life cycle, stack management, thread management, security and exception management, and garbage collection.

[0094] The system library can include a plurality of functional modules to provide various resources required by the Android operating system. Exemplarily, the system library can include media libraries. The media libraries support playback and recording of a plurality of commonly used audio, video formats, and static image files. The media libraries can support a plurality of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0095] Specifically, in the scheme of the present application, the system library can include a media codec management service MediaCodecService and a media codec MediaCodec; of course, it can also include other possible functions related to coding and decoding, which can be determined according to actual use requirements, and the embodiments of the present application are not limited.

[0096] The media codec MediaCodec in the system library is responsible for coding and decoding work, and the media codec MediaCodec is composed of an encoder coder (for video encoding) and a decoder decoder (for video decoding). It should be noted that the media codec MediaCodec in the system library is implemented by C++ programming, and the media codec MediaCodec in the APP framework layer is implemented by Java programming. The two media codecs MediaCodec play different roles in the coding and decoding process, and can interact and cooperate to complete the coding and decoding task.

[0097] The media codec management service MediaCodecService can provide services such as information query, information storage, policy selection, feature query, information management, policy management, and capability calculation.

[0098] The hardware abstraction layer is an interface layer between the system library and the hardware circuit, which is used to abstract the hardware and provide a virtual hardware use platform for the operating system. Specifically, in the scheme of the present application, the hardware abstraction layer can include a coding and decoding driver interface.

[0099] The kernel layer is the operating system layer of the Android operating system, and belongs to the lowest layer of the software hierarchy of the Android operating system. The kernel layer provides core system services and hardware-related drivers based on the Linux kernel for the Android operating system, and the drivers are used to trigger the hardware to perform corresponding actions. Specifically, in the scheme of the present application, the kernel layer can include a codec driver.

[0100] As shown in Figure 2 , under the system architecture of the Android operating system is the hardware layer, and the hardware (such as chips, cameras, display screens, etc.) in the hardware layer can perform corresponding actions according to the drivers provided by the kernel layer. Specifically, in the scheme of the present application, the hardware layer can include a codec chip, which can perform codec processing on media data according to the drivers provided by the kernel layer.

[0101] Taking the Android operating system as an example, in the embodiments of the present application, a software program implementing the codec resource scheduling method provided in the embodiments of the present application can be developed based on the system architecture of the Android operating system as shown in Figure 2 , so that the codec resource scheduling method can run based on the Android operating system as shown in Figure 2 . That is, the processor or electronic device can implement the codec resource scheduling method provided in the embodiments of the present application by running the software program in the Android operating system.

[0102] The present application aims to increase the management capability of macro block resources or capabilities, provide an interface for application or business, and determine whether the currently available macro block resources or capabilities of the electronic device meet the application business requirements, and provide reasonable strategies for the application business according to the determination result, so as to ensure the normal operation of the program or prevent the program from crashing. The following will be described in detail. Figure 3 The codec resource scheduling method provided in the embodiments of the present application will be described in detail.

[0103] Figure 3 is an interaction diagram implemented based on the system architecture of Figure 2 provided in the embodiments of the present application.

[0104] As shown in Figure 3As shown, the camera application in the APP layer can request the media codec MediaCodec for coding processing through the video recording module MediaRecorder in the APP framework layer, the video recording module in the system runtime library layer, and the media codec MediaCodec in sequence. The video application in the APP layer can request the media codec MediaCodec for coding processing through the media player MediaPlayer in the APP framework layer, the video recording module in the system runtime library layer, and the media codec MediaCodec in sequence. The screen projection application in the APP layer can request the media codec MediaCodec for coding processing through the media codec MediaCodec in the APP framework layer and the media codec MediaCodec in the system runtime library layer in sequence.

[0105] The application scheme adds the media codec manager HnMediaCodecManager class in the Android APP framework layer, which is used to encapsulate an interface that can be used to query the remaining macro block resources or capabilities. The interface can be called by the application or service in the APP layer.

[0106] As shown in Figure 3 The application scheme also adds the codec management service module for the media codec MediaCodec in the system runtime library layer. The codec management service module can include the media codec manager MediaCodecManager class and the media codec service MediaCodecService class.

[0107] The media codec manager MediaCodecManager class is used to manage the storage, query, and policy selection of the macro block information. The media codec service MediaCodecService class is used for macro block information management, policy management, and macro block capability calculation.

[0108] The codec management service module can interact with the media codec, for example, the codec management service module provides the codec policy information for the media codec, and the media codec sends the occupied codec resource information to the codec management service module and saves the information by the codec management service module.

[0109] The codec management service module can interact with the hardware codec module, for example, the maximum number of macro blocks supported by the hardware codec is obtained from the hardware codec module, and then the current remaining hardware codec resources, i.e. the current remaining hardware codec macro block number, can be determined according to the difference between the maximum number of macro blocks supported by the hardware codec and the currently occupied macro block number.

[0110] Optionally, the MediaCodecManager class and the MediaCodecService class can be independent; or can be integrated as a media codec management service module, which can be determined according to actual use requirements, and the embodiments of the present application are not limited. For ease of illustration, the media codec management service module is exemplarily illustrated in the following.

[0111] It should be noted that the embodiments of the present application add a media codec service in the system framework, which can maintain the codec macro block information, manage the currently running codec capability (code rate, frame rate, resolution information, etc.), and calculate the available codec capability according to the hardware codec capability.

[0112] The media codec service can provide an interface for querying the dynamic codec capability, and the application can apply for the corresponding codec capability according to the current remaining macro block number to avoid direct error reporting.

[0113] The media codec service can query the remaining hardware codec capability and judge whether the dynamic codec capability meets the application codec requirement.

[0114] The following continues to combine Figure 3 The scheduling method of the codec resource provided by the embodiments of the present application is exemplarily illustrated.

[0115] Exemplarily, as Figure 3 shown, the target application (for example, a video editing application) in the APP layer can call the media codec manager HnMeidaCodecManager class in the APP framework layer, and send a request for codec processing to the media codec management service module. After receiving the request, the codec management service module can first query the stored hardware codec resource occupation information, and calculate the current remaining hardware codec resource according to the hardware codec resource occupation information, and then judge whether the current remaining hardware codec resource meets the application codec requirement.

[0116] In some optional embodiments, when the current remaining hardware codec resource meets the application codec requirement, the codec management service module directly calls the codec driver program through the codec driver interface, and performs the codec processing according to the application codec requirement.

[0117] In some optional embodiments, when the current remaining hardware codec resource does not meet the application codec requirement, the embodiments of the present application provide the following several possible implementation modes to ensure the normal operation of the application service or the absence of abnormal exit phenomenon.

[0118] First implementation mode:

[0119] When the current remaining hardware codec resources do not meet the codec requirement of the application, the codec management service module can determine whether to continue to use the current remaining hardware codec resources for codec processing on the premise of reducing the video resolution or frame rate according to a preset policy.

[0120] For example, the codec management service module can directly determine to continue to use the current remaining hardware codec resources for codec processing on the premise of reducing the video resolution or frame rate, and then feed back the result of the hardware codec processing to the target application.

[0121] The second implementation manner:

[0122] When the current remaining hardware codec resources do not meet the codec requirement of the application, the codec management service module can also select not to use hardware codec, but to use software codec to code the service data of the target application, and then feed back the result of the software codec processing to the target application.

[0123] Through the above scheme, when the hardware codec resources are insufficient, software codec can be provided as an alternative scheme according to the current hardware capability and limitation, so that the normal operation of the application service can be guaranteed.

[0124] The third implementation manner:

[0125] When the current remaining hardware codec resources do not meet the codec requirement of the application, the codec management service module can feed back to the target application that the current remaining hardware codec resources are insufficient, and prompt the target application whether to continue to use the current remaining hardware codec resources for codec processing on the premise of reducing the video resolution or frame rate.

[0126] Optionally, the target application can directly determine whether to continue to use the current remaining hardware codec resources for codec processing on the premise of reducing the video resolution or frame rate according to a preset policy.

[0127] Figure 4 An interface schematic diagram of automatic decision of the electronic device in the video playing scene of the embodiment of the application is shown. As shown in FIG. 11, Figure 4 As shown in (a) of FIG. 11, the user can trigger the video to start playing by clicking the play control in the video playing interface 11. It is assumed that the current video playing mode is set to the high definition mode 12, which corresponds to the requirement of more codec resources, that is, the video playing needs to occupy more macroblocks. During the video playing process, if the requested codec resources of the video application are insufficient, that is, the remaining macroblocks cannot meet the requirement of the video playing in the high definition mode, the video application can directly determine to continue to use the current remaining hardware codec resources for codec processing on the premise of reducing the video resolution or frame rate. As shown in (b) of FIG. 11, Figure 4As shown in (b), the electronic device displays a prompt message 13 during video playback: "Automatically switched from HD to smooth mode, please continue playing the video." In this case, the video playback mode switches to smooth mode 14. This prompt message 13 can be canceled after a preset duration.

[0128] Optionally, the target application may also prompt the user to confirm whether to continue using the remaining encoding and decoding resources of the current hardware for encoding and decoding processing while reducing the video resolution or frame rate.

[0129] Figure 5 This illustration shows a user-defined interface in an embodiment of this application where an electronic device plays a video, and the decision is made by the user. Figure 5 As shown in (a), assuming the current video playback mode is set to HD mode 12, which requires more encoding / decoding resources, meaning more macroblocks are needed for video playback. During video playback, if the video application requests insufficient encoding / decoding resources—that is, the remaining macroblocks cannot meet the requirements for HD playback—the video application can display message 15: "HD is not currently supported. Click to switch and continue playing the video; HD (high resolution), Smooth (low resolution)." The user decides whether to continue playing the video while reducing the video resolution or frame rate. When the user selects the Smooth (low resolution) option, the video application can instruct the hardware codec to use the remaining hardware encoding / decoding resources for encoding / decoding processing. Figure 5 As shown in (b), the electronic device can continue playing the video, and at this time the video playback mode is switched to smooth mode 14.

[0130] Specifically, upon receiving confirmation from the user, the target application sends a message to the codec management service module, instructing it to continue using the remaining hardware codec resources for encoding and decoding processing, even at a reduced video resolution or frame rate. Furthermore, the codec management service module can call the codec driver program via the codec driver interface to perform encoding and decoding processing, and then feed back the results of the hardware encoding and decoding processing to the target application.

[0131] Fourth implementation method:

[0132] When the remaining hardware codec resources are insufficient to meet the application's codec requirements, the codec management service module can report to the target application that the remaining hardware codec resources are insufficient and prompt the target application whether it needs to perform codec processing through software codec.

[0133] Optionally, the target application can directly determine whether to continue using the remaining hardware codec resources for encoding and decoding processing while reducing the video resolution or frame rate, based on a preset strategy.

[0134] Optionally, the target application can also prompt the user for confirmation. When receiving the user's confirmation operation, the target application sends information to the codec management service module to indicate that the codec processing is performed by the software codec. Further, the codec management service module instructs the software codec to perform the codec processing, and then feeds back the result of the software codec processing to the target application.

[0135] Fifth implementation mode:

[0136] When the current remaining hardware codec resources do not meet the application codec requirement, the codec management service module can feed back error information to the target application, the error information being used to indicate that the current remaining hardware codec resources are insufficient and cannot meet the application codec requirement, at this time, the target application can display information on the display screen to prompt the user that the current application service is terminated according to the error information.

[0137] Figure 6 The interface schematic diagram for prompting the failure of playing when the electronic device plays a video scene in the embodiment of the application is shown. It is still assumed that the current video playing mode is set to the high-definition mode, which corresponds to the requirement of more codec resources, that is, the video playing requires more macroblocks. If the requested codec resources of the video application are insufficient, that is, the remaining macroblocks cannot meet the requirement of playing the video in the high-definition mode, then as shown in Figure 6 , the video application can display prompt information 16: high-definition is not supported at present, and the video playing fails.

[0138] In this way, by timely reporting the error information, the calling party increases the adaptation logic, and the playing exception can be avoided.

[0139] The execution subject of the codec resource scheduling method provided in the embodiment of the application can be an electronic device, or can be a functional module and / or functional entity in the electronic device which can implement the codec resource scheduling method, and the scheme of the application can be implemented in a hardware and / or software manner, and specifically, it can be determined according to actual use requirements, and the embodiment of the application is not limited.

[0140] The codec resource scheduling method provided in the embodiment of the application is exemplarily described below with reference to the electronic device and in combination with the drawings.

[0141] Figure 7 is a flow schematic diagram of the codec resource scheduling method provided in the first embodiment of the application. As shown in Figure 7 , the method comprises the following steps S101-S111.

[0142] S101, in response to the operation of the user on the first application, the electronic device runs the first video service.

[0143] Optionally, the first video service can be a video playing, a video recording, a screen projection service, or a video editing service, etc. It can be understood that the first video service is exemplarily described herein, and can also be any other possible service that needs to be coded and decoded, and the embodiments of the present application do not limit this.

[0144] Exemplarily, the first video service is a video playing service. For example, the electronic device can run a video playing client, and then play a video requested from a network through the video playing client, or can run a web browser, and then play a video in the web browser, or the electronic device can play a locally stored video file through a video playing software.

[0145] S102, the electronic device codes and decodes the first video service through the hardware codec.

[0146] Optionally, the electronic device can first query the hardware codec resource occupation information through the codec management service module. At this time, since the hardware codec is not occupied, the remaining available codec resources of the hardware codec are the maximum number of macroblocks that can be supported by the hardware codec, which meets the coding and decoding requirements of the first video service. In this case, the hardware codec can be used to code and decode the first video service.

[0147] S103, the electronic device stores the hardware codec resource occupation information and / or the hardware codec resource available information through the codec management service module.

[0148] The hardware codec resource occupation information is used to indicate the number of macroblocks that have been occupied by the hardware codec. It can be understood that the number of macroblocks that have been occupied by the hardware codec includes the number of macroblocks that have been occupied by the first video service.

[0149] The hardware codec resource available information is used to indicate the remaining number of macroblocks of the hardware codec.

[0150] In actual implementation, the hardware codec resource available information is updated when the codec resources are occupied, and is updated when the occupied codec resources are released. Similarly, the hardware codec resource available information is updated when the codec resources are occupied, and is updated when the occupied codec resources are released.

[0151] S104, the electronic device runs a second video service of a second application.

[0152] Optionally, the first application and the second application can be the same application, or can be different applications.

[0153] Optionally, the second video service can be a video playing, a video recording, a screen projection service, or a video editing service, etc.

[0154] Optionally, the electronic device can run the second video service of the second application in response to a user operation on the second application. Alternatively, the electronic device can automatically trigger running of the second video service during running of the first video service.

[0155] In S105, the electronic device determines the remaining available coding resource of the hardware codec according to the hardware coding resource occupation information or the hardware coding resource available information by the coding management service module.

[0156] In actual implementation, the coding management service module can know in advance the maximum number of macroblocks that the hardware codec can support. In this way, the coding management service module can subtract the number of macroblocks that the hardware codec has been occupied from the maximum number of macroblocks that the hardware codec can support, to calculate the remaining available coding resource of the hardware codec, i.e., the remaining available number of macroblocks of the hardware codec.

[0157] Optionally, the coding management service module can determine the remaining available resource amount of the hardware codec according to the hardware coding resource available information.

[0158] Optionally, the coding management service module can obtain the hardware coding total resource information corresponding to the hardware codec. Then, the coding management service module can determine the remaining available resource amount of the hardware codec according to the hardware coding resource occupation information and the hardware coding total resource information. The hardware coding total resource information is used to indicate the maximum resource amount supported by the hardware codec.

[0159] It can be understood that the maximum resource amount supported by the hardware codec is subtracted from the occupied resource of the hardware coding resource, to calculate the remaining available resource of the hardware codec, i.e., the remaining number of macroblocks.

[0160] In S106, the electronic device determines whether the remaining available coding resource of the hardware codec meets the coding requirement of the second video service.

[0161] The coding requirement of the second video service can be determined according to the coding parameter requested by the second video service. Optionally, the coding parameter can include at least one of the code rate, the frame rate, and the resolution.

[0162] Exemplarily, the following equation can be used to calculate the coding macroblock number required by a video file:

[0163] N = (W x H) x F / S

[0164] Wherein, N represents the macroblock number required by the video file, W represents the width pixel number of the video file, H represents the height pixel number of the video file, (W x H) represents the resolution value of the video file, F represents the frame rate value of the video file, and S represents the pixel block size.

[0165] Exemplarily, the resolution value (WxH) of the video file can be 1080x720, the frame rate value F of the video file can be 60fps (frames per second) or 120fps, and the pixel block size S can be 16x16.

[0166] In S106, if the condition is met, S107 described below is continued to be executed; if the condition is not met, S108 or S109 or S110 described below is continued to be executed.

[0167] It can be understood that the remaining available coding resources of the hardware codec are greater than or equal to the coding resources required by the second video service, that is, the coding requirement of the second video service is met. The remaining available coding resources of the hardware codec are less than the coding resources required by the second video service, that is, the coding requirement of the second video service is not met.

[0168] S107, the electronic device encodes and decodes the second video service through the hardware codec according to the coding parameter requested by the second video service.

[0169] Optionally, the coding parameter of the second video service can be adjusted by the target application; or the coding parameter of the second video service is adjusted by the coding management service module.

[0170] Exemplarily, in the case where it is judged that the remaining available coding resources of the hardware codec do not meet the coding requirement of the second video service, the coding management service module sends a first message to the target application, the first message being used to indicate the remaining available resource amount of the hardware codec and the coding resource amount requested by the second video service being greater than the remaining available resource amount of the hardware codec. After receiving the first message, the target application adjusts the coding parameter of the video service, so that the coding resource amount requested by the second video service is less than or equal to the remaining available resource amount of the hardware codec.

[0171] As described above, the coding resource amount requested by the second video service can be determined according to the coding parameter of the second video service.

[0172] Exemplarily, adjusting the coding parameter of the second video service can include reducing at least one of video resolution, code rate and frame rate. The video display performance corresponding to the original coding parameter requested by the second video service is better than the video display performance corresponding to the adjusted coding parameter of the second video service.

[0173] The video display performance can be measured by the definition and smoothness of the video playing. The definition corresponds to the video resolution, and the smoothness corresponds to the code rate and frame rate. The higher the video resolution is, the higher the definition is. The greater the code rate or frame rate value is, the better the video smoothness is.

[0174] When the remaining available codec resources of the hardware codec satisfy the codec requirement of the second video service, the hardware codec can be used to perform the codec processing on the second video service according to the codec parameters requested by the second video service.

[0175] S108, the electronic device performs the codec processing on the second video service through the hardware codec according to the adjusted codec parameters.

[0176] It can be understood that, after the codec parameters are adjusted, the required hardware codec resources are correspondingly reduced, which are less than or equal to the remaining available codec resources of the hardware codec.

[0177] Optionally, the code rate value, the frame rate value and / or the resolution value in the codec parameters requested by the second video service can be reduced, so that the required hardware codec resources after the adjustment of the codec parameters are less than or equal to the remaining available codec resources of the hardware codec.

[0178] It can be understood that, after the codec processing on the second video service is performed through the hardware codec, the result of the codec processing (i.e., the video data) can be sent to the second application, and then the video data can be displayed on the display screen.

[0179] S109, the electronic device performs the codec processing on the second video service through the software codec according to the codec parameters requested by the second video service.

[0180] In the soft decoding process, the CPU is controlled by the operating system to perform data processing. Usually, the request for soft decoding is first sent to the operating system (which can also be understood as the electronic device), and the operating system can respond to the request.

[0181] S110, the electronic device feeds back error information to the second application through the codec management service module, and the error information is used to indicate that the hardware codec resources do not satisfy the codec requirement of the second video service.

[0182] S111, the electronic device displays prompt information according to the error information to prompt the user that the second video service is terminated.

[0183] In this embodiment, when an application requests encoding / decoding processing for video services, it can use the macroblock capability query interface provided in this embodiment to query the stored hardware encoding / decoding resource occupancy information through the encoding / decoding management service module. Based on this information, it calculates the remaining hardware encoding / decoding resources and then determines whether these resources meet the application's encoding / decoding requirements. The encoding / decoding management service module then feeds back the result to the application, allowing the application to obtain the current terminal's macroblock capabilities in real time and adjust its subsequent decoding capability requests accordingly.

[0184] Strategy 1: Reduce the resolution or bitrate, and reduce the required encoding and decoding capabilities to ensure that the business can continue to operate.

[0185] Strategy 2: Change the encoding / decoding method and then apply for software encoding / decoding to ensure that the business can continue to operate.

[0186] Strategy 3: If the business cannot reduce the resolution or bitrate, and cannot use software encoding and decoding, then the business should be terminated normally, and the user should be notified to ensure that the application does not crash.

[0187] The software code involved in the scheduling method of encoding and decoding resources provided in the embodiments of this application is given below as an example.

[0188]

[0189]

[0190]

[0191] The following is combined Figures 8 to 10 This application provides an exemplary method for scheduling encoding and decoding resources.

[0192] Figure 8 Figure (a) shows a flowchart illustrating the request for codec resources corresponding to scenario 1 above. When a video application on an electronic device is playing a video, the video application first obtains codec information from the media codec list, performs codec configuration, and prioritizes hardware codec. If another application that needs to perform hardware codec has already performed codec and occupied the hardware codec macroblock resources, it will cause the video playback to fail or even cause the program to exit abnormally.

[0193] Figure 8 Figure (b) shows a flowchart illustrating the request for encoding / decoding resources when the solution of this application is applied to scenario 1. Figure 8 The difference between (a) and (b) is that Figure 8In the method, the codec management service module is added, when a video service request of an application requires codec processing, the codec management service module first receives the request, then queries the stored hardware codec resource occupation information, calculates the current remaining hardware codec resources according to the hardware codec resource occupation information, and judges whether the current remaining hardware codec resources meet the application codec requirement. On this basis, the codec management service module can select strategy 1: continue to use the current remaining hardware codec resources for codec processing on the premise of reducing the video resolution or frame rate; or strategy 2: change to use the software codec module for codec processing, thereby ensuring normal video playback. In some cases, strategy 3: feedback of insufficient codec resources to the application can be used to trigger service termination, thereby avoiding program abnormal exit.

[0194] From Figure 8 In the method, the codec management service module is added, when a video service request of an application requires codec processing, the codec management service module first receives the request, then queries the stored hardware codec resource occupation information, calculates the current remaining hardware codec resources according to the hardware codec resource occupation information, and judges whether the current remaining hardware codec resources meet the application codec requirement. On this basis, the codec management service module can select strategy 1: continue to use the current remaining hardware codec resources for codec processing on the premise of reducing the video resolution or frame rate; or strategy 2: change to use the software codec module for codec processing, thereby ensuring normal video playback. In some cases, strategy 3: feedback of insufficient codec resources to the application can be used to trigger service termination, thereby avoiding program abnormal exit.

[0195] Figure 9 In the method, the codec management service module is added, when a video service request of an application requires codec processing, the codec management service module first receives the request, then queries the stored hardware codec resource occupation information, calculates the current remaining hardware codec resources according to the hardware codec resource occupation information, and judges whether the current remaining hardware codec resources meet the application codec requirement. On this basis, the codec management service module can select strategy 1: continue to use the current remaining hardware codec resources for codec processing on the premise of reducing the video resolution or frame rate; or strategy 2: change to use the software codec module for codec processing, thereby ensuring normal video playback. In some cases, strategy 3: feedback of insufficient codec resources to the application can be used to trigger service termination, thereby avoiding program abnormal exit.

[0196] Figure 9 In the method, the codec management service module is added, when a video service request of an application requires codec processing, the codec management service module first receives the request, then queries the stored hardware codec resource occupation information, calculates the current remaining hardware codec resources according to the hardware codec resource occupation information, and judges whether the current remaining hardware codec resources meet the application codec requirement. On this basis, the codec management service module can select strategy 1: continue to use the current remaining hardware codec resources for codec processing on the premise of reducing the video resolution or frame rate; or strategy 2: change to use the software codec module for codec processing, thereby ensuring normal video playback. In some cases, strategy 3: feedback of insufficient codec resources to the application can be used to trigger service termination, thereby avoiding program abnormal exit. Figure 9 Different from Figure 9The middle (b) increases the codec management service module, when the application screen projection service request is processed by coding and decoding, the codec management service module will first receive the request, then query the stored hardware codec resource occupation information or hardware codec resource available information, and determine the current remaining hardware codec resource according to the hardware codec resource occupation information or available information, and then judge whether the current remaining hardware codec resource meets the application codec demand. On this basis, the codec management service module can select strategy 1 according to the actual use demand: continue to use the current remaining hardware codec resource for coding and decoding processing under the premise of reducing the video resolution or frame rate; or strategy 2: change to process coding and decoding through the software codec module, thereby ensuring the success of screen projection. In some cases, strategy 3 can also be used: feedback the lack of codec resources to the application, trigger the service termination, thereby avoiding the abnormal exit of the program.

[0197] From Figure 9 In the middle (b), the screen projection application can obtain the available coding and decoding capability in time to adjust the screen projection resolution, or the codec management service module can prompt the screen projection application in time that the coding and decoding resource is insufficient.

[0198] Figure 10 The middle (a) shows the flow diagram of requesting coding and decoding resource corresponding to the above-mentioned scenario 3. When the electronic device is projecting, if the user triggers the electronic device to play the video, at this time, since the screen projection service has occupied the hardware codec macro block resource, it may cause that the sufficient hardware codec macro block resource cannot be obtained for decoding when playing the video, resulting in video playing failure, and even video application program abnormal exit.

[0199] Figure 10 The middle (b) shows the flow diagram of requesting coding and decoding resource corresponding to the application of the application scheme in scenario 3. Compared with Figure 10 Different from the middle (a), the middle (b) adds a codec management service module, when the application screen projection service request is processed by coding and decoding, the codec management service module will first receive the request, then query the stored hardware codec resource occupation information or hardware codec resource available information, and determine the current remaining hardware codec resource according to the hardware codec resource occupation information or available information, and then judge whether the current remaining hardware codec resource meets the application codec demand. On this basis, the codec management service module can select strategy 1 according to the actual use demand: continue to use the current remaining hardware codec resource for coding and decoding processing under the premise of reducing the video resolution or frame rate; or strategy 2: change to process coding and decoding through the software codec module, thereby ensuring the success of screen projection. In some cases, strategy 3 can also be used: feedback the lack of codec resources to the application, trigger the service termination, thereby avoiding the abnormal exit of the program. Figure 10The middle (b) increases the codec management service module, when the video playing service request of the application is requested for codec processing, the codec management service module will first receive the request, then query the stored hardware codec resource occupation information or hardware codec resource available information, and determine the current remaining hardware codec resource according to the hardware codec resource occupation information or available information, and then judge whether the current remaining hardware codec resource meets the application codec demand. On this basis, the codec management service module can select strategy 1 according to the actual use demand: continue to use the current remaining hardware codec resource for codec processing on the premise of reducing the video resolution or frame rate; or strategy 2: change to software codec module for codec processing, thereby ensuring normal video playing. In some cases, strategy 3 can also be used: feedback the insufficient codec resource to the application, trigger service termination, thereby avoiding program abnormal exit.

[0200] From Figure 10 The middle (b) can be known that, in the process of screen projection, the three-party application plays video, and the system can adaptively select a codec according to the codec resource condition. If the hardware codec capability is insufficient, a software decoder can be selected for playing.

[0201] Figure 11 The middle (a) shows a flow diagram of requesting codec resource corresponding to the above-mentioned scenario 4. When the electronic device is playing video through a video application, decoding is performed when playing video, if at this time the background occupies the hardware codec macro block resource due to parsing video thumbnail (for example, obtaining the I frame or a frame of the video as the cover), then it may cause insufficient hardware codec macro block resource when playing video, resulting in failure of the video playing, and even abnormal exit of the video application program.

[0202] Figure 11 The middle (b) shows a flow diagram of requesting codec resource corresponding to the application of the present application to scenario 4. Unlike Figure 11 The middle (a) is different from Figure 11In the method (b), the codec management service module is added, when the video playing service request of the application requires codec processing, the codec management service module first receives the request, then queries the stored hardware codec resource occupation information or hardware codec resource available information, and determines the current remaining hardware codec resource according to the hardware codec resource occupation information or available information, and then judges whether the current remaining hardware codec resource meets the application codec requirement. On this basis, the codec management service module can select strategy 1: continue to use the current remaining hardware codec resource for codec processing on the premise of reducing the video resolution or frame rate; or strategy 2: change to use the software codec module for codec processing, thereby ensuring normal video playing. In some cases, strategy 3: feedback of insufficient codec resource to the application can be used to trigger service termination, thereby avoiding program abnormal exit.

[0203] From Figure 11 In the method (b), the codec management service module is added, when the video playing service request of the application requires codec processing, the codec management service module first receives the request, then queries the stored hardware codec resource occupation information or hardware codec resource available information, and determines the current remaining hardware codec resource according to the hardware codec resource occupation information or available information, and then judges whether the current remaining hardware codec resource meets the application codec requirement. On this basis, the codec management service module can select strategy 1: continue to use the current remaining hardware codec resource for codec processing on the premise of reducing the video resolution or frame rate; or strategy 2: change to use the software codec module for codec processing, thereby ensuring normal video playing. In some cases, strategy 3: feedback of insufficient codec resource to the application can be used to trigger service termination, thereby avoiding program abnormal exit.

[0204] Figure 12 In the method (a), the flowchart of the request of the codec resource corresponding to the scenario 5 is shown. When the user edits the video through the electronic device, if multiple video editing is applied at the same time, multiple hardware codecs are instantiated, at this time, the multiple hardware codecs will simultaneously occupy the hardware codec macro block resource (macro block overflow occurs), which may cause insufficient hardware codec macro block resource during video editing, resulting in video editing failure, and even program abnormal exit.

[0205] Figure 12 In the method (b), the flowchart of the request of the codec resource corresponding to the scenario 5 is shown. Unlike Figure 12 In the method (a), the flowchart of the request of the codec resource corresponding to the scenario 5 is shown. When the user edits the video through the electronic device, if multiple video editing is applied at the same time, multiple hardware codecs are instantiated, at this time, the multiple hardware codecs will simultaneously occupy the hardware codec macro block resource (macro block overflow occurs), which may cause insufficient hardware codec macro block resource during video editing, resulting in video editing failure, and even program abnormal exit. Figure 12In the method (b), the codec management service module is added, when the video editing service request of the application requires codec processing, the codec management service module first receives the request, then queries the stored hardware codec resource occupation information or hardware codec resource available information, and determines the current remaining hardware codec resource according to the hardware codec resource occupation information or available information, and then judges whether the current remaining hardware codec resource meets the application codec requirement. On this basis, the codec management service module can select strategy 1: continue to use the current remaining hardware codec resource for codec processing on the premise of reducing the video resolution or frame rate; or strategy 2: change to use the software codec module for codec processing, thereby ensuring normal video editing. In some cases, strategy 3: feedback of insufficient codec resources to the application can be used to trigger service termination, thereby avoiding program abnormal exit.

[0206] From Figure 12 As can be seen from the method (b), through the scheme of the present application, the video editing uses the codec concurrency capability at the same time, at this time, the available codec capability can be obtained, so as to facilitate the use of the application itself strategy, for example, using the software codec to realize the editing function.

[0207] It should be noted that the scheme of the present application is universal and can be applied in multiple scenarios involving similar super macro blocks. The super macro block refers to a scenario or phenomenon in which the macro block resources requested by the application exceed the hardware codec macro block resources.

[0208] It should be noted that the scheme of the present application is not only applicable to the decoding process in the scenarios of video playing, video projection, video editing, etc., but also applicable to the encoding process in the scenarios of video editing, video projection, etc.

[0209] It should be further noted that in the embodiments of the present application, "greater than" can be replaced by "greater than or equal to", "less than or equal to" can be replaced by "less than", or "greater than or equal to" can be replaced by "greater than", and "less than" can be replaced by "less than or equal to".

[0210] Each of the embodiments described in the present application can be an independent scheme, or can be combined according to the inherent logic, and these schemes all fall within the protection scope of the present application.

[0211] It can be understood that the methods and operations realized by the electronic device in each of the above method embodiments can also be realized by components (such as chips or circuits) that can be used in the electronic device.

[0212] The method embodiments provided by the present application are described above, and the device embodiments provided by the present application will be described below. It should be understood that the description of the device embodiments corresponds to the description of the method embodiments, and therefore, the content not described in detail can be referred to the method embodiments described above, which will not be repeated here for brevity.

[0213] The solutions provided by the embodiments of the present application are mainly described above from the perspective of method steps. It can be understood that, in order to implement the above functions, the electronic device implementing the method comprises the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the protection scope of the present application.

[0214] The embodiments of the present application can divide the functional modules of the electronic device according to the above method examples, for example, each functional module can be divided corresponding to each function, or two or more functions can be integrated in one processing module. The above integrated module can be implemented in the form of hardware or software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical functional division. When actually implemented, there can be other feasible division manners. The following will be described taking the division of each functional module corresponding to each function as an example.

[0215] Figure 13 A schematic block diagram of the coding resource scheduling device 800 provided by the embodiments of the present application is shown. The device 800 can be used to perform the actions performed by the electronic device in the above method embodiments. The device 800 comprises a coding processing unit 810, a hardware codec, and a software codec.

[0216] The coding processing unit 810 is configured to, when receiving a coding request of a video service of a target application, determine a remaining available resource amount of a hardware codec; if a coding resource amount requested by the video service is greater than the remaining available resource amount of the hardware codec, adjust a coding parameter of the video service, and perform coding processing on the video service by the hardware codec according to the adjusted coding parameter of the video service; or perform coding processing on the video service by a software codec according to an original coding parameter requested by the video service. The video display performance corresponding to the original coding parameter requested by the video service is superior to the video display performance corresponding to the adjusted coding parameter of the video service.

[0217] Through the scheme of the present application, when the video service request is applied for codec processing, the remaining available resource amount of the hardware codec is queried through the codec management service module provided by the present application based on the macro block capability query interface provided by the present application, and then it is judged whether the current remaining hardware codec resource meets the application codec demand. According to the judgment result, the codec strategy is adjusted: the resolution or code rate is reduced, the application codec capability is reduced, and it is ensured that the video service can continue to execute; or the codec mode is changed, and then software codec is applied, and it is ensured that the video service can continue to execute; or if the service party cannot reduce the resolution or code rate, and cannot use software codec, the video service is normally terminated, and the user is prompted, and it is ensured that the application does not crash.

[0218] The apparatus 800 according to the embodiments of the present application can correspond to performing the method described in the embodiments of the present application, and the above and other operations and / or functions of the units in the apparatus 800 are respectively for realizing the corresponding flows of the method, and for brevity, will not be repeated here.

[0219] Figure 14 is a structural schematic diagram of an electronic device 900 provided by the embodiments of the present application. The electronic device 900 can include a processor 910, an external memory interface 920, an internal memory 921, a universal serial bus (USB) interface 930, a charge management module 940, a power management unit 941, a battery 942, an antenna 1, an antenna 2, a mobile communication module 950, a wireless communication module 960, an audio module 970, a loudspeaker 970A, a receiver 970B, a microphone 970C, a headset interface 970D, a sensor module 980, a key 990, a motor 991, an indicator 992, a camera 993, a display screen 994, and a subscriber identification module (SIM) card interface 995, etc.

[0220] Among them, the sensor module 980 can include a pressure sensor 980A, a gyroscope sensor 980B, an air pressure sensor 980C, a magnetic sensor 980D, an acceleration sensor 980E, a distance sensor 980F, a proximity light sensor 980G, a fingerprint sensor 980H, a temperature sensor 980I, a touch sensor 980J, an ambient light sensor 980K, and a bone conduction sensor 980L, etc.

[0221] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 900. In some other embodiments of the present application, the electronic device 900 can include more or fewer components than those shown, or combine some components, or split some components, or different arrangement of components. The components shown can be implemented in hardware, software or a combination of software and hardware.

[0222] The processor 910 can include one or more processing units, for example: the processor 910 can include an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a controller, a memory, a video codec, a DSP, a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors. Among them, the controller can be the nerve center and command center of the electronic device 900. The controller can generate operation control signals according to instruction operation codes and timing signals, complete the control of fetching instructions and executing instructions.

[0223] The processor 910 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 910 is a cache memory. The memory can save instructions or data that the processor 910 has just used or repeatedly uses. If the processor 910 needs to use the instructions or data again, it can be directly called from the memory. Avoiding repeated access, reducing the waiting time of the processor 910, thus improving the efficiency of the system.

[0224] In some embodiments, the processor 910 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative, and does not constitute a structural limitation on the electronic device 900. In other embodiments of the present application, the electronic device 900 can also use different interface connection methods or a combination of multiple interface connection methods in the above embodiments.

[0225] The charging management module 940 is configured to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 940 can receive charging input from a wired charger through the USB interface 930. In some wireless charging embodiments, the charging management module 940 can receive wireless charging input through a wireless charging coil of the electronic device 900. The charging management module 940 can charge the battery 942 and also supply power to the electronic device through the power management unit 941.

[0226] The power management unit 941 is configured to connect the battery 942, the charging management module 940, and the processor 910. The power management unit 941 receives input from the battery 942 and / or the charging management module 940 to supply power to the processor 910, the internal memory 921, the external memory, the display screen 994, the camera 993, and the wireless communication module 960, etc. The power management unit 941 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc. In other embodiments, the power management unit 941 can also be arranged in the processor 910. In other embodiments, the power management unit 941 and the charging management module 940 can also be arranged in the same device.

[0227] The wireless communication function of the electronic device 900 can be implemented through the antenna 1, the antenna 2, the mobile communication module 950, the wireless communication module 960, the modem processor, and the baseband processor, etc.

[0228] The antenna 1 and the antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 900 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0229] The mobile communication module 950 can provide a solution including 2G / 3G / 4G / 5G wireless communication applied to the electronic device 900. The mobile communication module 950 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 950 can receive electromagnetic waves by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 950 can also amplify the signals modulated by the modem processor, and convert the signals into electromagnetic waves radiated by the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 950 can be arranged in the processor 910. In some embodiments, at least part of the functional modules of the mobile communication module 950 and at least part of the modules of the processor 910 can be arranged in the same device.

[0230] The modem processor can include a modulator and a demodulator. The modulator is used to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 970A, the microphone 970B, etc.), or displays an image or a video through the display screen 994. In some embodiments, the modem processor can be an independent device. In some other embodiments, the modem processor can be independent of the processor 910, and arranged in the same device as the mobile communication module 950 or other functional modules.

[0231] The wireless communication module 960 can provide a solution for wireless communication including WLAN (such as Wi-Fi), BT, global navigation satellite system (GNSS), FM, NFC, IR, or general 2.4G / 5G wireless communication technology, etc. applied on the electronic device 900. The wireless communication module 960 can be one or more devices integrating at least one communication processing module. The wireless communication module 960 receives electromagnetic waves via the antenna 2, frequency-modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 910. The wireless communication module 960 can also receive signals to be transmitted from the processor 910, frequency-modulate and amplify them, and radiate them as electromagnetic waves via the antenna 2.

[0232] In some embodiments, the wireless communication module 960 can be a Wi-Fi and / or Bluetooth chip. The electronic device 900 can establish a connection between the chip and the chip of another electronic device such as a wireless earphone through the chip to realize wireless communication and service processing between the electronic device 900 and the other electronic device through the connection. The Bluetooth chip can generally support BR / EDR Bluetooth and BLE.

[0233] In some embodiments, antenna 1 and mobile communication module 950 of electronic device 900 are coupled, and antenna 2 and wireless communication module 960 are coupled, so that electronic device 900 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidu navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation systems (SBAS).

[0234] Electronic device 900 implements a display function through a GPU, display screen 994, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 994 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 910 can include one or more GPUs that execute program instructions to generate or change display information.

[0235] The display screen 994 is configured to display images, videos, and the like. The display screen 994 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the electronic device 900 can include one or N display screens 994, where N is a positive integer greater than 1.

[0236] The electronic device 900 can implement a photographing function through an ISP, the camera 993, a video codec, a GPU, the display screen 994, and an application processor, and the like.

[0237] The ISP is configured to process data fed back by the camera 993. For example, when taking a photo, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be disposed in the camera 993.

[0238] The camera 993 is configured to capture still images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the electronic device 900 can include one or N cameras 993, where N is a positive integer greater than 1.

[0239] The digital signal processor is used to process digital signals, in addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device 900 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0240] The video codec is used to compress or decompress digital video. The electronic device 900 can support one or more video codecs. In this way, the electronic device 900 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0241] The NPU is a neural-network (NN) calculation processor, which can quickly process input information by drawing on the structure of a biological neural network, such as drawing on the transmission mode between human brain neurons, and can also constantly self-learn. Through the NPU, the electronic device 900 can realize intelligent cognition applications such as image recognition, face recognition, voice recognition, text understanding, etc.

[0242] The external memory interface 920 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the electronic device 900. The external storage card communicates with the processor 910 through the external memory interface 920 to realize data storage functions. For example, files such as music and videos are saved in the external storage card.

[0243] The internal memory 921 can be used to store computer executable program codes, which include instructions. The processor 910 executes various function applications and data processing of the electronic device 900 by running the instructions stored in the internal memory 921. The internal memory 921 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created during the use of the electronic device 900 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 921 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0244] The processor 910 can be used to execute the above program codes, call related modules to realize the functions of the electronic device in the embodiments of the present application. For example, multiple communication links are established with another electronic device; when there is a preset service (such as a file transmission service, etc.), data of the preset service is transmitted with another electronic device through multiple communication links.

[0245] The electronic device 900 can implement audio functions through a speaker 970A, a receiver 970B, a microphone 970C, an earphone interface 970D, and an application processor, etc. in the audio module 970. For example, music playback, voice recording, etc.

[0246] The audio module 970 is configured to convert digital audio information into an analog audio signal output, and to convert an analog audio input into a digital audio signal. The audio module 970 can also be configured to encode and decode audio signals. In some embodiments, the audio module 970 can be disposed in the processor 910, or some functional modules of the audio module 970 can be disposed in the processor 910.

[0247] The speaker 970A, also referred to as a "loudspeaker", is configured to convert an audio electrical signal into a sound signal. The electronic device 900 can listen to music or listen to a hands-free call through the speaker 970A.

[0248] The receiver 970B, also referred to as a "earpiece", is configured to convert an audio electrical signal into a sound signal. When the electronic device 900 receives a call or a voice message, the receiver 970B can be held close to a person's ear to listen to the voice through the receiver 970B.

[0249] The microphone 970C, also referred to as a "microphone", is configured to convert a sound signal into an electrical signal. When making a call or sending a voice message, a user can speak into the microphone 970C close to the person's mouth to input a sound signal into the microphone 970C. The electronic device 900 can be provided with at least one microphone 970C. In other embodiments, the electronic device 900 can be provided with two microphones 970C, in addition to collecting sound signals, noise reduction functions can also be realized. In other embodiments, the electronic device 900 can also be provided with three, four or more microphones 970C, in addition to collecting sound signals, noise reduction, and can also identify the source of the sound, realize the function of directional recording, etc.

[0250] The earphone interface 970D is configured to connect a wired earphone. The earphone interface 970D can be a USB interface 930, or a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0251] Exemplarily, the electronic device 900 can further include one or more of a key 990, a motor 991, an indicator 992, a SIM card interface 995 (or an eSIM card), etc.

[0252] The electronic device 900 can be a mobile terminal or a non-mobile terminal. For example, the electronic device 900 can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle terminal, wearable device, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (PDA), wireless headset, wireless bracelet, wireless smart glasses, wireless watch, augmented reality (AR) / virtual reality (VR) device, desktop computer, smart home appliance (e.g., television, speaker, refrigerator, air purifier, air conditioner, rice cooker), etc. The electronic device 900 can also be collectively referred to as an Internet of Things (IoT) device. This application embodiment does not specifically limit the device type of the electronic device 900.

[0253] It should be understood that Figure 14 The electronic device 900 shown can correspond to Figure 13 The device 800 shown. Wherein, Figure 14 The processor 910 in the electronic device 900 shown can correspond to Figure 13 The encoding / decoding processing unit 810 in the device 800.

[0254] In actual implementation, when the electronic device 900 is running, the processor 910 executes computer execution instructions in the memory 921 to perform the operation steps of the above method through the electronic device 900.

[0255] Optionally, in some embodiments, this application provides a chip coupled to a memory, the chip being used to read and execute computer programs or instructions stored in the memory to perform the methods described in the above embodiments.

[0256] Optionally, in some embodiments, this application provides an electronic device including a chip for reading and executing computer programs or instructions stored in a memory, such that the methods in the various embodiments are performed.

[0257] Optionally, in some embodiments, this application also provides a computer-readable storage medium storing program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0258] Optionally, in some embodiments, the embodiments of the present application further provide a computer program product, which comprises computer program codes, and when the computer program codes are run on a computer, the computer is caused to execute the method in each of the above embodiments.

[0259] In the embodiments of the present application, the electronic device comprises a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer can comprise hardware such as a CPU, a memory management unit (MMU), and a memory (also referred to as a main memory). The operating system of the operating system layer can be any one or more computer operating systems that implement business processing through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a windows operating system. The application layer can comprise a browser, an address book, word processing software, instant messaging software, and the like.

[0260] The embodiments of the present application do not particularly limit the specific structure of the execution subject of the method provided by the embodiments of the present application, as long as the execution subject can communicate according to the method provided by the embodiments of the present application by running a program in which the code of the method provided by the embodiments of the present application is recorded. For example, the execution subject of the method provided by the embodiments of the present application can be an electronic device, or a functional module in the electronic device that can invoke and execute a program.

[0261] Various aspects or features of the present application can be implemented as methods, apparatus, or articles of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" as used herein can encompass a computer program which can be accessed from any computer readable device, carrier, or media. For example, computer readable media can include but are not limited to: magnetic storage devices (e.g., hard disk; floppy disk; magnetic strips in identification cards); optical disks (e.g., compact disk (CD), digital versatile disk (DVD)); smart cards; and flash memory devices (e.g., EPROM; card; stick; or key drive). Additionally, various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" can include, without being limited to, wireless channels and various other media capable of storing, containing, and / or carrying instruction and / or data.

[0262] Various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" can include, without being limited to, wireless channels and various other media capable of storing, containing, and / or carrying instruction and / or data.

[0263] It is to be understood that the processor mentioned in the embodiments of the present application can be a CPU, and can also be other general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0264] It should also be understood that the memory mentioned in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM). For example, the RAM can be used as an external cache. As an example but not limitation, the RAM can include the following various forms: static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink DRAM (SLDRAM) and direct memory bus random access memory (direct rambus RAM, DR RAM).

[0265] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated in the processor.

[0266] It should also be noted that the memory described herein is intended to include, but not limited to, these and any other suitable types of memory.

[0267] Those skilled in the art can clearly understand that the units and steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0268] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0269] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0270] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0271] In addition, each functional unit in each embodiment of the present application can be integrated into one unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0272] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, or the parts that essentially contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a computer software product stored in a storage medium, and the computer software product includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium can include, but is not limited to, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various media that can store program codes.

[0273] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0274] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for scheduling of a coded resource, applied to an electronic device, wherein a hardware codec is arranged in the electronic device, and the method comprises the steps of: The electronic device is provided with an HnMediaCodecManager class at an application framework layer, which is used to encapsulate a macroblock capability query interface for querying dynamic coding and decoding capability; the electronic device is provided with a MediaCodecManager class and a MediaCodecService class at a system runtime library layer, the MediaCodecManager class supports interaction with a media codec MediaCodec, acquires and stores hardware coding and decoding resource occupation information; the method comprises: ​ receiving a coding and decoding request of a first service; calling the macroblock capability query interface to query a remaining available resource amount of the hardware coding and decoding device; the calling of the macroblock capability query interface to query the remaining available resource amount of the hardware coding and decoding device comprises: calling the macroblock capability query interface encapsulated by the HnMediaCodecManager class, and transmitting the coding and decoding request of the first service to the MediaCodecManager class; the MediaCodecManager class queries the stored hardware coding and decoding resource occupation information; the MediaCodecService class interacts with a hardware coding and decoding module to acquire hardware coding and decoding total resource information; the MediaCodecService class calculates the remaining available resource amount of the hardware coding and decoding device according to the hardware coding and decoding total resource information and the hardware coding and decoding resource occupation information; when the coding and decoding resource amount requested by the first service is greater than the remaining available resource amount of the hardware coding and decoding device, acquiring a target coding and decoding strategy, the target coding and decoding strategy being used to instruct to reduce the coding and decoding demand of the first service or to perform software coding and decoding on the first service; performing coding and decoding processing on the first service according to the target coding and decoding strategy; the acquiring of the target coding and decoding strategy comprises: determining whether to continue to perform coding and decoding processing by using the current remaining hardware coding and decoding resource on the premise of reducing video resolution or frame rate; if it is determined that the hardware coding and decoding capability is insufficient, then determining to perform software coding and decoding on the first service data of the target application; in the case that resolution or code rate cannot be reduced and software coding and decoding cannot be used, determining to trigger termination of the first service.

2. The method of claim 1, wherein, the target coding and decoding strategy comprises: adjusting a coding and decoding parameter of the first service, so that the coding and decoding resource amount requested by the first service is less than or equal to the remaining available resource amount of the hardware coding and decoding device, the coding and decoding resource amount requested by the first service being determined according to the coding and decoding parameter of the first service; performing coding and decoding processing on the first service by using the hardware coding and decoding device according to the adjusted coding and decoding parameter of the first service; wherein the coding and decoding parameter of the first service comprises video resolution, code rate and / or frame rate.

3. The method according to claim 1 or 2, characterized in that, the target coding and decoding strategy comprises: performing coding and decoding processing on the first service by using a software coding and decoding device according to the original coding and decoding parameter requested by the first service.

4. The method of claim 1, wherein, the method further comprises: store hardware codec resource available information, the hardware codec resource available information being used to indicate a remaining available resource amount of the hardware codec, the hardware codec resource available information being updated when a codec resource is occupied and being updated when an occupied codec resource is released; wherein the querying the remaining available resource amount of the hardware codec comprises: querying the remaining available resource amount of the hardware codec according to the hardware codec resource available information.

5. The method of claim 1, wherein, The method further comprises: store hardware codec resource occupation information, the hardware codec resource occupation information being used to indicate an occupied resource amount of the hardware codec, the hardware codec resource occupation information being updated when a codec resource is occupied and being updated when an occupied codec resource is released; wherein the querying the remaining available resource amount of the hardware codec comprises: querying the remaining available resource amount of the hardware codec according to the hardware codec resource occupation information and the hardware codec total resource information; wherein the hardware codec total resource information is used to indicate a maximum resource amount supported by the hardware codec.

6. The method according to claim 4 or 5, characterized in that, The electronic device is further provided with a codec management service module; the codec management service module comprises the MediaCodecManager class and the MediaCodecService class; wherein the codec management service module is used to store hardware codec resource available information and / or hardware codec resource occupation information.

7. The method of claim 6, wherein, The querying the remaining available resource amount of the hardware codec comprises: The codec management service module queries the remaining available resource amount of the hardware codec according to the stored hardware codec resource available information; or, The codec management service module queries the remaining available resource amount of the hardware codec according to hardware codec total resource information and the stored hardware codec resource occupation information.

8. The method of claim 7, wherein, The method further comprises: The codec management service module acquires hardware codec total resource information corresponding to the hardware codec.

9. The method of claim 2, wherein, The electronic device is installed with the target application, and the first service is a service initiated by the target application; The adjusting the codec parameter of the first service comprises: adjusting the codec parameter of the first service through the target application; or, adjusting the codec parameter of the first service through the codec management service module.

10. The method of claim 9, wherein, Before the above adjusting the codec parameter of the first service through the target application, the method further comprises: The codec management service module sends a first message to the target application, the first message being used to indicate the remaining available resource amount of the hardware codec and to indicate that the codec resource amount requested by the first service is greater than the remaining available resource amount of the hardware codec; wherein the adjusting the codec parameter of the first service through the target application comprises: The target application adjusts the codec parameter of the first service after receiving the first message.

11. The method of claim 6, wherein, The method further comprises: The target application receives a user's opening operation on the first service; In response to the starting operation, the target application sends a codec request of the first service to the codec management service module; The codec request includes the codec parameters of the first service request.

12. The method of claim 11, wherein, After the target application sends the codec request of the first service to the codec management service module, the method further includes: The codec management service module receives the codec request of the first service; According to the codec parameters of the first service request, the amount of codec resources of the first service request is calculated.

13. The method of claim 6, wherein, The method further includes: After the first service is completed, the hardware codec releases the hardware codec resources occupied by the first service; The hardware codec sends a second message to the codec management service module, the second message being used to indicate that the hardware codec resources occupied by the first service have been released; According to the second message, the codec management service module updates the stored available information or occupied information of the hardware codec resources.

14. The method of claim 1, wherein, After the available amount of the hardware codec resources is queried, the method further includes: If the amount of codec resources of the first service request is less than or equal to the available amount of the hardware codec resources, the first service is processed by the hardware codec according to the codec parameters of the first service request.

15. The method of claim 1 or 2, wherein, After the first service is processed, the method further includes: The target application obtains the decoded data or encoded data obtained by processing the first service; The target application displays the decoded data or stores the encoded data.

16. The method of claim 1, wherein, The method further includes: If neither the hardware codec nor the software codec supports the processing of the first service, the first service is terminated normally; A prompt information is displayed on the interface of the target application, the prompt information being used to prompt that the first service is terminated due to insufficient codec resources.

17. The method of claim 16, wherein, The first service is a video playing, video recording or screen projection service.

18. The method of claim 1 or 2, wherein, The codec resources of the hardware codec are macro block resources, and the available amount of the hardware codec resources is the number of remaining macro blocks.

19. An electronic device, comprising: The processor is coupled with the memory, and is used to execute the computer program or instruction stored in the memory, so that the electronic device implements the method in any one of claims 1 to 18.

20. A chip system, characterized by The chip system is coupled with the memory, and is used to read and execute the computer program stored in the memory, so as to implement the method in any one of claims 1 to 18.

21. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, when the computer program runs on an electronic device, the electronic device executes the method in any one of claims 1 to 18.

22. A computer program product, characterised in that, The computer program product includes a computer program, when the computer program runs on a computer, the computer executes the method in any one of claims 1 to 18.

Images