Video coding method and device

Abstract

The invention discloses a video coding method and device. The method comprises the following steps: obtaining a target coding parameter of a video application; determining whether the encoding capability of the default encoder meets the target encoding parameter; the default encoder is a hardware encoder or a software encoder; under the condition that the coding capability of the default coder does not meet the target coding parameter, the hardware coder and the software coder are used for coding an image set which does not meet the target condition and a plurality of sub-image sets obtained by splitting the image set meeting the target condition, and the image set comprises at least two continuous video frames provided by the video application.

Description

Technical Field

[0001] The present application relates to the field of video technology, and in particular to a video encoding method and device. Background Art

[0002] Electronic devices often need to use an encoder (MediaCodec) to perform video encoding processing. Generally, electronic devices can use a software encoder or a hardware encoder for encoding.

[0003] When encoding using a software encoder, if the encoded video has a high image quality, frame rate, and resolution, the software encoder will cause an excessive load on the central processing unit (CPU) of the electronic device, affecting other functions of the electronic device.

[0004] When encoding using a hardware encoder, different hardware encoders are generally adapted to different video parameters. When the video parameters of the encoded video are not compatible with the hardware encoder, the encoding performance of the hardware encoder is poor and is difficult to meet the requirements. Summary of the invention

[0005] To this end, this application discloses the following technical solutions:

[0006] The first aspect of the present application provides a video encoding method, comprising:

[0007] Get target encoding parameters for video applications;

[0008] Determining whether the encoding capability of a default encoder meets the target encoding parameter; the default encoder is a hardware encoder or a software encoder;

[0009] When the encoding capability of the default encoder does not meet the target encoding parameters, the hardware encoder and the software encoder are used to encode an image set that does not meet the target conditions and multiple sub-image sets split from the image set that meets the target conditions, wherein the image set includes at least two consecutive video frames provided by the video application.

[0010] Optionally, also include:

[0011] In a case where the video application does not belong to a video editing application, encoding the image set provided by the video application using the default encoder;

[0012] The determining whether the encoding capability of the default encoder meets the target encoding parameter includes:

[0013] In the case that the video application belongs to a video editing application, it is determined whether the encoding capability of the default encoder meets the target encoding parameters.

[0014] Optionally, a method of obtaining the image set includes:

[0015] Obtain each video frame provided by the video application one by one;

[0016] Whenever N video frames are obtained continuously or it is determined that the obtained video frame belongs to a changed frame, the currently obtained video frame is determined as an image set, N is the maximum number of video frames in the image set, and the maximum number of video frames is determined according to the target encoding parameter.

[0017] Optionally, the target condition includes that the target encoding parameter is less than a parameter threshold, the number of video frames of the image set is N, and the hardware encoder is in a high-load state when encoding the image set.

[0018] Optionally, also include:

[0019] determining real-time encoding parameters;

[0020] According to the real-time encoding parameter and the target encoding parameter, thread resources are allocated to the software encoder when encoding each of the image set or the sub-image set.

[0021] Optionally, allocating thread resources to the software encoder when encoding each of the image set or the sub-image set according to the real-time encoding parameter and the target encoding parameter comprises:

[0022] If the real-time encoding parameter satisfies the target encoding parameter when encoding any of the image set or the sub-image set, reducing thread resources allocated to the software encoder;

[0023] If the real-time encoding parameters do not satisfy the target encoding parameters when encoding any of the image sets or the sub-image sets, and there are allocatable thread resources, the thread resources allocated to the software encoder are increased.

[0024] Optionally, the image set and the sub-image set are both obtained by a hardware abstraction layer of an operating system and provided to the hardware encoder and the software encoder;

[0025] The thread resources of the software encoder are allocated by the hardware abstraction layer.

[0026] Optionally, encoding the image set or the sub-image set by using the hardware encoder and the software encoder includes:

[0027] The image set or the sub-image set is input to a video processing unit driver, and the video processing unit driver calls a hardware encoder and a software encoder to encode the image set or the sub-image set. The video processing unit driver has an interface of the hardware encoder and an interface of the software encoder.

[0028] Optionally, splitting the image set that meets the target condition to obtain multiple sub-image sets includes at least one of the following:

[0029] Splitting an image set that meets a target condition based on a preset number of frames to obtain a plurality of sub-image sets, wherein the sub-image sets include a plurality of continuous video frames whose number is less than or equal to the preset number of frames;

[0030] Splitting the image set that meets the target condition into a preset number of sub-image sets;

[0031] According to the similarity of each video frame in the image set that meets the target condition, the image set that meets the target condition is split into multiple sub-image sets.

[0032] A second aspect of the present application provides an electronic device, including a memory and a processor;

[0033] The memory is used to store computer programs;

[0034] The processor is used to execute the computer program to implement:

[0035] Get target encoding parameters for video applications;

[0036] Determining whether the encoding capability of a default encoder meets the target encoding parameter; the default encoder is a hardware encoder or a software encoder;

[0037] When the encoding capability of the default encoder does not meet the target encoding parameters, the hardware encoder and the software encoder are used to encode an image set that does not meet the target conditions and split the image set that meets the target conditions to obtain multiple sub-image sets, wherein the image set includes at least two consecutive video frames provided by the video application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0039] Figure 1is a flowchart of a video encoding method provided by an embodiment of the present application;

[0040] Figure 2 is a flowchart of another video encoding method provided by an embodiment of the present application;

[0041] Figure 3 is a flowchart of a method for processing an image set provided by an embodiment of the present application;

[0042] Figure 4 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0043] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] This application embodiment provides a video encoding method, see Figure 1 , is a flow chart of the method, and the method may include the following steps.

[0046] S101, obtaining target encoding parameters of a video application.

[0047] This embodiment can be executed by any electronic device using codec 2.0 and based on the Android operating system. The video application can be any video-related application installed on the electronic device, including but not limited to video editing applications for editing videos, short video applications for browsing videos, player applications, etc.

[0048] The target encoding parameter may be provided by a video application, and the target encoding parameter may indicate a parameter of video data that needs to be encoded.

[0049] Optionally, in this embodiment, the target encoding parameters may include the frame rate and resolution of the video to be encoded, and may also include other encoding-related parameters, which are not limited.

[0050] Exemplarily, the target encoding parameters obtained in S101 may include: 4k resolution, frame rate 90fps (ie, 90 frames per second).

[0051] S102, determining whether the encoding capability of the default encoder meets the target encoding parameters; the default encoder is a hardware encoder or a software encoder.

[0052] The default encoder can be a hardware encoder or a software encoder.

[0053] In S102, it can be determined whether the encoding capability of the hardware encoder meets the target encoding parameters. If not, it is determined that the encoding capability of the default encoder does not meet the target encoding parameters, and step S103 is executed. If so, it is determined that the encoding capability of the default encoder meets the target encoding parameters, and step S104 is executed.

[0054] It is also possible to determine whether the encoding capability of the software encoder meets the target encoding parameters. If not, it is determined that the encoding capability of the default encoder does not meet the target encoding parameters and step S103 is executed. If so, it is determined that the encoding capability of the default encoder meets the target encoding parameters and step S104 is executed.

[0055] It is also possible to first determine whether the encoding capability of the software encoder meets the target encoding parameters, and then determine whether the encoding capability of the hardware encoder meets the target encoding parameters, or first determine whether the encoding capability of the hardware encoder meets the target encoding parameters, and then determine whether the encoding capability of the software encoder meets the target encoding parameters;

[0056] If both of them are not satisfied, it is determined that the encoding capability of the default encoder does not meet the target encoding parameters, and step S103 is executed. If any one of them meets the target encoding parameters, it is determined that the encoding capability of the default encoder meets the target encoding parameters, and step S104 is executed.

[0057] The method for determining whether the encoding capability of the hardware encoder meets the target encoding parameters may be:

[0058] The capability set of the hardware encoder registered when the hardware encoder is installed is obtained. The capability set records the video parameters adapted by the installed hardware encoder. If the capability set of the hardware encoder does not contain the target encoding parameters, it is determined that the encoding capability of the hardware encoder does not meet the target encoding parameters. If the capability set contains the target encoding parameters, it is determined that the encoding capability of the hardware encoder meets the target encoding parameters.

[0059] Exemplarily, the capability set of the hardware encoder currently installed on the device can include a maximum of 1080P resolution and 60fps frame rate. If the target encoding parameters are the 4k resolution and 90fps frame rate in the aforementioned example, it can be determined that the capability set does not include the target encoding parameters and the encoding capabilities of the hardware encoder do not meet the target encoding parameters.

[0060] The method of determining whether the encoding capability of the software encoder meets the target encoding parameters may be:

[0061] The capability set of the pre-registered software encoder is obtained, and based on the capability set, it is determined whether the encoding capability of the software encoder can meet the target encoding parameters. The determination method can refer to the hardware encoder and is not described in detail.

[0062] In this embodiment, the system may have only one default encoder, which may be a hardware encoder or a software encoder. Generally, if there is an available hardware encoder, the vendor may register the private hardware encoder as the default encoder. The software encoder may correspond to a process named media.swcodec, and the hardware encoder may correspond to a process with an obvious company logo (vendor logo), such as a process with the following name, android.hardware.media.c2@1.2-xxx-service).

[0063] As an example, assuming that calling mediacodec needs to encode hevc video, and the default encoder is the hardware encoder, in S102, the system's framework layer can query whether the hardware encoder supports the target encoder and whether the target encoding parameters are supported. If supported, the hardware encoder is used for encoding. If not supported, the software encoder is queried whether it supports the target encoding parameters. If supported, the software encoder is used for encoding. If not supported, S103 is executed.

[0064] When calling mediacodec, you can specify to use a specific encoder, for example, specify to use c2.xxx.hevc.encoder, where xxx can be the ID of the hardware encoder manufacturer or the software encoder ID, usually android.

[0065] S103, when the encoding capability of the default encoder does not meet the target encoding parameters, use the hardware encoder and the software encoder to encode an image set that does not meet the target conditions and multiple sub-image sets split from the image set that meets the target conditions, where the image set includes at least two consecutive video frames provided by the video application.

[0066] The method of simultaneously encoding using the hardware encoder and the software encoder in S103 may be:

[0067] The image set or sub-image set is input to the video processing unit driver, and the hardware encoder and the software encoder are called by the video processing unit driver to encode the image set or sub-image set. The video processing unit driver has an interface of the hardware encoder and an interface of the software encoder.

[0068] In this embodiment, the hardware encoder may be implemented by a video processing unit (VPU) of the electronic device. Therefore, the electronic device may call the hardware encoder through a video processing unit driver (VPU Driver).

[0069] To encode using both the hardware encoder and the software encoder, you can perform the following preparations in advance:

[0070] The electronic device may set the interface of the hardware encoder as a synchronous interface in advance, and set certain delay parameters inside the hardware encoder according to the performance of the software encoder, so that the hardware encoder and the software encoder can work synchronously.

[0071] At the same time, a software encoder interface can be pre-created inside the video processing unit driver, so that the video processing unit driver can call various software encoders (such as ffmpeg or other open source codec libraries) through the software encoder interface.

[0072] Based on the above preparations, in S103, each image set or sub-image set that needs to be encoded can be input into the video processing unit driver. On the one hand, the video processing unit driver can call the hardware encoder provided by the VPU to encode part of the data of the input image set or sub-image set based on the original driving capability. On the other hand, it can call the software encoder to encode another part of the data of the input image set or sub-image set through a pre-created software encoder interface, thereby realizing that the hardware encoder and the software encoder simultaneously encode the video data output by the video application.

[0073] S104: When the encoding capability of the default encoder satisfies the target encoding parameter, the default encoder is used to encode the image set provided by the video application.

[0074] In step S104, the encoding process may be performed using the encoder whose encoding capability meets the target encoding parameters determined in S102. That is, if it is determined that the encoding capability of the hardware encoder meets the target encoding parameters, the hardware encoder is used to encode the image set provided by the video application to complete the video encoding task, and if it is determined that the encoding capability of the software encoder meets the target encoding parameters, the software encoder is used to encode the image set provided by the video application to complete the video encoding task.

[0075] The method of using a hardware encoder or a software encoder to perform video encoding can be found in the relevant technology and will not be described in detail.

[0076] The beneficial effects of this embodiment are:

[0077] When the encoding capability of a single default encoder does not meet the target encoding parameters, the method can utilize the hardware encoder and the software encoder to simultaneously encode the video frames provided by the video application, thereby improving the encoding performance.

[0078] See also Figure 2 The embodiment of the present application also provides a video encoding method, which may include Figure 1 The method shown in the figure may further include steps S101 to S104 and the following steps:

[0079] S201: Determine whether the video application belongs to a video editing application.

[0080] When the video application does not belong to a video editing application, step S202 is executed.

[0081] Whether a video application belongs to a video editing application can be determined in that the electronic device can obtain a pre-recorded application whitelist, which can be used to record the application names of the video editing applications installed in the electronic device. The electronic device can determine whether the application name of the video application that currently needs to be encoded is in the application whitelist. If it is in the list, it is determined to be a video editing application; if not, it is determined not to be a video editing application.

[0082] The method for determining whether a video application belongs to a video editing application may also be that the electronic device obtains type information of the video application in the application market software used to install the application program, and determines whether the video application currently requiring video encoding belongs to a video editing application based on the type information.

[0083] S202: Encode a set of images provided by a video application using a default encoder.

[0084] The encoding parameters required by non-video editing applications installed on electronic devices are generally within the encoding capabilities of the hardware encoder or software encoder installed on the electronic device. Therefore, if it is determined that the video application does not belong to a video editing application, it can be considered that the electronic device has a hardware encoder or software encoder with encoding capabilities that can meet the target encoding parameters, and the hardware encoder or software encoder can be directly used to encode the video data output by the video application.

[0085] Correspondingly, for video editing applications, the target encoding parameters of such applications can generally fluctuate within a large range according to user needs. For example, users can use video editing applications to edit videos with 4k resolution and 90fps frame rate, or they can edit videos with 720P resolution and 30fps frame rate.

[0086] Therefore, in the case where the video application belongs to a video editing application, step S102 of the aforementioned embodiment may be performed to determine whether the encoding capability of the default encoder meets the target encoding parameters.

[0087] Among them, step S201 can be as follows Figure 2 It is shown as being executed after step S101, or it may be executed before step S101, or it may be executed simultaneously with step S101, without limitation.

[0088] In S104, the electronic device may divide the multiple video frames to be encoded provided by the video application into several image sets (Group of Picture, gop), and use a hardware encoder and a software encoder to jointly encode each image set in units of image sets, or jointly encode each sub-image set split from the image set.

[0089] See also Figure 3 , the electronic device may obtain the image set in the following manner:

[0090] Obtain each video frame provided by the video application one by one;

[0091] Whenever N video frames are obtained continuously or it is determined that the obtained video frame belongs to a changed frame, the currently obtained video frame is determined as an image set, N is the maximum number of video frames in the image set, and the maximum number of video frames is determined according to the target encoding parameters.

[0092] In this embodiment, the electronic device can continuously obtain video frames that need to be encoded output by the video application, and cache the obtained video frames in a pre-allocated cache area. Each time a video frame is obtained, the electronic device determines whether the video frame is a changed frame, and updates the cumulative number of obtained video frames.

[0093] Among them, for each video frame, the electronic device can first decode the video frame, and then use various algorithms or models based on machine vision to process the decoded video frame to determine whether a scene change occurs in the video frame based on the processing result. If a scene change occurs, it is determined that the video frame belongs to a changed frame; if no scene change occurs, it is determined that the video frame does not belong to a changed frame. The method for determining whether a scene change occurs can be referred to the relevant technology and will not be repeated here.

[0094] If the electronic device determines that the currently obtained video frame belongs to a changing frame, the electronic device can determine all the video frames cached in the cache area and obtained up to the current moment as an image set, store the image set in a pre-established first image set queue (i.e., old gopN queue), wait for subsequent encoding of the image set, and at the same time clear the video frames cached in the cache area, reset the cumulative number of video frames to zero, and continue to cache the video frames after the image set.

[0095] If the electronic device determines that the currently obtained video frame does not belong to a changing frame, but the cumulative number of video frames obtained is equal to N, the electronic device can determine the N video frames cached in the cache area and obtained up to the current moment as an image set, store the image set in a pre-established first image set queue (i.e., old gopN queue), wait for subsequent encoding of the image set, and at the same time clear the video frames cached in the cache area, reset the cumulative number of video frames to zero, and continue to cache the video frames after the image set.

[0096] If the electronic device determines that the currently obtained video frame does not belong to a changing frame and the cumulative number of obtained video frames is less than N, the electronic device can continue to obtain the next video frame until either the currently obtained video frame belongs to a changing frame or the cumulative number of obtained video frames is equal to N.

[0097] Among them, the maximum number of video frames N can be determined according to the target encoding parameters. Generally, if the target encoding parameters include a larger resolution or frame rate, a smaller maximum number of video frames can be set to reduce the load of the encoder when encoding each image set. If the resolution and frame rate of the target encoding parameters are both small, a larger maximum number of video frames can be set to improve encoding efficiency.

[0098] Exemplarily, assuming that the first N video frames output by the video application are not changing frames, the electronic device can, after accumulating the cache of the first N video frames, store the first N video frames as the first image set in the first image set queue, and continue to obtain the N+1th video frame; assuming that the obtained N+mth video frame is a changing frame (m is less than N and greater than 1), the electronic device will determine the cached N+1th to N+mth video frames as the second image set, continue to obtain the N+m+1th video frame, and so on.

[0099] Continue to see Figure 3 When the hardware encoder and the software encoder are used to jointly encode the image set in S103, the electronic device can read the image sets from the first image set queue one by one in the order of entering the queue, and the image sets that enter the queue first are read first, and the image sets that enter the queue later are read later.

[0100] Each time an image set is read, the electronic device may determine whether the image set meets the target condition.

[0101] If the read image set does not meet the target condition, the electronic device directly writes the image set into the third image set queue (ie, real gopN queue), and the hardware encoder and the software encoder can read the written image set from the third image set queue and encode the image set.

[0102] If the read image set meets the target condition, the electronic device can split the image set into multiple sub-image sets. During the splitting process, the image set can be temporarily stored in the second image set queue (i.e., new gopN queue). After the splitting is completed, the electronic device transfers each split sub-image set from the second image set queue to the third image set queue one by one. The hardware encoder and the software encoder can read the written sub-image sets one by one from the third image set queue, and encode each read sub-image set one by one.

[0103] Each sub-image set includes a portion of continuous video frames in the image set to which it belongs, and the number of video frames included in different sub-image sets may be equal or unequal. Exemplarily, assuming that N is equal to 20, an image set including 20 video frames may be split into two sub-image sets, one of which includes the first 12 video frames and the other includes the last 8 video frames; or, an image set including 20 video frames may be split into four sub-image sets, each of which includes 5 continuous video frames.

[0104] The method of encoding the sub-image set can refer to the aforementioned step S103 and will not be described in detail.

[0105] The beneficial effects of this embodiment are:

[0106] When an image set contains too many video frames, encoding the image set may generate too much load on the hardware encoder and software encoder, affecting the operation of other applications on the electronic device. Splitting the image set that meets the target conditions into multiple smaller sub-image sets and encoding them in units of sub-image sets can appropriately reduce the load of the encoder and allow other applications to run more smoothly.

[0107] In this embodiment, the target condition can be set as needed without limitation.

[0108] As an example, the target condition may include that the target encoding parameter is less than a parameter threshold, the number of video frames of the image set is N, and the hardware encoder is in a high load state when encoding the image set.

[0109] The parameter threshold can be determined according to the maximum encoding performance of the hardware encoder. For example, the maximum encoding performance of the hardware encoder is 4k@30fps, 10M. If the resolution of the video currently to be encoded is 1080p, since the 4k resolution is 4 times the 1080p resolution, the parameter threshold can be determined to be 1080p@120fps.

[0110] Different hardware encoders have different encoding performances. Therefore, the above parameter thresholds may be adjusted according to different hardware encoders actually used by the device, and are not limited to the above examples.

[0111] Whether the hardware encoder is in a high-load state can be determined based on the output frame rate of the hardware encoder. The output frame rate of the hardware encoder can represent the number of video frames encoded by the hardware encoder in a unit of time (for example, within 1 second). If the output frame rate is greater than a preset output frame rate threshold, it can be determined that the hardware encoder is in a high-load state. If the output frame rate is less than or equal to the output frame rate threshold, it can be determined that the hardware encoder is in a low-load state.

[0112] The electronic device can obtain the output frame rate of the hardware encoder when reading an image set from the first image set queue. If the output frame rate at this time is greater than the output frame rate threshold, it is determined that the hardware encoder is in a high-load state when encoding the image set. If the output frame rate at this time is less than or equal to the output frame rate threshold, it is determined that the hardware encoder is in a low-load state when encoding the image set.

[0113] The electronic device may split an image set that meets the target condition in a variety of ways. As some examples, the image set that meets the target condition may be split in at least one of the following splitting ways.

[0114] First, based on a preset number of frames, the image set that meets the target condition is split into multiple sub-image sets, where the sub-image sets include multiple continuous video frames whose number is less than or equal to the preset number of frames;

[0115] Second, split the image set that meets the target condition into a preset number of sub-image sets;

[0116] Third, according to the similarity of each video frame in the image set that meets the target condition, the image set that meets the target condition is split into multiple sub-image sets.

[0117] In splitting mode 1, the electronic device can obtain a preset number of frames of the sub-image set, which indicates how many video frames each sub-image set obtained by splitting can contain at most, and the electronic device can split the image set that meets the target condition into multiple sub-image sets containing at most the preset number of video frames according to the preset number of frames. Exemplarily, assuming that N is equal to 50, the preset number of frames can be 10, in which case the electronic device can split the image set that meets the target condition into 5 sub-image sets containing 10 video frames; the preset number of frames can also be 20, in which case the electronic device can split the image set that meets the target condition into 2 sub-image sets containing 20 video frames and 1 sub-image set containing 10 video frames.

[0118] In the second splitting mode, the electronic device can obtain a preset number of sub-image sets, which indicates how many sub-image sets are to be split into an image set that meets the target condition, and based on the preset number, the electronic device can evenly split the image set that meets the target condition into the preset number of sub-image sets. Exemplarily, assuming that N is equal to 50, the preset number can be 2, and the electronic device can split the image set that meets the target condition into 2 sub-image sets containing 25 video frames.

[0119] In the third splitting method, the electronic device can determine the similarity between every two adjacent video frames in the image set that meets the target condition, and then divide the video frames in the image set whose similarity is greater than the similarity threshold into the same sub-image set, and divide the video frames whose similarity is less than or equal to the similarity threshold into different sub-image sets. Exemplarily, assuming that N is equal to 50, wherein the similarity of adjacent video frames in the 1st to 20th video frames is greater than the similarity threshold, the similarity of adjacent video frames in the 21st to 50th video frames is greater than the similarity threshold, and the similarity between the 20th and 21st video frames is less than the similarity threshold, then the image set can be split into two sub-image sets, the first sub-image set contains the 1st to 20th video frames, and the second sub-image set contains the 21st to 50th video frames.

[0120] In some optional embodiments, the electronic device may dynamically allocate thread resources to the software encoder as follows during simultaneous encoding of a video using the hardware encoder and the software encoder:

[0121] determining real-time encoding parameters;

[0122] According to the real-time encoding parameters and the target encoding parameters, thread resources are allocated to the software encoder when encoding each picture set or sub-picture set.

[0123] The real-time encoding parameters may include the sum of the frame rates output by the hardware encoder and the software encoder at the current moment.

[0124] The amount of thread resources can be expressed by the number of threads.

[0125] In the above allocation method, if the real-time encoding parameter is greater than the target encoding parameter, the electronic device can reduce the thread resources allocated to the software encoder, that is, reduce the number of threads occupied by the software encoder. If the real-time encoding parameter is less than the target encoding parameter, the electronic device can increase the thread resources allocated to the software encoder, that is, increase the number of threads occupied by the software encoder.

[0126] For example, assuming that the target encoding parameters include a frame rate of 90fps, if the real-time encoding parameter is 80fps, the electronic device can increase the threads occupied by the software encoder from 10 threads to 12 threads; if the real-time encoding parameter is 100fps, the electronic device can reduce the threads occupied by the software encoder from 10 threads to 8 threads.

[0127] Through the above allocation method, the electronic device can reduce the thread resources occupied by the software encoder when the real-time encoding parameters are high, and increase the thread resources occupied by the software encoder when the real-time encoding parameters are low to meet the target encoding parameters, thereby meeting the encoding requirements of the target encoding parameters and minimizing resource occupation.

[0128] Optional, see Figure 3 The electronic device may allocate thread resources to the software encoder according to whether there are allocatable thread resources and whether the real-time encoding parameters meet the target encoding parameters. The allocation method may be:

[0129] If the real-time encoding parameters satisfy the target encoding parameters when encoding any image set or sub-image set, the thread resources allocated to the software encoder are reduced;

[0130] If the real-time encoding parameters do not meet the target encoding parameters when encoding any image set or sub-image set, and there are allocatable thread resources, the thread resources allocated to the software encoder are increased.

[0131] Whether there are allocatable thread resources can be determined by comparing the current processor occupancy with the preset maximum processor occupancy. If the current processor occupancy is less than the maximum processor occupancy, it is determined that there are allocatable thread resources. If the current processor occupancy is greater than or equal to the maximum processor occupancy, it is determined that there are no allocatable thread resources.

[0132] Generally, a software encoder uses a central processing unit (CPU) for encoding, and a hardware encoder may use a VPU for encoding. The processor occupancy rate may be the occupancy rate of the CPU occupied by the software encoder.

[0133] The current processor occupancy rate of the software encoder can be represented by the proportion of threads currently occupied by the software encoder in the threads allocatable by the CPU. For example, if the CPU has 100 allocatable threads and the software encoder currently occupies 10 of them, the current processor occupancy rate is 10%.

[0134] The maximum processor occupancy rate may be a parameter preset based on the performance of the processor and the requirements of the software encoder. The maximum processor occupancy rate of the software encoder may indicate how many CPU thread resources the software encoder can occupy at most.

[0135] The real-time encoding parameter satisfies the target encoding parameter, which may include that the real-time encoding parameter is greater than or equal to the target encoding parameter, and the real-time encoding parameter does not satisfy the target encoding parameter, which may include that the real-time encoding parameter is less than the target encoding parameter.

[0136] In the allocation method of this embodiment, if the real-time encoding parameters meet the target encoding parameters, the thread resources allocated to the software encoder can be reduced, so that the hardware encoder bears more encoding work and the software encoder bears less encoding work. This can not only give full play to the performance of the hardware encoder, but also reduce the thread resources occupied by the software encoder, thereby improving the fluency of other applications that require thread resources.

[0137] If the real-time encoding parameters do not meet the target encoding parameters, it means that the performance of the hardware encoder is no longer sufficient to meet the target encoding parameters. At this time, if there are allocatable thread resources, the thread resources allocated to the software encoder can be increased, thereby improving the performance of the joint encoding of the hardware encoder and the software encoder, so as to improve the real-time encoding parameters and meet the requirements of the target encoding parameters.

[0138] The electronic device used to execute the method of this embodiment may have the following Figure 4 The system architecture shown in Figure 1 can be divided into the application layer, the framework layer (Java FrameWork), the local framework layer (Native FrameWork), the hardware abstraction layer (Native Hal) and the driver layer (Driver).

[0139] The application layer may include the aforementioned video application. The video application may provide the video frames to be encoded and target encoding parameters to the encoder (MediaCodec) through at least one of the media recorder (MediaRecorder) and the media player (MediaPlayer) of the framework layer.

[0140] The encoder can be located at the local framework level of the system.

[0141] Optionally, both the image set and the sub-image set are obtained by a hardware abstraction layer of an operating system and provided to the hardware encoder and the software encoder;

[0142] The thread resources of the software encoder are allocated by the hardware abstraction layer.

[0143] like Figure 4 As shown, the video frame obtained by the encoder can be detected as a changed frame through a scene change detection method after being decoded by the hardware abstraction layer, so that the hardware abstraction layer can divide the image set based on the changed frame, and then can split the sub-image set based on the image set that meets the target condition.

[0144] The obtained image sets and sub-image sets can be stored in the image set queue of the hardware abstraction layer (including at least one of the aforementioned first to third image set queues), and the hardware abstraction layer provides the image sets in the queue to the video processing unit driver of the driving layer, so that the video processing unit driver can call the hardware encoder and the software encoder to simultaneously encode each image set and / or sub-image set of the video data.

[0145] And, if Figure 4 As shown, the hardware abstraction layer can also compare the current processor occupancy rate and the maximum processor occupancy rate when the hardware encoder and the software encoder are encoding at the same time. Figure 3 The thread resources of the software encoder are allocated by the method, and then the image set to be encoded is allocated to the hardware encoder and the software encoder according to the amount of allocated thread resources. That is to say, when more thread resources are allocated to the software encoder, more encoding work can be allocated to the software encoder, and when fewer thread resources are allocated to the software encoder, less encoding work can be allocated to the software encoder.

[0146] Based on the above system architecture, when implementing the encoding method of this embodiment, only the hardware abstraction layer of the electronic device can be modified without changing the application layer, framework layer and local framework layer of the electronic device, making the encoding method of this embodiment easier to implement.

[0147] The present application also provides an electronic device. Figure 5 , is a schematic diagram of the structure of the electronic device, which may include a memory 501 and a processor 502.

[0148] The memory is used to store computer programs;

[0149] The processor is used to execute computer programs to:

[0150] Get target encoding parameters for video applications;

[0151] Determine whether the encoding capability of the default encoder meets the target encoding parameters; the default encoder is a hardware encoder or a software encoder;

[0152] When the encoding capability of the default encoder does not meet the target encoding parameters, a hardware encoder and a software encoder are used to encode an image set that does not meet the target conditions and split the image set that meets the target conditions to obtain multiple sub-image sets, where the image set includes at least two consecutive video frames provided by the video application.

[0153] The working principle of the electronic device of this embodiment can refer to the relevant steps of the video encoding method provided in the above-mentioned embodiment, which will not be described in detail.

[0154] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0155] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0156] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0157] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0158] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A video encoding method, comprising: Get target encoding parameters for video applications; Determining whether the encoding capability of the default encoder meets the target encoding parameters; The default encoder is a hardware encoder or a software encoder; When the encoding capability of the default encoder does not meet the target encoding parameters, the hardware encoder and the software encoder are used to encode an image set that does not meet the target conditions and multiple sub-image sets split from the image set that meets the target conditions, wherein the image set includes at least two consecutive video frames provided by the video application.

2. The method according to claim 1, further comprising: In a case where the video application does not belong to a video editing application, encoding the image set provided by the video application using the default encoder; The determining whether the encoding capability of the default encoder meets the target encoding parameter includes: In the case that the video application belongs to a video editing application, it is determined whether the encoding capability of the default encoder meets the target encoding parameters.

3. According to the method of claim 1, the manner of obtaining the image set comprises: Obtain each video frame provided by the video application one by one; Whenever N video frames are obtained continuously or it is determined that the obtained video frame belongs to a changed frame, the currently obtained video frame is determined as an image set, N is the maximum number of video frames in the image set, and the maximum number of video frames is determined according to the target encoding parameter.

4. The method according to claim 1, wherein the target condition includes that the target encoding parameter is less than a parameter threshold, the number of video frames of the image set is N, and the hardware encoder is in a high load state when encoding the image set.

5. The method according to claim 1, further comprising: determining real-time encoding parameters; According to the real-time encoding parameter and the target encoding parameter, thread resources are allocated to the software encoder when encoding each of the image set or the sub-image set.

6. The method according to claim 5, wherein the allocating thread resources to the software encoder when encoding each of the picture set or the sub-picture set according to the real-time encoding parameter and the target encoding parameter comprises: If the real-time encoding parameter satisfies the target encoding parameter when encoding any of the image set or the sub-image set, reducing thread resources allocated to the software encoder; If the real-time encoding parameters do not satisfy the target encoding parameters when encoding any of the image sets or the sub-image sets, and there are allocatable thread resources, the thread resources allocated to the software encoder are increased.

7. The method according to claim 5, wherein the image set and the sub-image set are both obtained by a hardware abstraction layer of an operating system and provided to the hardware encoder and the software encoder; The thread resources of the software encoder are allocated by the hardware abstraction layer.

8. The method according to claim 1, encoding the set of images or the set of sub-images using the hardware encoder and the software encoder, comprising: The image set or the sub-image set is input to a video processing unit driver, and the video processing unit driver calls a hardware encoder and a software encoder to encode the image set or the sub-image set. The video processing unit driver has an interface of the hardware encoder and an interface of the software encoder.

9. The method according to claim 1, splitting the image set that meets the target condition to obtain multiple sub-image sets, including at least one of the following: Splitting an image set that meets a target condition based on a preset number of frames to obtain a plurality of sub-image sets, wherein the sub-image sets include a plurality of continuous video frames whose number is less than or equal to the preset number of frames; Splitting the image set that meets the target condition into a preset number of sub-image sets; According to the similarity of each video frame in the image set that meets the target condition, the image set that meets the target condition is split into multiple sub-image sets.

10. An electronic device comprising a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement: Get target encoding parameters for video applications; Determining whether the encoding capability of a default encoder meets the target encoding parameter; the default encoder is a hardware encoder or a software encoder; When the encoding capability of the default encoder does not meet the target encoding parameters, the hardware encoder and the software encoder are used to encode an image set that does not meet the target conditions and split the image set that meets the target conditions to obtain multiple sub-image sets, wherein the image set includes at least two consecutive video frames provided by the video application.

Images