Video processing method, electronic equipment, storage medium and chip

Abstract

The invention provides a video processing method, electronic equipment, a storage medium and a chip. According to the method, when the video begins to be decoded, under the condition that the current hardware macro block resources are not enough to create the hard decoder, the video is decoded by using the soft decoder, and then the hard decoder is created by using the hardware macro block resources released by other hard decoders, so that the video can be decoded by combining the hard decoder and the soft decoder, and the video decoding efficiency is improved. According to the invention, the problems of hard decoder creation failure, blank screen and flash back phenomena of the electronic equipment, or system crash or errors of the electronic equipment and the like caused by the fact that the electronic equipment does not create the hard decoder with the current hardware macro block resources are not enough in related schemes are avoided, the continuity when a user edits a video is ensured, and the user experience is improved. In addition, the method combines the hard decoder and the soft decoder to decode the video, and the decoding duration of the soft decoder is shortened, so that the fluency of video playing is ensured.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to a video processing method, electronic equipment, storage medium and chip. Background Art

[0002] Currently, camera functions are common services provided by electronic devices, allowing users to take photos and record videos. If users need to process the captured images or videos, such as stitching multiple videos or combining videos and images, adding picture-in-picture, adding special effects, or adding music, they typically need to use electronic devices or third-party image processing software to perform the processing.

[0003] After processing an image or video in image processing software, users can preview the processed video. During preview, the electronic device will create a hardware decoder for each video or image by occupying the electronic device's hardware macroblock resources to decode the video or image, allowing preview playback. However, the electronic device's hardware macroblock resources are limited. If the hardware macroblock resources are insufficient to create the hardware decoder, the hardware decoder creation and video or image decoding will fail. This may cause the image processing software to display a black screen or flash back, or may cause the electronic device to experience system crashes or errors, affecting the user experience. Summary of the Invention

[0004] The present application provides a video processing method, electronic device, storage medium and chip, which can avoid problems such as black screen and flash back of image processing software due to failure to create a hardware decoder, or system crash or error in electronic equipment, thereby ensuring the continuity of users when editing videos or pictures and improving the user experience.

[0005] In a first aspect, a video processing method is provided, which is applied to an electronic device, wherein the electronic device is configured with hardware macroblock resources, and the hardware macroblock resources are used by the electronic device to create a hard decoder to decode the video. The method includes:

[0006] In response to a clipping operation, a first video is acquired; in response to a preview operation, when it is determined that the electronic device does not have sufficient hardware macroblock resources to create a hard decoder for the first video, the first video is decoded based on a soft decoder; in the process of decoding the first video based on the soft decoder, it is determined whether the hardware macroblock resources released by the hard decoders of other videos are sufficient to create a hard decoder for the first video, the other videos being different from the first video; when it is determined that the hardware macroblock resources released by the other hard decoders are sufficient to create a hard decoder for the first video, the first video is decoded based on the soft decoder and the hard decoder of the first video.

[0007] In the embodiment of the present application, the electronic device may first decode the first video based on a soft decoder that does not occupy the hardware macroblock resources (for example, Figure 14 Second, during the decoding of the first video based on the soft decoder, the hardware decoders of other videos are decoded and the hardware macroblock resources are released (e.g. Figure 14 The hardware decoder of video A shown releases hardware macroblock resources at T3), determines whether the hardware macroblock resources released by the hardware decoders of other videos are sufficient to create a hard decoder for the first video, and creates a hard decoder based on the released hardware macroblock resources (for example, Figure 14 The hard decoder is created between T3 and TA as shown), and the first video is decoded based on the soft decoder and the hard decoder (eg Figure 14 As shown, between T3 and TA, video B continues to be decoded with the soft decoder, and after TA, video B is decoded based on the hard decoder of video B). In this way, when the hardware macroblock resources are insufficient to create a hard decoder, the video can be decoded together by combining the soft decoder and the hard decoder. This avoids problems such as black screen and flash back of the image processing software due to failure to create a hard decoder, or system crash or error in the electronic device, thereby ensuring the continuity of users when editing videos or pictures and improving the user experience.

[0008] In a related solution, an electronic device decodes a video using a soft decoder when the hardware macroblock resources are insufficient to create a hard decoder, e.g. Figure 15 In the embodiment of the present application, if the electronic device determines that the current remaining hardware macroblock resources are insufficient to create a hard decoder at T2, then the entire video B (the first video) will be decoded based on the soft decoder. However, since the soft decoder is a decoding program running on the CPU and does not rely on dedicated hardware, it may consume more CPU resources, resulting in slower decoding speed and increased power consumption. In addition, the performance of the soft decoder is limited to a certain extent by the processing power of the CPU, and the video playback is not as smooth as hard decoding, resulting in freezes and delays in the video decoded by the soft decoder. In the embodiment of the present application, when it is determined that the current remaining hardware macroblock resources are insufficient to create a hard decoder to decode the first video, the video is decoded together by combining the soft decoder and the hard decoder (for example, Figure 14Video B is decoded based on the soft decoder between T2 and TA, and video B is decoded based on the hard decoder between TA and T4). Since the time for decoding the video based on the soft decoder is shortened, it avoids consuming more CPU resources, thereby increasing the speed of decoding the video, reducing power consumption, and improving decoding performance. It also reduces the probability of freezes and delays in the decoded video, allowing users to have a better experience when previewing the decoded video.

[0009] In conjunction with the first aspect, in a possible implementation of the first aspect, when it is determined that the electronic device does not have sufficient hardware macroblock resources to create a hard decoder for the first video, decoding the first video based on a soft decoder includes:

[0010] At the start moment of decoding of the first video, determine whether the electronic device has sufficient current hardware macroblock resources to create a hard decoder for the first video; if it is determined that the electronic device has insufficient current hardware macroblock resources to create a hard decoder for the first video, decode the first video based on a soft decoder; and, in the process of decoding the first video based on the soft decoder, determine whether the hardware macroblock resources released by the hard decoders of other videos are sufficient to create a hard decoder for the first video, including: in the process of decoding the first video based on the soft decoder, at the end moment of decoding of other videos, release the hardware macroblock resources occupied by the hard decoders of other videos; determine whether the hardware macroblock resources released by the hard decoders of other videos are sufficient to create a hard decoder for the first video.

[0011] In an embodiment of the present application, the electronic device can execute the corresponding task under the time node based on the time node (for example, the decoding start time of the first video, the decoding end time of other videos) (for example, determining at the decoding start time of the first video whether the electronic device uses the current hardware macroblock resources to create a hard decoder for the first video, etc., and determining at the decoding end time of other videos whether the hardware macroblock resources after the hard decoders of other videos are released are sufficient to create a hard decoder for the first video, etc.). Executing the corresponding task based on the time node can reduce the complexity brought about by concurrent execution of multiple tasks or simultaneous execution of multiple tasks, and can improve the execution efficiency of the tasks.

[0012] In conjunction with the first aspect, in a possible implementation of the first aspect, when it is determined that hardware macroblock resources released by other hardware decoders are sufficient to create a hardware decoder for the first video, decoding the first video based on the soft decoder and the hardware decoder of the first video includes:

[0013] When it is determined that the hardware macroblock resources released by other hard decoders are sufficient to create a hard decoder for the first video, the first video between the decoding end moment of the other videos and the first moment is decoded based on the soft decoder, the first moment being the moment when the hard decoder for the first video is completed, and the first moment is separated from the decoding end moment of the other videos by a preset time length; the first video between the first moment and the decoding end moment of the first video is decoded based on the hard decoder for the first video.

[0014] In an embodiment of the present application, the first moment is the moment when the hard decoder of the first video is created. The first moment is separated from the decoding end moments of other videos by a preset time length. The electronic device can decode the first video between the decoding end moment of other videos and the first moment based on the soft decoder, and decode the first video between the first moment and the decoding end moment of the first video based on the hard decoder of the first video. Since the preset time length is usually short, when decoding the first video, the time length for decoding the first video based on the soft decoder is relatively short, and the time length for decoding the first video based on the hard decoder is relatively long. The preset time length can make the soft decoding time length shorter, further improving the speed of decoding the video, reducing power consumption, and further improving the decoding performance.

[0015] In conjunction with the first aspect, in a possible implementation of the first aspect, the other videos include a second video and a third video, the decoding start time of the second video is earlier than the decoding start time of the third video, and the decoding start time of the third video is earlier than the decoding start time of the first video; and, if it is determined that the electronic device is insufficient to create a hard decoder for the first video using current hardware macroblock resources, before decoding the first video based on the soft decoder, the method further includes:

[0016] At the start moment of decoding the second video, determine whether the electronic device uses sufficient hardware macroblock resources to create a hard decoder for the second video; if it is determined that the electronic device uses sufficient hardware macroblock resources to create a hard decoder for the second video, decode the second video based on the hard decoder of the second video; at the start moment of decoding the third video, determine whether the electronic device uses sufficient hardware macroblock resources to create a hard decoder for the third video; if it is determined that the electronic device uses sufficient hardware macroblock resources to create a hard decoder for the third video, decode the third video based on the hard decoder of the third video; and if it is determined that the electronic device does not use sufficient hardware macroblock resources to create a hard decoder for the first video, decode the first video based on a soft decoder, including: at the start moment of decoding the first video, determine whether the electronic device uses sufficient hardware macroblock resources to create a hard decoder for the first video; if it is determined that the electronic device does not use sufficient hardware macroblock resources to create a hard decoder for the first video, decode the first video based on the soft decoder.

[0017] In conjunction with the first aspect, in a possible implementation of the first aspect, the decoding end time of the second video is earlier than the decoding end time of the third video, and the decoding end time of the third video is earlier than the decoding end time of the first video; and, during decoding of the first video based on the soft decoder, determining whether hardware macroblock resources released by hard decoders of other videos are sufficient to create a hard decoder for the first video includes:

[0018] During decoding of the first video based on the soft decoder, at the end of decoding of the second video, the hardware macroblock resources occupied by the hard decoder of the second video are released; it is determined whether the hardware macroblock resources after the release of the hard decoder of the second video are sufficient to create the hard decoder of the first video; and, when it is determined that the hardware macroblock resources after the release of other hard decoders are sufficient to create the hard decoder of the first video, the first video is decoded based on the soft decoder and the hard decoder of the first video, including: when it is determined that the hardware macroblock resources after the release of the hard decoder of the second video are sufficient to create the hard decoder of the first video, the first video is decoded based on the soft decoder and the hard decoder of the first video.

[0019] In an embodiment of the present application, the video acquired by the electronic device is three-channel video (for example, the three-channel video is a first video, a second video, and a third video). When the hardware macroblock resources of the electronic device are sufficient to create a hard decoder for the second video and the third video, but insufficient to create a hard decoder for the first video, a soft decoder is first created to decode the first video. Then, during the process of soft decoding of the video, when it is determined that the hardware macroblock resources released by the hard decoder of the second video are sufficient to create a hard decoder for the first video, the first video is decoded based on the soft decoder and the hard decoder of the first video. In this way, when decoding the three-channel video, if the hardware macroblock resources of the electronic device are insufficient to create a hard decoder for one of the three videos (for example, the first video), a hard decoder is created using the hardware macroblock resources released by the other video (for example, the second video) in the three videos. Then, the soft decoder and the hard decoder are combined to decode the video together. This avoids the failure of creating the hard decoder due to insufficient hardware resources to create the hard decoder for the three videos when decoding the three videos, resulting in a black screen and flash back of the image processing software, or causing a system crash or error in the electronic device. This ensures the continuity of the user when editing videos or pictures, and improves the user experience.

[0020] In other related solutions for three-way video decoding, electronic devices decode the video through a soft decoder when the hardware macroblock resources are insufficient to create a hard decoder, e.g. Figure 25In the example, the electronic device determines at T3 that the currently remaining hardware macroblock resources are insufficient to create a hard decoder for the first video (video B), and then decodes the entire video B based on the soft decoder. However, since the soft decoder is a decoding program running on the CPU and does not rely on specialized hardware, it may consume more CPU resources, resulting in slower decoding speeds and increased power consumption. In addition, the performance of the soft decoder is limited to a certain extent by the processing power of the CPU, and the video playback is not as smooth as hard decoding, resulting in freezes and delays in the video decoded by the soft decoder. In the embodiment of the present application, when it is determined that the currently remaining hardware macroblock resources are insufficient to create a hard decoder to decode the first video, the video is decoded together by combining the soft decoder and the hard decoder (for example, Figure 23 (a) between T3 and TA, the first video (video B) is decoded based on the soft decoder, and between TA and T6, video B is decoded based on the hard decoder). Since the time for decoding the video based on the soft decoder is shortened, the consumption of more CPU resources is avoided, thereby increasing the speed of decoding the video, reducing power consumption, and improving decoding performance. The probability of freezes and delays in the decoded video is reduced, so that users have a better experience when previewing the decoded video.

[0021] In conjunction with the first aspect, in a possible implementation of the first aspect, after determining whether the hardware macroblock resources released by the hardware decoder of the second video are sufficient to create the hardware decoder of the first video, the method further includes:

[0022] When it is determined that the hardware macroblock resources after the release of the hard decoder of the second video are insufficient to create the hard decoder of the first video, at the end of decoding of the third video, the hardware macroblock resources occupied by the hard decoder of the third video are released; it is determined whether the hardware macroblock resources after the release of the hard decoder of the third video are sufficient to create the hard decoder of the first video; when it is determined that the hardware macroblock resources after the release of the hard decoder of the third video are sufficient to create the hard decoder of the first video, the first video is decoded based on the soft decoder and the hard decoder of the first video.

[0023] In an embodiment of the present application, the video acquired by the electronic device is three-channel video (for example, the three-channel video is a first video, a second video, and a third video). When the hardware macroblock resources of the electronic device are sufficient to create hard decoders for the second video and the third video, but are insufficient to create a hard decoder for the first video, a soft decoder is first created to decode the first video. Then, in the process of soft decoding the video, it is first determined whether the hardware macroblock resources released by the hard decoder of one video other than the first video in the three-channel video (for example, the second video) are sufficient to create a hard decoder for the first video. When it is determined that the hardware macroblock resources released by the hard decoder of the second video are insufficient to create a hard decoder for the first video, it is determined whether the hardware macroblock resources released by the hard decoder of another video other than the first video in the three-channel video (for example, the third video) are sufficient to create a hard decoder for the first video. When it is determined that the hardware macroblock resources released by the hard decoder of the third video are sufficient to create a hard decoder for the first video, the first video is decoded based on the soft decoder and the hard decoder of the first video. This ensures that there are sufficient hardware macroblock resources to create a hard decoder for the first video to perform hard decoding on the first video.

[0024] In combination with the first aspect, in a possible implementation of the first aspect, an overlay transition is configured between the second video and the third video, the start time of the overlay transition is the start time of decoding of the third video, and the end time of the overlay transition is the end time of decoding of the second video.

[0025] In the embodiment of the present application, when three-way video is applied to a scene with superimposed transitions, if the hardware macroblock resources of the electronic device are insufficient to create a hard decoder for the first video in the three-way video, the electronic device can perform soft decoding processing on the first video during the superimposed transition period. For example: Figure 23 T3 to T4 shown in (a) is the duration of the superimposed transition. The electronic device performs soft decoding on video B (the first video) between T3 and T4. Since the duration of the superimposed transition is relatively short, when the hardware macroblock resources are insufficient to create a hard decoder for the first video, the duration of soft decoding of the first video is relatively short, which can shorten the duration of video decoding based on the soft decoder and avoid consuming more CPU resources. This can increase the speed of video decoding, reduce power consumption, and improve decoding performance, reduce the probability of freezes and delays in the decoded video, and enable users to have a better experience when previewing the decoded video.

[0026] In combination with the first aspect, in a possible implementation of the first aspect, the electronic device displays an editing interface, the editing interface includes a main video track and a picture-in-picture track, and the first video is a video on the main video track, or a video on the picture-in-picture track.

[0027] In conjunction with the first aspect, in a possible implementation of the first aspect, during decoding of the first video based on the soft decoder, determining whether hardware macroblock resources released by hard decoders of other videos are sufficient to create a hard decoder for the first video includes:

[0028] During decoding of the first video based on the soft decoder, based on the resolution of the first video and the hardware macroblock resources released by the hard decoders of other videos, it is determined whether the hardware macroblock resources released by the hard decoders of other videos are sufficient to create a hard decoder for the first video.

[0029] In a second aspect, an electronic device is provided, wherein the electronic device is configured to execute the method provided in the first aspect. Specifically, the electronic device may include a processing unit configured to execute any possible implementation of the first aspect.

[0030] In a third aspect, an electronic device is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs comprising instructions, which, when executed by the one or more processors, enable the electronic device to execute a method in any possible implementation of the first aspect.

[0031] In a fourth aspect, a computer-readable storage medium is provided, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in the first aspect.

[0032] In a fifth aspect, a chip is provided, comprising a memory for storing instructions; and a processor for calling and executing instructions from the memory, so that an electronic device equipped with the chip executes the method described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is an example diagram of a failed attempt to create a hard decoder.

[0034] Figure 2 This is an example diagram of decoding by combining a hard decoder and a soft decoder provided in an embodiment of the present application.

[0035] Figure 3 1 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0036] Figure 4 It is a software structure block diagram of an electronic device 100 provided in an embodiment of the present application.

[0037] Figure 5 This is a schematic diagram of a user opening a clipping interface in an electronic device provided by an embodiment of the present application.

[0038] Figure 6 This is a schematic diagram of another embodiment of the present application provided by a user opening a clipping interface in an electronic device.

[0039] Figure 7 This is a schematic diagram of adding a video or picture in an editing interface provided by an embodiment of the present application.

[0040] Figure 8 This is another schematic diagram of adding videos or pictures in the editing interface provided by an embodiment of the present application.

[0041] Figure 9 It is a schematic diagram of a video processing method provided in an embodiment of the present application.

[0042] Figure 10 This is an example diagram of a video provided in an embodiment of the present application.

[0043] Figure 11 This is an example diagram provided by an embodiment of the present application in which a specific video does not exist in two-channel videos.

[0044] Figure 12 This is another example diagram of a specific video existing in two-channel video provided by an embodiment of the present application.

[0045] Figure 13 This is another example diagram of a specific video existing in two-channel video provided by an embodiment of the present application.

[0046] Figure 14 This is another example diagram of a specific video existing in two-channel video provided by an embodiment of the present application.

[0047] Figure 15 The following is an example diagram of a related solution.

[0048] Figure 16 This is a schematic diagram of three-channel video processing provided by an embodiment of the present application.

[0049] Figure 17 This is an example diagram provided by an embodiment of the present application in which a specific video does not exist in three-channel video.

[0050] Figure 18 This is an example diagram of a three-channel video with a specific video present in an embodiment of the present application.

[0051] Figure 19 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application.

[0052] Figure 20 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application.

[0053] Figure 21 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application.

[0054] Figure 22 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application.

[0055] Figure 23 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application.

[0056] Figure 24 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application.

[0057] Figure 25 is an example diagram of another related solution provided.

[0058] Figure 26 This is a timing diagram of a video processing method provided in an embodiment of the present application.

[0059] Figure 27 This is a timing diagram of another video processing method provided in an embodiment of the present application.

[0060] Figure 28 It is a schematic diagram of a video processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] Before introducing the technical solutions in the embodiments of the present application, the relevant terms involved in the embodiments of the present application are explained.

[0062] Hardware macroblock resources typically refer to the macroblock resources used by electronic devices to process video or image data during the encoding and decoding process. These resources can be memory areas, hardware logic, or processors. Macroblocks are the basic processing units in video or image encoding. They are composed of multiple pixel blocks and are the smallest unit for encoding and decoding operations. In hardware implementations, hardware macroblock resources may include dedicated processing units, caches, registers, etc., used to perform operations such as motion compensation, DCT (discrete cosine transform), quantization, inverse quantization, and IDCT (inverse discrete cosine transform).

[0063] A hardware decoder is a specialized hardware device or hardware-integrated decoding function used to decode video and audio data. Creating a software decoder typically consumes hardware macroblock resources. Hardware decoders are typically integrated into graphics cards, GPUs, dedicated decoding chips, or video playback devices. Because hardware decoders are specifically designed for decoding tasks, they offer faster decoding speeds, improved video playback performance (for example, enabling smoother video playback), and lower CPU utilization, which can reduce power consumption.

[0064] A soft decoder, also known as a software decoder, is a decoding program that runs on the CPU. Soft decoders do not rely on specialized hardware, but instead use the CPU's computing power to decode video and audio data. Creating a soft decoder typically does not occupy hardware macroblock resources. The advantage of soft decoders is that they are more flexible, can support more encoding formats, and can support new formats through software updates. However, soft decoders may consume more CPU resources, resulting in slower decoding speeds and increased power consumption. The performance of soft decoders is also limited to a certain extent by the CPU's processing power, making video playback less smooth than hard decoding, resulting in video freezes and delays after being decoded by the soft decoder.

[0065] Transitions are transition effects used to connect two clips in a video clip, enhancing visual quality and narrative flow. Transitions are primarily categorized into two types: dissolve and non-dissolve. A dissolve transition is a common transition effect that creates a transition by gradually fading one clip out while another gradually fades in. Dissolve transitions are characterized by the two clips being displayed simultaneously for part of the transition, visually "superimposing" them. Non-dissolve transitions, on the other hand, do not involve the two clips being superimposed. They can be simple cuts or more complex effects such as wipes, slides, and zooms. Non-dissolve transitions are typically used to maintain scene continuity or emphasize a direct connection between two clips. The characteristic of this type of transition is that the two clips do not appear simultaneously during the transition; instead, one clip switches directly to the other.

[0066] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0067] Currently, camera functions are common services provided by electronic devices, allowing users to take photos and record videos. If users need to process the captured images or videos, such as stitching multiple videos or combining videos and images, adding picture-in-picture, adding special effects, or adding music, they typically need to use electronic devices or third-party image processing software to perform the processing.

[0068] After processing an image or video in image processing software, users can preview the processed video. During preview, the electronic device will create a hardware decoder for each video or image by occupying the electronic device's hardware macroblock resources to decode the video or image, allowing preview playback. However, the electronic device's hardware macroblock resources are limited. If the hardware macroblock resources are insufficient to create the hardware decoder, the hardware decoder creation and video or image decoding will fail. This may cause the image processing software to display a black screen or flash back, or may cause the electronic device to experience system crashes or errors, affecting the user experience.

[0069] For example, please refer to Figure 1 , Figure 1 This is an example diagram of a failed attempt to create a hard decoder. Figure 1 In the video, Video A and Video B are two videos imported into the image processing software by the user, Video C is a picture-in-picture added by the user, and the transition is the overlay transition set by the user.

[0070] It should be understood that the timeline represents the order in which the video or audio is played during the video editing process. The timeline usually appears in the form of a time axis, showing the start and end time nodes of all the clips in the editing. For example Figure 1 In the example, Video A starts at T1 and ends at T4 on the timeline. Video B starts at T3 and ends at T6. Video C starts at T2 and ends at T5. The overlay transition starts at T3, the start time of Video B, and ends at T4, the end time of Video A. Figure 1 In the example, video A plays first, followed by video C, and finally video B. Video A ends first, followed by video C, and finally video B. The timeline and time nodes involved in the following embodiments can be referred to the description here and will not be repeated here.

[0071] Assume that the hardware macroblock resources of the electronic device can process a total of 79632 macroblocks of video or picture data, one macroblock consists of 16×16 pixels, and the resolutions of video A, video B, and video C are all 4K resolution (3840 (pixels) × 2160 (pixels)). Then, when previewing, the electronic device Figure 1At the starting time node T1 of the video A shown, a hard decoder is created by occupying hardware macroblock resources to decode the video A. The hard decoder can process (3840×2160) / (16×16)=32400 macroblocks in the video A through the hardware macroblock resources.

[0072] At the start time node T2 of video C, the electronic device needs to determine whether the currently remaining hardware macroblock resources are sufficient to create a hard decoder to decode video C. For example, at T2, since the hard decoder of video A has not yet released the hardware macroblock resources (released at time node T4), the electronic device calculates that the currently remaining hardware macroblock resources can still process 79632-32400=47232 macroblocks, while only (3840×2160) / (16×16)=32400 macroblocks are required for hard decoding of video C. Therefore, the electronic device determines that the currently remaining hardware macroblock resources are sufficient to create a hard decoder to decode video C. Therefore, the electronic device can create a hard decoder to decode video C by occupying hardware macroblock resources at T2.

[0073] At the start time node T3 of video B, the electronic device still needs to determine whether the remaining hardware macroblock resources are sufficient to create a hard decoder to decode video B. For example, at T3, since video A has not yet released its hardware macroblock resources (released at time node T4), and the hard decoder of video C has not yet released its hardware macroblock resources (released at time node T5), the electronic device calculates that the remaining hardware macroblock resources can still process 79632-32400-32400=14832 macroblocks. However, hard decoding of video B requires (3840×2160) / (16×16)=32400 macroblocks, and 14832 is less than 32400. Therefore, the electronic device determines that the remaining hardware macroblock resources are insufficient to create a hard decoder to decode video B. This may result in failure to create a hard decoder for video B and failure to decode video B, which may cause the image processing software to display a black screen or flash back, or may cause the electronic device to experience system crashes or errors, affecting the user experience.

[0074] In order to solve the above problems, an embodiment of the present application provides a video processing method. When an electronic device determines that the currently remaining hardware macroblock resources are insufficient to create a hard decoder to decode the video or picture, it can first create a soft decoder that does not occupy the hardware macroblock resources to decode the video or picture. Secondly, after the hard decoder completes the decoding and releases the hardware macroblock resources, a hard decoder is created based on the released hardware macroblock resources, and the video or picture is decoded by the hard decoder. In this way, when the hardware macroblock resources are insufficient to create a hard decoder, the video can be decoded by combining the soft decoder and the hard decoder, thereby avoiding the black screen and flash back of the image processing software due to the failure to create the hard decoder, or the system crash or error of the electronic device, thereby improving the user experience.

[0075] For example, please refer to Figure 2 , Figure 2 This is an example diagram of decoding by combining a hard decoder and a soft decoder provided in an embodiment of the present application. Figure 2 In the example, it is assumed that the hardware macroblock resources of the electronic device can process a total of 79632 macroblocks of video or picture data, one macroblock includes 16×16 pixels, and the resolutions of video A, video B, and video C are all 4K resolution (3840 (pixels) × 2160 (pixels)). At the starting time node T1 of video A and the starting time node T2 of video B, the electronic device can use the hardware macroblock resources to create a hard decoder to decode videos A and C. The process of this part can be referred to Figure 1 , I will not go into details here.

[0076] At time node T3, the start of video B, the electronic device needs to determine whether the remaining hardware macroblock resources are sufficient to create a hard decoder to decode video B. For example, at T3, because the hardware decoder of video A has not yet released the hardware macroblock resources (released at time node T4), and the hardware decoder of video C has not yet released the hardware macroblock resources (released at time node T5), the electronic device calculates that the remaining hardware macroblock resources can still process 79632-32400-32400=14832 macroblocks. However, hard decoding of video B requires (3840×2160) / (16×16)=32400 macroblocks, and 14832 is less than 32400. Therefore, the electronic device determines that the remaining hardware macroblock resources are insufficient to create a hard decoder to decode video B. At this time, the electronic device creates a soft decoder at T3 that does not occupy hardware macroblock resources to decode video B.

[0077] At the end time node T4 of Video A, the hardware decoder for Video A has completed decoding Video A and released hardware macroblock resources. At this point, the electronic device needs to determine whether the remaining hardware macroblock resources are sufficient to create a hardware decoder for decoding Video B. For example, at T3, the remaining hardware macroblock resources can process 79632-32400-32400 = 14832 macroblocks. At T4, because Video A has released hardware macroblock resources, the remaining macroblock resources can now process 14832+32400 = 47232 macroblocks. Since hard decoding Video B requires (3840×2160) / (16×16) = 32400 macroblocks, 47232 is greater than 32400. Therefore, the electronic device determines that the remaining hardware macroblock resources are sufficient to create a hardware decoder for Video B. At T4, the electronic device begins creating a hardware decoder for Video B.

[0078] Since it takes a certain amount of time to create a hard decoder, the electronic device can create a hard decoder at a time node (e.g. Figure 2 At the time node TA shown, the hardware decoder for video B is considered to have been created. The preset duration can be 200 milliseconds to 1000 milliseconds, such as 300 milliseconds, 500 milliseconds, 700 milliseconds, etc., and is not limited in this embodiment of the present application. For ease of description, the process of creating a hardware decoder is referred to as hardware decoding preparation. After TA, the electronic device decodes video B using the hardware decoder.

[0079] The video processing method provided in the embodiments of the present application is applied to electronic devices. The electronic device may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The electronic device may be a mobile phone, a tablet computer (Pad), a laptop computer, a desktop computer, a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the electronic device.

[0080] In order to better understand the embodiments of the present application, the structure of the electronic device according to the embodiments of the present application is introduced below.

[0081] For example, Figure 3 1 shows a schematic structural diagram of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 130, an antenna 1, an antenna 2, a mobile communication module 140, a wireless communication module 150, an audio module 160, a speaker 160A, a receiver 160B, a microphone 160C, an earphone interface 160D, a sensor module 170, a camera 180, a display screen 190, and the like. The sensor module 170 may include a pressure sensor 170A, a touch sensor 170B, and the like.

[0082] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0083] The processor 110 may include one or more processing units, for example, a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0084] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0085] A hardware decoder can be configured in the GPU. Creating a hardware decoder will occupy the hardware macroblock resources of the GPU.

[0086] The decoding program of the soft decoder can be run on the CPU, and creating a soft decoder will not occupy the hardware macroblock resources of the GPU.

[0087] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0088] In an embodiment of the present application, the processor 110 can determine whether the currently remaining hardware macroblock resources are sufficient to create a hard decoder. When it is determined that the currently remaining hardware macroblock resources are insufficient to create a hard decoder to decode the video, a soft decoder that does not occupy hardware macroblock resources is first created to decode the video. Secondly, after the hard decoder completes decoding and releases the hardware macroblock resources, a hard decoder is created based on the released hardware macroblock resources, and the video is decoded by the hard decoder.

[0089] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 140, the wireless communication module 150, the modem processor and the baseband processor.

[0090] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 140 , and antenna 2 is coupled to wireless communication module 150 , so that electronic device 100 can communicate with a network and other devices through wireless communication technology.

[0091] The electronic device 100 implements a display function through a GPU, a display screen 190 , and an application processor.

[0092] The display screen 190 is used to display images, videos, etc. In the embodiment of the present application, the screen of the application can be displayed in the display screen 190 through a floating window.

[0093] The electronic device 100 can implement a shooting function through an ISP, a camera 180, a video codec, a GPU, a display screen 190, and an application processor.

[0094] The ISP is used to process data fed back by the camera 180.

[0095] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100.

[0096] The internal memory 130 can be used to store computer executable program codes, which include instructions. The processor 110 executes the instructions stored in the internal memory 130 to execute various functional applications and data processing of the electronic device 100. The internal memory 130 can include a program storage area and a data storage area.

[0097] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 160, the speaker 160A, the receiver 160B, the microphone 160C, the headphone jack 160D, and the application processor.

[0098] The pressure sensor 170A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 170A can be provided on the display screen 190. When a touch operation is applied to the display screen 190, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 170A.

[0099] The touch sensor 170B, also known as a "touch panel," can be disposed on the display screen 190. The touch sensor 170B and the display screen 190 form a touch screen, also known as a "touch screen." The touch sensor 170B is used to detect touch operations applied thereto or near it.

[0100] This concludes the introduction to the hardware structure of the electronic device 100. It is understood that Figure 3The components included in the illustrated hardware structure do not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different component configuration. Figure 3 The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0101] Furthermore, an operating system runs on top of the aforementioned components, such as the iOS operating system, the Android open-source operating system, the Windows operating system, and the Hongmeng operating system. Applications can be installed and run on these operating systems.

[0102] The operating system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android operating system with a layered architecture is used as an example to illustrate the operating system of the electronic device 100.

[0103] Figure 4 This is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application. A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided from top to bottom into an application layer, an application framework layer, a system library layer, a kernel layer, and a hardware layer. The application layer may include a series of application packages.

[0104] like Figure 4 As shown, the application layer includes applications such as camera, gallery, calculator, and map.

[0105] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0106] like Figure 4 As shown, the application framework layer may include a window manager, a view system, a resource manager, a notification manager, a media codec (MediaCodec), etc.

[0107] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0108] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0109] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0110] The notification manager enables applications to display notification information in the status bar, which can be used to convey informational messages and disappear automatically after a short stay without user interaction.

[0111] MediaCodec is used to encode and decode audio and video data. It supports multiple codec formats.

[0112] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0113] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0114] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0115] System libraries can include multiple functional modules, such as media libraries, 3D graphics processing libraries (e.g. OpenGL ES), 2D graphics engines (e.g. SGL), drawing components (Surface Flinger, SF), audio and video processing tools (FFmpeg), etc.

[0116] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0117] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0118] A 2D graphics engine is a drawing engine for 2D drawings.

[0119] The drawing component is responsible for the system's frame buffer, mainly used to allocate graphics buffers, synthesize graphics buffers, manage vertical refresh synchronization (VSync) events, etc.

[0120] The audio and video processing tool FFmpeg is a widely used multimedia framework that provides a complete solution for recording, converting, and streaming audio and video. In the Android system, FFmpeg can be located at different levels. For example, in the embodiment of the present application, FFmpeg can be located in the system library. In other embodiments, FFmpeg can also be integrated into the gallery application as part of the application and interact directly with the user. In other embodiments, some hardware devices in the hardware layer may already have certain FFmpeg functions built in to accelerate tasks such as video encoding and decoding.

[0121] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0122] The hardware layer includes CPU, GPU, etc.

[0123] It should be understood that Figure 4 The software and hardware architecture diagram shown is for illustrative purposes only and should not be construed as limiting the embodiments of the present application.

[0124] The above embodiment introduces the hardware structure and software architecture of the electronic device 100. The following introduces the application scenario of the video processing method provided in the embodiment of the present application.

[0125] The video processing method provided in the embodiment of the present application can be applied to the scene of video editing. Figure 5 , Figure 5 This is a schematic diagram of a user opening a clipping interface in an electronic device provided by an embodiment of the present application.

[0126] Please refer to Figure 5 The interface 510 shown in (a) includes a "Gallery" icon 511, which the user can click. In response to the user clicking the icon 511, the electronic device displays the following Figure 5 The interface 520 shown in (b) includes an "album" control 521, which the user can click. In response to the user clicking the control 521, the electronic device displays the following Figure 5 The interface 530 shown in (c) of FIG. The interface includes a "video" window 531, and the user can click on the window 531. In response to the user clicking on the window 531, the electronic device displays the following Figure 5The interface 540 shown in (d) includes multiple videos, such as video 541, and the user can click on video 541. In response to the user clicking on video 541, the electronic device displays the following Figure 5 The interface 550 shown in (e) of FIG. The interface includes an "Edit" control 551, and the user can click on the control 551. In response to the user clicking on the control 551, the electronic device displays the following Figure 5 In the interface 560 shown in (f) of FIG, the user can edit the video in the interface 560. For example, the user can add a filter to the video using the "Filter" control 563 and add a picture-in-picture using the "Picture-in-Picture" control 562. If the user wants to add a video, the user can also add a video using the control 561. After the user completes the video editing process through the interface 560, the user can preview the edited video using the "Play" control 564 and save the edited video to the gallery 511 using the "Export" control 565.

[0127] Please refer to Figure 6 , Figure 6 This is a schematic diagram of another embodiment of the present application provided by a user opening a clipping interface in an electronic device.

[0128] Please refer to Figure 6 In the interface 610 shown in (a), the interface 610 includes a "Gallery" icon 611, which the user can click. In response to the user clicking the icon 611, the electronic device displays the following Figure 6 The interface 620 shown in (b) includes a "Create" control 621, which the user can click. In response to the user clicking the control 621, the electronic device displays the following Figure 6 The interface 630 shown in (c) of FIG. The interface includes a "Clip" control 631, which the user can click. In response to the user clicking the control 631, the electronic device displays the following Figure 6 The interface 640 shown in (d) includes a "start creating" control 641, and the user can click the control 641. In response to the user clicking the control, the electronic device displays the following Figure 6 The interface 650 shown in (e) of FIG. The interface includes multiple videos and pictures, such as video 651. The user can click on video 651 and click on control 652. In response to the user clicking on control 652, the electronic device displays the following Figure 6In the interface 660 shown in (f) of FIG, the user can edit the video in the interface 660. For example, the user can add a filter to the video through the "Filter" control 663 and add a picture-in-picture through the "Picture-in-Picture" control 662. If the user wants to add a video, the user can also add a video through the control 661. After the user completes the video editing process through the interface 660, the user can preview the edited video through the "Play" control 664 and save the edited video to the gallery 611 through the "Export" control 665.

[0129] The video processing method provided in the embodiment of the present application requires the above Figure 5 and Figure 6 The multiple videos or pictures in the editing interface shown are processed, and the following embodiment introduces how the user adds multiple videos or pictures in the editing interface.

[0130] In one way, users can Figure 5 In the interface 520 shown in (b) of FIG, multiple videos or pictures are selected, so that the electronic device can Figure 5 The interface 560 shown in (f) may display multiple videos or pictures, or the user may Figure 6 Select multiple videos or pictures in the interface 650 shown in (e) in the figure, so that the electronic device can Figure 6 Multiple videos or pictures can be displayed in the interface 660 shown in (f).

[0131] For another method, please refer to Figure 7 , Figure 7 This is a schematic diagram of adding a video or picture in an editing interface provided by an embodiment of the present application.

[0132] Please refer to Figure 7 The "Clip" interface 710 shown in (a) of FIG. 7 shows a video 711. The interface also includes a control 712, and the user can click the control 712. In response to the user clicking 712, the electronic device displays Figure 7 The interface 720 shown in (b) includes multiple videos and pictures, such as video 721. The user can click on the video 721 and the control 722. In response to the user clicking on the control 722, the electronic device displays Figure 7 In the interface 730 shown in (c), the interface 730 displays a video 731 and a video 732, and includes a transition 733 between the video 731 and the video 732. The user can click the transition 733. In response to the user clicking the transition 733, the electronic device displays Figure 7In the interface 740 shown in (d), the interface 740 includes a pop-up window 741, which includes a plurality of transition classification options, such as a "hot" classification option 742, and a plurality of transition style options, such as a "romantic floating" style option 742. The user can select the style option 742 and click the control 744. In response to the user clicking the control 744, the electronic device displays Figure 7 In (e) of FIG. 7 , the interface 750 is shown, and the interface 750 displays a transition 751 set to a “romantic floating” style.

[0133] For another method, please refer to Figure 8 , Figure 8 This is another schematic diagram of adding videos or pictures in the editing interface provided by an embodiment of the present application.

[0134] Please refer to Figure 8 The "Clip" interface 810 shown in (a) of FIG. 8 includes a "play head" 811 and a "picture-in-picture" control 812. The user can click the control 812. In response to the user clicking the control 812, the electronic device displays Figure 8 In the interface 820 shown in (b), the interface 820 includes a "new picture-in-picture" control 821, and the user can click the control 821. In response to the user clicking the control 821, the electronic device displays an interface 830, which includes multiple videos and pictures, such as video 831. The user can select video 831 and click the control 832. In response to the user clicking the control 832, the electronic device displays Figure 8 In the interface 840 shown in (d), a “picture-in-picture” 842 is displayed at the position of the “play head” 841 in the interface 840 .

[0135] In some embodiments, the user can click Figure 8 The “add picture-in-picture” control 821 shown in (b) adds multiple picture-in-pictures in the interface 840 .

[0136] In this embodiment of the application, the user Figure 5 The "Clip" interface 560 shown in (f) of FIG. 1 , or in Figure 6 After the video is edited in the "Edit" interface 660 shown in (f), for example, Figure 7 The operation shown adds multiple videos or pictures to splice multiple videos or pictures, for example, by Figure 8 Shown operation to add a picture in picture, the user can click Figure 5 The "Play" control 564 in the "Clip" interface 560 shown in (f) of FIG. 1 , or Figure 6The "play" control 664 in the "edit" interface 660 shown in (f) previews and plays the edited video, wherein, after the user clicks the "play" control 664, the electronic device can execute the video processing method provided in the embodiment of the present application.

[0137] The above embodiments introduce the application scenarios of the video processing method provided by the embodiments of the present application. The following embodiments introduce the implementation process of the video processing method provided by the embodiments of the present application.

[0138] Please refer to Figure 9 , Figure 9 FIG. 1 is a schematic diagram of a video processing method provided in an embodiment of the present application. The method includes steps 910 to 990.

[0139] Step 910: In response to user operation 1, the electronic device obtains an input video.

[0140] It should be understood that user operation 1 refers to the operation of the user importing the video in the image processing software. For example, user operation 1 can be Figure 5 The user operations shown in (a) to (e) can also be Figure 6 The user operations shown in (a) to (e) can also be performed by the user through Figure 7 The operation of adding a video shown can also be performed by the user through Figure 8 The operation of adding a picture-in-picture is shown, and the embodiment of the present application does not limit this. The user operation can be a click, a slide, a gesture, a voice input, etc., and the embodiment of the present application does not limit this.

[0141] It should also be understood that the input video can be a video on the main video track in the editing interface, such as Figure 7 The video 731 and video 732 shown in (c) can also be the video on the picture-in-picture track in the editing interface, for example, Figure 8 Video 842 is shown in (d).

[0142] Step 920, in response to user operation 2, when the acquired input video is a one-channel video, the electronic device determines whether the one-channel video is a specific video, where the specific video refers to a video for which the current remaining hardware macroblock resources are insufficient to create a hard decoder for decoding.

[0143] It should be understood that user operation 2 may refer to the user's operation of previewing the edited video. Figure 5 Clicking the "play" control 564 in the interface 560 shown in (f) of FIG. 5 may also be a user's Figure 6 Click the "Play" control 664 in the interface 660 shown in (f).

[0144] It should also be understood that, for example, the user Figure 5 or Figure 6 Import a video into the image processing software, and the video can be called a video. Figure 7 The operation can add video in the image processing software, for example, the video after adding is Figure 7 The video 731 and the video 732 shown in (c) can be called two-way video. Figure 8 The operation shown can add a picture-in-picture video in the image processing software, then you can Figure 8 The two videos on the main video track and one video on the picture-in-picture track shown in (d) are called three-way video, etc.

[0145] It can also be understood that since the video processing method provided in the embodiment of the present application is similar in the implementation process of processing two-channel videos, three-channel videos, and more than three-channel videos, the implementation process of processing one-channel video is different from the implementation process of processing two or more channels of video. Therefore, the embodiment of the present application divides the number of input videos into two categories, namely one-channel video and two or more channels of video.

[0146] During implementation, the electronic device may determine whether the video is a specific video based on the video resolution and the currently remaining hardware macroblock resources.

[0147] It should be understood that the resolution of a video refers to the clarity of the video image, and is usually expressed as the number of horizontal pixels multiplied by the number of vertical pixels. Table 1 below lists some common resolutions from low to high.

[0148] Table 1

[0149]

[0150]

[0151] It should also be understood that the number of macroblocks of video or image data that the hardware macroblock resources of an electronic device can process is determined by the hardware device. For example, the total number of macroblocks of video or image data that the hardware macroblock resources of an electronic device can process is 79,632, and each macroblock includes 16×16 pixels. It should be noted that in the following embodiments, unless the number of macroblocks of video or image data that the hardware macroblock resources of an electronic device can process and the number of pixels in a single macroblock are specifically specified, the number of macroblocks refers to 79,632, and each macroblock includes 16×16 pixels.

[0152] For example, assuming that the resolution of the video acquired by the electronic device is 4K resolution (3840×2160), when the input video is one video, the electronic device does not need to occupy hardware macroblock resources to create a hard decoder for other videos. Therefore, the number of macroblocks that can be processed by the current remaining hardware macroblock resources for video or picture data is 79632. Based on the resolution of one video, the electronic device can determine that only (3840×2160) / (16×16)=32400 macroblocks are needed for hard decoding of one video. 32400 is less than 79632. Therefore, the electronic device determines that the current remaining hardware macroblock resources are sufficient to create a hard decoder to decode one video, thereby determining that the one video is not a specific video. Assuming that the resolution of the video obtained by the electronic device is 8K resolution (7680x4320), the electronic device can determine based on the resolution of the video that (7680x4320) / (16×16)=129600 macroblocks are required for hard decoding processing of the video. 129600 is greater than 79632. Therefore, the electronic device determines that the current remaining hardware macroblock resources are insufficient to create a hard decoder to decode the video, thereby determining that the video is a specific video.

[0153] It should be understood that when the hardware macroblock resources of an electronic device can process a total of 79,632 macroblocks of video or image data, and one macroblock includes 16×16 pixels, when the resolution of the video channel obtained by the electronic device is the resolution corresponding to standard definition, high definition, full HD, quad HD, 2K, 4K, or 6K in Table 1 (for ease of description, these resolutions are referred to as 6K and below in this embodiment of the application), the electronic device determines that the current remaining hardware macroblock resources are sufficient to create a hard decoder to decode the video channel based on the resolution of the video and the current remaining hardware macroblock resources, thereby determining that the video channel is not a specific video. When the resolution of the video channel obtained by the electronic device is the resolution corresponding to 8K, 10K, or 12K in Table 1 (for ease of description, these resolutions are referred to as 8K and above in this embodiment of the application), the electronic device determines that the current remaining hardware macroblock resources are insufficient to create a hard decoder to decode the video channel based on the resolution of the video and the current remaining hardware macroblock resources, thereby determining that the video channel is a specific video.

[0154] In step 930 , when the electronic device determines that the video is not a specific video, the electronic device decodes the video based on a hardware decoder.

[0155] For example, please refer to Figure 10 , Figure 10 This is an example diagram of a video provided in an embodiment of the present application. Figure 10The figure shows a video track (e.g., video A) in the editing interface of the image processing software (the editing interface can be referred to the above embodiment and will not be described here in detail) and the start time node (e.g., T1) and end time node (e.g., T2) of the video. When the electronic device determines that the video track is not a specific video, the electronic device can Figure 10 At T1 shown in (a), a hard decoder is created, and then one channel of video is decoded based on the hard decoder. At T2, the hardware macroblock resources occupied by the hard decoder can be released.

[0156] In step 940 , when the electronic device determines that one video channel is a specific video channel, the electronic device decodes the video channel based on the soft decoder.

[0157] For example, when the electronic device determines that a video is a specific video, the electronic device may Figure 10 At T1 shown in (b), a soft decoder that does not occupy hardware macroblock resources is created, and then one channel of video is decoded based on the soft decoder.

[0158] Step 950: When the electronic device determines that the input video is two or more channels of video, it determines whether any one of the two or more channels of video is a specific video. A specific video refers to a video for which the current remaining hardware macroblock resources are insufficient to create a hard decoder for decoding.

[0159] It should be understood that two-channel and above videos include two-channel videos and videos with a greater number than two-channel videos, such as three-channel videos, four-channel videos, and the like.

[0160] It should also be understood that for each of the two or more video channels, the electronic device needs to determine whether the video channel is a specific video through step 950. For any video channel, the implementation method of determining whether the video channel is a specific video can refer to step 920 and will not be repeated here.

[0161] It should be noted that each of the two or more video channels is played in the image processing software in the order of arrangement on the timeline. Therefore, when determining whether a video channel is a specific video, it is necessary to determine it based on the resolution of the video channel and the current remaining hardware macroblock resources. For the meaning of the current remaining hardware macroblock resources, please refer to other embodiments and will not be repeated here.

[0162] Assume that two or more channels of video are two-channel videos, and the two-channel videos include video A and video B. Since the resolutions of video A and video B can be any resolution shown in Table 1, the electronic device needs to determine whether each channel of video in the two-channel video is a specific video. The following four results can be obtained as shown in Table 2:

[0163] Table 2

[0164] Video A Video B Result A1 Non-specific video Non-specific video Result A2 Specific videos Specific videos Result A3 Specific videos Non-specific video Result A4 Non-specific video Specific videos

[0165] In step 960 , when the electronic device determines that the video is not a specific video, the electronic device decodes the video based on a hardware decoder.

[0166] For example, assuming that the electronic device determines through step 950 that video A is a non-specific video shown in Table 2, the electronic device may decode video A based on the hardware decoder. If the electronic device determines through step 950 that video B is a non-specific video shown in Table 2, the electronic device may decode video B based on the hardware decoder.

[0167] The following section introduces the implementation process of step 950 and step 960 in conjunction with result A1 in Table 2.

[0168] For example, please refer to Figure 11 , Figure 11 This is an example diagram provided by an embodiment of the present application in which a specific video does not exist in two-channel videos. Figure 11 The user-set transition shown in (a) is a non-overlapping transition. Video A and Video B are both videos on the main video track. The starting time node of Video A is T1 and the ending time node is T2. The starting time node of Video B is T2 and the ending time node is T3. Assuming that the resolution of Video A is the resolution corresponding to 4K shown in Table 1, then at the starting time node T1 of Video A, the electronic device determines through step 950 that Video A is not a specific video, and then creates a hard decoder for Video A, and decodes Video A based on the hard decoder. At the starting time node T2 of Video B, since Video A will also end at T2, the hard decoder of Video A will release the occupied hardware macroblock resources. Then, at T2, assuming that the resolution of Video B is the resolution corresponding to 4K shown in Table 1, the electronic device determines through step 950 that Video B is not a specific video, and then creates a hard decoder for Video B, and decodes Video B based on the hard decoder. It should be understood that the resolution of video A and video B in this example can also be other resolutions, such as the resolutions corresponding to standard definition, high definition, full HD, quad HD, 2K or 6K shown in Table 1, and the embodiments of the present application are not limited to this.

[0169] For example, Figure 11The transition set by the user shown in (b) is an overlay transition. Video A and Video B are both videos on the main video track. The starting time node of Video A is T1 and the ending time node is T3. The starting time node of Video B is T2 and the ending time node is T4. The starting time node of the overlay transition is T2 and the ending time node is T3. Assuming that the resolution of Video A is the resolution corresponding to 4K shown in Table 1, then at the starting time node T1 of Video A, the electronic device determines through step 950 that Video A is not a specific video, and then creates a hard decoder for Video A, and decodes Video A based on the hard decoder. At the starting time node T2 of Video B, assuming that the resolution of Video B is the resolution corresponding to 4K shown in Table 1, then the electronic device determines through step 950 that Video B is not a specific video, and then creates a hard decoder for Video B, and decodes Video B based on the hard decoder. It should be understood that the resolution of video A and video B in this example can also be other resolutions, such as the resolutions corresponding to standard definition, high definition, full HD, quad HD or 2K shown in Table 1, and the embodiments of the present application are not limited to this.

[0170] For example, Figure 11 The video A shown in (c) is a video on the main video track, and the video B is a video on the picture-in-picture track. The starting time node of video A is T1 and the ending time node is T4. The starting time node of video B is T2 and the ending time node is T3. Assuming that the resolution of video A is the resolution corresponding to 4K shown in Table 1, then at the starting time node T1 of video A, the electronic device determines through step 950 that video A is not a specific video, and then creates a hard decoder for video A, and decodes video A based on the hard decoder. At the starting time node T2 of video B, assuming that the resolution of video B is the resolution corresponding to 4K shown in Table 1, then the electronic device determines through step 950 that video B is not a specific video, and then creates a hard decoder for video B, and decodes video B based on the hard decoder. It should be understood that the resolutions of video A and video B in this example can also be other resolutions, such as the resolutions corresponding to standard definition, high definition, full HD, quad HD or 2K shown in Table 1, and the embodiments of the present application are not limited to this.

[0171] Step 970: When the electronic device determines that the video is a specific video, it first decodes the specific video based on the soft decoder. Secondly, during the process of decoding the specific video based on the soft decoder, if other hard decoders release hardware macroblock resources, it determines whether the released hardware macroblock resources are sufficient to create a hard decoder to decode the specific video.

[0172] It should be understood that the release of hardware macroblock resources by other hardware decoders refers to the release of hardware macroblock resources by the hardware decoders of the videos other than the specific video in the two-channel or above video when the specific video is decoded by the soft decoder. For example, please refer to Figure 12 , Figure 12 This is another example diagram of a specific video existing in two-channel video provided by an embodiment of the present application. Figure 12 In the embodiment, assuming that video A is a non-specific video and video B is a specific video, video A is played before video B. The electronic device can create a hard decoder for video A at T1 and a soft decoder for video B at T2. In the process of video B being decoded by the soft decoder, the electronic device releases the hardware macroblock resources occupied by the hard decoder of video A at T3. Other hardware decoders may refer to the hardware macroblock resources released by the hard decoder of video A.

[0173] It should also be understood that the released hardware macroblock resources refer to the currently remaining hardware macroblock resources. For example, Figure 12 In the figure, the resolutions of video A and video B are both 4K resolution (3840×2160), and the hardware macroblock resources can process a total of 79632 macroblocks of video or picture data. Then at T1, the hardware macroblock resources occupied by the hard decoder of video A created by the electronic device can process (3840×2160) / (16×16)=32400 macroblocks in video A, and the remaining hardware macroblock resources can process 79632-32400=47232 macroblocks in the video or picture data. After the hardware macroblock resources occupied by the hard decoder of video A are released at T3, the current remaining hardware macroblock resources (the released hardware macroblock resources) can process 47232+32400=79632 macroblocks in the video or picture data.

[0174] When decoding a specific video using a soft decoder, the electronic device determines whether the released hardware macroblock resources are sufficient to create a hard decoder for decoding the specific video when other hard decoders release hardware macroblock resources. This can be understood as follows: if the specific video exists in two or more video channels, and the hardware decoders of other videos release hardware macroblock resources while the specific video is being decoded by the soft decoder, then the electronic device determines whether the released hardware macroblock resources are sufficient to create a hard decoder for decoding the specific video. That is, if the hardware decoders of other videos do not release hardware macroblock resources while the specific video is being decoded by the soft decoder (for ease of description, this embodiment of the present application refers to this situation as situation one), and although the hardware decoders of other videos do release hardware macroblock resources while the specific video is being decoded by the soft decoder, the released hardware macroblock resources are insufficient to create a hardware decoder (for ease of description, this embodiment of the present application refers to this situation as situation two), then the electronic device determines in step 970 that the released hardware macroblock resources are insufficient to create a hard decoder for decoding the specific video, and then executes step 980. If the hardware macroblock resources of other videos are released by the hard decoder during the decoding of a specific video by the soft decoder, and the released hardware macroblock resources are sufficient to create a hard decoder (for the sake of convenience of description, this embodiment of the present application refers to this situation as situation three), then the electronic device determines through step 970 that the released hardware macroblock resources are sufficient to create a hard decoder to decode the specific video, and the electronic device executes step 990.

[0175] In step 980 , the electronic device decodes the specific video based on the soft decoder when determining that the released hardware macroblock resources are insufficient to create a hard decoder to decode the specific video.

[0176] It should be understood that when the specific video belongs to the above-mentioned case 1 and case 2, the electronic device decodes the specific video based on the soft decoder.

[0177] The following section introduces the implementation process of step 980 in conjunction with results A2 to A4 shown in Table 2.

[0178] For example, please refer to Figure 13 , Figure 13 This is another example diagram of a specific video existing in two-channel video provided by an embodiment of the present application.

[0179] Figure 13 The specific video shown in (a) belongs to the above situation 1, and the result of the electronic device determining the two-way video in step 950 is the result A2 in the above table 2. For example, assuming Figure 13The resolution of video A shown in (a) is the resolution corresponding to 8K shown in Table 1. Then, at T1, the electronic device determines that video A is a specific video (such as the result A2 of video A shown in Table 2) through step 950. Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 970, and decodes video A based on the soft decoder. And when video A is decoded by the soft decoder, there is no hardware macroblock resource released by the hard decoder of other videos (such as Figure 13 (a) shows that there is no hard decoder of other videos between T1 and T2 to release hardware macroblock resources), which belongs to the above-mentioned situation one. Therefore, the electronic device determines through step 970 that the released hardware macroblock resources are insufficient to create a hard decoder, and then the electronic device continues to decode video A with the soft decoder until the decoding is completed at T2.

[0180] Then at T2, assuming Figure 13 The resolution of video B shown in (a) is the resolution corresponding to 8K shown in Table 1. Then the electronic device determines that video B is a specific video (such as the result A2 of video B shown in Table 2) through step 950. Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 970, and decodes video B based on the soft decoder. And when video B is decoded by the soft decoder, there is no hardware macroblock resource released by the hard decoder of other videos (such as Figure 13 (a) shows that there are no hardware macroblock resources released by the hardware decoder for other videos between T2 and T4, which belongs to the first situation described above. Therefore, step 970 determines that the released hardware macroblock resources are insufficient to create a hardware decoder. Then, the electronic device continues to decode video B using the soft decoder until decoding ends at T3. In other embodiments, the resolutions of videos A and B shown in this example can also be other resolutions, such as 8K and above as shown in Table 1.

[0181] Figure 13 The specific video shown in (b) belongs to the above situation 1, and the result of the electronic device determining the two-way video in step 950 is the result A3 in the above table 2. For example, assuming Figure 13 The resolution of video A shown in (b) is the resolution corresponding to 8K shown in Table 1. Then the electronic device determines at step 950 at T1 that video A is a specific video (such as the result A3 of video A shown in Table 2). Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 970, and decodes video A based on the soft decoder. And when video A is decoded by the soft decoder, there is no hardware macroblock resource released by the hard decoder of other videos (such as Figure 13(b) shows that there is no hard decoder of other videos between T1 and T3 to release hardware macroblock resources), which belongs to the above-mentioned situation one. Therefore, the electronic device determines through step 970 that the released hardware macroblock resources are insufficient to create a hard decoder, and then the electronic device continues to decode video A with the soft decoder until it ends at T3.

[0182] Then, at T2, assuming that the resolution of video B is the 4K resolution shown in Table 1, the electronic device determines through step 950 that video B is not a specific video (such as result A3 of video B shown in Table 2). Then, at T2, the electronic device creates a hard decoder for video B through step 960, and then decodes video B based on the hard decoder until it ends at T4. In other embodiments, video A and video B can also have other resolutions shown in Table 1. For example, in this example, the resolution of video A can be 8K or higher resolutions shown in Table 1. The resolution of video B can be 6K or lower resolutions shown in Table 1.

[0183] Figure 13 The specific video shown in (c) belongs to the above situation 1, and the result of the electronic device determining the two-way video in step 950 is the result A4 in the above table 2. For another example, assuming Figure 13 The resolution of video A shown in (c) is the resolution corresponding to 6K shown in Table 1, then the electronic device determines through step 950 that video A is not a specific video (such as result A4 of video A shown in Table 2), then the electronic device creates a hard decoder for video A through step 960 at T1, and then decodes video A based on the hard decoder until the decoding is completed at T4, and releases the hardware macroblock resources occupied by the hard decoder of video A.

[0184] Then, at T2, assuming that the resolution of video B is the 4K resolution shown in Table 1, the electronic device determines that video B is a specific video (such as the result A4 of video B shown in Table 2) through step 950. Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 970, and decodes video B based on the soft decoder. In addition, when video B is decoded by the soft decoder, there is no hardware macroblock resource released by the hardware decoder of other videos (such as Figure 13(c) shows that there is no hardware macroblock resource released by the hard decoder of other videos between T2 and T3), which belongs to the above-mentioned situation one. Therefore, the electronic device determines through step 970 that the released hardware macroblock resources are insufficient to create a hard decoder, and then the electronic device continues to decode video B with the soft decoder until it ends at T3. In other embodiments, video A and video B can also be other resolutions in Table 1. For example, when the resolution of video A is the resolution corresponding to 6K shown in Table 1, the resolution of video B shown in this example can also be the resolution of 6K and below in Table 1. When the resolution of video B is the resolution corresponding to 6K shown in Table 1, the resolution of video A can be the resolution of 6K and below in Table 1.

[0185] Figure 13 The specific video shown in (d) belongs to the above situation 2, and the electronic device determines the result of the two-way video in step 950 as result A4 in the above table 2. For example: Figure 13 The resolution of video A shown in (d) is the resolution corresponding to 4K shown in Table 1, then the electronic device determines through step 950 that video A is not a specific video (such as result A4 of video A shown in Table 2), then the electronic device creates a hard decoder for video A through step 960 at T1, and then decodes video A based on the hard decoder until the decoding is completed at T2, and releases the hardware macroblock resources occupied by the hard decoder of video A.

[0186] Then at T2, assuming that the resolution of video B is the resolution corresponding to 8K shown in Table 1, the electronic device determines that video B is a specific video (such as the result A4 of video B shown in Table 2) through step 950, and thus the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 970, and decodes video B based on the soft decoder. And during the decoding process of video B by the soft decoder, although there are hardware macroblock resources released by the hardware decoder of other videos (such as Figure 13 (d) shows that at T2, the hardware decoder of video A releases the hardware macroblock resources. However, the released hardware macroblock resources are insufficient to create a hard decoder, which belongs to the second situation described above. Therefore, the electronic device determines at T2 through step 970 that the released hardware macroblock resources are insufficient to create a hard decoder. Then, the electronic device continues to decode video B with the soft decoder until it ends at T3. In other embodiments, the resolutions of video A and video B shown in this example can also be other resolutions. For example, video A can be a resolution of 6K or below as shown in Table 1, and video B can be a resolution of 8K or above as shown in Table 1.

[0187] In step 990 , the electronic device decodes the specific video based on the soft decoder and the hard decoder when determining that the released hardware macroblock resources are sufficient to create a hard decoder to decode the specific video.

[0188] It should be understood that when a specific video meets the above-mentioned third situation, the electronic device can decode the specific video based on the soft decoder and the hard decoder.

[0189] The following section introduces the implementation process of step 980 in conjunction with result A4 shown in Table 2.

[0190] For example, please refer to Figure 14 , Figure 14 This is another example diagram of a specific video existing in two-channel video provided by an embodiment of the present application. Figure 14 The specific video shown in FIG has the above situation three, and the result of the electronic device determining the two-way video through step 950 is result A4 in the above Table 2.

[0191] for example, Figure 14 Video A and Video B shown in (a) are videos on the main track in the image processing software. Video A is played before Video B, and Video A ends playing before Video B. Assuming that the resolution of Video A is the resolution corresponding to 4K shown in Table 1, the electronic device determines through step 950 that Video A is not a specific video (such as the result A4 of Video A shown in Table 2), and thus the electronic device creates a hard decoder for Video A through step 960 at T1, and then decodes Video A based on the hard decoder (such as Figure 14 (a) shows the hard decoding of video A between T1 and T3).

[0192] Then, at T2, assuming that the resolution of video B is the resolution corresponding to 6K shown in Table 1, the electronic device determines that video B is a specific video (such as the result A4 of video B shown in Table 2) through step 950. Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 970, and decodes video B based on the soft decoder (such as Figure 14 (a) in the figure shows the soft decoding of video B between T2 and T4). Later, when video B is decoded by the soft decoder, the hard decoder of video A releases the hardware macroblock resources at T3. The released hardware macroblock resources are sufficient to create a hard decoder, which belongs to the third situation mentioned above. Therefore, the electronic device determines at T3 through step 970 that the released hardware macroblock resources are sufficient to create a hard decoder. Then the electronic device creates a hard decoder for video B at T3 through step 990. Since it takes a certain amount of time to create a hard decoder, the electronic device can consider that the hard decoder of video B has been created at a time node TA with a preset time interval from T3 (for example Figure 14(a) shows that the hard decoding preparation stage of video B between T3 and TA is the stage of creating a hard decoder. Between T3 and TA, video B is still decoded based on the soft decoder). After TA, the electronic device decodes video B through the hard decoder. In other embodiments, video A and video B can also be other resolutions in Table 1. For example, when the resolution of video B is the resolution corresponding to 6K shown in Table 1, the resolution of video A shown in this example can also be the resolution of 6K and below in Table 1. When the resolution of video A is the resolution corresponding to 6K shown in Table 1, the resolution of video B can be the resolution of 6K and below in Table 1.

[0193] For example, Figure 14 Video A shown in (b) is a video on the main video track in the image processing software, and video B is a video on the picture-in-picture track in the image processing software. Video A plays before video B, and video A finishes playing before video B. Assume Figure 14 The resolution of video A and video B shown in (b) is Figure 14 The video A and video B shown in (a) have the same resolution, so the video processing method provided by the embodiment of the present application is Figure 14 The implementation process in the scenario shown in (b) can be referred to Figure 14 The implementation process in the scenario shown in (a) will not be repeated here.

[0194] In the embodiment of the present application, when the electronic device determines that the current remaining hardware macroblock resources are insufficient to create a hard decoder to decode the video or picture, it can first create a soft decoder that does not occupy the hardware macroblock resources to decode the video or picture (for example, Figure 14 The video B is soft-decoded between T2 and T3 as shown), and secondly, after the hardware decoder completes decoding and releases the hardware macroblock resources (e.g. Figure 14 The hardware decoder of the video A shown in FIG3 releases the hardware macroblock resources at T3, and creates a hardware decoder based on the released hardware macroblock resources (eg Figure 14 As shown, a hard decoder is created between T3 and TA), and a video or picture is decoded by the hard decoder (eg Figure 14 As shown, video B is decoded by the hard decoder of video B after TA). In this way, when the hardware macroblock resources are insufficient to create a hard decoder, the video can be decoded by combining the soft decoder and the hard decoder. This avoids the failure to create a hard decoder, which may cause the image processing software to have a black screen and flash back, or cause the electronic device to have a system crash or error. This ensures the continuity of users when editing videos or pictures and improves the user experience.

[0195] In a related solution, electronic devices decode videos using a soft decoder when the hardware macroblock resources are insufficient to create a hard decoder. For example, see Figure 15 , Figure 15 The following is an example diagram of a related solution. Figure 15 The scene shown is similar to Figure 14 The same scenario is shown in Figure 15 In the example, if the electronic device determines at T2 that the currently remaining hardware macroblock resources are insufficient to create a hard decoder, then it will decode the entire video B based on the soft decoder. However, since the soft decoder is a decoding program running on the CPU and does not rely on specialized hardware, it may consume more CPU resources, resulting in slower decoding speed and increased power consumption. In addition, the performance of the soft decoder is limited to a certain extent by the processing power of the CPU, and the video playback is not as smooth as hard decoding, resulting in freezes and delays in the video decoded by the soft decoder. In the embodiment of the present application, when it is determined that the currently remaining hardware macroblock resources are insufficient to create a hard decoder to decode the video or picture, the video is decoded together by combining the soft decoder and the hard decoder (for example, Figure 14 Video B is decoded based on the soft decoder between T2 and TA, and video B is decoded based on the hard decoder between TA and T4). Since the time for decoding the video based on the soft decoder is shortened, it avoids consuming more CPU resources, thereby increasing the speed of decoding the video, reducing power consumption, and improving decoding performance. It also reduces the probability of freezes and delays in the decoded video, allowing users to have a better experience when previewing the decoded video.

[0196] Furthermore, when the embodiments of the present application are applied to a scene with superimposed transitions, if the hardware macroblock resources of the electronic device are insufficient to create a hard decoder, the electronic device can only perform soft decoding processing on the video during the superimposed transition and the preset duration, and perform hard decoding processing on the video during the rest of the duration. For example: Figure 14 In (a), T2 to T3 is the duration of the superimposed transition, T3 to TA is the preset duration, and TA to T4 is another duration. The electronic device performs soft decoding on video B between T2 and TA, and hard decoding on video B between TA and T4. Since the duration of the superimposed transition is usually short, and the preset duration is 200 milliseconds to 1000 milliseconds, the duration is short, so the duration of soft decoding of the video is relatively short, and the duration of hard decoding of the video is relatively long. Compared with the related scheme of soft decoding of the entire video when the hardware macroblock resources are insufficient to create a hard decoder, the duration of video decoding based on the soft decoder can be shortened, avoiding the consumption of more CPU resources, thereby improving the speed of decoding the video, reducing power consumption, and improving decoding performance, reducing the probability of freezes and delays in the decoded video, and allowing users to have a better experience when previewing the decoded video.

[0197] The above embodiments are combined Figures 9 to 15 The video processing method provided in the embodiment of the present application is introduced. However, the above embodiment uses one-channel video and two-channel video as examples. The following embodiment introduces the video processing method provided in the embodiment of the present application using three-channel video as an example.

[0198] For example, please refer to Figure 16 , Figure 16 This is a schematic diagram of a three-channel video processing method provided by an embodiment of the present application, including steps 1010 to 1060.

[0199] Step 1010: In response to user operation 1, the electronic device obtains an input video.

[0200] The implementation of this step can refer to the implementation of step 910 above, and will not be repeated here.

[0201] Step 1020, in response to user operation 2, the electronic device, when obtaining the input video as three-channel video, determines whether any one of the three-channel video is a specific video. The specific video refers to a video for which the current remaining hardware macroblock resources are insufficient to create a hard decoder for decoding.

[0202] It should be understood that for each of the three video channels, the electronic device needs to determine whether the video channel is a specific video in step 1020. The electronic device determines whether the three video channels are specific videos in step 1020, and can obtain the eight results shown in the following Table 3:

[0203] Table 3

[0204] Video A Video C Video B Result B1 Non-specific video Non-specific video Non-specific video Result B2 Specific videos Specific videos Specific videos Result B3 Specific videos Specific videos Non-specific video Result B4 Specific videos Non-specific video Non-specific video Result B5 Specific videos Non-specific video Specific videos Result B6 Non-specific video Specific videos Specific videos Result B7 Non-specific video Specific videos Non-specific video Result B8 Non-specific video Non-specific video Specific videos

[0205] In step 1030 , when the electronic device determines that the video is not a specific video, the electronic device decodes the video based on a hardware decoder.

[0206] Exemplarily, assuming that the electronic device determines through step 1020 that video A, video B, or video C is a non-specific video shown in Table 3, the electronic device may decode video A, video B, or video C based on a hard decoder.

[0207] The following section introduces the implementation process of step 1020 and step 1030 in conjunction with result B1 in Table 3.

[0208] For example, please refer to Figure 17 , Figure 17 This is an example diagram provided by an embodiment of the present application in which a specific video does not exist in three-channel video. Figure 17The user-set transition shown in (a) is an overlay transition. Video A and Video B are both videos on the main video track in the image processing software. Video C is a video on the picture-in-picture track. The start time node of Video A is T1 and the end time node is T4. The start time node of Video B is T3 and the end time node is T6. The start time node of Video C is T2 and the end time node is T4. Video A plays first, followed by Video C, and finally Video B. Video A ends first, followed by Video C, and finally Video B. Assuming that the resolution of Video A is 2K as shown in Table 1, then at the start time node T1 of Video A, the electronic device determines that Video A is not a specific video through step 1020. Thereafter, the electronic device creates a hardware decoder for Video A through step 1030, decodes Video A based on the hardware decoder, and releases the hardware decoder resources occupied by Video A at the end time node T4 of Video A.

[0209] At the start node T2 of video C, assuming that the resolution of video C is 2K as shown in Table 1, the electronic device determines through step 1020 that video C is not a specific video. Thereafter, the electronic device creates a hard decoder for video C through step 1030, decodes video C based on the hard decoder, and releases the hardware decoder resources occupied by video C at the end time node T5 of video C.

[0210] At the start time T3 of video B, assuming that the resolution of video B is 2K as shown in Table 1, the electronic device determines in step 1020 that video B is not a specific video. Thereafter, the electronic device creates a hardware decoder for video B in step 1030, decodes video B based on the hardware decoder, and releases the hardware decoder resources occupied by video B at the end time T6 of video B. In other embodiments, the resolutions of video A, video B, and video C in this example may also be other resolutions, such as 2K and below as shown in Table 1.

[0211] For example, Figure 17 The video A and video B shown in (b) are videos on the main video track in the image processing software, and video C is a video on the picture-in-picture track. The time nodes of video A, video B and video C are the same as those of video A, video B and video C. Figure 17 The time nodes shown in (a) are the same. Assume Figure 17 The resolution of video A, video B and video C shown in (b) is the same as Figure 17 The video A, video B, and video C shown in (a) have the same resolution. Therefore, the video processing method provided by the embodiment of the present application is Figure 17 The implementation process in the scenario shown in (b) can be referred to Figure 17 The implementation process in the scenario shown in (a) will not be repeated here.

[0212] Step 1040: When the electronic device determines that the video is a specific video, it first decodes the specific video based on the soft decoder. Secondly, during the process of decoding the specific video based on the soft decoder, if other hard decoders release hardware macroblock resources, it determines whether the released hardware macroblock resources are sufficient to create a hard decoder to decode the specific video.

[0213] It should be understood that during the process of decoding a specific video using a soft decoder, the specific video may also fall into Case 1, Case 2, or Case 3 mentioned in step 970. If the specific video falls into Case 1 or Case 2, the electronic device determines that the released hardware macroblock resources are insufficient to create a hard decoder to decode the specific video, and then executes step 1050. If the specific video falls into Case 3, the electronic device determines that the released hardware macroblock resources are sufficient to create a hard decoder to decode the specific video, and then executes step 1060.

[0214] Step 1050 : When the electronic device determines that the released hardware macroblock resources are insufficient to create a hard decoder to decode the specific video, the electronic device decodes the specific video based on the soft decoder.

[0215] The following section introduces the implementation process of step 1050 in conjunction with the results B2 to B4 shown in Table 3.

[0216] Please refer to Figure 18 , Figure 18 This is an example diagram of a three-channel video with a specific video present in an embodiment of the present application.

[0217] Figure 18 The specific video shown in (a) belongs to the above situation 1, and the electronic device determines the result of the three-way video in step 1020 as the result B2 in the above table 3. For example, assuming Figure 18 The video A shown in (a) is the resolution corresponding to 8K shown in Table 1. Then, the electronic device determines that the video A is a specific video (such as the result B2 of the video A shown in Table 3) through step 1020 at T1. Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 1040, and decodes the video A based on the soft decoder. And when the video A is decoded by the soft decoder, there is no other video's hard decoder releasing hardware macroblock resources (such as in Figure 18 (a) shows that there is no hard decoder of other videos between T1 and T4 to release hardware macroblock resources), which belongs to the above-mentioned situation one. Therefore, the electronic device determines through step 1040 that the released hardware macroblock resources are insufficient to create a hard decoder, and then the electronic device continues to decode video A with the soft decoder until the decoding is completed at T4.

[0218] Then at T2, assuming that the resolution of video C is the resolution corresponding to 8K shown in Table 1, the electronic device determines that video C is a specific video (such as the result B2 of video C shown in Table 3) through step 1020, and thus the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 1040, and decodes video C based on the soft decoder. In addition, when video C is decoded by the soft decoder, there is no hardware macroblock resource released by the hard decoder of other videos (such as Figure 18 (a) shows that there is no hardware macroblock resource released by the hard decoder of other videos between T2 and T5, which belongs to the above-mentioned situation 1. Therefore, it is determined through step 1040 that the released hardware macroblock resources are insufficient to create a hard decoder. Then the electronic device continues to decode video C with the soft decoder until the decoding is completed at T5.

[0219] Then at T3, assuming that the resolution of video C is the resolution corresponding to 8K shown in Table 1, the electronic device determines that video B is a specific video (such as the result B2 of video B shown in Table 3) through step 1020. Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 1040, and decodes video B based on the soft decoder. In addition, when video B is decoded by the soft decoder, there is no hardware macroblock resource released by the hard decoder of other videos (such as Figure 18 (a) shows that there are no hardware macroblock resources released by the hardware decoder for other videos between T3 and T6, which corresponds to the first scenario described above. Therefore, step 1040 determines that the released hardware macroblock resources are insufficient to create a hardware decoder. The electronic device then continues to decode video B using the soft decoder until decoding ends at T6. In other embodiments, the resolutions of videos A, B, and C shown in this example can also be other resolutions, such as 8K and above as shown in Table 1.

[0220] Figure 18 The specific video shown in (b) belongs to the above situation 1, and the electronic device determines the result of the three-way video in step 1020 as result B3 in the above table 3. For example, assuming Figure 18 The resolutions of video A and video C shown in (a) are both the resolutions corresponding to 8K shown in Table 1. The implementation process of the electronic device processing video A and video C through steps 1020 and 1040 can refer to Figure 18 The implementation process of the electronic device shown in (a) processing video A and video C through steps 1020 and 1040 will not be repeated here.

[0221] Then, at T3, assuming that the resolution of video B is the resolution corresponding to 4K shown in Table 1, the electronic device determines through step 1020 that video B is not a specific video (such as the result B3 of video B shown in Table 3). Then, the electronic device creates a hard decoder for video B through step 1030 at T3, and then decodes video B based on the hard decoder until the decoding is completed at T6, and releases the hardware macroblock resources occupied by the hard decoder of video B. In other embodiments, the resolutions of video A, video B, and video C shown in this example can also be other resolutions. For example, video A and video C can be 8K or higher resolutions shown in Table 1. Video B can be 6K or lower resolutions shown in Table 1.

[0222] Figure 18 The specific video shown in (c) belongs to the above situation 1, and the electronic device determines the result of the three-way video in step 1020 as result B4 in Table 3. For example, assuming Figure 18 The resolution of video A shown in (a) is the resolution corresponding to 8K shown in Table 1. The implementation process of the electronic device processing video A through steps 1020 and 1040 can refer to Figure 18 The implementation process of the electronic device shown in (a) processing video A through steps 1020 and 1040 will not be repeated here.

[0223] Then at T2, assuming that the resolution of video C is the resolution corresponding to 4K shown in Table 1, the electronic device determines through step 1020 that video C is not a specific video (such as the result B4 of video C shown in Table 3), then the electronic device creates a hard decoder for video C through step 1030 at T2, and then decodes video C based on the hard decoder until the decoding is completed at T5, and releases the hardware macroblock resources occupied by the hard decoder of video C.

[0224] Then, at T3, assuming that the resolution of video B is the resolution corresponding to 4K shown in Table 1, the electronic device determines through step 1020 that video B is not a specific video (such as the result B4 of video B shown in Table 3). Then, the electronic device creates a hard decoder for video B through step 1030 at T3, and then decodes video B based on the hard decoder until the decoding is completed at T6, and releases the hardware macroblock resources occupied by the hard decoder of video B. In other embodiments, the resolutions of videos A, B, and C shown in this example can also be other resolutions. For example, video A can be a resolution of 8K or above shown in Table 1. Videos B and C can be a resolution of 4K or below shown in Table 1.

[0225] Figure 18The specific video shown in (d) belongs to the above situation 2, and the electronic device determines the result of the three-way video in step 1020 as result B5 in the above table 3. For example, assuming Figure 18 The resolution of video A shown in (a) is the resolution corresponding to 8K shown in Table 1. The implementation process of the electronic device processing video A through steps 1020 and 1040 can refer to Figure 18 The implementation process of the electronic device shown in (a) processing video A through steps 1020 and 1040 will not be repeated here.

[0226] Then at T2, assuming that the resolution of video C is the resolution corresponding to 4K shown in Table 1, the electronic device determines through step 1020 that video C is not a specific video (such as the result B5 of video C shown in Table 3), then the electronic device creates a hard decoder for video C through step 1030 at T2, and then decodes video C based on the hard decoder until the decoding is completed at T5, and releases the hardware macroblock resources occupied by the hard decoder of video C.

[0227] Then, at T3, assuming the resolution of video B is the 8K resolution shown in Table 1, the electronic device determines at T3, through step 1020, that video B is a specific video (e.g., result B5 for video B shown in Table 3). Therefore, the electronic device first creates a soft decoder for video B, which does not occupy hardware macroblock resources, in step 1040, and decodes video B using the soft decoder. Furthermore, while video B is being decoded by the soft decoder, at T5, the hardware decoder of video C completes decoding and releases the hardware macroblock resources occupied by the hardware decoder. However, the released hardware macroblock resources are insufficient to create the hardware decoder for video B, which corresponds to the second scenario described above. Therefore, the electronic device determines at step 1040 that the released hardware macroblock resources are insufficient to create the hardware decoder. The electronic device then continues decoding video B using the soft decoder until decoding is completed at T6. In other embodiments, the resolutions of videos A, B, and C shown in this example can also be other resolutions. For example, videos A and B can have resolutions of 8K or higher as shown in Table 1. Video C can have resolutions of 6K or lower as shown in Table 1.

[0228] In step 1060 , the electronic device decodes the specific video based on the soft decoder and the hard decoder when determining that the released hardware macroblock resources are sufficient to create a hard decoder to decode the specific video.

[0229] It should be understood that when a specific video meets the above-mentioned third situation, the electronic device can decode the specific video based on the soft decoder and the hard decoder.

[0230] The following section introduces the implementation process of step 1060 in conjunction with results B5 to B8 shown in Table 3.

[0231] Please refer to Figure 19 , Figure 19 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application. Figure 19 The specific video shown belongs to the above situation three, and the result of the electronic device determining the three-way video in step 1020 is result B5 in the above Table 3.

[0232] Figure 19 The specific video shown in (a) belongs to the above situation 1 and situation 3, and the electronic device determines the result of the three-way video in step 1020 as result B5 in the above table 3. For example, assuming Figure 19 The resolution of video A shown in (a) is the resolution corresponding to 8K shown in Table 1. Then, the electronic device determines that video A is a specific video (such as the result B5 of video A shown in Table 3) through step 1020 at T1. Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 1040, and decodes video A based on the soft decoder. And when video A is decoded by the soft decoder, there is no hardware macroblock resource released by the hard decoder of other videos (such as Figure 19 (a) shows that there is no hard decoder of other videos between T1 and T4 to release hardware macroblock resources), which belongs to the above-mentioned situation one. Therefore, the electronic device determines through step 1040 that the released hardware macroblock resources are insufficient to create a hard decoder, and then the electronic device continues to decode video A with the soft decoder until the decoding is completed at T4.

[0233] Then at T2, assuming that the resolution of video C is the resolution corresponding to 6K shown in Table 1, the electronic device determines through step 1020 that video C is not a specific video (such as the result B5 of video C shown in Table 3), then the electronic device creates a hard decoder for video C through step 1030 at T2, and then decodes video C based on the hard decoder until the decoding is completed at T5, and releases the hardware macroblock resources occupied by the hard decoder of video C.

[0234] Then, at T3, assuming that the resolution of video B is the resolution corresponding to 6K shown in Table 1, the electronic device determines in step 1020 that video B is a specific video (such as the result B5 of video B shown in Table 3), and thus the electronic device first creates a soft decoder that does not occupy hardware macroblock resources in step 1040, and decodes video B based on the soft decoder (such as Figure 19(a) in the figure shows the soft decoding of video B between T3 and T5). Later, during the decoding of video B by the soft decoder, the hard decoder of video C releases the hardware macroblock resources at T5. The released hardware macroblock resources are sufficient to create the hard decoder of video B, which belongs to the third situation mentioned above. Therefore, the electronic device determines at T5 through step 1040 that the released hardware macroblock resources are sufficient to create the hard decoder. Then the electronic device creates the hard decoder of video B at T5 through step 1060. Since it takes a certain amount of time to create a hard decoder, the electronic device can consider that the hard decoder of video B has been created at the time node TA with a preset time interval from T5 (for example Figure 19 (a) in FIG. 5 shows the hard decoding preparation stage of video B between T5 and TA, i.e., the stage of creating a hard decoder. Between T5 and TA, video B is still decoded based on the soft decoder. After TA, the electronic device decodes video B using the hard decoder.

[0235] In other embodiments, video A, video B, and video C may also have other resolutions shown in Table 1. For example, if the resolution of video A is 8K or higher as shown in Table 1, and if the resolution of video B is the resolution corresponding to 6K shown in Table 1, the resolution of video C shown in this example may also be 6K or lower as shown in Table 1. When the resolution of video C is the resolution corresponding to 6K shown in Table 1, the resolution of video B may be 6K or lower as shown in Table 1.

[0236] Figure 19 The specific video shown in (b) belongs to the above situation 1 and situation 3, and the electronic device determines the result of the three-way video in step 1020 as result B5 in Table 3. Figure 19 The resolutions of video A, video B, and video C shown in (b) are the same as those of Figure 19 The video A, video B, and video C shown in (a) have the same resolution. Therefore, the video processing method provided by the embodiment of the present application is Figure 19 The implementation process in the scenario shown in (b) can be referred to Figure 19 The implementation process in the scenario shown in (a) will not be repeated here.

[0237] Please refer to Figure 20 , Figure 20 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application. Figure 20 The specific video shown belongs to the above situation 3, and the electronic device determines the result of the three-way video in step 1020 as result B6 in the above table 3. For example, assuming Figure 20The resolution of video A shown in (a) is the resolution corresponding to 6K shown in Table 1, then the electronic device determines through step 1020 at T1 that video A is not a specific video (such as result B6 of video A shown in Table 3), then the electronic device creates a hard decoder for video A through step 1030 at T1, and then decodes video A based on the hard decoder until the decoding is completed at T4, and releases the hardware macroblock resources occupied by the hard decoder of video A.

[0238] Then, at T2, assuming that the resolution of video C is the 4K resolution shown in Table 1, the electronic device determines in step 1020 that video C is a specific video (such as the result B6 of video C shown in Table 3). Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources in step 1040, and decodes video C based on the soft decoder (such as Figure 20 (a) in the figure shows the soft decoding of video C between T2 and T4). Later, when video C is decoded by the soft decoder, the hard decoder of video A releases the hardware macroblock resources at T4. The released hardware macroblock resources are sufficient to create the hard decoder of video B, which belongs to the third situation mentioned above. Therefore, the electronic device determines at T4 through step 1040 that the released hardware macroblock resources are sufficient to create the hard decoder. Then the electronic device creates the hard decoder of video C at T4 through step 1060. Since it takes a certain amount of time to create a hard decoder, the electronic device can consider that the hard decoder of video C has been created at the time node TA with a preset time interval from T4 (for example Figure 20 (a) in FIG. 4 shows the hard decoding preparation stage of video C between T4 and TA, i.e., the stage of creating a hard decoder. Between T4 and TA, video C is still decoded based on the soft decoder. Between TA and T5, the electronic device decodes video C using the hard decoder.

[0239] Then, at T3, assuming that the resolution of video B is the 4K resolution shown in Table 1, the electronic device determines in step 1020 that video B is a specific video (such as the result B6 of video B shown in Table 3). Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources in step 1040, and decodes video B based on the soft decoder (such as Figure 20(a) in the figure shows the soft decoding of video B between T3 and T4). Later, during the decoding of video B by the soft decoder, the hard decoder of video A releases the hardware macroblock resources at T4. The released hardware macroblock resources are sufficient to create the hard decoder of video C and the hard decoder of video B, which belongs to the above situation three. Therefore, the electronic device determines at T4 through step 1040 that the released hardware macroblock resources are sufficient to create the hard decoder. Then the electronic device creates the hard decoder of video B at T4 through step 1060. Since it takes a certain amount of time to create a hard decoder, the electronic device can consider that the hard decoder of video B has been created at the time node TA with a preset time interval from T4 (for example Figure 20 (a) in FIG. 5 shows the hard decoding preparation stage of video B between T5 and T1, i.e., the stage of creating a hard decoder. Between T5 and T1, video B is still decoded based on the soft decoder. Between T1 and T6, the electronic device decodes video B using the hard decoder.

[0240] In other embodiments, video A, video B, and video C may also be other resolutions in Table 1. For example, the resolution of video A is the resolution corresponding to 5K shown in Table 1, and the resolutions of video B and video C may be 5K and below in Table 1.

[0241] For example, suppose Figure 20 The resolution of video A shown in (b) is the resolution corresponding to 6K shown in Table 1. Then the processing process of video A by the electronic device at T1 can refer to Figure 20 The processing of video A at T1 in (a) will not be repeated here.

[0242] Then at T2, assuming that the resolution of video C is the resolution corresponding to 6K shown in Table 1, the processing process of the electronic device on video C at T2 can refer to Figure 20 The processing of video B at T2 in (a) will not be repeated here.

[0243] Then, at T3, assuming that the resolution of video B is the resolution corresponding to 6K shown in Table 1, the electronic device determines in step 1020 that video B is a specific video (such as the result B6 of video B shown in Table 3). Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources in step 1040, and decodes video B based on the soft decoder (such as Figure 20(a) shows the soft decoding of video B between T3 and T4). Later, when video B is decoded by the soft decoder, although the hard decoder of video A releases the hardware macroblock resources at T4, the released hardware macroblock resources are only sufficient to create the hard decoder of video C, and are not sufficient to create the hard decoder of video B. This belongs to the second situation mentioned above. Therefore, the electronic device determines through step 1040 at T4 that the released hardware macroblock resources are not sufficient to create the hard decoder. Then the electronic device continues to decode video B with the soft decoder. In other embodiments, video A, video B, and video C can also be other resolutions in Table 1. For example, video A, video B, and video C can all be 5K resolution.

[0244] For example, suppose Figure 20 The resolution of video A shown in (c) is the resolution corresponding to 6K shown in Table 1. Then the processing process of video A by the electronic device at T1 can refer to Figure 20 The processing of video A at T1 in (a) will not be repeated here.

[0245] Then, at T2, assuming the resolution of Video C is the 8K resolution shown in Table 1, the electronic device determines, in step 1020, that Video C is a specific video (such as result B6 for Video C shown in Table 3). Therefore, in step 1040, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources and decodes Video C using the soft decoder. Later, while Video C is being decoded by the soft decoder, the hardware decoder of Video A releases hardware macroblock resources at T4. However, the released hardware macroblock resources are insufficient to create the hardware decoder for Video C, representing the second scenario described above. Therefore, at T4, the electronic device determines, in step 1040, that the released hardware macroblock resources are insufficient to create the hardware decoder for Video C. The electronic device then continues decoding Video C using the soft decoder.

[0246] Then, at T3, assuming that the resolution of video B is the resolution corresponding to 6K shown in Table 1, assuming that the resolution of video B is the resolution corresponding to 4K shown in Table 1, the electronic device determines in step 1020 that video B is a specific video (such as the result B6 of video B shown in Table 3), and thus the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 1040, and decodes video B based on the soft decoder (such as Figure 20(a) shows the soft decoding of video B between T3 and T4). Later, during the decoding of video B by the soft decoder, the hard decoder of video A releases the hardware macroblock resources at T4. Although the released hardware macroblock resources are not enough to create the hard decoder of video C, they are enough to create the hard decoder of video B, which belongs to the third situation mentioned above. Therefore, the electronic device determines at T4 through step 1040 that the released hardware macroblock resources are enough to create the hard decoder. Then the electronic device creates the hard decoder of video B at T4 through step 1060. Since it takes a certain amount of time to create a hard decoder, the electronic device can consider that the hard decoder of video B has been created at the time node TA with a preset time interval from T4 (for example Figure 20 (a) in FIG. 5 shows the hard decoding preparation stage of video B between T5 and T1, i.e., the stage of creating a hard decoder. Between T5 and T1, video B is still decoded based on the soft decoder. Between T1 and T6, the electronic device decodes video B using the hard decoder.

[0247] In other embodiments, Video A, Video B, and Video C may also have other resolutions shown in Table 1. For example, the resolution of Video C is 8K or higher as shown in Table 1, and both Video A and Video B have 5K resolutions. For another example, when the resolution of Video A is 6K, the resolution of Video B may be 6K or lower; and when the resolution of Video B is 6K, the resolution of Video A may be 6K or lower.

[0248] Please refer to Figure 21 , Figure 21 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application. Figure 21 The specific video shown belongs to the above situation 3, and the electronic device determines the result of the three-way video in step 1020 as result B6 in the above table 3. Figure 21 The resolutions of Video A, Video B, and Video C in (a), (b), and (c) are the same as Figure 20 The resolutions of video A, video B, and video C in (a), (b), and (c) are the same, so Figure 21 The implementation process of (a), (b) and (c) can be referred to Figure 20 The implementation process of (a), (b) and (c) will not be repeated here.

[0249] Please refer to Figure 22 , Figure 22 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application. Figure 22 The specific video shown belongs to the above situation 3, and the electronic device determines the result of the three-way video in step 1020 as result B7 in the above table 3. For example, assuming Figure 22The resolution of video A shown in (a) is the resolution corresponding to 6K shown in Table 1, then the electronic device determines at T1 that video A is not a specific video (such as result B7 of video A shown in Table 3), then the electronic device creates a hard decoder for video A at T1 through step 1030, and then decodes video A based on the hard decoder until the decoding is completed at T3, and releases the hardware macroblock resources occupied by the hard decoder of video A.

[0250] Then, at T2, assuming that the resolution of video B is the resolution corresponding to 6K shown in Table 1, the electronic device determines in step 1020 that video B is a specific video (such as the result B7 of video B shown in Table 3). Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources in step 1040, and decodes video B based on the soft decoder (such as Figure 22 (a) in the figure shows the soft decoding of video B between T2 and T3). Later, during the decoding of video B by the soft decoder, the hard decoder of video A releases the hardware macroblock resources at T3. The released hardware macroblock resources are sufficient to create the hard decoder of video B, which belongs to the above situation three. Therefore, the electronic device determines at T3 through step 1040 that the released hardware macroblock resources are sufficient to create the hard decoder. Then the electronic device creates the hard decoder of video B at T3 through step 1060. Since it takes a certain amount of time to create a hard decoder, the electronic device can consider that the hard decoder of video B has been created at the time node TA with a preset time interval from T3 (for example Figure 22 (a) in FIG. 3 shows the hard decoding preparation stage of video B between T3 and TA, i.e., the stage of creating a hard decoder. Between T3 and TA, video B is still decoded based on the soft decoder. Between TA and T5, the electronic device decodes video B using the hard decoder.

[0251] Then, at T5, assuming that the resolution of video C is the resolution corresponding to 6K shown in Table 1, the electronic device determines at T5 that video C is not a specific video (such as result B7 for video C shown in Table 3). Then, the electronic device creates a hard decoder for video C at T5 through step 1030, and then decodes video C based on the hard decoder until decoding is completed at T6, and releases the hardware macroblock resources occupied by the hard decoder of video C. In other embodiments, video A, video B, and video C can also have other resolutions shown in Table 1. For example, the resolution of video A is 6K or lower as shown in Table 1, the resolution of video B can be 6K or lower, and the resolution of video C can be 6K or lower.

[0252] Assumptions Figure 22 The resolutions of video A, video B, and video C shown in (b) and (c) are the same as those in Figure 22The resolution of video A shown in (a) and (c) is the same, then Figure 22 The implementation of (b) and (c) can refer to Figure 22 The implementation of (a) in the above will not be described in detail here.

[0253] Please refer to Figure 23 , Figure 23 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application. Figure 23 The specific video shown belongs to the above situation 3, and the electronic device determines the result of the three-way video in step 1020 as result B8 in Table 3. For example, assuming Figure 23 The resolution of video A shown in (a) is the resolution corresponding to 4K shown in Table 1, then the electronic device determines at T1 that video A is not a specific video (such as result B8 of video A shown in Table 3), then the electronic device creates a hard decoder for video A through step 1030 at T1, and then decodes video A based on the hard decoder until the decoding is completed at T4, and releases the hardware macroblock resources occupied by the hard decoder of video A.

[0254] Then at T2, assuming that the resolution of video C is the resolution corresponding to 4K shown in Table 1, the electronic device determines at T2 that video A is not a specific video (such as the result B8 of video C shown in Table 3). Then, the electronic device creates a hard decoder for video A through step 1030 at T2, and then decodes video C based on the hard decoder until the decoding is completed at T5, and releases the hardware macroblock resources occupied by the hard decoder of video C.

[0255] Then, at T3, assuming that the resolution of video B is the resolution corresponding to 4K shown in Table 1, the electronic device determines at T3 that video B is a specific video (such as the result B8 of video B shown in Table 3). Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 1040, and decodes video B based on the soft decoder (such as Figure 23 (a) in the figure shows the soft decoding of video B between T3 and T4). Later, during the decoding of video B by the soft decoder, the hard decoder of video A releases the hardware macroblock resources at T4. The released hardware macroblock resources are sufficient to create the hard decoder of video B, which belongs to the above situation three. Therefore, the electronic device determines at T4 through step 1040 that the released hardware macroblock resources are sufficient to create the hard decoder. Then the electronic device creates the hard decoder of video B at T4 through step 1060. Since it takes a certain amount of time to create a hard decoder, the electronic device can consider that the hard decoder of video B has been created at the time node TA with a preset time interval from T4 (for example Figure 23(a) in FIG. 4 shows the hard decoding preparation stage of video B between T4 and T5, i.e., the stage of creating a hard decoder. Between T4 and T5, video B is still decoded based on the soft decoder. Between T5 and T6, the electronic device decodes video B using the hard decoder.

[0256] In other embodiments, video A, video B, and video C may also be other resolutions in Table 1. For example, the resolution of video A is the 2K resolution shown in Table 1, the resolution of video C may be 2K resolution, and the resolution of video B may be 5K resolution.

[0257] For example, suppose Figure 23 The resolution of video A shown in (b) is the resolution corresponding to 4K shown in Table 1. Then the electronic device can refer to the processing method of video A at T1. Figure 23 The way in which the electronic device shown in (a) processes the video A at T1 will not be described in detail here.

[0258] Then at T2, assuming that the resolution of video C is the 4K resolution shown in Table 1, the electronic device can process video C at T2 by referring to Figure 23 The way in which the electronic device shown in (a) processes the video C at T2 will not be described in detail here.

[0259] Then, at T3, assuming that the resolution of video B is the resolution corresponding to 5K shown in Table 1, the electronic device determines at T3 that video B is a specific video (such as the result B8 of video B shown in Table 3). Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 1040, and decodes video B based on the soft decoder (such as Figure 23 (a) in FIG. 2 shows the soft decoding of video B between T3 and T4.) Later, while video B is being decoded by the soft decoder, the hardware decoder of video A releases hardware macroblock resources at T4. However, the released hardware macroblock resources are insufficient to create a hardware decoder for video B, which corresponds to the second scenario described above. Therefore, the electronic device determines at T4 in step 1040 that the released hardware macroblock resources are insufficient to create a hardware decoder. Thereafter, the electronic device continues to decode video B using the soft decoder.

[0260] Afterwards, while video B continues to be decoded by the soft decoder, the hardware decoder of video C releases the hardware macroblock resources at T5. The released hardware macroblock resources are sufficient to create the hardware decoder of video B, which belongs to the above situation three. Therefore, the electronic device determines at T5 through step 1040 that the released hardware macroblock resources are sufficient to create the hardware decoder. Then, the electronic device creates the hardware decoder of video B at T5 through step 1060. Since it takes a certain amount of time to create the hardware decoder, the electronic device can consider that the hardware decoder of video B has been created at time node TA with a preset time interval from T5 (for example, Figure 23 (b) in FIG. 5 shows the hard decoding preparation stage of video B between T5 and T1A, i.e., the stage of creating a hard decoder. Between T5 and T1A, video B is still decoded based on the soft decoder. Between T1A and T6, the electronic device decodes video B using the hard decoder.

[0261] In other embodiments, video A, video B, and video C may also be other resolutions in Table 1. For example, the resolution of video A is 4K or lower as shown in Table 1, the resolution of video C may be 4K or lower, and the resolution of video B may be 6K.

[0262] For example, suppose Figure 23 The resolution of video A shown in (c) is the resolution corresponding to 4K shown in Table 1. Then the electronic device can refer to the processing method of video A at T1. Figure 23 The way in which the electronic device shown in (a) processes the video A at T1 will not be described in detail here.

[0263] Then at T2, assuming that the resolution of video C is the 4K resolution shown in Table 1, the electronic device can process video C at T2 by referring to Figure 23 The way in which the electronic device shown in (a) processes the video C at T2 will not be described in detail here. Figure 23 The video C shown in (c) is Figure 23 The difference of video C shown in (a) is that Figure 23 The end time node of video C shown in (c) is before the end time node of video A. Figure 23 The end time node of video C shown in (a) is after the end time node of video A.

[0264] Then, at T3, assuming that the resolution of video B is the resolution corresponding to 4K shown in Table 1, the electronic device determines at T3 that video B is a specific video (such as the result B8 of video B shown in Table 3). Therefore, the electronic device first creates a soft decoder that does not occupy hardware macroblock resources based on step 1040, and decodes video B based on the soft decoder (such as Figure 23(a) shows the soft decoding of video B between T3 and T4). Later, during the decoding of video B by the soft decoder, the hard decoder of video C releases the hardware macroblock resources at T4. The released hardware macroblock resources are sufficient to create the hard decoder of video B, which belongs to the above situation three. Therefore, the electronic device determines at T4 through step 1040 that the released hardware macroblock resources are sufficient to create the hard decoder. Then the electronic device creates the hard decoder of video B at T4 through step 1060. Since it takes a certain amount of time to create a hard decoder, the electronic device can consider that the hard decoder of video B has been created at the time node TA with a preset time interval from T4 (for example Figure 23 (a) in FIG. 4 shows the hard decoding preparation stage of video B between T4 and T5, i.e., the stage of creating a hard decoder. Between T4 and T5, video B is still decoded based on the soft decoder. Between T5 and T6, the electronic device decodes video B using the hard decoder.

[0265] In other embodiments, video A, video B, and video C may also be other resolutions in Table 1. For example, the resolution of video A is the 2K resolution shown in Table 1, the resolution of video C may be 2K resolution, and the resolution of video B may be 5K resolution.

[0266] Please refer to Figure 24 , Figure 24 This is another example diagram of a specific video existing in three-way video provided by an embodiment of the present application. Figure 24 The specific video shown belongs to the above situation 3, and the electronic device determines the result of the three-way video in step 1020 as result B8 in the above table 3. Figure 24 The resolutions of video A, video B, and video C shown in (a) are the same as those of Figure 23 The resolutions of video A, video B, and video C shown in (a) are the same, so Figure 24 The implementation of (a) can refer to Figure 23 The implementation of (a) in the above will not be described here. Similarly, Figure 24 The implementation of (b) and (c) can refer to Figure 23 The implementation of (b) and (c) in the above will not be described in detail here.

[0267] In the embodiment of the present application, when the electronic device determines in step 1020 that the remaining hardware macroblock resources are insufficient to create a hard decoder to decode the video or picture, a soft decoder that does not occupy the hardware macroblock resources is first created to decode the video or picture (for example, Figure 24 (a) shows the soft decoding of video B between T3 and T4), and secondly, after the hardware decoder completes the decoding and releases the hardware macroblock resources (such as Figure 24The hard decoder of video A shown in (a) releases the hardware macroblock resources at T4), and creates a hard decoder based on the released hardware macroblock resources (such as Figure 24 (a) shows that a hard decoder is created between T4 and TA), and a video or picture is decoded by a hard decoder (such as Figure 24 (a) shows that video B is decoded by the hard decoder of video B between TA and T6). In this way, when the hardware macroblock resources are insufficient to create a hard decoder, the video can be decoded by combining the soft decoder and the hard decoder. This avoids the problem of black screen and flash back of image processing software due to failure to create a hard decoder, or system crash or error in electronic equipment, thereby ensuring the continuity of users when editing videos or pictures and improving user experience.

[0268] In a related solution, electronic devices decode videos using a soft decoder when the hardware macroblock resources are insufficient to create a hard decoder. For example, see Figure 25 , Figure 25 is an example diagram of another related solution provided. Figure 25 The scene shown in (a) is similar to Figure 23 The scenes shown in (a) and (b) are the same. Figure 25 The scene shown in (b) is similar to Figure 23 The same scenario is shown in (c). Figure 25 In the example, if the electronic device determines at T3 that the currently remaining hardware macroblock resources are insufficient to create a hard decoder for video B, then it will decode the entire video B based on the soft decoder. However, since the soft decoder is a decoding program running on the CPU and does not rely on specialized hardware, it may consume more CPU resources, resulting in slower decoding speed and increased power consumption. In addition, the performance of the soft decoder is limited to a certain extent by the processing power of the CPU, and the video playback is not as smooth as hard decoding, resulting in freezes and delays in the video decoded by the soft decoder. In the embodiment of the present application, when it is determined that the currently remaining hardware macroblock resources are insufficient to create a hard decoder to decode the video or picture, the video is decoded together by combining the soft decoder and the hard decoder (for example, Figure 23 (a) shows that video B is decoded based on the soft decoder between T3 and TA, and video B is decoded based on the hard decoder between TA and T6). Since the time for decoding the video based on the soft decoder is shortened, the consumption of more CPU resources is avoided, thereby improving the speed of decoding the video, reducing power consumption, and improving decoding performance, reducing the probability of freezes and delays in the decoded video, so that users have a better experience when previewing the decoded video.

[0269] Furthermore, when the embodiments of the present application are applied to a scene with superimposed transitions, if the hardware macroblock resources of the electronic device are insufficient to create a hard decoder, the electronic device can only perform soft decoding processing on the video during the superimposed transition and the preset duration, and perform hard decoding processing on the video during the rest of the duration. For example: Figure 23 In (a), T3 to T4 is the duration of the superimposed transition, T4 to TA is the preset duration, and TA to T6 is another duration. The electronic device performs soft decoding on video B between T3 and TA, and hard decoding on video B between TA and T6. Since the duration of the superimposed transition is usually short, and the preset duration is 200 milliseconds to 1000 milliseconds, the duration is short, so the duration of soft decoding of the video is relatively short, and the duration of hard decoding of the video is relatively long. Compared with the related scheme of soft decoding of the entire video when the hardware macroblock resources are insufficient to create a hard decoder, the duration of video decoding based on the soft decoder can be shortened, avoiding the consumption of more CPU resources, thereby improving the speed of decoding the video, reducing power consumption, and improving decoding performance, reducing the probability of freezes and delays in the decoded video, and allowing users to have a better experience when previewing the decoded video.

[0270] The above embodiments are combined Figures 16 to 25 The implementation process of three-channel video is introduced. For the implementation process of four-channel video or more, you can refer to the implementation process of three-channel video, which will not be repeated here.

[0271] The above embodiments are combined Figures 9 to 25 The implementation process of the video processing method provided by the embodiment of the present application is introduced. Figure 4 The software architecture shown here again introduces the implementation process of the video processing method provided in the embodiment of the present application.

[0272] For example, please refer to Figure 26 , Figure 26 This is a timing diagram of a video processing method provided in an embodiment of the present application. This timing diagram uses the two-channel video mentioned in the above embodiment as an example to illustrate the implementation process of the video processing method provided in an embodiment of the present application. This timing diagram includes steps 261 to 2630.

[0273] Step 261 : The editing software in the application layer receives user operation 1 .

[0274] For the explanation of user operation 1, please refer to the above embodiment and will not be repeated here.

[0275] In step 262 , the editing software in the application layer obtains at least one video and a time node of the at least one video on the timeline in response to user operation 1 .

[0276] It should be understood that the editing software can obtain one channel video, two channels video, three channels video, three channels and above video, etc. mentioned in the above embodiments in response to user operation 1. The embodiment of the present application uses two channels video as an example. For example, two channels video can be Figure 14 (a) shows video A and video B. The starting time node of video A is T1 and the ending time node is T3. The starting time node of video B is T2 and the ending time node is T4. Figure 14 In the scene shown in (a), the user also sets an overlay transition, the start time node of the overlay transition is T2, and the end time node is T3.

[0277] It should be noted that in the following embodiments, unless otherwise specified, Figure 14 The scenario shown in (a) is used as an example.

[0278] In step 263 , the editing software determines the timing of at least one video on the timeline based on the time nodes of at least one video on the timeline.

[0279] for example, Figure 14 In the scenario shown in (a), the editing software determines that video A is played before video B and video A ends before video B based on the time nodes of video A and video B.

[0280] Step 264, the editing software responds to user operation 2, sends video A to MediaCodec in the application framework layer at the start time node T1 of video A, and sends request A to create hard decoder A (request A carries the end time node T3 of video A).

[0281] It should be understood that the explanation of user operation 2 can refer to the above embodiment and will not be repeated here.

[0282] In step 265 , MediaCodec determines whether the currently remaining hardware macroblock resource 1 is sufficient to create a hardware decoder A based on the resolution of video A.

[0283] It should be understood that video A is the video that is sorted first on the timeline in the editing software as determined by step 263. Therefore, before processing video A, the hardware macroblock resources of the electronic device have not been occupied. Therefore, assuming that the hardware macroblock resources of the electronic device can process a total of 79,632 macroblocks of video or picture data, and one macroblock includes 16×16 pixels, then the current remaining hardware macroblock resources 1 can process 79,632 macroblocks of video or picture data.

[0284] Regarding determining whether the current remaining hardware macroblock resources 1 are sufficient to create a hard decoder A based on the resolution of video A, reference may be made to the above embodiment, which will not be repeated here.

[0285] In step 266 , when MediaCodec determines that the currently remaining hardware macroblock resources 1 are sufficient to create the hardware decoder A, it creates the hardware decoder A and determines the currently remaining hardware macroblock resources 2 .

[0286] It should be understood that, assuming that the resolution of video A is the 4K resolution (3840×2160) shown in Table 1, the hard decoder A created by occupying hardware macroblock resources can process (3840×2160) / (16×16)=32400 macroblocks in video A, and the determined current remaining hardware macroblock resources 2 can also process 79632-32400=47232 macroblocks.

[0287] In some embodiments, MediaCodec can also determine, based on the resolution of frequency A, that the currently remaining hardware macroblock resources 1 are insufficient to create a hard decoder A. For the implementation method of determining that the currently remaining hardware macroblock resources are insufficient to create a hard decoder, please refer to other embodiments and will not be repeated here.

[0288] Step 267 , after determining the current remaining hardware macroblock resource 2 , MediaCodec returns notification 1 (carrying the current remaining hardware macroblock resource 2 ) to the editing software.

[0289] In step 268 , MediaCodec decodes video A based on hardware decoder A.

[0290] In step 269 , based on notification 1 , the editing software sends video B to MediaCodec at the start time node T2 of video B, and sends a request B1 to create a hard decoder B1 (the request B1 carries the end time node T4 of video B).

[0291] In step 2610 , MediaCodec determines whether the currently remaining hardware macroblock resources 2 are sufficient to create a hard decoder B1 based on the resolution of video B.

[0292] In step 2611 , when MediaCodec determines that there is not enough hardware decoder B1 to create, it returns notification 2 (carrying the current remaining hardware macroblock resources 2) to the editing software.

[0293] For example, the current remaining hardware macroblock resources 2 can also process 79632-32400=47232 macroblocks. The resolution of video B is 6K (6016x3384) as shown in Table 1. The hardware macroblock resources occupied by the hard decoder B1 for creating video B can process 6016x3384) / (16×16)=79524 macroblocks. 47232 is less than 79524. Therefore, MediaCodec determines that the current remaining hardware macroblock resources 2 are not sufficient to create the hard decoder B1.

[0294] Step 2612: Based on Notification 2, the editing software sends Video B to FFmpeg in the system library at the starting time node T2 of Video B, and sends a request B2 for creating a soft decoder B2.

[0295] Step 2613, FFmpeg creates soft decoder B2.

[0296] In step 2614, FFmpeg decodes video B based on the soft decoder B2.

[0297] In step 2615, MediaCodec releases the hardware macroblock resources occupied by hardware decoder A at the end time node of video A.

[0298] Step 2616: During the decoding process of the hardware decoder A, MediaCodec sends a decoded frame to the editing software for each frame of image in the decoded video A, and returns notification 3 (carrying the current remaining hardware macroblock resources 3) to the editing software after the hardware decoder A releases the hardware macroblock resources.

[0299] For example, the current remaining hardware macroblock resources 2 can still process 79632-32400=47232 macroblocks. The hard decoder A created by occupying the hardware macroblock resources can process (3840×2160) / (16×16)=32400 macroblocks in video A. After the hard decoder A releases the hardware macroblock resources, the current remaining hardware macroblock resources 3 can still process 79632 macroblocks.

[0300] In step 2617 , the editing software displays each decoded frame it receives, until the decoded frame of each image frame of video A is displayed.

[0301] In step 2618, based on notification 3, the editing software sends a request B3 to MediaCodec to create a hard decoder B1 for video B at the end time node T2 of video A (request B3 carries the end time node T4 of video B and the resolution of video B (for example, 6K resolution)).

[0302] In step 2619 , MediaCodec determines whether the currently remaining hardware macroblock resources 3 are sufficient to create a hard decoder B1 based on the resolution of video B.

[0303] For example, the current remaining hardware macroblock resources 3 can still process 32400+47232=79632 macroblocks. The hardware macroblock resources required to create the hard decoder B1 of video B can process 6016x3384) / (16×16)=79524 macroblocks. 79524 is less than 79632, thus determining that the current remaining hardware macroblock resources 3 are sufficient to create the hard decoder B1.

[0304] In step 2620 , when MediaCodec determines that the current remaining hardware macroblock resources are sufficient to create the hard decoder B1 , it creates the hard decoder B1 and determines the current remaining hardware macroblock resources 4 .

[0305] For example, the remaining hardware macroblock resource 4 can still process 79632-79524=108 macroblocks.

[0306] Step 2621, MediaCodec returns notification 4 (carrying the current remaining hardware macroblock resource 4) to the editing software.

[0307] Step 2622: Based on Notification 4, the editing software sends a request B4 to FFmpeg to stop decoding video B based on the soft decoder B2 at a time point TA that is a preset time interval from the end time point of video A.

[0308] In step 2623, FFmpeg stops decoding video B based on soft decoder B2.

[0309] In step 2624, when FFmpeg decodes video B based on the soft decoder B2, each time a frame of image in partial image 1 in video B is decoded, a decoded frame is sent to the editing software.

[0310] It should be understood that the partial image 1 can be understood as multiple image frames between the start time node of video B and the time node with a preset time interval from the end time node of video A. For example: Figure 14 (a) shows multiple image frames between T2 and TA in video B.

[0311] In step 2625 , the editing software displays each decoded frame it receives until the decoded frame of each image in partial image 1 in video B is displayed, and partial image 2 in video B is determined.

[0312] It should be understood that partial image 2 refers to the image frames other than partial image 1 in video B. For example: Figure 14 (a) shows multiple image frames between TA and T4 in video B.

[0313] Step 2626: The editing software sends partial image 2 in video B to MediaCodec.

[0314] In step 2627 , MediaCodec decodes the partial image 2 based on the hard decoder B1 .

[0315] In step 2628 , MediaCodec releases the hardware macroblock resources occupied by the hardware decoder B1 at the end time node of video B.

[0316] Step 2629: During the decoding process of the hard decoder B1, MediaCodec sends a decoded frame to the editing software every time it decodes a frame of the partial image 2 in the video B, and returns a notification 5 (carrying the current remaining hardware macroblock resources 5) to the editing software after the hard decoder B1 releases the hardware macroblock resources.

[0317] For example, the hardware macroblock resources required to create video B by the hard decoder B1 can process 79524 macroblocks, and the current remaining hardware macroblock resources 4 can also process 79632-79524=108 macroblocks. Then, after the hard decoder B1 releases the hardware macroblock resources, the current remaining hardware macroblock resources 5 can still process 108+79524=79632 macroblocks.

[0318] In step 2630 , the editing software displays each decoded frame received, until the decoded frame of each image in the partial image 2 in the video B is displayed.

[0319] It should be noted that Figure 26 The embodiment shown is based on the scenario of two-way video. Figure 14 The scene shown in (a) is used as an example to illustrate the implementation process of the embodiment of the present application. When the scene of the two-way video is Figure 14 When the scenario shown in (a) is different, the implementation process can refer to Figure 26 The implementation process shown is not repeated here.

[0320] The above embodiments are combined Figure 4 The software architecture shown in the figure is used to introduce the implementation process of the video processing method provided by the embodiment of the present application by taking two-way video as an example. Figure 4 The software architecture shown takes three-channel video as an example to introduce the implementation process of the video processing method provided in the embodiment of the present application.

[0321] For example, please refer to Figure 27 , Figure 27 This is a timing diagram of another video processing method provided by an embodiment of the present application. The timing diagram includes steps 271 to 2738.

[0322] Step 271 : The editing software in the application layer receives user operation 1 .

[0323] In step 272 , the editing software obtains video A, video B, and video C, as well as the time nodes of video A, video B, and video C on the timeline.

[0324] It should be understood that Figure 27 The scenario is based on three-way video. Figure 23 Taking the scenario shown in (a) as an example, the implementation process of the embodiment of the present application is described. For example, Figure 23 In (a), the starting time node of video A is T1 and the ending time node is T4, the starting time node of video C is T2 and the ending time node is T5, and the starting time node of video B is T3 and the ending time node is T6.

[0325] In step 273 , the editing software determines the timing of video A, video B, and video C on the timeline based on the time nodes of video A, video B, and video C on the timeline.

[0326] For example, the editing software determines the timing of video A, video B, and video C on the timeline as video A plays first, followed by video C, and finally video B. Video A ends first, followed by video C, and finally video B.

[0327] Step 274, the editing software responds to user operation 2, sends video A to MediaCodec in the application framework layer at the start time node T1 of video A, and sends request A to create hard decoder A (request A carries the end time node T4 of video A).

[0328] In step 275 , MediaCodec determines whether the currently remaining hardware macroblock resources 1 are sufficient to create a hard decoder A based on the resolution of video A.

[0329] It should be understood that the explanation of the current remaining hardware macroblock resource 1 can refer to the above embodiment and will not be repeated here. For example, the current remaining hardware macroblock resource 1 can process 79632 macroblocks of video or picture data.

[0330] In step 276 , when MediaCodec determines that the currently remaining hardware macroblock resources 1 are sufficient to create a hard decoder A, it creates a hard decoder A and determines the currently remaining hardware macroblock resources 2 .

[0331] For example, assuming that the video 1 has a 4K resolution, the currently determined remaining hardware macroblock resource 2 can still process 79632-32400=47232 macroblocks.

[0332] Step 277 , MediaCodec returns notification 1 (carrying the current remaining hardware macroblock resources 2 ) to the editing software.

[0333] In step 278 , MediaCodec decodes video A based on hardware decoder A.

[0334] In step 279 , based on notification 1 , the editing software sends video C to MediaCodec at the start time node T2 of video C, and sends a request C for creating a hard decoder C (the request C carries the end time node T5 of video C).

[0335] In step 2710 , MediaCodec determines whether the currently remaining hardware macroblock resources 2 are sufficient to create a hardware decoder C based on the resolution of video C.

[0336] For example, assuming that the resolution of video C is 4K (3840×2160), the hard decoder A created by occupying hardware macroblock resources can process (3840×2160) / (16×16)=32400 macroblocks in video A. The current remaining hardware macroblock resources 2 can also process 79632-32400=47232 macroblocks. 47232 is greater than 32400. Therefore, based on the resolution of video C, MediaCodec determines that the current remaining hardware macroblock resources 2 are sufficient to create hard decoder C.

[0337] In step 2711 , when MediaCodec determines that the currently remaining hardware macroblock resources 2 are sufficient to create a hardware decoder C, it creates a hardware decoder C and determines the currently remaining hardware macroblock resources 3.

[0338] For example, the remaining hardware macroblock resource 3 can still process 47232-32400=14832 macroblocks.

[0339] Step 2712, MediaCodec returns notification 2 (carrying the current remaining hardware macroblock resources 3) to the editing software.

[0340] Step 2713, MediaCodec decodes video C based on hardware decoder C.

[0341] In step 2714, based on notification 2, the editing software sends video B to MediaCodec at the start time node T3 of video B, and sends a request B1 to create a hard decoder B1 (the request B1 carries the end time node T6 of video B).

[0342] In step 2715 , MediaCodec determines whether the currently remaining hardware macroblock resources 3 are sufficient to create a hard decoder B1 based on the resolution of video B.

[0343] For example: the current remaining hardware macroblock resources 3 can also process 47232-32400=14832 macroblocks. The resolution of video B is 4K (3840×2160). Then the hard decoder B1 created by occupying the hardware macroblock resources can process (3840×2160) / (16×16)=32400 macroblocks in video B. 14832 is less than 32400. Therefore, based on the resolution of video B, MediaCodec determines that the current remaining hardware macroblock resources 3 are not sufficient to create the hard decoder B1.

[0344] Step 2716 , when MediaCodec determines that the current remaining hardware macroblock resources 3 are insufficient to create the hard decoder B1 , it returns a notification 3 (carrying the current remaining hardware macroblock resources 3 ) to the editing software.

[0345] Step 2717: Based on Notification 3, the editing software sends Video B to FFmpeg in the system library at the starting time node T3 of Video B, and sends a request B2 to create a soft decoder B2.

[0346] Step 2718, FFmpeg creates soft decoder B2.

[0347] In step 2719, FFmpeg decodes video B based on the soft decoder B2.

[0348] Step 2720, FFmpeg releases the hardware macroblock resources occupied by the hardware decoder A at the end time node T4 of video A.

[0349] Step 2721, during the decoding process of the hardware decoder A, FFmpeg sends a decoded frame to the editing software for each frame of image in the decoded video A, and returns notification 4 (carrying the current remaining hardware macroblock resources 4) after the hardware decoder A releases the hardware macroblock resources.

[0350] For example: hardware macroblock resource 3 can also process 47232-32400=14832 macroblocks, and the created video A hard decoder A can process (3840×2160) / (16×16)=32400 macroblocks. Therefore, after the hard decoder A of video A releases the hardware macroblock resources, the current remaining hardware macroblock resources 4 can still process 32400+14832=47232 macroblocks.

[0351] In step 2722 , the editing software displays each decoded frame it receives until the decoded frame of each image frame of video A is displayed.

[0352] In step 2723, based on notification 4, the editing software sends a request B3 to MediaCodec to create a hard decoder B1 for video B at the end time node T4 of video A (request B3 carries the end time node of video B and the resolution of video B).

[0353] In step 2724 , MediaCodec determines whether the currently remaining hardware macroblock resources 4 are sufficient to create a hard decoder B1 based on the resolution of video B.

[0354] For example: the current remaining hardware macroblock resources 4 can also process 32400+14832=47232 macroblocks. Assuming that the resolution of video B is 4K (3840×2160), the hardware macroblock resources occupied by the hard decoder B1 for creating video B can process (3840×2160) / (16×16)=32400 macroblocks. 47232 is greater than 32400, so MediaCodec determines that the current remaining hardware macroblock resources 4 are sufficient to create the hard decoder B1.

[0355] In step 2725 , MediaCodec creates the hard decoder B1 and determines the current remaining hardware macroblock resources 5 when determining that the current remaining hardware macroblock resources are sufficient to create the hard decoder B1 .

[0356] For example, the currently remaining hardware macroblock resource 5 can process 47232-32400=14832 macroblocks.

[0357] Step 2726, MediaCodec returns notification 5 (carrying the current remaining hardware macroblock resource 5) to the editing software.

[0358] In step 2727 , based on notification 5 , the editing software sends a request B4 to FFmpeg to stop decoding video B based on the soft decoder B2 at a time point TA that is a preset time interval from the end time point of video A.

[0359] In step 2728, FFmpeg stops decoding video B based on soft decoder B2.

[0360] In step 2729, during the decoding process of the soft decoder B2, FFmpeg sends a decoded frame to the editing software every time it decodes a frame of image 1 in the partial image of video B.

[0361] For example: Part of image 1 can be Figure 23 (a) shows multiple image frames between T3 and TA in video B.

[0362] In step 2730 , the editing software displays each decoded frame it receives until the decoded frame of each image in partial image 1 in video B is displayed, and partial image 2 in video B is determined.

[0363] For example: Part of image 1 can be Figure 23 (a) shows multiple image frames between TA and T6 in video B.

[0364] Step 2731: The editing software sends partial image 2 in video B to MediaCodec.

[0365] In step 2732 , MediaCodec decodes the partial image 2 based on the hard decoder B1 .

[0366] In step 2733 , MediaCodec releases the hardware macroblock resources occupied by hardware decoder C at the end time node T5 of video C.

[0367] In step 2734, during the decoding process of the hardware decoder C, MediaCodec sends a decoded frame for each frame of image in the decoded video C, and returns notification 6 (carrying the current remaining hardware macroblock resources 6) after the hardware decoder C releases the hardware macroblock resources.

[0368] For example, the currently remaining hardware macroblock resource 6 can process 4832+32400=47232 macroblocks.

[0369] In step 2735 , the editing software displays each decoded frame it receives, until the decoded frame of each image frame of video C is displayed.

[0370] In step 2736 , MediaCodec releases the hardware macroblock resources occupied by the hardware decoder B1 at the end time node T6 of video B.

[0371] Step 2737: During the decoding process of the hard decoder B1, MediaCodec sends a decoded frame to the editing software every time it decodes a frame of image 2 in the partial image B. After the hard decoder B1 releases the hardware macroblock resources, it returns notification 7 (carrying the current remaining hardware macroblock resources 7).

[0372] For example, the remaining hardware macroblock resource 7 can process 47232+32400=79632 macroblocks.

[0373] In step 2738 , the editing software displays each decoded frame it receives, until the decoded frame of each image in the partial image 2 in the video B is displayed.

[0374] It should be noted that Figure 27 The embodiment shown is based on a three-way video scene. Figure 23 The scene shown in (a) is used as an example to illustrate the implementation process of the embodiment of the present application. When the scene of the three-way video is Figure 23 When the scenario shown in (a) is different, the implementation process can refer to Figure 27 Furthermore, when the electronic device processes four or more channels of video, it can also refer to Figure 27 or Figure 26 The illustrated embodiments will not be described in detail.

[0375] The following summarizes the relevant solutions involved in the above embodiments.

[0376] Please refer to Figure 28 , Figure 28 is a schematic diagram of a video processing method provided by an embodiment of the present application. Figure 3 The electronic device 100 shown here may also be executed by a processor or chip in the electronic device 100, and the present application does not impose any limitation thereto. For ease of description, the method is described in detail using an electronic device as an example. The method includes steps 281 to 284.

[0377] Step 281: The electronic device obtains a first video in response to a clipping operation.

[0378] It should be understood that the clipping operation can refer to the user operation 1 in the above embodiment, which will not be repeated here.

[0379] It should also be understood that the electronic device displays a clipping interface, such as Figure 7 The "editing interface" 750 shown in (e) of FIG. Figure 8 The "editing interface" 840 shown in (d) in FIG. The editing interface includes a main video track and a picture-in-picture track. The first video can be a video on the main video track. Figure 14 The video B shown in (a) Figure 19 Video B shown in (a) Figure 23 Video B etc. is shown.

[0380] The first video can also be a video on a picture-in-picture track. Figure 14 Video B shown in (b) Figure 19 Video B shown in (b), and Figure 24 Video B and so on are shown.

[0381] Step 282 : In response to the preview operation, the electronic device decodes the first video based on a soft decoder if it is determined that the electronic device does not have sufficient hardware macroblock resources to create a hard decoder for the first video.

[0382] It should be understood that the preview operation can refer to the user operation 2 in the above embodiment, which will not be described in detail here.

[0383] It should also be understood that the electronic device is equipped with hardware macroblock resources, which are used by the electronic device to create a hard decoder to decode the video. For an explanation of the hardware macroblock resources and the hard decoder, please refer to the above embodiments and will not be repeated here. For an explanation of the soft decoder, please refer to the above embodiments and will not be repeated here.

[0384] The current hardware macroblock resources refer to the hardware macroblock resources currently remaining in the electronic device when the first video starts to be decoded. For example, Figure 14Video B shown in (a) is the first video, and video A plays before video B, and video A ends playing before video B. Assuming that the hardware macroblock resources of the electronic device can process a total of 79,632 macroblocks of video or picture data, and the resolution of video A is the 4K resolution (3840×2160) shown in Table 1, then, at the starting time node T1 of video A shown in (a) of 14, the electronic device occupies hardware macroblock resources to create a hard decoder to decode video A. The hard decoder can process (3840×2160) / (16×16)=32,400 macroblocks in video A through hardware macroblock resources. Then, when decoding of the first video starts (at T2), the currently remaining hardware macroblock resources can process 79,632-32,400=47,232 macroblocks.

[0385] In implementation, the electronic device may determine whether the current hardware macroblock resources are sufficient to create a hard decoder for the first video based on the resolution of the first video and the current hardware macroblock resources.

[0386] For example: assuming that the resolution of the first video is 5K (5120x2880), the hardware macroblock resources required to create a hard decoder for the first video can process 57,600 macroblocks, but the current hardware macroblock resources can only process 47,232 macroblocks, and 47,232 is less than 57,600. Therefore, the electronic device determines that the current hardware macroblock resources are insufficient to create a hard decoder for the first video.

[0387] Step 283 , during decoding of the first video based on the soft decoder, the electronic device determines whether hardware macroblock resources released by hard decoders of other videos are sufficient to create a hard decoder for the first video, where the other videos are different from the first video.

[0388] It should be understood that the process of decoding the first video based on the soft decoder can refer to Figure 14 (a) shows the process of soft decoding of video B between T2 and T3. For other videos, please refer to Figure 14 Video A shown, Figure 23 Video A and Video C are shown.

[0389] In the implementation, when decoding the first video based on the soft decoder, the electronic device can determine whether the hardware macroblock resources released by the hard decoders of other videos are sufficient to create the hard decoder of the first video based on the resolution of the first video and the hardware macroblock resources released by the hard decoders of other videos. For example, please refer to Figure 14In (a), assuming that the resolution of the first video is 5K (5120x2880), the hardware macroblock resources of the electronic device can process a total of 79,632 macroblocks of video or picture data. At T1, the hardware macroblock resources occupied by the hard decoder of the created video A can process 32,400 macroblocks. Then, at T3, after the hard decoder of video A releases the hardware macroblock resources, the released hardware macroblock resources can still process 79,632 macroblocks. Based on the resolution of the first video at T3, it can be determined that the hardware macroblock resources required to create the hard decoder of the first video can process 57,600 macroblocks, and the released hardware macroblock resources can still process 79,632 macroblocks. 79,632 is greater than 57,600, so the electronic device determines that the hardware macroblock resources are sufficient to create the hard decoder of the first video.

[0390] Step 284 : When the electronic device determines that the hardware macroblock resources released by other hard decoders are sufficient to create a hard decoder for the first video, the electronic device decodes the first video based on the soft decoder and the hard decoder for the first video.

[0391] For example, please refer to Figure 14 In (a), after the electronic device releases the hardware macroblock resources of video A at T3, and determines that the hardware macroblock resources after the release of the hard decoder of video A are sufficient to create the hard decoder of video B, it starts to create the hard decoder of video B. After the hard decoder of video B is created, video B is decoded based on the hard decoder, that is, the hard decoder of video B is created between T3 and TA, the hard decoder of video B is created at TA, and video B is decoded based on the hard decoder between TA and T4.

[0392] In the embodiment of the present application, the electronic device may first decode the first video based on a soft decoder that does not occupy the hardware macroblock resources (for example, Figure 14 Second, during the decoding of the first video based on the soft decoder, the hardware decoders of other videos are decoded and the hardware macroblock resources are released (e.g. Figure 14 The hardware decoder of video A shown releases hardware macroblock resources at T3), determines whether the hardware macroblock resources released by the hardware decoders of other videos are sufficient to create a hard decoder for the first video, and creates a hard decoder based on the released hardware macroblock resources (for example, Figure 14 The hard decoder is created between T3 and TA as shown), and the first video is decoded based on the soft decoder and the hard decoder (eg Figure 14As shown, between T3 and TA, video B continues to be decoded with the soft decoder, and after TA, video B is decoded based on the hard decoder of video B). In this way, when the hardware macroblock resources are insufficient to create a hard decoder, the video can be decoded together by combining the soft decoder and the hard decoder. This avoids problems such as black screen and flash back of the image processing software due to failure to create a hard decoder, or system crash or error in the electronic device, thereby ensuring the continuity of users when editing videos or pictures and improving the user experience.

[0393] In a related solution, an electronic device decodes a video using a soft decoder when the hardware macroblock resources are insufficient to create a hard decoder, e.g. Figure 15 In the embodiment of the present application, if the electronic device determines that the current remaining hardware macroblock resources are insufficient to create a hard decoder at T2, then the entire video B (the first video) will be decoded based on the soft decoder. However, since the soft decoder is a decoding program running on the CPU and does not rely on dedicated hardware, it may consume more CPU resources, resulting in slower decoding speed and increased power consumption. In addition, the performance of the soft decoder is limited to a certain extent by the processing power of the CPU, and the video playback is not as smooth as hard decoding, resulting in freezes and delays in the video decoded by the soft decoder. In the embodiment of the present application, when it is determined that the current remaining hardware macroblock resources are insufficient to create a hard decoder to decode the first video, the video is decoded together by combining the soft decoder and the hard decoder (for example, Figure 14 Video B is decoded based on the soft decoder between T2 and TA, and video B is decoded based on the hard decoder between TA and T4). Since the time for decoding the video based on the soft decoder is shortened, it avoids consuming more CPU resources, thereby increasing the speed of decoding the video, reducing power consumption, and improving decoding performance. It also reduces the probability of freezes and delays in the decoded video, allowing users to have a better experience when previewing the decoded video.

[0394] In some embodiments, when the electronic device determines that the current hardware macroblock resources are insufficient to create a hard decoder for the first video, the electronic device decodes the first video based on a soft decoder, including: at the start moment of decoding the first video, determining whether the electronic device is sufficient to create a hard decoder for the first video with the current hardware macroblock resources; when it is determined that the electronic device is insufficient to create a hard decoder for the first video with the current hardware macroblock resources, decoding the first video based on the soft decoder; and, in the process of decoding the first video based on the soft decoder, determining whether the hardware macroblock resources released by the hard decoders of other videos are sufficient to create a hard decoder for the first video, including: in the process of decoding the first video based on the soft decoder, at the end moment of decoding of other videos, releasing the hardware macroblock resources occupied by the hard decoders of other videos; determining whether the hardware macroblock resources released by the hard decoders of other videos are sufficient to create a hard decoder for the first video.

[0395] It should be understood that after the electronic device obtains the first video and the other videos, it needs to sort the first video and the other videos based on the start time node and the end time node of the first video and the start time node and the end time node of the other videos, such as Figure 14 In (a), the first video is video B, which starts at time T2 and ends at time T3. The other video is video A, which starts at time T1 and ends at time T3. After sorting, video A plays before video B and ends before video B.

[0396] In implementation, the electronic device can execute the task of the time node based on the sorted time node. For example, at the decoding start time of the first video (the starting time node T2 of video B), the electronic device determines whether the electronic device is sufficient to create a hard decoder for the first video with the current hardware macroblock resources; if it is determined that the electronic device is not sufficient to create a hard decoder for the first video with the current hardware macroblock resources, the electronic device decodes the first video based on the soft decoder. In the process of decoding the first video based on the soft decoder, the electronic device releases the hardware macroblock resources occupied by the hard decoders of other videos at the end time of decoding of other videos (the end time node T3 of video A); and determines whether the hardware macroblock resources after the release of the hard decoders of other videos are sufficient to create a hard decoder for the first video.

[0397] In an embodiment of the present application, the electronic device can execute the corresponding task under the time node based on the time node (for example, the decoding start time of the first video, the decoding end time of other videos) (for example, determining at the decoding start time of the first video whether the electronic device uses the current hardware macroblock resources to create a hard decoder for the first video, etc., and determining at the decoding end time of other videos whether the hardware macroblock resources after the hard decoders of other videos are released are sufficient to create a hard decoder for the first video, etc.). Executing the corresponding task based on the time node can reduce the complexity brought about by concurrent execution of multiple tasks or simultaneous execution of multiple tasks, and can improve the execution efficiency of the tasks.

[0398] In some embodiments, when the electronic device determines that the hardware macroblock resources after the release of other hard decoders are sufficient to create a hard decoder for the first video, the electronic device decodes the first video based on the soft decoder and the hard decoder of the first video, including: when it is determined that the hardware macroblock resources after the release of other hard decoders are sufficient to create a hard decoder for the first video, decoding the first video between the decoding end moment of the other videos and the first moment based on the soft decoder, the first moment being the moment when the hard decoder of the first video is completed, and the first moment and the decoding end moment of the other videos are separated by a preset time length; decoding the first video between the first moment and the decoding end moment of the first video based on the hard decoder of the first video.

[0399] It should be understood that the electronic device can create a hard decoder for the first video when it determines that the hardware macroblock resources released by other hard decoders are sufficient to create a hard decoder for the first video. Since it takes a certain amount of time to create a hard decoder, the electronic device can create a hard decoder for the first video at the same time as the decoding of other videos is completed (for example, Figure 14 (a) shows T3) a time node with a preset time interval (such as Figure 14 The hardware decoder of the first video is considered to have been created at TA shown in (a). In the process of creating the hardware decoder of the first video, the electronic device still decodes the first video based on the soft decoder, such as the end time of decoding other videos based on the soft decoder ( Figure 14 (a) shows T3) to the first moment (such as Figure 14 After the electronic device completes the creation of the hard decoder for the first video, the electronic device decodes the first video based on the hard decoder for the first video from the first moment (TA) to the end moment of decoding the first video (for example, Figure 14 The first video decoding between T3) shown in (a).

[0400] It should also be understood that the preset duration can be 200 milliseconds to 1000 milliseconds, and this embodiment of the present application does not limit this.

[0401] In an embodiment of the present application, the first moment is the moment when the hard decoder of the first video is created. The first moment is separated from the decoding end moments of other videos by a preset time length. The electronic device can decode the first video between the decoding end moment of other videos and the first moment based on the soft decoder, and decode the first video between the first moment and the decoding end moment of the first video based on the hard decoder of the first video. Since the preset time length is usually short, when decoding the first video, the time length for decoding the first video based on the soft decoder is relatively short, and the time length for decoding the first video based on the hard decoder is relatively long. The preset time length can make the soft decoding time length shorter, further improving the speed of decoding the video, reducing power consumption, and further improving the decoding performance.

[0402] In some embodiments, the other videos include a second video and a third video, the decoding start time of the second video is earlier than the decoding start time of the third video, and the decoding start time of the third video is earlier than the decoding start time of the first video; and before the electronic device decodes the first video based on the soft decoder when it determines that the electronic device is insufficient to create a hard decoder for the first video with the current hardware macroblock resources, the method further includes: the electronic device determines at the decoding start time of the second video whether the electronic device is sufficient to create a hard decoder for the second video with the current hardware macroblock resources; when it is determined that the electronic device is sufficient to create a hard decoder for the second video with the current hardware macroblock resources, the electronic device decodes the second video based on the hard decoder of the second video; At the start moment of video decoding, determining whether the electronic device is sufficient to create a hard decoder for the third video with current hardware macroblock resources; decoding the third video based on the hard decoder of the third video when it is determined that the electronic device is sufficient to create a hard decoder for the third video with current hardware macroblock resources; and decoding the first video based on a soft decoder when it is determined that the electronic device is insufficient to create a hard decoder for the first video with current hardware macroblock resources, including: at the start moment of decoding the first video, determining whether the electronic device is sufficient to create a hard decoder for the first video with current hardware macroblock resources; decoding the first video based on a soft decoder when it is determined that the electronic device is insufficient to create a hard decoder for the first video with current hardware macroblock resources.

[0403] For example, please refer to Figure 23 In (a), the second video is Video A, the third video is Video C, and the first video is Video B. The decoding of the second video starts at T1 and ends at T4. The decoding of the third video starts at T2 and ends at T5. The decoding of the first video starts at T3 and ends at T6. Arrange these time nodes in chronological order, from earliest to latest, as T1 to T6.

[0404] In progress, please refer to Figure 23In (a), for example, at decoding start time T1 of the second video (video A), the electronic device determines whether the current hardware macroblock resources are sufficient to create a hard decoder for the second video. If it is determined that the current hardware macroblock resources are sufficient to create a hard decoder for the second video, the electronic device decodes the second video based on the hard decoder for the second video. The implementation of determining whether the current hardware macroblock resources are sufficient to create a hard decoder can be found in the above embodiment and will not be further described here.

[0405] At the decoding start time T2 of the third video (video A), determine whether the electronic device has sufficient current hardware macroblock resources to create a hard decoder for the third video; if it is determined that the electronic device has sufficient current hardware macroblock resources to create a hard decoder for the third video, decode the third video based on the hard decoder of the third video.

[0406] At the decoding start time T3 of the first video (video B), it is determined whether the electronic device has sufficient current hardware macroblock resources to create a hard decoder for the first video; if it is determined that the electronic device does not have sufficient current hardware macroblock resources to create a hard decoder for the first video, the first video is decoded based on a soft decoder.

[0407] In some embodiments, the decoding end time of the second video is earlier than the decoding end time of the third video, and the decoding end time of the third video is earlier than the decoding end time of the first video; and, in the process of decoding the first video based on the soft decoder, determining whether the hardware macroblock resources after the hard decoders of other videos are released are sufficient to create a hard decoder for the first video, including: in the process of decoding the first video based on the soft decoder, the electronic device releases the hardware macroblock resources occupied by the hard decoder of the second video at the end time of decoding the second video; determining whether the hardware macroblock resources after the hard decoder of the second video are sufficient to create a hard decoder for the first video; and, in the case of determining that the hardware macroblock resources released by other hard decoders are sufficient to create a hard decoder for the first video, decoding the first video based on the soft decoder and the hard decoder of the first video, including: in the case of determining that the hardware macroblock resources released by the hard decoder of the second video are sufficient to create a hard decoder for the first video, decoding the first video based on the soft decoder and the hard decoder of the first video.

[0408] In progress, please refer to Figure 23In (a), for example, the electronic device releases the hardware macroblock resources occupied by the hard decoder of the second video (video A) at the end time T4 of decoding the second video during the process of decoding the first video based on the soft decoder (for example, the decoding process between T3 and T4); and determines whether the hardware macroblock resources after the release of the hard decoder of the second video are sufficient to create the hard decoder of the first video. When it is determined that the hardware macroblock resources after the release of the hard decoder of the second video are sufficient to create the hard decoder of the first video, the electronic device decodes the first video based on the soft decoder and the hard decoder of the first video (for example, decoding based on the soft decoder between T4 and TA, and decoding based on the hard decoder between TA and T6).

[0409] In an embodiment of the present application, the video acquired by the electronic device is three-channel video (for example, the three-channel video is a first video, a second video, and a third video). When the hardware macroblock resources of the electronic device are sufficient to create a hard decoder for the second video and the third video, but insufficient to create a hard decoder for the first video, a soft decoder is first created to decode the first video. Then, during the process of soft decoding of the video, when it is determined that the hardware macroblock resources released by the hard decoder of the second video are sufficient to create a hard decoder for the first video, the first video is decoded based on the soft decoder and the hard decoder of the first video. In this way, when decoding the three-channel video, if the hardware macroblock resources of the electronic device are insufficient to create a hard decoder for one of the three videos (for example, the first video), a hard decoder is created using the hardware macroblock resources released by the other video (for example, the second video) in the three videos. Then, the soft decoder and the hard decoder are combined to decode the video together. This avoids the failure of creating the hard decoder due to insufficient hardware resources to create the hard decoder for the three videos when decoding the three videos, resulting in a black screen and flash back of the image processing software, or causing a system crash or error in the electronic device. This ensures the continuity of the user when editing videos or pictures, and improves the user experience.

[0410] In other related solutions for three-way video decoding, electronic devices decode the video through a soft decoder when the hardware macroblock resources are insufficient to create a hard decoder, e.g. Figure 25In the example, the electronic device determines at T3 that the currently remaining hardware macroblock resources are insufficient to create a hard decoder for the first video (video B), and then decodes the entire video B based on the soft decoder. However, since the soft decoder is a decoding program running on the CPU and does not rely on specialized hardware, it may consume more CPU resources, resulting in slower decoding speeds and increased power consumption. In addition, the performance of the soft decoder is limited to a certain extent by the processing power of the CPU, and the video playback is not as smooth as hard decoding, resulting in freezes and delays in the video decoded by the soft decoder. In the embodiment of the present application, when it is determined that the currently remaining hardware macroblock resources are insufficient to create a hard decoder to decode the first video, the video is decoded together by combining the soft decoder and the hard decoder (for example, Figure 23 (a) between T3 and TA, the first video (video B) is decoded based on the soft decoder, and between TA and T6, video B is decoded based on the hard decoder). Since the time for decoding the video based on the soft decoder is shortened, the consumption of more CPU resources is avoided, thereby increasing the speed of decoding the video, reducing power consumption, and improving decoding performance. The probability of freezes and delays in the decoded video is reduced, so that users have a better experience when previewing the decoded video.

[0411] In some embodiments, after determining whether the hardware macroblock resources released by the hard decoder of the second video are sufficient to create the hard decoder of the first video, the method further includes: when the electronic device determines that the hardware macroblock resources released by the hard decoder of the second video are not sufficient to create the hard decoder of the first video, at the end of decoding of the third video, releasing the hardware macroblock resources occupied by the hard decoder of the third video; determining whether the hardware macroblock resources released by the hard decoder of the third video are sufficient to create the hard decoder of the first video; and decoding the first video based on the soft decoder and the hard decoder of the first video when it is determined that the hardware macroblock resources released by the hard decoder of the third video are sufficient to create the hard decoder of the first video.

[0412] In progress, please refer to Figure 23 As shown in (b), for example, assuming that the electronic device releases the hardware macroblock resources occupied by the hard decoder of the second video at the end time T4 of decoding of the second video, but the electronic device determines that the released hardware macroblock resources are not sufficient to create the hard decoder of the first video, then the electronic device continues to decode the first video with the soft decoder after T4.

[0413] At the end time T5 of the decoding of the third video, the hardware macroblock resources occupied by the hard decoder of the third video are released. The electronic device determines whether the released hardware macroblock resources are sufficient to create a hard decoder for the first video. If it is determined that the resources are sufficient to create a hard decoder for the first video, the electronic device creates a hard decoder for the first video, and decodes the first video based on the soft decoder and the hard decoder of the first video (for example, decoding based on the soft decoder between T5 and TA, and decoding based on the hard decoder between TA and T6).

[0414] In an embodiment of the present application, the video acquired by the electronic device is three-channel video (for example, the three-channel video is a first video, a second video, and a third video). When the hardware macroblock resources of the electronic device are sufficient to create hard decoders for the second video and the third video, but are insufficient to create a hard decoder for the first video, a soft decoder is first created to decode the first video. Then, in the process of soft decoding the video, it is first determined whether the hardware macroblock resources released by the hard decoder of one video other than the first video in the three-channel video (for example, the second video) are sufficient to create a hard decoder for the first video. When it is determined that the hardware macroblock resources released by the hard decoder of the second video are insufficient to create a hard decoder for the first video, it is determined whether the hardware macroblock resources released by the hard decoder of another video other than the first video in the three-channel video (for example, the third video) are sufficient to create a hard decoder for the first video. When it is determined that the hardware macroblock resources released by the hard decoder of the third video are sufficient to create a hard decoder for the first video, the first video is decoded based on the soft decoder and the hard decoder of the first video. This ensures that there are sufficient hardware macroblock resources to create a hard decoder for the first video to perform hard decoding on the first video.

[0415] In some embodiments, an overlay transition is configured between the second video and the third video, the start time of the overlay transition is the decoding start time of the third video, and the end time of the overlay transition is the decoding end time of the second video.

[0416] For example, please refer to Figure 23 In the superimposed transition shown in (a), the starting time of the superimposed transition is the decoding starting time T3 of the third video, and the ending time of the superimposed transition is the decoding ending time T4 of the second video.

[0417] In the embodiment of the present application, when three-way video is applied to a scene with superimposed transitions, if the hardware macroblock resources of the electronic device are insufficient to create a hard decoder for the first video in the three-way video, the electronic device can perform soft decoding processing on the first video during the superimposed transition period. For example: Figure 23T3 to T4 shown in (a) is the duration of the superimposed transition. The electronic device performs soft decoding on video B (the first video) between T3 and T4. Since the duration of the superimposed transition is relatively short, when the hardware macroblock resources are insufficient to create a hard decoder for the first video, the duration of soft decoding of the first video is relatively short, which can shorten the duration of video decoding based on the soft decoder and avoid consuming more CPU resources. This can increase the speed of video decoding, reduce power consumption, and improve decoding performance, reduce the probability of freezes and delays in the decoded video, and enable users to have a better experience when previewing the decoded video.

[0418] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0419] The present application provides a computer program product that, when executed on an electronic device, enables the electronic device to execute the technical solution in the above embodiment. The implementation principle and technical effects are similar to those of the above method-related embodiments and will not be described in detail here.

[0420] The embodiment of the present application provides a readable storage medium, which contains instructions. When the instructions are executed on an electronic device, the electronic device executes the technical solution of the above embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0421] The present application provides a chip for executing instructions. When the chip is running, the technical solution of the above embodiment is executed. The implementation principle and technical effect are similar and will not be described here.

[0422] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0423] It should be understood that the “embodiment” mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0424] Those skilled in the art will understand that the various numerical numbers such as first and second involved in this application are only for the convenience of description and are not used to limit the scope of the embodiments of this application, and also indicate the order of precedence.

[0425] In this application, elements expressed in the singular are intended to mean "one or more" rather than "one and only one" unless otherwise specified. In this application, unless otherwise specified, "at least one" is intended to mean "one or more" and "a plurality" is intended to mean "two or more."

[0426] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. A can be singular or plural, and B can be singular or plural.

[0427] The term "at least one of..." in this document refers to all or any combination of the listed items. For example, "at least one of A, B and C" can mean: A exists alone, B exists alone, C exists alone, A and B exist at the same time, B and C exist at the same time, and A, B and C exist at the same time. A can be singular or plural, B can be singular or plural, and C can be singular or plural.

[0428] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0429] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0430] The same or similar parts between the various embodiments in this application can refer to each other. In the various embodiments in this application, and the various implementation methods / implementation methods / implementation methods in each embodiment, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment are consistent and can be referenced to each other. The technical features in different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment can be combined to form new embodiments, implementation methods, implementation methods, or implementation methods according to their inherent logical relationships. The above-described implementation methods of this application do not constitute a limitation on the scope of protection of this application.

[0431] The above is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims. In short, the above is only a preferred embodiment of the technical solution of the present application, and is not used to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A video processing method, characterized in that: Applied to an electronic device, the electronic device is configured with hardware macroblock resources, the hardware macroblock resources are used by the electronic device to create a hard decoder to decode a video, the method comprising: In response to the clipping operation, obtaining a first video; In response to the preview operation, if it is determined that the electronic device is insufficient to create a hard decoder for the first video using current hardware macroblock resources, decoding the first video based on a soft decoder; During decoding of the first video based on the soft decoder, determining whether hardware macroblock resources released by hard decoders of other videos are sufficient to create a hard decoder for the first video, the other videos being different from the first video; In a case where it is determined that the hardware macroblock resources released by the other hard decoders are sufficient to create a hard decoder for the first video, the first video is decoded based on the soft decoder and the hard decoder for the first video.

2. The method according to claim 1, characterized in that The step of decoding the first video based on a soft decoder when determining that the electronic device does not have sufficient hardware macroblock resources to create a hard decoder for the first video includes: At a decoding start time of the first video, determining whether the electronic device has sufficient hardware macroblock resources to create a hard decoder for the first video; Decoding the first video based on the soft decoder when determining that the electronic device is insufficient to create a hard decoder for the first video using current hardware macroblock resources; and The step of determining, during the process of decoding the first video based on the soft decoder, whether hardware macroblock resources released by hard decoders of other videos are sufficient to create a hard decoder for the first video includes: During decoding of the first video based on the soft decoder, at the end of decoding of the other videos, releasing hardware macroblock resources occupied by the hard decoder of the other videos; Determine whether the hardware macroblock resources released by the hard decoder of the other video are sufficient to create the hard decoder of the first video.

3. The method according to claim 1 or 2, characterized in that The step of decoding the first video based on the soft decoder and the hard decoder of the first video, when it is determined that the hardware macroblock resources released by the other hard decoders are sufficient to create the hard decoder of the first video, includes: If it is determined that the hardware macroblock resources released by the other hardware decoders are sufficient to create a hardware decoder for the first video, decoding the first video between a time when decoding of the other videos by the soft decoder ends and a first time, where the first time is a time when the hardware decoder for the first video is completed, and a preset time interval is between the first time and the time when decoding of the other videos ends; The hard decoder based on the first video decodes the first video between the first moment and the decoding end moment of the first video.

4. The method according to any one of claims 1 to 3, characterized in that The other videos include a second video and a third video, the decoding start time of the second video is earlier than the decoding start time of the third video, and the decoding start time of the third video is earlier than the decoding start time of the first video; and, in the case of determining that the electronic device is insufficient to create a hard decoder for the first video with current hardware macroblock resources, before decoding the first video based on the soft decoder, the method further includes: At a decoding start time of the second video, determining whether the electronic device has sufficient hardware macroblock resources to create a hard decoder for the second video; If it is determined that the electronic device has sufficient hardware macroblock resources to create a hard decoder for the second video, decoding the second video based on the hard decoder for the second video; At a decoding start time of the third video, determining whether the electronic device has sufficient hardware macroblock resources to create a hard decoder for the third video; If it is determined that the electronic device has sufficient hardware macroblock resources to create a hard decoder for the third video, decoding the third video based on the hard decoder for the third video; and The step of decoding the first video based on a soft decoder when determining that the electronic device does not have sufficient hardware macroblock resources to create a hard decoder for the first video includes: At a decoding start time of the first video, determining whether the electronic device has sufficient hardware macroblock resources to create a hard decoder for the first video; In a case where it is determined that the electronic device is insufficient to create a hard decoder for the first video using current hardware macroblock resources, the first video is decoded based on the soft decoder.

5. The method according to claim 4, characterized in that The decoding end time of the second video is earlier than the decoding end time of the third video, and the decoding end time of the third video is earlier than the decoding end time of the first video; and, in the process of decoding the first video based on the soft decoder, determining whether the hardware macroblock resources released by the hard decoders of other videos are sufficient to create the hard decoder of the first video, includes: In a process of decoding the first video based on the soft decoder, at a moment when decoding of the second video is completed, releasing hardware macroblock resources occupied by the hard decoder of the second video; determining whether the hardware macroblock resources released by the hardware decoder of the second video are sufficient to create the hardware decoder of the first video; and The step of decoding the first video based on the soft decoder and the hard decoder of the first video, when it is determined that the hardware macroblock resources released by the other hard decoders are sufficient to create the hard decoder of the first video, includes: In a case where it is determined that the hardware macroblock resources released by the hard decoder of the second video are sufficient to create the hard decoder of the first video, the first video is decoded based on the soft decoder and the hard decoder of the first video.

6. The method according to claim 5, characterized in that After determining whether the hardware macroblock resources released by the hard decoder of the second video are sufficient to create the hard decoder of the first video, the method further includes: If it is determined that the hardware macroblock resources released by the hardware decoder of the second video are insufficient to create the hardware decoder of the first video, releasing the hardware macroblock resources occupied by the hardware decoder of the third video at the end of decoding of the third video; determining whether hardware macroblock resources released by the hardware decoder of the third video are sufficient to create the hardware decoder of the first video; In a case where it is determined that the hardware macroblock resources released by the hard decoder of the third video are sufficient to create the hard decoder of the first video, the first video is decoded based on the soft decoder and the hard decoder of the first video.

7. The method according to any one of claims 4 to 6, characterized in that An overlay transition is configured between the second video and the third video, the start time of the overlay transition is the start time of decoding of the third video, and the end time of the overlay transition is the end time of decoding of the second video.

8. The method according to any one of claims 1 to 7, characterized in that The electronic device displays an editing interface, which includes a main video track and a picture-in-picture track. The first video is a video on the main video track or a video on the picture-in-picture track.

9. The method according to any one of claims 1 to 8, characterized in that The step of determining, during the process of decoding the first video based on the soft decoder, whether hardware macroblock resources released by hard decoders of other videos are sufficient to create a hard decoder for the first video includes: In the process of decoding the first video based on the soft decoder, based on the resolution of the first video and the hardware macroblock resources released by the hard decoders of the other videos, it is determined whether the hardware macroblock resources released by the hard decoders of the other videos are sufficient to create the hard decoder of the first video.

10. An electronic device, characterized in that: include: one or more processors; one or more memories; The one or more memories store one or more computer programs, and the one or more computer programs include instructions. When the instructions are executed by the one or more processors, the electronic device performs the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 9.

12. A chip, characterized in that: The chip includes: a memory for storing instructions; A processor is configured to call and execute the instructions from the memory, so that an electronic device equipped with the chip executes the method according to any one of claims 1 to 9.

Images