Video processing method and device and storage medium

Abstract

The embodiment of the invention provides a video processing method and device and a storage medium, a timeline of a video editing draft is segmented to obtain a plurality of time slices, and multimedia materials included in different time slices in the video editing draft do not have a dependency relationship. Obtaining a complexity index of the coding process corresponding to each time slice; resource use condition information of the encoding equipment is obtained, an encoding task is generated on the encoding equipment according to the resource use condition information of the encoding equipment, and the encoding task is used for selecting one time slice of the video editing draft according to the complexity index to be encoded to obtain a corresponding video slice; and fusing the video segments corresponding to the plurality of time segments to obtain a target video. According to the embodiment of the invention, the video editing draft is divided into a plurality of time slices, and segmented coding can be performed in parallel, so that the coding efficiency of the video editing draft is improved, and resources on coding equipment can be fully utilized.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and in particular to a video processing method, device, and storage medium. Background Art

[0002] With the development of network technology and the improvement of terminal device hardware performance, the demand for video creation and video editing on terminal devices is becoming more and more urgent.

[0003] However, in the prior art, video editing on a terminal device takes a long time, especially video encoding and exporting, which seriously affects the user experience. Summary of the Invention

[0004] The embodiments of the present disclosure provide a video processing method, device, and storage medium to improve the efficiency of video encoding and enable full utilization of resources on the encoding device.

[0005] In a first aspect, an embodiment of the present disclosure provides a video processing method, including:

[0006] Segmenting the timeline of the video editing draft to obtain a plurality of time segments, wherein multimedia materials included in different time segments in the video editing draft do not have dependencies, and obtaining a complexity index of the encoding process corresponding to each time segment;

[0007] Obtaining resource usage status information of an encoding device, and generating an encoding task on the encoding device according to the resource usage status information of the encoding device, wherein the encoding task is used to select a time segment of the video editing draft according to a complexity index to encode a corresponding video segment;

[0008] The video segments corresponding to the multiple time segments are fused to obtain a target video.

[0009] In a second aspect, an embodiment of the present disclosure provides a video processing device, including:

[0010] a segmentation unit configured to segment the timeline of the video editing draft into a plurality of time segments, wherein multimedia materials included in different time segments in the video editing draft do not have dependencies with each other, and to obtain an encoding process complexity index corresponding to each time segment;

[0011] a task management unit, configured to obtain resource usage status information of an encoding device, and generate an encoding task on the encoding device based on the resource usage status information of the encoding device, wherein the encoding task is configured to select a time segment of the video editing draft according to a complexity index and encode the selected time segment to obtain a corresponding video segment;

[0012] The fusion unit is configured to fuse the video segments corresponding to the multiple time segments to obtain a target video.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0014] The memory stores computer-executable instructions;

[0015] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video processing method described in the first aspect and various possible designs of the first aspect.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the video processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0017] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the video processing method as described in the first aspect and various possible designs of the first aspect is implemented.

[0018] The video processing method, device, and storage medium provided by the embodiments of the present disclosure divide the timeline of a video editing draft into multiple time segments, wherein there is no dependency between the multimedia materials included in different time segments in the video editing draft, and obtains a complexity index of the encoding process corresponding to each time segment; obtains resource usage status information of the encoding device, generates an encoding task on the encoding device based on the resource usage status information of the encoding device, and the encoding task is used to select a time segment of the video editing draft according to the complexity index to encode to obtain a corresponding video segment; and fuses the video segments corresponding to the multiple time segments to obtain a target video. By dividing the video editing draft into multiple time segments, the embodiments of the present disclosure can perform segmented encoding in parallel, thereby improving the encoding efficiency of the video editing draft and enabling the resources on the encoding device to be fully utilized. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1A schematic diagram of a scenario of a video processing method provided by an embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of a video processing method according to an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of segmenting a video to be processed according to an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of generating a coding task provided by an embodiment of the present disclosure;

[0024] Figure 5 A structural block diagram of a video processing device provided in one embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0027] In the existing technology, the video editing process on the terminal device is time-consuming, especially the video encoding and exporting process, which is usually carried out in a serial encoding manner, that is, a complete video is encoded using one process or one thread, which takes a long time and seriously affects the user experience; and serial encoding is only performed on the GPU or CPU, while the terminal device usually has multiple GPUs and / or multiple CPUs, and serial encoding cannot bring into play the hardware performance, resulting in insufficient utilization of resources.

[0028] In order to solve the above technical problems, the present disclosure considers dividing the timeline of the video editing draft into multiple time segments without any dependency, and making full use of the resources of multiple encoding devices such as GPU and CPU, and encoding the time segments in parallel on one or more encoding devices in a multi-process or multi-threaded manner. Taking into account that the performance of different encoding devices may be different, the encoding efficiency of time segments with different complexities will be different. Therefore, after dividing into multiple time segments, the complexity index of the encoding process corresponding to each time segment is also obtained, so that the time segments with higher complexity are encoded on the encoding device with better performance, and the time segments with lower complexity are encoded on the encoding device with lower performance. On the one hand, the resources of encoding devices with different performance can be more fully utilized, and on the other hand, the encoding efficiency can be further improved and the encoding time can be shortened.

[0029] The video processing method provided by the embodiment of the present disclosure can be applied to any electronic device such as a terminal device or a server, for example Figure 1 As shown, taking the terminal device as an example, the timeline of the video editing draft is divided into multiple time segments, wherein there is no dependency between the multimedia materials included in different time segments in the video editing draft, and the complexity index of the encoding process corresponding to each time segment is obtained; the resource usage status information of the encoding device is obtained, and an encoding task is generated on the encoding device according to the resource usage status information of the encoding device, and the encoding task is used to select a time segment of the video editing draft according to the complexity index to encode and obtain a corresponding video segment; the video segments corresponding to the multiple time segments are fused to obtain the target video.

[0030] It should be noted that the user information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] The video processing method disclosed herein will be described in detail below with reference to specific embodiments.

[0032] refer to Figure 2 , Figure 2 This is a flow chart of a video processing method provided by an embodiment of the present disclosure. The method of this embodiment can be applied to electronic devices such as terminal devices or servers. The video processing method includes:

[0033] S201. Segment the timeline of the video editing draft to obtain multiple time segments, wherein there is no dependency between the multimedia materials included in different time segments in the video editing draft, and obtain a complexity index of the encoding process corresponding to each time segment.

[0034] In this embodiment, in some video editing scenarios (such as video editing applications) or other scenarios that require video encoding, the video editing draft includes one or more multimedia materials such as original video materials (one or more), picture materials, audio materials, and editing information for the multimedia materials, and the encoding process (export process) is to synthesize these multimedia materials into a video according to the editing information. In order to improve the encoding efficiency of the video editing draft and reduce the time consumption of video export, the timeline of the video editing draft can be divided into multiple time segments, so that the multiple time segments can be encoded in parallel, which greatly shortens the encoding time compared to the serial encoding method. Considering that there may be certain dependencies in the video editing draft, such as context dependencies in certain time segments, and that certain video special effects depend on previous and next frames, in this embodiment, when segmenting the timeline of the video editing draft, it is necessary to consider the dependencies between the multimedia materials included in the video editing draft, so that there are no dependencies between the multimedia materials included in the different time segments obtained by segmentation.

[0035] Optionally, when the timeline of the video editing draft is divided into multiple time segments, the dependency information between the multimedia materials included in the video editing draft can be obtained first, and the segmentation point position can be determined in the time range where no dependency exists based on the dependency information; the video editing draft can be divided according to the segmentation point position.

[0036] In this embodiment, the dependency information between the multimedia materials included in the video editing draft is first obtained, and then the time range corresponding to the dependency information is determined based on the dependency information. When segmenting, the time range corresponding to the dependency information can be divided into a time segment. Therefore, when determining the segmentation point position, the segmentation point position can be determined in the time range where there is no dependency. The time range where there is no dependency is the time range outside the time range corresponding to the dependency information, including the time range before the time range corresponding to the dependency and the time range after the time range corresponding to the dependency. Therefore, the segmentation point position can be determined before the time range corresponding to the dependency (including the start time of the time range corresponding to the dependency) and after the time range corresponding to the dependency (including the end time of the time range corresponding to the dependency). The video editing draft can be segmented based on the segmentation point so that the time range corresponding to the dependency information can be divided into a time segment.

[0037] Among them, the dependency information existing in the video editing draft may be relevant information of the video effects that depend on the previous and next frames, and the relevant information may be the time information and / or relevant video frames corresponding to the video effects, etc. After obtaining the relevant information of the video effects that depend on the previous and next frames included in the video editing draft, the time range corresponding to the video effects can be determined based on the relevant information of the video effects, and then the time range outside the time range corresponding to the video effects can be determined as the time range without dependency, that is, the time range before the start time of the video effects and after the end time of the video effects is determined as the time range without dependency, and the split point position is determined before the start time of the video effects and after the end time of the video effects, so as to ensure that the multimedia material corresponding to the video effects will not be divided into different time segments. If there are multiple video effects in the video editing draft, the video content corresponding to each video effect can be divided into different time segments, or the video content corresponding to two or more video effects can be divided into the same time segment. Figure 3 As shown, the video editing draft may have been originally spliced ​​together by different video material fragments (original segment 1, original segment 2), but in this embodiment, we do not need to pay attention to the original splicing method of the video editing draft, but instead segment it based on dependency information such as related information of video special effects to obtain time segments 1 to 4, ensuring that the video content corresponding to the video special effects will not be divided into different time frequency segments.

[0038] Based on the above embodiment, when determining the segmentation point position in a time range where no dependency exists, the video frame corresponding to the segmentation point position may be an intra-frame coded frame, i.e., an I-frame (Intra Picture), also known as a key frame. The I-frame is an independent frame that carries all information and can be decoded independently without referring to other images. This also ensures that there is no dependency between the multimedia materials included in different time segments after segmentation.

[0039] After the timeline of the video editing draft is divided into multiple time segments, the complexity index of the encoding process corresponding to each time segment can be obtained to characterize the complexity of each time segment of the video editing draft in the subsequent encoding process. Specifically, for any time segment, multiple video attribute information of the time segment can be obtained. The video attribute information is attribute information related to the complexity of the encoding process, including but not limited to resolution, frame rate, bit rate, and related parameters of the video special effects involved in the time segment. Then, based on the multiple video attribute information and the corresponding preset weights, the complexity index of the encoding process corresponding to the time segment can be obtained. For example, a weighted average or weighted sum of the multiple video attribute information can be performed, and the weighted average or weighted sum can be used as the complexity index corresponding to the time segment.

[0040] Optionally, before obtaining the complexity index of the time segment encoding process based on the multiple video attribute information and the corresponding preset weights, the video attribute information may be normalized.

[0041] S202. Obtain resource usage status information of the encoding device, and generate an encoding task on the encoding device according to the resource usage status information of the encoding device, wherein the encoding task is used to select a time segment of the video editing draft according to a complexity index for encoding to obtain a corresponding video segment.

[0042] In this embodiment, an electronic device such as a terminal device or a server generally has multiple encoding devices that can be used for encoding, such as a CPU, a GPU (including an integrated graphics card and / or a discrete graphics card), etc., and resource usage status information of each encoding device can be obtained, wherein the resource usage status information of the encoding device may include resource usage rate, resource remaining amount, etc., and then an encoding task of at least one thread or at least one process is generated on at least one encoding device based on the resource usage status information of each encoding device, including generating encoding tasks of multiple processes or multiple threads on one or more encoding devices among all encoding devices, and also including generating encoding tasks of one process or one thread on at least two encoding devices among all encoding devices, that is, generating multiple encoding tasks in all encoding devices (of course, if there is only one time segment, only one encoding task is generated), for example Figure 4 As shown, the resource management center obtains resource usage information of multiple encoding devices (such as GPU and CPU), generates encoding tasks on the encoding devices and adds them to the task queue, so that there are multiple encoding tasks in the task queue, so that the resources on each encoding device can be fully utilized. At the same time, the encoding tasks of more threads or processes can encode each time segment of the video editing draft in parallel, which can also shorten the encoding time.

[0043] Based on the resource usage information of each encoding device, an encoding task for at least one thread or at least one process is generated on at least one encoding device. For example, for a particular encoding device, if the resource usage rate is low and the remaining resources are large, encoding tasks for more threads or processes may be generated, allowing for a greater number of time segments to be encoded simultaneously. Conversely, if the resource usage rate is high and the remaining resources are small, encoding tasks for fewer threads or processes may be generated, allowing for a smaller number of time segments to be encoded simultaneously. Therefore, based on the resource usage information of any encoding device, the number of encoding task threads or processes that the current remaining resources of the encoding device can support can be determined. Based on the number of threads or processes, encoding tasks for a corresponding number of threads or processes are generated on the encoding device, thereby fully utilizing the resources of the encoding device. Of course, the number of time segments can also be considered. If the total number of encoding task threads or the total number of processes that can be supported by the current remaining resources of each encoding device exceeds the number of time segments, the number of time segments can be used as the total number of threads or the total number of processes, and then allocated to each encoding device based on the performance of each encoding device and / or the resource usage status information of each encoding device; if the total number of encoding task threads or the total number of processes that can be supported by the current remaining resources of each encoding device is less than the number of time segments, then in the subsequent encoding process, some time segments will need to wait for one or some encoding tasks to complete the encoding of the current time segment before encoding.

[0044] In this embodiment, each encoding task can encode a time segment. After completing the encoding of one time segment, it can continue to encode another time segment. Therefore, if there is an encoding task of at least one thread or at least one process on at least one encoding device, a time segment can be selected from multiple time segments for encoding. It should be noted that due to the different performance of different encoding devices, for example, for video encoding, the GPU performance is better than the CPU. Therefore, the GPU can be used to encode time segments with higher complexity, and the CPU can be used to encode time segments with lower complexity. That is, for any encoding task on any encoding device, according to the performance of the encoding device and the complexity index corresponding to each time segment, mutually matching time segments to be encoded are selected for encoding. In particular, assuming that the multiple encoding devices include a first encoding device and a second encoding device, if the performance of the first encoding device is higher than that of the second encoding device, the complexity index corresponding to the time segment encoded by the encoding task on the first encoding device is higher than the complexity index corresponding to the time segment encoded by the encoding task on the second encoding device, so that resources can be maximized.

[0045] S203: Fusing the video segments corresponding to the multiple time segments to obtain a target video.

[0046] In this embodiment, the video segments obtained by encoding the time segments may be fused, that is, the video segments may be spliced ​​into a complete video, ie, the target video.

[0047] Optionally, before merging multiple video clips, the multiple video clips may be tested to ensure that the relevant parameters of the multiple video clips are consistent before the fusion of the multiple video clips. Optionally, when generating encoding tasks, parameter testing may also be performed to ensure that each encoding task encodes each time segment based on the same encoding parameters. The relevant parameters of the multiple video clips obtained are consistent before the fusion of the multiple video clips.

[0048] The video processing method provided in this embodiment divides the timeline of a video editing draft into multiple time segments, wherein the multimedia materials included in different time segments in the video editing draft do not have dependencies between each other, and obtains a complexity index of the encoding process corresponding to each time segment; obtains resource usage status information of an encoding device, generates an encoding task on the encoding device based on the resource usage status information of the encoding device, and uses the encoding task to select a time segment of the video editing draft based on the complexity index to encode the corresponding video segment; and fuses the video segments corresponding to the multiple time segments to obtain a target video. By dividing the video editing draft into multiple time segments, segmented encoding can be performed in parallel, thereby improving the encoding efficiency of the video editing draft and enabling the resources on the encoding device to be fully utilized.

[0049] Of course, the video processing method of this embodiment is not limited to the scenario of video encoding tasks, and can also be used for other video processing tasks, such as video decoding tasks, etc., just by replacing the above-mentioned video encoding tasks with other video processing tasks.

[0050] Based on any of the above embodiments, after obtaining the complexity index of the encoding process corresponding to each time segment in S201, the following steps may also be included:

[0051] According to the complexity index corresponding to each of the time segments, one or more time segments of the multiple time segments of the video editing draft are adjusted to finally obtain multiple adjusted time segments.

[0052] In this embodiment, after obtaining the complexity index of the encoding process corresponding to each time segment, the complexity index corresponding to each time segment may be different in size, and there may be large differences. For example, the complexity index corresponding to time segment 1 is 3, the complexity index corresponding to time segment 2 is 9, the complexity index corresponding to time segment 3 is 4, and the complexity index corresponding to time segment 4 is 8. The segmentation method of the video editing draft can be adjusted, especially the currently segmented time segments can be adjusted, such as splitting again or merging adjacent time segments, so that the complexity index of the adjusted time segments is relatively uniform, or meets the performance of the encoding device, so that the encoding tasks can be completed almost at the same time, avoiding the encoding of some time segments taking too long, while other time segments still need to wait for a long time after the encoding is completed.

[0053] Optionally, adjusting one or more time segments of the multiple time segments of the video editing draft may include:

[0054] For a time segment whose complexity index exceeds a first preset threshold, that is, the coding complexity of the time segment may be too high and the encoding time is too long, the time segment may be further segmented. Of course, the segmentation process must also ensure that there is no dependency between the multimedia materials included in the different time segments obtained by segmentation, as in S201 above; and / or

[0055] At least two adjacent time segments whose complexity index is lower than the second preset threshold are merged, that is, the compilation complexity of the at least two adjacent time segments is low and the encoding time is short. The at least two adjacent time segments can be merged to avoid the need for subsequent encoding tasks to frequently pull time segments for encoding, thereby reducing interaction time. Optionally, the second preset threshold may be lower than the first preset threshold. Of course, the complexity index corresponding to the merged time segment is not likely to be too high. For example, if the complexity index corresponding to the merged time segment exceeds the first preset threshold, it may not be merged or other segmentation strategies may be considered.

[0056] After adjusting one or more time segments among the multiple time segments of the video editing draft, multiple adjusted time segments are obtained, and then encoding can be performed on the multiple adjusted time segments in subsequent steps.

[0057] Based on any of the above embodiments, after encoding tasks are generated on encoding devices according to resource usage information of encoding devices in S202, encoding tasks may be dynamically added or deleted according to resource usage information of each encoding device, as follows:

[0058] If the resource usage information of any encoding device does not exceed the preset usage indicator, for example, the resource usage rate of the encoding device is lower than the first preset resource usage rate threshold, it means that there are relatively many remaining resources, and a new thread or process encoding task can be generated on the encoding device to fully utilize the resources of the encoding device; or

[0059] If the resource usage status information of any encoding device exceeds the preset usage status indicator, for example, the resource usage rate of the encoding device exceeds the second preset resource usage rate threshold, it means that the remaining resources are small and the resources are relatively tight. In this case, the encoding task of at least one idle thread or process in the encoding task on the encoding device can be destroyed. If there is currently no idle thread or process encoding task, the encoding task can be destroyed and the resources can be recycled after any encoding task completes the encoding of the current time segment.

[0060] Based on any of the above embodiments, after the timeline of the video editing draft is segmented to obtain multiple time segments, the multiple time segments can be added to a queue and consumed from the time segment queue by encoding tasks of at least one thread or at least one process on at least one encoding device. When the encoding of a time segment fails, it can be re-added to the time segment queue. When it is detected that all time segments in the time segment queue have completed encoding, the corresponding video segments of each time segment are fused to obtain the target video; or when it is detected that any adjacent time segment has completed encoding, the corresponding video segments of the adjacent time segment are fused until all the video segments corresponding to all time segments are fused to obtain the target video.

[0061] Based on the above embodiments, after generating an encoding task of at least one thread or at least one process on at least one encoding device, the encoding task can be added to another queue, and an idle encoding task can be taken out from the encoding task queue to consume the time segment in the time segment queue for encoding.

[0062] Corresponding to the video processing method of the above embodiment, Figure 5 This is a structural block diagram of a video processing device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 5 The video processing device 500 includes: a segmentation unit 501, a task management unit 502, and a fusion unit 503.

[0063] The segmentation unit 501 is configured to segment the timeline of the video editing draft into a plurality of time segments, wherein there is no dependency between the multimedia materials included in different time segments in the video editing draft, and obtain an encoding process complexity index corresponding to each time segment;

[0064] The task management unit 502 is configured to obtain resource usage status information of the encoding device and generate an encoding task on the encoding device based on the resource usage status information of the encoding device, wherein the encoding task is configured to select a time segment of the video editing draft according to the complexity index and encode the corresponding video segment;

[0065] The fusion unit 503 is configured to fuse the video segments corresponding to the multiple time segments to obtain a target video.

[0066] In one or more embodiments of the present disclosure, when segmenting the timeline of the video editing draft to obtain multiple time segments, the segmentation unit 501 is configured to:

[0067] Obtaining dependency information between multimedia materials included in the video editing draft, and determining a segmentation point position in a time range where no dependency exists according to the dependency information;

[0068] The timeline of the video editing draft is segmented according to the segmentation point positions.

[0069] In one or more embodiments of the present disclosure, when obtaining dependency information between multimedia materials included in the video editing draft and determining the position of a segmentation point in a time range where no dependency exists based on the dependency information, the segmentation unit 501 is configured to:

[0070] Obtaining relevant information of video special effects dependent on previous and next frames included in the video editing draft;

[0071] Determine a time range corresponding to the video special effect according to relevant information of the video special effect, and determine a time range other than the time range corresponding to the video special effect as a time range without a dependency relationship;

[0072] The split point positions are determined in the time range in which no dependency exists.

[0073] In one or more embodiments of the present disclosure, when determining the position of a segmentation point in a time range in which no dependency exists, the segmentation unit 501 is configured to:

[0074] Any intra-frame coding frame is selected in the time range where no dependency relationship exists, and is determined as the segmentation point position.

[0075] In one or more embodiments of the present disclosure, when obtaining the complexity index of the encoding process corresponding to each time segment, the segmentation unit 501 is configured to:

[0076] For any time segment, obtaining multiple video attribute information of the time segment;

[0077] According to the plurality of video attribute information and corresponding preset weights, a complexity index of a process corresponding to the time segment encoding is obtained.

[0078] In one or more embodiments of the present disclosure, the video attribute information includes one or more of the following:

[0079] Parameters related to resolution, frame rate, bit rate, and video effects.

[0080] In one or more embodiments of the present disclosure, after obtaining the complexity index of the encoding process corresponding to each time segment, the segmentation unit 501 is further configured to:

[0081] According to the complexity index corresponding to each of the time segments, one or more time segments of the multiple time segments of the video editing draft are adjusted to finally obtain multiple adjusted time segments.

[0082] In one or more embodiments of the present disclosure, when the segmentation unit 501 adjusts one or more time segments of the plurality of time segments of the video editing draft according to the complexity index corresponding to each time segment, it is configured to:

[0083] Segmenting the time segments whose complexity index exceeds a first preset threshold, wherein multimedia materials included in different segmented time segments have no dependency relationship; and / or

[0084] At least two adjacent time segments whose complexity index is lower than a second preset threshold are merged.

[0085] In one or more embodiments of the present disclosure, when generating an encoding task on an encoding device according to resource usage status information of the encoding device, the task management unit 502 is configured to:

[0086] Based on the resource usage status information of any encoding device among multiple encoding devices, the number of encoding task threads or processes that the current remaining resources of the encoding device can support is determined, and encoding tasks of a corresponding number of threads or processes are generated on the encoding device based on the number of threads or processes.

[0087] In one or more embodiments of the present disclosure, after generating an encoding task on an encoding device according to the resource usage status information of the encoding device, the task management unit 502 is further configured to:

[0088] If the resource usage status information of any of the encoding devices does not exceed the preset usage status indicator, generating an encoding task of a new thread or process on the encoding device; or

[0089] If the resource usage status information of any of the encoding devices exceeds a preset usage status indicator, the encoding task of at least one idle thread or process in the encoding tasks on the encoding device is destroyed.

[0090] In one or more embodiments of the present disclosure, after generating an encoding task on an encoding device according to resource usage status information of the encoding device, the task management unit 502 is configured to:

[0091] For any encoding task on any encoding device, a time segment to be encoded is selected for encoding according to the performance of the encoding device and a complexity index corresponding to each time segment.

[0092] In one or more embodiments of the present disclosure, if the performance of a first encoding device among the multiple encoding devices is higher than that of a second encoding device, the complexity index corresponding to the time segment encoded by the encoding task on the first encoding device is higher than the complexity index corresponding to the time segment encoded by the encoding task on the second encoding device.

[0093] The video processing device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0094] refer to Figure 6 , which shows a schematic structural diagram of an electronic device 600 suitable for implementing an embodiment of the present disclosure. The electronic device 600 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0095] like Figure 6As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0096] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0097] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0098] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0099] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0100] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0101] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0104] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0105] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] In a first aspect, according to one or more embodiments of the present disclosure, a video processing method is provided, including:

[0107] Segmenting the timeline of the video editing draft to obtain a plurality of time segments, wherein multimedia materials included in different time segments in the video editing draft do not have dependencies, and obtaining a complexity index of the encoding process corresponding to each time segment;

[0108] Obtaining resource usage status information of an encoding device, and generating an encoding task on the encoding device according to the resource usage status information of the encoding device, wherein the encoding task is used to select a time segment of the video editing draft according to a complexity index to encode a corresponding video segment;

[0109] The video segments corresponding to the multiple time segments are fused to obtain a target video.

[0110] According to one or more embodiments of the present disclosure, segmenting the timeline of the video editing draft to obtain multiple time segments includes:

[0111] Obtaining dependency information between multimedia materials included in the video editing draft, and determining a segmentation point position in a time range where no dependency exists according to the dependency information;

[0112] The timeline of the video editing draft is segmented according to the segmentation point positions.

[0113] According to one or more embodiments of the present disclosure, obtaining dependency information between multimedia materials included in the video editing draft, and determining the segmentation point position in a time range where no dependency exists based on the dependency information, includes:

[0114] Obtaining relevant information of video special effects dependent on previous and next frames included in the video editing draft;

[0115] Determine a time range corresponding to the video special effect according to relevant information of the video special effect, and determine a time range other than the time range corresponding to the video special effect as a time range without a dependency relationship;

[0116] The split point positions are determined in the time range in which no dependency exists.

[0117] According to one or more embodiments of the present disclosure, determining the split point position in a time range in which no dependency exists includes:

[0118] Any intra-frame coding frame is selected in the time range where no dependency relationship exists, and is determined as the segmentation point position.

[0119] According to one or more embodiments of the present disclosure, obtaining a complexity index of the encoding process corresponding to each time segment includes:

[0120] For any time segment, obtaining multiple video attribute information of the time segment;

[0121] According to the plurality of video attribute information and corresponding preset weights, a complexity index of a process corresponding to the time segment encoding is obtained.

[0122] According to one or more embodiments of the present disclosure, the video attribute information includes one or more of the following:

[0123] Parameters related to resolution, frame rate, bit rate, and video effects.

[0124] According to one or more embodiments of the present disclosure, after obtaining the complexity index of the encoding process corresponding to each time segment, the method further includes:

[0125] According to the complexity index corresponding to each of the time segments, one or more time segments of the multiple time segments of the video editing draft are adjusted to finally obtain multiple adjusted time segments.

[0126] According to one or more embodiments of the present disclosure, adjusting one or more time segments of the plurality of time segments of the video editing draft according to the complexity index corresponding to each time segment includes:

[0127] Segmenting the time segments whose complexity index exceeds a first preset threshold, wherein multimedia materials included in different segmented time segments have no dependency relationship; and / or

[0128] At least two adjacent time segments whose complexity index is lower than a second preset threshold are merged.

[0129] According to one or more embodiments of the present disclosure, generating an encoding task on an encoding device according to resource usage status information of the encoding device includes:

[0130] Based on the resource usage status information of any encoding device among multiple encoding devices, the number of encoding task threads or processes that the current remaining resources of the encoding device can support is determined, and encoding tasks of a corresponding number of threads or processes are generated on the encoding device based on the number of threads or processes.

[0131] According to one or more embodiments of the present disclosure, after generating an encoding task on the encoding device according to the resource usage status information of the encoding device, the method further includes:

[0132] If the resource usage status information of any of the encoding devices does not exceed the preset usage status indicator, generating an encoding task of a new thread or process on the encoding device; or

[0133] If the resource usage status information of any of the encoding devices exceeds a preset usage status indicator, the encoding task of at least one idle thread or process in the encoding tasks on the encoding device is destroyed.

[0134] According to one or more embodiments of the present disclosure, after generating an encoding task on an encoding device according to resource usage status information of the encoding device, the method includes:

[0135] For any encoding task on any encoding device, a time segment to be encoded is selected for encoding according to the performance of the encoding device and a complexity index corresponding to each time segment.

[0136] According to one or more embodiments of the present disclosure, if the performance of a first encoding device among the multiple encoding devices is higher than that of a second encoding device, the complexity index corresponding to the time segment encoded by the encoding task on the first encoding device is higher than the complexity index corresponding to the time segment encoded by the encoding task on the second encoding device.

[0137] In a second aspect, according to one or more embodiments of the present disclosure, a video processing device is provided, including:

[0138] a segmentation unit configured to segment the timeline of the video editing draft into a plurality of time segments, wherein multimedia materials included in different time segments in the video editing draft do not have dependencies with each other, and to obtain an encoding process complexity index corresponding to each time segment;

[0139] a task management unit, configured to obtain resource usage status information of an encoding device, and generate an encoding task on the encoding device based on the resource usage status information of the encoding device, wherein the encoding task is configured to select a time segment of the video editing draft according to a complexity index and encode the selected time segment to obtain a corresponding video segment;

[0140] The fusion unit is configured to fuse the video segments corresponding to the multiple time segments to obtain a target video.

[0141] According to one or more embodiments of the present disclosure, when the segmentation unit segments the timeline of the video editing draft to obtain multiple time segments, it is configured to:

[0142] Obtaining dependency information between multimedia materials included in the video editing draft, and determining a segmentation point position in a time range where no dependency exists according to the dependency information;

[0143] The timeline of the video editing draft is segmented according to the segmentation point positions.

[0144] According to one or more embodiments of the present disclosure, when the segmentation unit obtains dependency information between multimedia materials included in the video editing draft and determines the location of a segmentation point in a time range where no dependency exists based on the dependency information, it is configured to:

[0145] Obtaining relevant information of video special effects dependent on previous and next frames included in the video editing draft;

[0146] Determine a time range corresponding to the video special effect according to relevant information of the video special effect, and determine a time range other than the time range corresponding to the video special effect as a time range without a dependency relationship;

[0147] The split point positions are determined in the time range in which no dependency exists.

[0148] According to one or more embodiments of the present disclosure, when determining the position of a segmentation point in a time range in which no dependency exists, the segmentation unit is configured to:

[0149] Any intra-frame coding frame is selected in the time range where no dependency relationship exists, and is determined as the segmentation point position.

[0150] According to one or more embodiments of the present disclosure, when obtaining the complexity index of the encoding process corresponding to each time segment, the segmentation unit is configured to:

[0151] For any time segment, obtaining multiple video attribute information of the time segment;

[0152] According to the plurality of video attribute information and corresponding preset weights, a complexity index of a process corresponding to the time segment encoding is obtained.

[0153] According to one or more embodiments of the present disclosure, the video attribute information includes one or more of the following:

[0154] Parameters related to resolution, frame rate, bit rate, and video effects.

[0155] According to one or more embodiments of the present disclosure, after obtaining the complexity index of the encoding process corresponding to each time segment, the segmentation unit is further configured to:

[0156] According to the complexity index corresponding to each of the time segments, one or more time segments of the multiple time segments of the video editing draft are adjusted to finally obtain multiple adjusted time segments.

[0157] According to one or more embodiments of the present disclosure, when the segmentation unit adjusts one or more time segments of the multiple time segments of the video editing draft according to the complexity index corresponding to each time segment, it is configured to:

[0158] Segmenting the time segments whose complexity index exceeds a first preset threshold, wherein multimedia materials included in different segmented time segments have no dependency relationship; and / or

[0159] At least two adjacent time segments whose complexity index is lower than a second preset threshold are merged.

[0160] According to one or more embodiments of the present disclosure, when the task management unit generates an encoding task on the encoding device according to the resource usage status information of the encoding device, it is configured to:

[0161] Based on the resource usage status information of any encoding device among multiple encoding devices, the number of encoding task threads or processes that the current remaining resources of the encoding device can support is determined, and encoding tasks of a corresponding number of threads or processes are generated on the encoding device based on the number of threads or processes.

[0162] According to one or more embodiments of the present disclosure, after generating an encoding task on an encoding device according to resource usage status information of the encoding device, the task management unit is further configured to:

[0163] If the resource usage status information of any of the encoding devices does not exceed the preset usage status indicator, generating an encoding task of a new thread or process on the encoding device; or

[0164] If the resource usage status information of any of the encoding devices exceeds a preset usage status indicator, the encoding task of at least one idle thread or process in the encoding tasks on the encoding device is destroyed.

[0165] According to one or more embodiments of the present disclosure, after generating an encoding task on an encoding device according to resource usage status information of the encoding device, the task management unit is configured to:

[0166] For any encoding task on any encoding device, a time segment to be encoded is selected for encoding according to the performance of the encoding device and a complexity index corresponding to each time segment.

[0167] According to one or more embodiments of the present disclosure, if the performance of a first encoding device among the multiple encoding devices is higher than that of a second encoding device, the complexity index corresponding to the time segment encoded by the encoding task on the first encoding device is higher than the complexity index corresponding to the time segment encoded by the encoding task on the second encoding device.

[0168] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0169] The memory stores computer-executable instructions;

[0170] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video processing method described in the first aspect and various possible designs of the first aspect.

[0171] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When the processor executes the computer execution instructions, the video processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0172] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the video processing method as described in the first aspect and various possible designs of the first aspect is implemented.

[0173] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0174] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0175] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A video processing method, characterized in that: include: Segmenting the timeline of the video editing draft to obtain a plurality of time segments, wherein there is no dependency between the multimedia materials included in different time segments in the video editing draft, and obtaining a complexity index of the encoding process corresponding to each time segment; Obtaining resource usage status information of an encoding device, and generating an encoding task on the encoding device according to the resource usage status information of the encoding device, wherein the encoding task is used to select a time segment of the video editing draft according to a complexity index to encode a corresponding video segment; The video segments corresponding to the multiple time segments are fused to obtain a target video.

2. The method according to claim 1, characterized in that The timeline of the video editing draft is segmented to obtain multiple time segments, including: Obtaining dependency information between multimedia materials included in the video editing draft, and determining the position of the segmentation point in a time range where no dependency exists according to the dependency information; The timeline of the video editing draft is segmented according to the segmentation point positions.

3. The method according to claim 2, characterized in that The step of obtaining dependency information between multimedia materials included in the video editing draft, and determining the position of a segmentation point in a time range where no dependency exists according to the dependency information, comprises: Obtaining relevant information of video special effects dependent on previous and next frames included in the video editing draft; Determine the time range corresponding to the video special effect according to the relevant information of the video special effect, and determine the time range outside the time range corresponding to the video special effect as the time range without dependency relationship; The split point positions are determined in the time range in which no dependency exists.

4. The method according to claim 2 or 3, characterized in that: The step of determining the split point position in a time range where no dependency exists comprises: Any intra-frame coding frame is selected in the time range where no dependency relationship exists, and is determined as the segmentation point position.

5. The method according to claim 1, characterized in that The obtaining of the complexity index of the encoding process corresponding to each time segment includes: For any time segment, obtaining multiple video attribute information of the time segment; According to the plurality of video attribute information and corresponding preset weights, a complexity index of a process corresponding to the time segment encoding is obtained.

6. The method according to claim 5, characterized in that The video attribute information includes one or more of the following: Parameters related to resolution, frame rate, bit rate, and video effects.

7. The method according to claim 5 or 6, characterized in that: After obtaining the complexity index of the encoding process corresponding to each time segment, the method further includes: According to the complexity index corresponding to each of the time segments, one or more time segments among the multiple time segments of the video editing draft are adjusted to finally obtain multiple adjusted time segments.

8. The method according to claim 7, characterized in that The step of adjusting one or more time segments of the plurality of time segments of the video editing draft according to the complexity index corresponding to each of the time segments includes: Segmenting the time segments whose complexity index exceeds a first preset threshold, wherein there is no dependency relationship between the multimedia materials included in the different time segments obtained by segmentation; and / or At least two adjacent time segments whose complexity index is lower than a second preset threshold are merged.

9. The method according to claim 1, characterized in that: The generating of the encoding task on the encoding device according to the resource usage status information of the encoding device comprises: Based on the resource usage status information of any encoding device among multiple encoding devices, determine the number of encoding task threads or processes that the current remaining resources of the encoding device can support, and generate encoding tasks of a corresponding number of threads or processes on the encoding device based on the number of threads or processes.

10. The method according to claim 1 or 9, characterized in that: After the encoding task is generated on the encoding device according to the resource usage status information of the encoding device, the method further includes: If the resource usage status information of any of the encoding devices does not exceed the preset usage status indicator, generating an encoding task of a new thread or process on the encoding device; or If the resource usage status information of any of the encoding devices exceeds a preset usage status indicator, the encoding task of at least one idle thread or process among the encoding tasks on the encoding device is destroyed.

11. The method according to claim 1, 5 or 6, characterized in that: After the encoding task is generated on the encoding device according to the resource usage status information of the encoding device, the method includes: For any encoding task on any encoding device, a time segment to be encoded is selected for encoding according to the performance of the encoding device and a complexity index corresponding to each time segment.

12. The method according to claim 11, characterized in that If the performance of a first encoding device among the multiple encoding devices is higher than that of a second encoding device, the complexity index corresponding to the time segment encoded by the encoding task on the first encoding device is higher than the complexity index corresponding to the time segment encoded by the encoding task on the second encoding device.

13. A video processing device, characterized in that: include: A segmentation unit is used to segment the timeline of the video editing draft to obtain a plurality of time segments, wherein there is no dependency between the multimedia materials included in different time segments in the video editing draft, and obtain a complexity index of the encoding process corresponding to each time segment; A task management unit, used to obtain resource usage status information of an encoding device, and generate an encoding task on the encoding device according to the resource usage status information of the encoding device, wherein the encoding task is used to select a time segment of the video editing draft according to a complexity index to encode a corresponding video segment; The fusion unit is used to fuse the video segments corresponding to the multiple time segments to obtain a target video.

14. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product, characterized in that The method comprises computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.

Images