Video processing method and apparatus, and electronic device
By dividing the video editing timeline into multiple time segments and caching the rendering data, the problem of lag in the preview effect during video editing was solved, thus improving the user experience.
Patent Information
- Application Number
- PCT/CN2024/100288
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-30
- Filing Date
- 2024-06-20
- Publication Date
- 2025-12-26
AI Technical Summary
The video editing process experienced stuttering when previewing the edited content, resulting in a poor user experience.
During video editing, the timeline is divided into multiple time segments based on the material track segments on the editing track. The material track segments within the target time segment are composited and rendered to generate video rendering data, which is then cached. When responding to preview commands, the rendered data is read from the cache and displayed.
It reduces stuttering when previewing editing effects during video editing, thus improving the user experience.
Smart Images
Figure CN2024100288_26122025_PF_FP_ABST
Abstract
Description
Video processing method, device and electronic device
[0001] The present application claims priority to the Chinese patent application No. 202310804891.5, filed on June 30, 2023, entitled "Video processing method, device and electronic device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD
[0002] Embodiments of the present disclosure relate to the technical field of computer, and particularly relate to a video processing method, device and electronic device. BACKGROUND
[0003] Video editing generally includes a process of adding special effects, scenes to the already recorded video, or mixing video clips from different video sources to obtain a new video.
[0004] The above video editing and editing effect preview process needs to occupy a large amount of system resources. When the electronic device displays the editing effect during video editing, the phenomenon of frame freezing usually occurs. And the user usually frequently previews the editing effect during the video editing process, and encounters the problem of frame freezing when previewing the editing effect multiple times, therefore, the user's video editing experience is poor.
[0005] SUMMARY
[0006] Embodiments of the present disclosure provide a video processing method, device and electronic device to overcome the problem of poor experience caused by frame freezing when previewing the editing effect during video editing in the related art.
[0007] In a first aspect, the embodiments of the present disclosure provide an image processing method, which includes: in a video editing process, dividing a time axis for video editing into a plurality of time periods according to material track segments on an edit track; performing synthetic rendering on material track segments on the edit track located in a target time period of the plurality of time periods, to generate video rendering data of the target time segment and cache; in response to receiving a video editing preview instruction, if a preview position is located in the target time period, reading rendering data of the preview position in the cached video rendering data of the target time period and displaying.
[0008] In a second aspect, the embodiments of the present disclosure provide an image processing apparatus, comprising: a segmentation unit configured to segment a timeline for video editing into a plurality of time periods according to material track segments on an edit track in a video editing process; a rendering unit configured to perform synthetic rendering on material track segments on the edit track within a target time period of the plurality of time periods, to generate video rendering data of the target time period and cache; and a display unit configured to, in response to receiving a video editing preview instruction, read rendering data of a preview position from the video rendering data of the target time period if the preview position is within the target time period and display the rendering data.
[0009] In a third aspect, the embodiments of the present disclosure provide an electronic device, comprising: a processor and a memory; the memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the video processing method as described in the first aspect and various possible aspects related to the first aspect.
[0010] In a fourth aspect, the embodiments of the present disclosure provide a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the video processing method as described in the first aspect and various possible aspects related to the first aspect is implemented.
[0011] In a fifth aspect, the embodiments of the present disclosure provide a computer program product, comprising a computer program, which, when executed by a processor, implements the video processing method as described in the first aspect and various possible aspects related to the first aspect. BRIEF DESCRIPTION OF DRAWINGS
[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present disclosure, and those skilled in the art can obtain other drawings according to these drawings without creative labor.
[0013] FIG. 1 is a schematic diagram of a video editing interface according to an embodiment of the present disclosure;
[0014] FIG. 2 is a schematic diagram of a video processing method according to an embodiment of the present disclosure;
[0015] FIG. 3 is a schematic diagram of segmentation of a timeline according to an embodiment of the present disclosure;
[0016] FIG. 4 is a schematic diagram of a video processing method according to an embodiment of the present disclosure;
[0017] FIG. 5 is a schematic diagram of a scene according to an embodiment of the present disclosure.
[0018] FIG. 6 is a schematic diagram of one scenario provided by the present disclosure;
[0019] FIG. 7 is a structural block diagram of a video processing apparatus provided by an embodiment of the present disclosure;
[0020] FIG. 8 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some but not all of the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present disclosure.
[0022] Please refer to FIG. 1, which shows a schematic diagram of a video editing. The schematic diagram is a process of mixing two video materials. The video editing mixes video material 10 and video material 20. The video editing interface can include track 11 and track 12 below the time axis 13. Track 11 includes a material track segment composed of multiple frames selected from video material 10. Track 12 can display a material track segment composed of multiple frames selected from video material 20.
[0023] The user drags the material track segment at track 11 and drags the material track segment at video frame area 12, so as to select the frames at which video material 10 and video material 20 start to be mixed. After multiple mixed frames of the two are determined, effect preview can be performed to browse the editing effect.
[0024] FIG. 1 is only one example of video editing. In practice, multiple video materials can be mixed, or text, special effect graphics, etc. can be added to a video. During the video editing process, the user usually previews the editing effect to watch the editing effect. If the electronic device used by the user to perform the video editing has a low resource configuration, or the material used by the user to perform the video editing is large itself, when the editing effect is previewed, since each frame needs to be rendered and a preview picture is generated according to the rendering result, since each frame corresponding to the rendering editing result itself needs more system resources, the preview editing result display will appear a lag phenomenon, resulting in poor user experience.
[0025] To improve the above problems, the scheme provided by the present disclosure divides the time axis into multiple time periods according to the material track segments on the edit track, generates video rendering data for a target time period in the multiple time periods and caches the video rendering data, and can generate a preview picture using the video rendering data of the target time period during preview. In addition, the video rendering data processed in the above manner can be reused, which can reduce the resources required for previewing the video editing effect and thus reduce the lag phenomenon.
[0026] Please refer to FIG. 2, which is a flowchart of a video processing method according to an embodiment of the present disclosure. The execution subject of the video processing method in the present disclosure can be a terminal device, and the method comprises:
[0027] S201: In the video editing process, the time axis for video editing is divided into multiple time periods according to the material track segments on the edit track.
[0028] In the present embodiment, various video editing applications can be run in the terminal device. The video material to be edited can be imported in the video editing application, and then editing operation is performed on the video material.
[0029] The video material to be edited can include video material obtained from various methods, including but not limited to: video material recorded using a video recording device, video material obtained from other electronic devices based on authorization, etc. The video material can also include one of the following: a preset special effect, a preset background, etc. The video material to be edited is only described by way of example here, and all video materials are not exhaustively listed, and the above examples do not limit the protection scope of the present disclosure.
[0030] Generally, in the video editing application, the time axis can be displayed in the video editing interface. The time axis generally refers to a thumbnail horizontal list that appears below the video preview and moves with the video playback during video editing. Through the time axis, the content of the picture displayed at a certain time point of the video can be controlled. In addition, by using the time axis and network technology, multiple events are connected in time sequence to form a relatively complete recording system.
[0031] For a video material in the video editing material, multiple video frames corresponding to the video material and arranged according to the display time are displayed below the time axis after transcoding processing of the video material. As shown in FIG. 1, for video 1, multiple video frames of video material 10 can be displayed below time axis 13.
[0032] In the video editing process, at least one material track segment can be included on the edit track.
[0033] The material track segment includes a text material track segment, a graphic special effect material track segment, or a video material track segment.
[0034] The timeline can be divided based on clips from the editing track, resulting in multiple time segments.
[0035] One implementation method is to use the start and end points of the source track segments as the timeline's dividing points, and then segment the timeline accordingly. For multiple source track segments, the start and end points of each segment can be used as dividing points to segment the timeline. For source track segments in different tracks, the start and end points of each track's source track segments can be used as the dividing points to segment the timeline.
[0036] By dividing the time axis using the aforementioned dividing points, multiple time periods can be obtained.
[0037] In this way, the starting point of each material track segment is the starting point of a time period, and at the same time, the ending point of the previous time period.
[0038] In the above time segmentation method, the end point of each material track segment is the end point of a time segment, and at the same time, the start point of the next time segment.
[0039] Within each time period, no more than one source track segment will exist within the same track during that time period.
[0040] Please refer to Figure 3, which illustrates a timeline segmentation diagram. Different video clips can correspond to different tracks; for example, video clip 1 and video clip 2 can correspond to track 1 and track 2, respectively. Assume that the first frame of track clip 1 appears at time 00:00 (t0) on track 1 (i.e., t0 is the starting point of track clip 1). Starting at t1, the first frame of track clip 2 appears in video clip 2 (time t1 is the starting point of track clip 2). Track clip 1 ends at time t2 (t2 is the ending point of track clip 1); and at time t3, track clip 2 ends (t3 is the ending point of track clip 2). Thus, since the timeline can be segmented using t0, t1, t2, and t3 as segmentation points, we obtain time period 1 (t0~t1), time period 2 (t1~t2), and time period 3 (t2~t3). t1 is the ending point of time period 1 and also the starting point of time period 2. t2 is the starting point of time period 3 and also the ending point of time period 2.
[0041] The plurality of time periods obtained by the above segmentation manner have good continuity of the material track segments in each track in the time period. In addition, each time period obtained by the above segmentation manner does not include a plurality of material track segments, so that the amount of calculation required for generating video rendering data for each time period is small, which helps to quickly generate video rendering data for each time period.
[0042] S202: Perform compositing rendering on the material track segments in the target time period in the plurality of time periods on the edit track, generate video rendering data of the target time period and cache.
[0043] After obtaining the plurality of time periods in step S101, a target time period in the plurality of time periods can be determined. The target time period here can be a time period corresponding to a larger number of material track segments in the plurality of time periods, or a time period corresponding to material track segments with more changes in picture information.
[0044] For the target time period, the corresponding material track segments in the target time period can be determined, and then a compositing rendering task is generated. The compositing rendering task is then executed. Specifically, the corresponding material track segments in the target time period can be composited according to a preset compositing rule, and then rendered to obtain video rendering data corresponding to the target time period.
[0045] For example, for each video frame corresponding to each material track segment aligned on the time axis in the target time period, the video frames are composited in units of frames, and then rendered. Or the video frames and graphic frames in the material track segments aligned on the time axis are composited in units of frames, and then rendered. Therefore, the video rendering data corresponding to the target time period can include compositing rendering data corresponding to a plurality of frames belonging to the target time period.
[0046] In practice, the compositing rendering task can be executed by invoking a rendering control management object provided by a video editing application.
[0047] For the compositing rendering tasks corresponding to the plurality of target time periods, the compositing rendering tasks can be executed in sequence according to a serial task execution manner, or can be executed in parallel according to a parallel task execution manner.
[0048] For each target time period, the video rendering data corresponding to the target time period can be cached.
[0049] In some embodiments, the above video processing method further includes: caching the video rendering data using a global storage path.
[0050] In these embodiments, the global storage path here can be a storage path other than a draft corresponding to the video material being edited.
[0051] After setting the global storage path, the video rendering data can be associated with the draft of the video material described above. In this way, when the draft name is modified, the video rendering data is not affected by the change of the draft name.
[0052] S203: In response to receiving the video editing preview instruction, if the preview position is located in the target time period, read the rendering data of the preview position in the video rendering data cached in the target time period and display.
[0053] During the editing of the video, the user often needs to browse the editing effect. The user can click the preview control that can be used to preview the editing effect during the editing operation, thereby sending the preview instruction to the execution subject described above.
[0054] After receiving the preview instruction, the preview position indicated by the preview instruction can be determined. If the preview position indicated by the preview instruction is located in the target time period, the rendering data corresponding to the preview position in the video rendering data cached in the target time period is read and displayed.
[0055] In this embodiment, during the video editing, the timeline for video editing is divided into multiple time periods according to the material track segments on the editing track; the material track segments on the editing track located in the target time period in the multiple time periods are synthesized and rendered to generate video rendering data of the target time period and cache; in response to receiving a video editing preview instruction, if the preview position is located in the target time period, the rendering data of the preview position in the video rendering data cached in the target time period is read and displayed. Since the timeline is divided into multiple time periods according to the material track segments on the editing track, and the material track segments in the target time period in the multiple time periods are synthesized and rendered, the preview picture is generated according to the video rendering data corresponding to the target time period when the video editing effect is previewed. The lag phenomenon caused by real-time rendering when the editing effect is previewed during video editing can be reduced, thereby improving the user experience during video editing.
[0056] Please refer to FIG. 4, which is a flowchart of a video processing method provided by an embodiment of the present disclosure. The execution subject of the video processing method in the present disclosure can be a terminal device, and the method comprises:
[0057] S401: In the video editing process, the timeline for video editing is divided into multiple time periods according to the material track segments on the editing track.
[0058] S402: The material track segments on the editing track located in the target time period in the multiple time periods are synthesized and rendered to generate video rendering data of the target time period and cache.
[0059] The implementation of steps S401-S402 can refer to the related description of the embodiment shown in FIG. 2. Details are not described herein.
[0060] S403: For each target time period, generate the first identifier corresponding to the target time period according to the picture data of the material track segment corresponding to the target time period; and set the first identifier for the cached video rendering data corresponding to the target time period.
[0061] The target time period can be a time period in the plurality of time periods obtained in step S401. Illustratively, the target time period can be a time period determined from the plurality of time periods according to a selection operation of a user. It can also be a time period determined according to the size of the picture information transformation in each time period.
[0062] The first identifier herein can be a first identifier obtained using various digital signature algorithms.
[0063] Illustratively, for the time period 2 (target time period) in FIG. 3, the digital signature algorithm can be used on the picture data in the material track segment 1 and the material track segment 2 corresponding to the time period 2, so as to obtain the first identifier of the time period 2. The first identifier obtained by the digital signature algorithm is usually a unique identifier.
[0064] Illustratively, the digital signature algorithm can be a message digest algorithm (MD5).
[0065] The first identifier has uniqueness and irreversibility. When the video editing data input to the digital signature algorithm changes, for example, the video frame is increased or decreased, or special effects or backgrounds are added, etc., the first identifier obtained by the digital signature algorithm will change.
[0066] Through the above steps, for a target time period in the plurality of time periods, the track material segment in the target time period can be synthesized and rendered to obtain video rendering data. The first identifier is cached and set for the video rendering data of the target time period. Since the first identifier has uniqueness, the video rendering data can be accurately found in the cache by the first identifier corresponding to the target time period.
[0067] For example, for the time period 2 in FIG. 2, the frame picture data of each frame of the material track segment 1 of the corresponding video material 1 in the time period 2 and the frame picture data of each frame of the material track segment 2 of the corresponding video material 2 can be pre-combined and rendered to obtain the video rendering data of the time period 2. The digital signature algorithm can be used to determine the first identifier of the time period 2 by using the multiple frame picture data of the material track segment 1 in the time period 2 and the multiple frame picture data of the material track segment 2, and the first identifier is set for the cached video rendering data of the time period 2.
[0068] Therefore, for the same time period, as long as the picture data of the material track segment of each track in the time period is not changed, the first identifier calculated by using the same digital signature algorithm on the picture data in the material track segment in the time period is the same. Based on this, when the editing effect is previewed, the video rendering data corresponding to each target time period can be accurately found according to the first identifier.
[0069] In addition, the first identifier is added to the identifier of the video rendering data of the target time period, which can avoid performing multiple combined rendering on the same material track segment in the same time period.
[0070] S404: In response to receiving the video editing preview instruction, if the preview position is located in the target time period, the first identifier corresponding to the target time period is determined.
[0071] S405: The rendering data corresponding to the preset preview position is determined from the video rendering data corresponding to the first identifier read from the cache.
[0072] After receiving the video editing preview instruction, the time period in which the video editing preview instruction falls can be determined. For example, the first identifier of the time period can be generated according to the picture data of each material editing segment of the time period according to the algorithm for generating the first identifier, and then the first identifier is matched in the cache. If the matching is successful, it indicates that the time period is the target time period. Then the video rendering data corresponding to the target time period is extracted from the cache.
[0073] Then the rendering data corresponding to the preview position is extracted from the video rendering data corresponding to the target time period.
[0074] In the embodiment, compared with the scheme of the embodiment shown in FIG. 2, the first identifier related to the picture data of the material track segment is generated for the target time period, and the video rendering data of the target time period is cached according to the first identifier. Therefore, when the preview instruction is received, the video rendering data of the target time period can be accurately and quickly found from the cache according to the first identifier, and the rendering data corresponding to the preview position is extracted. The accuracy of the preview effect displayed by the preview instruction can be improved.
[0075] In some embodiments, the video processing method further comprises: for the video rendering data of the target time period, establishing a mapping relationship between the video rendering data and frame information of the material track segment in the target time period; and the step S405 further comprises the following steps:
[0076] Firstly, according to the first identifier of the target time period, the video rendering data corresponding to the target time period is determined from the cache.
[0077] Secondly, according to the mapping relationship, the rendering data of the preview position is determined from the video rendering data corresponding to the target time period.
[0078] Here, the mapping relationship between the video rendering data and the frame information of the material track segment is established, which can determine which frames in the material correspond to a video rendering data. When the preview instruction indicates the rendering data corresponding to a target time period, the video rendering data corresponding to the target time period can be located according to the first identifier, and the rendering data corresponding to the preview position indicated by the preview instruction can be extracted from the video rendering data according to the frame information, so that the editing effect of the preview position can be accurately and quickly displayed.
[0079] In some embodiments, the step S404 comprises the following steps:
[0080] The frame information of the material track segment corresponding to the target time period in the corresponding material is determined; and the second identifier of the video rendering data of the target time period generated by the frame information is used to establish a mapping relationship between the video rendering data and the frame information corresponding to the target time period.
[0081] Illustratively, for the video rendering data in the time period 1 in FIG. 3, the frame information of the time period 1 can be the frame information "1-5 frame video images" of the material track segment in the video material 1. The second identifier, for example, can be "video material 1 video rendering data corresponding frame (1-5.mp4)". By generating the second identifier including the frame information for the target time period, and setting the second identifier as part of the identifier of the video rendering data of the target time period, a mapping relationship between the video rendering data of the target time period and the frame information corresponding to the target time period is established.
[0082] In these embodiments, the second identifier generated by the frame information can be set for the video rendering data. Then the rendering data corresponding to the video frame in the target time period can be quickly determined according to the second identifier, so that the rendering data corresponding to the preview position can be quickly determined from the cache according to the second identifier, and the speed of previewing the editing effect can be further improved.
[0083] In some embodiments, the determining the frame information of the material track segment corresponding to the target time period in the corresponding material comprises: for the multi-track material track segment in the target time period, determining the frame information of the material track segment corresponding to the target time period in the corresponding material according to the frame information of the material track segment appearing latest on the time axis in the corresponding material.
[0084] In these embodiments, the edit track can comprise a plurality of tracks, and the material can comprise a plurality of materials.
[0085] The following is described in conjunction with FIG. 5, which is a schematic diagram of an application scenario provided by the present disclosure. In the editing process, the material track segment 3 corresponding to the target time period D on the time axis is a segment from the 100th frame to the 200th frame of the video material 1. The material track segment 4 is a material track segment extracted from the video material 2. The material track segment of the video material 2 corresponding to the target time period D is a segment from the 75th frame to the 175th frame of the video material 2. The user performs a mixing editing operation on the material track segment 3 and the material track segment 4. On the edit track, the appearance time of the material track segment 4 is later than that of the material track segment 3. The first frame of the video material 4 can be used as the starting position for determining the frame information of the target time period D. This is because the video material 1 and the video material 2 are mixed at most from the first frame of the video material 2.
[0086] In addition, a first identifier E can be generated according to the picture data from the 100th frame to the 200th frame of the video material 1 and the 75th frame to the 175th frame of the video material 2. A second identifier can be generated according to the frame information from the 75th frame to the 175th frame of the video material 2. The video rendering data of the material track segment 3 and the material track segment 4 corresponding to the above-mentioned target time period D can be set with the first identifier and the second identifier. The mapping relationship between the frame information 75-175 and the video rendering data is established by using the second identifier. The second identifier can be, for example, “75-175.mp4”.
[0087] In these embodiments, for the editing scenario of the multi-track video material, for the target time period, the frame mapping information between the video rendering data of the target time period and the track material segment is determined according to the frame information of the track material segment appearing latest on the time axis in the corresponding material, so that the video rendering data generated by the historical editing operation can be reused when the user performs subsequent editing operation on the same video material. Therefore, system resources can be saved.
[0088] In some embodiments, the video processing method further comprises the following steps:
[0089] First, in response to receiving a re-edit operation performed on the material track segment, the timeline is segmented according to information of the material track segment on the edit track.
[0090] Second, a new target time period that has no overlap with the historical target time period is determined from the segmentation result, and video rendering data is generated for the material track segment corresponding to the new target time period and is cached.
[0091] In these embodiments, video editing usually requires multiple editing operations on video materials to finally generate a video editing result that the user is satisfied with.
[0092] After the video rendering data is generated according to the previous editing operation and previewed, the user can continue to perform editing operations on the video editing materials.
[0093] When receiving a re-edit operation of the user on the video editing materials, the timeline is segmented again according to the segmentation process of step S401, and multiple segmentation results can be obtained.
[0094] For each segmentation result, it can be judged whether the segmentation result overlaps with the historical target time period. If so, no new target time period and corresponding video rendering data are generated. If part of the segmentation result overlaps with the historical target time period, a new target time period can be generated according to the non-overlapping part. If the segmentation result has no overlapping part with the historical target time period, the segmentation result can be used as a new target time period.
[0095] For the new target time period, video rendering data is generated, and the video rendering data is cached. When caching, the video rendering data is set with a first identifier and a second identifier. The first identifier is generated from the picture data of the new target time period, and the second identifier is generated from the frame information of the material track segment corresponding to the new target time period in the corresponding material.
[0096] When receiving a preview instruction for previewing the re-edit operation, for the target time period overlapping with the historical target time period, the video rendering data corresponding to the historical target time period can be reused for preview presentation.
[0097] In combination with FIG. 5 and FIG. 6, FIG. 6 is a schematic diagram of an application scenario provided by the present disclosure. For the mixing editing operation performed by the user in FIG. 5 on the 100th frame to the 200th frame of the video material 1 and the 75th frame to the 175th frame of the video material 2. In the subsequent editing operation, the video material 1 is moved to the left on the time axis, and the number of frames in which the video material 1 is mixed with the video material 2 is increased. As shown in FIG. 6, after the re-editing operation of the user, the 75th frame to the 200th frame of the video material 1 is mixed with the 50th frame to the 175th frame of the video material 2. That is, the video material 1 corresponds to the material track segment 3', and the video material 2 corresponds to the material track segment 4, as shown in FIG. 6.
[0098] For the editing operation shown in FIG. 6, the new target time period F corresponding to the 75th frame to the 99th frame of the video material 1 can be generated in the above manner. For the new target time period F, the first identifier P is generated according to the picture data of the 75th frame to the 99th frame of the video material 1 and the 50th frame to the 74th frame of the video material 2. The second identifier of the new target time period is generated according to the frame information of the track material segment 4 in the video material 2 corresponding to the new target time period F. The video rendering data of the new target time period is generated, and the mapping relationship between the video rendering data and the frame information corresponding to the new target time period is established by using the second identifier. The video rendering data information is cached, and the identifier including the first identifier and the second identifier is used for the video rendering data.
[0099] For the 100th frame to the 200th frame of the video material 1 and the 75th frame to the 175th frame of the video material 2, the video rendering data corresponding to the original target time period D can be reused. In this way, when previewing the editing operation in FIG. 6, the video rendering data corresponding to the target time period D generated by the historical editing operation shown in FIG. 5 can be directly used for preview. When the preview position is located in the target time period F, the newly generated video rendering data can also be used for preview.
[0100] In these embodiments, for multiple editing operations, the video rendering data corresponding to the original historical target time period can be reused, thereby reducing the resources required for rendering respectively required by multiple editing operations. Further, the user can display the preview picture generated according to the video rendering data more smoothly when previewing each time.
[0101] The video processing method, device and electronic equipment provided in this embodiment divide a time axis for video editing into multiple time periods according to material track segments on an edit track in a video editing process; perform composite rendering on material track segments on the edit track in a target time period among the multiple time periods, generate video rendering data of the target time period and cache; in response to receiving a video editing preview instruction, if a preview position is in the target time period, read rendering data of the preview position from the cached video rendering data of the target time period and display. Since the time axis for video editing is divided into multiple time periods according to material track segments on an edit track, and then the composite rendering is performed on material track segments in a target time period, when the video editing effect is previewed, a preview video is generated according to video rendering data corresponding to different target time periods. The phenomenon of lag caused by real-time rendering when the editing effect is previewed in the video editing process can be reduced, and thus the user experience in the video editing process can be improved.
[0102] Corresponding to the video processing method of the above embodiment, FIG. 7 is a structural block diagram of a video processing device provided in an embodiment of the present disclosure. For ease of illustration, only parts related to the embodiments of the present disclosure are shown. Referring to FIG. 7, the device 70 includes a division unit 701, a rendering unit 702 and a display unit 703. Among them:
[0103] The division unit 701 is configured to divide a time axis for video editing into multiple time periods according to material track segments on an edit track in a video editing process.
[0104] The rendering unit 702 is configured to perform composite rendering on material track segments on an edit track in a target time period among the multiple time periods, generate video rendering data of the target time period and cache.
[0105] The display unit 703 is configured to, in response to receiving a video editing preview instruction, if a preview position is in the target time period, read rendering data of the preview position from the cached video rendering data of the target time period and display.
[0106] In some embodiments of the present disclosure, the start point of each material track segment is the start point of a time period and the end point of a previous time period of the time period; and the end point of each material track segment is the end point of a time period and the start point of a next time period of the time period.
[0107] In some embodiments of the present disclosure, the apparatus 70 further comprises an identification configuration unit (not shown in the figure). The identification configuration unit is configured to: for each target time period, generate a first identification corresponding to the target time period according to the picture data of the material track segment corresponding to the target time period; and set the first identification for the cached video rendering data corresponding to the target time period.
[0108] In some embodiments of the present disclosure, the display unit 703 is further configured to: determine the first identification corresponding to the target time period; and determine the rendering data corresponding to the preset preview position from the video rendering data corresponding to the first identification read from the cache.
[0109] In some embodiments of the present disclosure, the apparatus 70 further comprises a mapping unit (not shown in the figure), which is configured to: for the video rendering data of the target time period, establish a mapping relationship between the video rendering data and the frame information of the material track segment in the target time period; and the display unit 703 is further configured to: determine the video rendering data corresponding to the target time period from the cache according to the first identification of the target time period; and determine the rendering data of the preview position from the video rendering data corresponding to the target time period according to the mapping relationship.
[0110] In some embodiments of the present disclosure, the mapping unit is further configured to: determine the frame information of the material track segment corresponding to the target time period in the corresponding material; and establish a mapping relationship between the video rendering data of the target time period and the frame information corresponding to the target time period by using the second identification of the video rendering data generated by the frame information.
[0111] In some embodiments of the present disclosure, the mapping unit is further configured to: for the multi-track material track segment in the target time period, determine the frame information of the material track segment corresponding to the target time period in the corresponding material according to the frame information of the material track segment appearing latest on the time axis in the corresponding material.
[0112] In some embodiments of the present disclosure, the splitting unit is further configured to: in response to receiving a re-editing operation performed on the material track segment, split the time axis according to the information of the material track segment on the edit track; and the rendering unit is further configured to: determine a new target time period from the splitting result without overlapping with the historical target time period, generate video rendering data for the material track segment corresponding to the new target time period, and cache the video rendering data.
[0113] In some embodiments of the present disclosure, the apparatus 70 further comprises a storage unit (not shown in the figure), which is configured to: store the video rendering data using a global storage path.
[0114] To implement the above-mentioned embodiments, the present disclosure further provides an electronic device.
[0115] Referring to FIG. 8, a structural diagram of an electronic device 800 suitable for implementing embodiments of the present disclosure is shown, which can be various terminal devices, servers. Among them, the terminal device can include an extended reality device. The extended reality device may, for example, include various wearable devices, such as a head-mounted extended reality device, etc. The electronic device shown in FIG. 8 is merely an example and should not impose any limitation on the functions and use range of embodiments of the present disclosure.
[0116] As shown in FIG. 8, the electronic device 800 can include a processing device (such as a central processor, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0117] Generally, the following devices can be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication devices 809 can allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 8 shows the electronic device 800 with various devices, it should be understood that it is not required to implement or have all the shown devices. More or fewer devices can be alternatively implemented or provided.
[0118] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 809, or installed from the storage devices 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.
[0119] It is noted that the aforementioned computer-readable medium of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example and without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wire, cable, RF, etc., or any suitable combination of the foregoing.
[0120] The aforementioned computer-readable medium can be included in the aforementioned electronic device; or can exist separately from the electronic device and be not assembled in the electronic device.
[0121] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the aforementioned embodiments.
[0122] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).
[0123] The flow diagrams and the block diagrams in the drawings are meant as possible implementations of the systems, methods and computer program products according to the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that carries out specified functions or operations (e.g., a combination of analog and digital circuitry), or can be implemented by a combination of dedicated hardware and computer instructions.
[0124] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself. For example, the dividing unit can also be described as a unit that divides a time axis for video editing into multiple time periods according to the material track segments on the edit track in a video editing process.
[0125] The functionality described herein above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.
[0126] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0127] In a first aspect, according to one or more embodiments of the present disclosure, a video processing method is provided, including: in a video editing process, dividing a time axis for video editing into a plurality of time periods according to material track segments on an edit track; performing synthetic rendering on material track segments on the edit track within a target time period in the plurality of time periods to generate video rendering data of the target time segment and cache; in response to receiving a video editing preview instruction, if a preview position is located in the target time period, reading rendering data of the preview position from the cached video rendering data of the target time period and displaying.
[0128] According to one or more embodiments of the present disclosure, the start point of each material track segment is the start point of a time period, and is also the end point of a previous time period of the time period; the end point of each material track segment is the end point of a time period, and is also the start point of a next time period of the time period.
[0129] According to one or more embodiments of the present disclosure, the method further includes: for each target time period, generating a first identifier corresponding to the target time period according to picture data of a material track segment corresponding to the target time period; and setting the first identifier for the cached video rendering data corresponding to the target time period.
[0130] According to one or more embodiments of the present disclosure, in response to receiving the video editing preview instruction, if the preview position is located in the target time period, reading and displaying the rendering data of the preview position in the video rendering data cached in the target time period comprises: determining a first identifier corresponding to the target time period; and determining the rendering data corresponding to the preset preview position from the video rendering data corresponding to the first identifier read from the cache.
[0131] According to one or more embodiments of the present disclosure, the method further comprises: for the video rendering data of the target time period, establishing a mapping relationship between the video rendering data and frame information of the material track segment in the target time period; and determining the rendering data corresponding to the preview position from the video rendering data corresponding to the first identifier read from the cache comprises: determining the video rendering data corresponding to the target time period from the cache according to the first identifier of the target time period; and determining the rendering data of the preview position from the video rendering data corresponding to the target time period according to the mapping relationship.
[0132] According to one or more embodiments of the present disclosure, for the rendering data of each time period, establishing a mapping relationship between the rendering data and frame information of the material track segment in the time period comprises: determining frame information of the material track segment corresponding to the target time period in the corresponding material; and establishing a mapping relationship between the video rendering data of the target time period and the frame information corresponding to the target time period by using a second identifier of the video rendering data of the target time period generated from the frame information.
[0133] According to one or more embodiments of the present disclosure, determining the frame information of the material track segment corresponding to the target time period in the corresponding material comprises: for the multi-track material track segment in the target time period, determining the frame information of the material track segment corresponding to the target time period in the corresponding material according to the frame information of the material track segment appearing latest on the time axis in the corresponding material.
[0134] According to one or more embodiments of the present disclosure, the method further comprises: in response to receiving a re-editing operation performed on the material track segment, dividing the time axis according to information of the material track segment on the edit track; determining a new target time period from the division result without overlapping with the historical target time period, and generating video rendering data for the material track segment corresponding to the new target time period and caching the video rendering data.
[0135] According to one or more embodiments of the present disclosure, the method further comprises: storing the video rendering data using a global storage path.
[0136] In a second aspect, according to one or more embodiments of the present disclosure, a video processing apparatus is provided, comprising: a segmentation unit configured to segment a timeline for video editing into a plurality of time periods according to material track segments on an edit track in a video editing process; a rendering unit configured to perform a composite rendering on the material track segments on the edit track within a target time period of the plurality of time periods, to generate video rendering data of the target time period and cache the video rendering data; and a display unit configured to, in response to receiving a video editing preview instruction, read rendering data of a preview position from the video rendering data of the target time period cached if the preview position is within the target time period and display the rendering data.
[0137] According to one or more embodiments of the present disclosure, a start point of each material track segment is a start point of a time period and a terminal point of a previous time period of the time period; and a terminal point of each material track segment is a terminal point of a time period and a start point of a next time period of the time period.
[0138] According to one or more embodiments of the present disclosure, the apparatus further comprises an identification configuration unit. The identification configuration unit is configured to: for each target time period, generate a first identification corresponding to the target time period according to picture data of a material track segment corresponding to the target time period; and set the first identification for the video rendering data corresponding to the target time period cached.
[0139] According to one or more embodiments of the present disclosure, the display unit is further configured to: determine the first identification corresponding to the target time period; and determine rendering data corresponding to a preset preview position from the video rendering data corresponding to the first identification read from the cache.
[0140] According to one or more embodiments of the present disclosure, the video processing apparatus further comprises a mapping unit. The mapping unit is configured to: for the video rendering data of the target time period, establish a mapping relationship between the video rendering data and frame information of a material track segment in the target time period; and the display unit is further configured to: determine the video rendering data corresponding to the target time period from the cache according to the first identification of the target time period; and determine the rendering data of the preview position from the video rendering data corresponding to the target time period according to the mapping relationship.
[0141] According to one or more embodiments of the present disclosure, the mapping unit is further configured to: determine frame information of the material track segment corresponding to the target time period in a corresponding material; and establish a mapping relationship between the video rendering data of the target time period and the frame information corresponding to the target time period by using a second identification of the video rendering data of the target time period generated from the frame information.
[0142] According to one or more embodiments of the present disclosure, the mapping unit is further configured to: for the multi-track material track segment within the target time period, determine frame information of the target time period corresponding material track segment in the corresponding material according to frame information of the latest appearing material track segment on the timeline in the corresponding material.
[0143] According to one or more embodiments of the present disclosure, the splitting unit is further configured to: in response to receiving a re-edit operation performed on the material track segment, split the timeline according to information of the material track segment on the edit track; and the rendering unit is further configured to: determine a new target time period from the splitting result without overlapping with the historical target time period, and generate video rendering data and cache for the new target time period corresponding material track segment.
[0144] The video processing apparatus further includes a storage unit configured to: store the video rendering data using a global storage path.
[0145] In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, including: at least one processor and a memory; the memory stores computer-executable instructions; and the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video processing method of the first aspect and various possible aspects related to the first aspect.
[0146] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer-executable instructions, when a processor executes the computer-executable instructions, the video processing method of the first aspect and various possible aspects related to the first aspect is implemented.
[0147] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the video processing method of the first aspect and various possible aspects related to the first aspect.
[0148] The above description is merely preferred embodiments of the present disclosure and a description of the principles of the technology applied. Those skilled in the art should understand that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the disclosed concept. For example, the above technical features can be replaced with technical features disclosed in the present disclosure (but not limited to) having similar functions to form technical solutions.
[0149] Moreover, while operations are depicted in a particular order, this should not be understood as requiring such an order nor infringing on the scope of the disclosure. Certain of the operations described in the discussion are combinable into a single operation, and certain operations can be separated into several operations. In some embodiments, the operations described in the discussion can be performed in an order different than presented in the discussion. In some embodiments, the operations described in the discussion can be performed concurrently. Also, while several specific implementation details are discussed in the discussion, these should not be interpreted as limiting the scope of the disclosure. Rather, certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.
[0150] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
Claims
1. A video processing method, comprising: During video editing, the timeline used for video editing is divided into multiple time segments based on the material track segments on the editing track; The material track segments located within the target time period in the plurality of time periods on the editing track are composited and rendered to generate video rendering data for the target time segment and cached. In response to receiving a video editing preview command, if the preview location is within the target time period, the rendering data of the preview location is read from the cached video rendering data of the target time period and displayed.
2. The method according to claim 1, wherein, The starting point of each material track segment is the starting point of a time period and the ending point of the previous time period. The end point of each material track segment is the end point of a time period, and at the same time, the start point of the next time period.
3. The method according to claim 1, wherein, The method further includes: For each target time period, a first identifier corresponding to the target time period is generated based on the image data of the material track segment corresponding to the target time period; and the first identifier is set for the cached video rendering data corresponding to the target time period.
4. The method according to claim 3, wherein, The step of responding to a received video editing preview command, if the preview location is within the target time period, reading and displaying the rendering data of the preview location from the cached video rendering data of the target time period, includes: Determine the first identifier corresponding to the target time period; The rendering data corresponding to the preview position is determined from the video rendering data corresponding to the first identifier read from the cache.
5. The method according to claim 4, wherein, The method further includes: For video rendering data within a target time period, establish a mapping relationship between this video rendering data and the frame information of the source track segments within that target time period; and Determining the rendering data corresponding to the preview position from the video rendering data corresponding to the first identifier read from the cache includes: Based on the first identifier of the target time period, determine the corresponding time period from the cache. Video rendering data; The rendering data for the preview position is determined from the video rendering data corresponding to the target time period based on the mapping relationship.
6. The method according to claim 5, wherein, The process of establishing a mapping relationship between the rendering data for each time period and the frame information of the material track segments within that time period includes: Determine the frame information of the media track segment corresponding to the target time period within the corresponding media; Using the second identifier of the video rendering data for the target time period generated from the frame information, a mapping relationship is established between the video rendering data and the frame information corresponding to the target time period.
7. The method according to claim 6, wherein, The step of determining the frame information of the material track segment corresponding to the target time period in the corresponding material includes: For multi-track material segments within the target time period, the frame information of the material track segment corresponding to the target time period in the corresponding material is determined based on the frame information of the material track segment that appears latest on the timeline in the corresponding material.
8. The method according to claim 1, wherein, The method further includes: In response to receiving a re-edit operation performed on the material track segment, the timeline is segmented according to the information of the material track segment on the edit track; From the segmentation results, a new target time period that does not overlap with the historical target time period is determined. For the material track segment corresponding to the new target time period, video rendering data is generated and cached.
9. The method according to any one of claims 1 to 8, wherein, The method further includes: The video rendering data is stored using a global storage path.
10. A video processing apparatus, comprising: A segmentation unit is used to divide the timeline for video editing into multiple time segments based on the material track segments on the editing track during the video editing process; The rendering unit is used to composite and render the material track segments located in the target time period of the multiple time periods on the editing track, generate video rendering data of the target time segment and cache it; The display unit is configured to respond to a received video editing preview instruction, and if the preview position is within the target time period, read the rendering data of the preview position from the cached video rendering data of the target time period and display it.
11. An electronic device, comprising: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the video processing method as described in any one of claims 1 to 9.
12. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the video processing method as described in any one of claims 1 to 9.
13. A computer program product comprising a computer program that, when executed by a processor, implements the video processing method as described in any one of claims 1 to 9.