A video enhancement method, device, storage medium and computer program product

Abstract

Embodiments of the present application provide a video enhancement method and device, a storage medium and a computer program product. The method comprises: obtaining a first video stream and a first data stream; wherein the first data stream comprises transcoding operation information and data enhancement information; the data enhancement information is used for enhancement processing of a first video frame; a second video stream comprises a video stream obtained by transcoding the first video stream; the first video frame is a newly added video frame in the second video stream; a first label is assigned to the first video frame based on the transcoding operation information; first position information of the first video frame is determined based on the first label; wherein the first position information comprises a predicted timing position of the first video frame in the second video stream; a mapping relationship between the first video frame and the data enhancement information is established based on the first position information; and the first video frame is processed based on the mapping relationship, thereby improving the playing quality of the video stream.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video enhancement method, device, storage medium, and computer program product. Background Technology

[0002] Currently, video enhancement capabilities for on-demand content are primarily achieved by inserting SEI (Supplemental Enhancement Information) into video frames. Specifically, the SEI information of a video frame carries a specific identifier field. The corresponding supplemental enhancement information is then searched in a separate extended data stream to establish a connection between the video frame and the extended data. During playback, when the video renders a particular frame, the corresponding enhancement data is matched against the identifier in the frame's SEI in the extended data stream to complete enhancement services such as video effects, interactive elements, and intelligent subtitles.

[0003] However, data synchronization between video frames and external metadata streams is a key bottleneck in the aforementioned solutions. Due to the inherent limitations of I-frames, P-frames, and B-frames in video encoding, and the need for decoding rearrangement, coupled with the further transcoding required by various business scenarios (such as frame interpolation and frame rate conversion), the frame order in the final playback stream often differs from the order in which SEI information was inserted. This leads to discontinuities and misalignments in the tag indexes carried by the SEI in continuously played frames, disrupting the accurate synchronization between video frames and the enhancement information stream. The enhancement data cannot be precisely applied to the target frames, affecting the final enhancement effect and ultimately degrading the quality of video playback. Summary of the Invention

[0004] This application provides a video enhancement method, device, storage medium, and computer program product that can ensure accurate synchronization between video frames and enhanced information streams, thereby improving the playback quality of the video stream.

[0005] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a video enhancement method, the method comprising: Acquire a first video stream and a first data stream; wherein, the first data stream includes transcoding operation information and data enhancement information; the transcoding operation information is used to mark the type of the first video frame in the second video stream, the data enhancement information is used to enhance the first video frame, the second video stream includes a video stream after transcoding the first video stream, and the first video frame includes newly added video frames in the second video stream. A first tag is assigned to the first video frame based on the transcoding operation information, and a first position information of the first video frame is determined based on the first tag; wherein, the first position information includes the predicted temporal position of the first video frame in the second video stream, and the first tag is used to characterize the feature information corresponding to the first video frame. Based on the first location information, a mapping relationship is established between the first video frame and the data enhancement information, and the first video frame is enhanced based on the mapping relationship.

[0006] Secondly, embodiments of this application provide a video enhancement device, the video enhancement device comprising: a processor and a memory; wherein, The memory is used to store computer programs that can run on the processor; The processor is configured to execute the video enhancement method described above when running the computer program.

[0007] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer program code, which, when executed by a computer, implements the video enhancement method described above.

[0008] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the video enhancement method described above.

[0009] This application provides a video enhancement method, device, storage medium, and computer program product. The method includes: acquiring a first video stream and a first data stream; wherein the first data stream includes transcoding operation information and data enhancement information; the transcoding operation information is used to mark the type of a first video frame in a second video stream, and the data enhancement information is used to enhance the first video frame; the second video stream includes a video stream after transcoding the first video stream, and the first video frame includes newly added video frames in the second video stream; assigning a first tag to the first video frame based on the transcoding operation information, and determining a first position information of the first video frame based on the first tag; wherein the first position information includes the predicted temporal position of the first video frame in the second video stream, and the first tag is used to characterize the feature information corresponding to the first video frame; establishing a mapping relationship between the first video frame and the data enhancement information based on the first position information, and enhancing the first video frame based on the mapping relationship. Therefore, in the embodiment of this application, during the stage of transcoding the first video stream to generate the second video stream, the first data stream synchronously carries transcoding operation information, directly marking the type of the newly added first video frame in advance. Then, a first tag can be assigned to the first video frame based on the transcoding operation information. The first tag represents the feature information corresponding to the first video frame, and the first position information of the first video frame can be determined based on the first tag. That is, after assigning the first tag to the first video frame, this embodiment of the application directly calculates the predicted temporal position of this frame in the final transcoded stream, using dynamic position instead of traditional sequential index. Even if the original sequence of the original frame is disrupted due to B-frame rearrangement or frame insertion, the position information remains accurate. Indeed; finally, based on the first location information, a mapping relationship between the first video frame and the data enhancement information can be established, and the first video frame can be enhanced based on the mapping relationship. That is, the embodiments of this application can directly establish a mapping relationship between enhancement information and video frames based on dynamically determined location information, thereby getting rid of the dependence on traditional SEI sequential indexing. No matter how many interpolated frames are added after transcoding or whether the original frame order is disrupted, it can ensure that each video frame and the corresponding enhancement information are accurately aligned, thus improving the playback quality of the video stream; and there is no need to store additional index tag information in the SEI area of ​​the video frame, avoiding increasing the redundant volume of the video bitstream and not occupying additional bandwidth and storage space. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the video enhancement method proposed in an embodiment of this application; Figure 2 This is a schematic diagram of the structural composition of the terminal video enhancement playback system proposed in the embodiments of this application; Figure 3 This is a schematic diagram of the composition structure of the video enhancement device proposed in the embodiments of this application; Figure 4This is a schematic diagram of the composition structure of the video enhancement device proposed in the embodiments of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the accompanying drawings.

[0012] Currently, video enhancement capabilities for on-demand content are primarily achieved by inserting SEI (Supplemental Enhancement Information) into video frames. Specifically, the SEI information of a video frame carries a specific identifier field. The corresponding supplemental enhancement information is then searched in a separate extended data stream to establish a connection between the video frame and the extended data. During playback, when the video renders a particular frame, the corresponding enhancement data is matched against the identifier in the SEI of that frame in the extended data stream to complete enhancement services such as video effects, interactive effects, and intelligent subtitles.

[0013] However, data synchronization between video frames and external metadata streams is a key bottleneck in the aforementioned solutions. Due to the inherent limitations of I-frames, P-frames, and B-frames in video encoding, and the need for decoding rearrangement, coupled with the further transcoding required by various business scenarios (such as frame interpolation and frame rate conversion), the frame order in the final playback stream often differs from the order in which SEI information was inserted. This leads to discontinuities and misalignments in the tag indexes carried by the SEI in continuously played frames, disrupting the accurate synchronization between video frames and the enhancement information stream. The enhancement data cannot be precisely applied to the target frames, affecting the final enhancement effect and ultimately degrading the quality of video playback.

[0014] To address the problem that current video enhancement methods negatively impact the final enhancement effect, leading to a decrease in video playback quality, this application provides a video enhancement method, device, storage medium, and computer program product. The method includes: acquiring a first video stream and a first data stream; wherein the first data stream includes transcoding operation information and data enhancement information; the transcoding operation information is used to mark the type of a first video frame in a second video stream, and the data enhancement information is used to enhance the first video frame; the second video stream includes a video stream after transcoding the first video stream, and the first video frame includes newly added video frames in the second video stream; assigning a first tag to the first video frame based on the transcoding operation information, and determining first position information of the first video frame based on the first tag; wherein the first position information includes the predicted temporal position of the first video frame in the second video stream, and the first tag is used to characterize the feature information corresponding to the first video frame; establishing a mapping relationship between the first video frame and the data enhancement information based on the first position information, and enhancing the first video frame based on the mapping relationship. Therefore, in the embodiment of this application, during the stage of transcoding the first video stream to generate the second video stream, the first data stream synchronously carries transcoding operation information, directly marking the type of the newly added first video frame in advance. Then, a first tag can be assigned to the first video frame based on the transcoding operation information. The first tag represents the feature information corresponding to the first video frame, and the first position information of the first video frame can be determined based on the first tag. That is, after assigning the first tag to the first video frame, this embodiment of the application directly calculates the predicted temporal position of this frame in the final transcoded stream, using dynamic position instead of traditional sequential index. Even if the original sequence of the original frame is disrupted due to B-frame rearrangement or frame insertion, the position information remains accurate. Indeed; finally, based on the first location information, a mapping relationship between the first video frame and the data enhancement information can be established, and the first video frame can be enhanced based on the mapping relationship. That is, the embodiments of this application can directly establish a mapping relationship between enhancement information and video frames based on dynamically determined location information, thereby getting rid of the dependence on traditional SEI sequential indexing. No matter how many interpolated frames are added after transcoding or whether the original frame order is disrupted, it can ensure that each video frame and the corresponding enhancement information are accurately aligned, thus improving the playback quality of the video stream; and there is no need to store additional index tag information in the SEI area of ​​the video frame, avoiding increasing the redundant volume of the video bitstream and not occupying additional bandwidth and storage space.

[0015] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0016] This application provides a video enhancement method. Figure 1 This is a schematic diagram of the video enhancement method proposed in the embodiments of this application, such as... Figure 1 As shown, a video enhancement method may include the following steps: Step 101: Obtain the first video stream and the first data stream; wherein, the first data stream contains transcoding operation information and data enhancement information; the transcoding operation information is used to mark the type of the first video frame in the second video stream, and the data enhancement information is used to enhance the first video frame; the second video stream contains the video stream after transcoding the first video stream, and the first video frame contains the newly added video frame in the second video stream.

[0017] In embodiments of this application, the video enhancement device can acquire a first video stream and a first data stream.

[0018] It should be noted that, in the embodiments of this application, the video enhancement device may include a mobile terminal, a tablet, or other devices, and this application does not specifically limit the types of devices included in the video enhancement device.

[0019] It should be noted that, in the embodiments of this application, the first data stream may include a metadata stream, which may contain transcoding operation information and data enhancement information. This application does not specifically limit the type and quantity of information included in the first data stream.

[0020] It should be noted that, in the embodiments of this application, the data enhancement information may include one or more metadata blocks, which are used to carry corresponding special effects, subtitles, interactive and other enhancement content, so as to enhance the first video frame. This application does not specifically limit the information type included in the data enhancement information.

[0021] It should be noted that, in the embodiments of this application, the first video frame includes newly added video frames in the second video stream, and may also include reconstructed video frames, and may also include other types of video frames. This application does not specifically limit the types of video frames included in the first video frame.

[0022] For example, in the embodiments of this application, assuming the video enhancement device is a mobile terminal, when acquiring the first video stream and the first data stream, the video application (APP) in the mobile terminal can download the first video stream and the first data stream from the server side for subsequent playback services to complete the video enhancement capability.

[0023] Step 102: Assign a first tag to the first video frame based on the transcoding operation information, and determine the first position information of the first video frame based on the first tag; wherein, the first position information includes the predicted temporal position of the first video frame in the second video stream, and the first tag is used to characterize the feature information corresponding to the first video frame.

[0024] In the embodiments of this application, after acquiring the first video stream and the first data stream, the video enhancement device can assign a first tag to the first video frame based on transcoding operation information, and determine the first position information of the first video frame based on the first tag.

[0025] It should be noted that, in the embodiments of this application, the first tag is used to characterize the feature information corresponding to the first video frame. The feature information may include information such as the generation method, position features, and operation attributes of the first video frame during the transcoding process, which is used to distinguish it from the original retained frame. This application does not specifically limit the number and type of information included in the feature information.

[0026] It should be noted that, in the embodiments of this application, the first position information includes the predicted temporal position of the first video frame in the second video stream, indicating that the playback order position of the frame in the transcoded video stream (i.e., the second video stream) is calculated based on the information recorded by the first tag, and is used for subsequent synchronization and alignment with metadata.

[0027] Optionally, in embodiments of this application, when the video enhancement device assigns a first tag to a first video frame based on transcoding operation information, it can determine the type information corresponding to the first video frame based on the transcoding operation information. If the type information represents a first type, a fourth tag is assigned to the first video frame; wherein the fourth tag is at least used to represent the position offset information and spatiotemporal interleaving information corresponding to the first video frame, and the first tag includes the fourth tag; or, if the type information represents a second type, a fifth tag is assigned to the first video frame; wherein the fifth tag is at least used to represent the recursion level information and hierarchical dependency information corresponding to the first video frame, and the first tag includes the fifth tag; or, if the type information represents a third type, a sixth tag is assigned to the first video frame; wherein the sixth tag is at least used to represent the insertion condition corresponding to the first video frame, and the first tag includes the sixth tag; or, if the type information represents a fourth type, a seventh tag is assigned to the first video frame; wherein the seventh tag is at least used to represent the mirror transformation information corresponding to the first video frame, and the first tag includes the seventh tag.

[0028] It should be noted that, in the embodiments of this application, the first type can refer to the type in which the first video frame is inserted simultaneously according to the time order and a complex frame insertion and reconstruction is performed according to the interweaving pattern of spatial information (such as picture content, inter-frame differences, etc.) and time information during the transcoding process of the first video stream. This application does not specifically limit the first type.

[0029] It should be noted that, in the embodiments of this application, the second type can represent the type in which the first video frame undergoes multiple stages of recursive operations during the transcoding process of the first video stream. This application does not specifically limit the second type.

[0030] It should be noted that, in the embodiments of this application, the third type may refer to the video frame type used to restore the quality of the video stream during the transcoding process of the first video stream, such as I-frame or P-frame. This application does not specifically limit the third type.

[0031] It should be noted that, in the embodiments of this application, the fourth type can represent the video frame type involved in frame fusion or mirroring operations during the transcoding process of the first video stream. This application does not specifically limit the fourth type.

[0032] It should be noted that, in the embodiments of this application, the video enhancement device determines the type information corresponding to the first video frame based on the transcoding operation information. If the type information represents the first type, a fourth tag can be assigned to the first video frame. The format of the fourth tag can be represented as: {Frame_Type, GOP_Index, Temporal_Position, Spatial_Index, Time_Space_Interaction, Drift_Factor}; where Spatial_Index represents the spatial index of the frame (i.e., the first video frame), Time_Space_Interaction represents the degree of interaction between the frame in time and space, reflecting the spatiotemporal interleaving complexity when it is inserted, Drift_Factor represents the time drift factor of the frame, that is, the time offset of the frame relative to the normal insertion position, Frame_Type represents the frame type, GOP_Index represents the Group of Pictures (GOP) number to which the frame belongs, and Temporal_Position represents the temporal position of the frame in its GOP.

[0033] For example, in an embodiment of this application, assuming that during the transcoding of the first video stream, the spatial structure of frames P1 and P2 is reorganized, and their insertion is not only based on time, but also spatially optimized according to changes in picture quality and content, then the temporary spatiotemporal interleaving reconstruction label of the video frame is: (P, GOP_2, 5, 3, 0.9, 2), indicating that the spatial and temporal interleaving degree of the frame is 0.9, the temporal drift factor is 2, the insertion position is the 5th frame in GOP 2, and the spatial mapping is position 3.

[0034] It should be noted that, in the embodiments of this application, the video enhancement device determines the type information corresponding to the first video frame based on the transcoding operation information. If the type information represents a second type, a fifth tag can be assigned to the first video frame. The format of this fifth tag can be represented as: {Frame_Type, GOP_Index, Recursive_Position, Recursive_Level, Temporal_Shift, Layer_Dependency}; where Recursive_Position indicates the final insertion position of the frame (i.e., the first video frame), but it is affected by the recursion level; Recursive_Level indicates the recursive insertion level of the frame; Temporal_Shift indicates the time offset of the frame relative to the original frame during recursive insertion; and Layer_Dependency indicates the dependency relationship between the frame and its upper or lower layer frames, indicating the insertion order between recursive levels.

[0035] It should be noted that, in the embodiments of this application, the video enhancement device determines the type information corresponding to the first video frame based on the transcoding operation information. If the type information represents the third type, a sixth tag can be assigned to the first video frame. The format of the sixth tag can be represented as: {Frame_Type, GOP_Index, Position_in_GOP, State_Change, Change_Threshold, Insert_Strategy}; where, State_Change: indicates the state change experienced by the original video frame (such as quality degradation, motion intensity increase, etc.), Change_Threshold: indicates the threshold of the state change, indicating when to insert, Insert_Strategy: indicates the strategy used when inserting the frame (i.e., the first video frame), such as based on quality change, noise change, etc., and Position_in_GOP indicates the position number in the GOP to which the first video frame belongs.

[0036] It should be noted that, in the embodiments of this application, the video enhancement device determines the type information corresponding to the first video frame based on the transcoding operation information. When the type information represents the fourth type, a seventh tag can be assigned to the first video frame. The format of the seventh tag can be represented as: {Frame_Type, GOP_Index, Position_in_GOP, Fusion_Level, Source_Frames, Mirror_Factor}; where Fusion_Level: indicates the fusion level of the frame (i.e., the first video frame), indicating from which level of original frames the frame was fused; Mirror_Factor: indicates the degree of mirror inversion of the frame, indicating the mirror change of the frame relative to the original frame (e.g., orientation flip, timing reversal, etc.); and Source_Frames: indicates the set of identifiers of all original frames on which the generation of this fused new frame depends.

[0037] For example, in an embodiment of this application, assuming that frames P1 and P3 are merged into a new P-frame and this frame is inserted into the video stream, but its time order is reversed during insertion, a temporary mirror insertion label can be assigned to this video frame: (P, GOP_5, 2, 2, [P1, P3], -1), indicating that this is a new frame generated by merging frames P1 and P3, with a level 2 fusion, and is inserted in reverse mirror order.

[0038] It should be noted that, in the embodiments of this application, after the video enhancement device assigns a first tag to the first video frame based on the transcoding operation information, it can determine the first position information of the first video frame based on the first tag; wherein, the first tag includes a fourth tag, a fifth tag, a sixth tag, and a seventh tag.

[0039] It should be noted that, in the embodiments of this application, the recursive hierarchy information may include the time offset information and spatial offset information of the first video frame recursively at the Kth level, where K is a positive integer. This application does not specifically limit the types of information included in the recursive hierarchy information.

[0040] It should be noted that, in the embodiments of this application, the insertion conditions may include one or more of the following: the intensity of the state change corresponding to the second video frame, the time information of the duration of the state change, and the spatial information of the duration of the state change. This application does not specifically limit the type of information included in the insertion conditions.

[0041] It should be noted that, in the embodiments of this application, the second video frame is included in the original video frame in the second video stream, and this application does not specifically limit the type of video frame included in the second video frame.

[0042] Optionally, in embodiments of this application, when the video enhancement device determines the first position information of the first video frame based on the first tag, it may determine the position information of the first video frame based on position offset information and spatiotemporal interleaving information; or, based on time offset information, spatial offset information, and hierarchical dependency information; or, based on one or more of state change intensity, time information, and spatial information; or, based on mirror transformation information.

[0043] For example, in the embodiments of this application, when the video enhancement device determines the position information of the first video frame based on the position offset information and the spatiotemporal interleaving information, and the type information corresponding to the first video frame represents the first type, after assigning a fourth tag to the first video frame, it can obtain information such as the spatial features of the video frame, the amount of change of the spatial features between the frame and the previous frame, and the time difference from the fourth tag. Then, based on the above information, the position information (i.e., the first position information) of the first video frame can be predicted according to the following formula (1).

[0044] (1) in, Indicates the first location information. Indicates the time position in the fourth tag. The spatial characteristics representing a frame (i.e., the first video frame). This represents the amount of change in spatial features between a frame (i.e., the first video frame) and the previous frame. Represents a spatial distance metric. Indicates time difference, This indicates the adjustment factor that controls the relative influence of time and space on location estimation. represents a constant, and represents the time decay factor, used to control the nonlinear effect of time differences on the predicted location.

[0045] For example, in the embodiments of this application, the video enhancement device can also determine the position information of the first video frame based on time offset information, spatial offset information, and hierarchical dependency information. When the type information corresponding to the first video frame represents the second type, after assigning a fifth tag to the first video frame, the device can obtain information such as the time position, the backoff factor of the recursion level k, the time offset and spatial offset in the kth recursion from the fifth tag. Then, based on the above information, the device can predict the relative position (i.e., the first position information) of the video frame according to the following formula (2).

[0046] (2) in, Indicates the time position in the fifth tag. This represents the time offset in the k-th level of recursion, reflecting the backtracking operation in the time sequence. This represents the spatial adjustment factor for the recursion level k, controlling the impact of spatial offset on the prediction results. This represents the spatial offset information in the k-th level of recursion, indicating the positional adjustment caused by spatial backtracking. This represents the backoff factor for the recursion level k.

[0047] For example, in the embodiments of this application, the video enhancement device can also determine the position information of the first video frame based on one or more of the intensity of state change, time information, and spatial information. When the type information corresponding to the first video frame represents the third type, after assigning a sixth tag to the first video frame, the device can obtain information such as time position, intensity of frame state change, and nonlinear adjustment factor of state change on time and space based on the information of the sixth tag. Then, based on the above information, the device can predict the relative position of the video frame (i.e., the first position information) according to the following formula (3).

[0048] (3) in, Indicates the intensity of the state change in the frame (i.e., the second video frame). It represents the amount of change over time (i.e., the duration of the state change). It represents the amount of spatial change (i.e., spatial information about the duration of state changes). A moderating factor representing the interaction between time and space.

[0049] For example, in the embodiments of this application, the video enhancement device can also determine the position information of the first video frame based on the mirror transformation information. When the type information corresponding to the first video frame represents the fourth type, after assigning the seventh tag to the first video frame, the relative position (i.e. the first position information) of the video frame can be predicted based on the spatial quality, time offset and mirror reversal effect of the frame (i.e. the first video frame). It can be predicted by the following formula (4).

[0050] (4) in, This indicates the time position of the first video frame after it has been mirrored and reversed. and This indicates the adjustment factor set, which controls the impact of spatial and mirror reversal on position estimation. Indicates the spatial features of the first video frame. This represents the adjustment factor for mirror reverse insertion.

[0051] Step 103: Based on the first location information, establish a mapping relationship between the first video frame and the data enhancement information, and perform enhancement processing on the first video frame based on the mapping relationship.

[0052] In the embodiments of this application, after the video enhancement device assigns a first tag to the first video frame based on transcoding operation information and determines the first position information of the first video frame based on the first tag, it can establish a mapping relationship between the first video frame and data enhancement information based on the first position information, and perform enhancement processing on the first video frame based on the mapping relationship.

[0053] Optionally, in embodiments of this application, when the video enhancement device establishes a mapping relationship between a first video frame and data enhancement information based on the first location information, it can determine a second tag based on the first location information and a first tag; wherein the second tag includes the second location information after the first location information has been corrected; then, it can assign a corresponding third tag to each metadata block based on the second location information and a spatiotemporal relationship graph; wherein the third tag is used to characterize the mapping relationship between the second location information of the first video frame and the corresponding metadata block; wherein the spatiotemporal relationship graph is used to characterize the correlation between the first video frame and the metadata block.

[0054] It should be noted that, in the embodiments of this application, the spatiotemporal relationship graph can be used to connect each video frame with its corresponding multiple metadata block nodes. There is an edge between each video frame and the metadata block, and the weight of this edge is determined by the spatiotemporal correlation strength, representing the spatiotemporal dependency relationship between the video frame and the metadata stream. This application does not specifically limit the structure of the spatiotemporal relationship graph.

[0055] Optionally, in embodiments of this application, when the video enhancement device determines the second tag based on the first location information and the first tag, it may determine the second tag based on one or more of the first location information, the first tag, the first adjustment factor, and the third location information corresponding to the first video frame; wherein, the third location information includes the actual temporal position of the first video frame in the second video stream, and the first adjustment factor is used to control the magnitude information of the correction of the first tag.

[0056] For example, in the embodiments of this application, when the video enhancement device determines the second tag based on one or more of the first location information, the first tag, the first adjustment factor, and the third location information corresponding to the first video frame, it can use the following formula (5).

[0057] (5) in, Indicates the second tag. This represents the quantization value of the first label. This indicates a preset dynamic adjustment factor for the label (i.e., the first adjustment factor). This indicates the relative position of the predicted video frame (i.e., the first position information). This indicates the current display position of the video frame within the GOP (i.e., the third position information).

[0058] For example, in the embodiments of this application, (i.e., the first adjustment factor) can be calculated using the following formula (6).

[0059] (6) in, This indicates the magnitude control factor for label correction. Indicates the spatiotemporal correlation strength of video frames. This represents a balance factor that modulates the strength of the influence of spatiotemporal correlation on label adjustment; where, It can be calculated using the following formula (7).

[0060] (7) in, This indicates changes in frame state, primarily including changes in video content. Represents spatial difference feature vectors, such as the previous frame (i.e., ... ) and the currently inserted frame ( The direction and magnitude of changes in spatial attributes (such as texture, brightness, color distribution, etc.) of the image. Represents the temporal difference vector, such as the currently inserted frame ( ) and the previous frame ( The interval or offset on the timeline position (such as timestamp, playback sequence).

[0061] It should be noted that in the embodiments of this application, for recursively inserted frames (i.e., the type information corresponding to the first video frame is the second type), the final label (i.e. the second label) will be corrected layer by layer until the top-level frame finally determines its label. The correction formula is shown in the following formula (8).

[0062] (8) in, Indicates the second tag. This represents the final label mapped from the temporary label (i.e., the first label) at level k. This represents the preset dynamic adjustment label factor for the k-th level.

[0063] It should be noted that, in the embodiments of this application, after the video enhancement device determines the second tag based on the first location information and the first tag, it can assign a corresponding third tag to each metadata block based on the second location information and the spatiotemporal relationship graph. In this way, the mapping relationship between the second location information of the first video frame and the corresponding metadata block can be obtained, and the first video frame can be enhanced based on the mapping relationship. This can get rid of the dependence on the traditional SEI sequential index, ensure that each video frame and the corresponding enhancement information can be accurately aligned, and improve the playback quality of the video stream.

[0064] For example, in an embodiment of this application, when the video enhancement device assigns a corresponding third tag to each metadata block based on the second location information and the spatiotemporal relationship graph, it can find the corresponding metadata block in the spatiotemporal relationship graph according to the second location information of the first video frame, and then assign a corresponding third tag to the metadata block; wherein, the format of the third tag can be as follows: Meta_Tag=(Frame_Type, Temporal_Position, Spatial_Position, Metadata_Type, Video_Frame_Association, In this context, Meta_Tag represents the third tag, Frame_Type represents the frame (i.e., the first video frame) type, Temporal_Position represents the temporal position of the frame (i.e., the first video frame) within its GOP, Spatial_Position represents the spatial position of the frame (i.e., the first video frame), Metadata_Type represents the type of metadata block, and Video_Frame_Association represents the association attribute between the metadata block and the corresponding video frame. The second tag represents the second location information, which may include Temporal_Position and Spatial_Position. The third tag contains metadata block information and the second location information of the first video frame, thereby establishing a mapping relationship between the second location information and the corresponding metadata block.

[0065] Furthermore, in the embodiments of this application, the video enhancement device can, based on the synchronization information (i.e., mapping relationship) of video frames, apply each metadata block in the metadata stream during the rendering of a video frame (i.e., the first video frame) to complete video enhancement processing such as image enhancement, color correction, and motion enhancement. The video frame and its corresponding one or more metadata blocks will undergo synchronized enhancement processing during the rendering of each frame, ensuring that all video enhancement tasks are accurately completed, greatly improving the playback quality of the video.

[0066] It should be noted that, in the embodiments of this application, during the playback of the second video stream, the mapping relationship between the first video frame and the data enhancement information is updated in real time according to the position shift information and spatiotemporal correlation information of the first video frame to obtain the updated mapping relationship; wherein, the spatiotemporal correlation information is used to characterize the spatiotemporal correlation strength between the first video frame and the corresponding metadata block; then, the corresponding first video frame can be enhanced according to the updated mapping relationship.

[0067] For example, in the embodiments of this application, during the playback of the video (i.e., the second video stream), the position of the video frame (i.e., the first video frame) will continuously shift, and the content of the metadata stream also needs to be continuously aligned with the video frame. At this time, the mapping relationship between the predicted position (i.e., the second position information) of each video frame and the corresponding metadata stream (i.e., the metadata block) can be adjusted and updated in real time according to the spatiotemporal correlation strength, as shown in the following formula (9), to ensure that the video frame and the corresponding metadata are always synchronized during playback, and then the corresponding first video frame can be enhanced according to the updated mapping relationship.

[0068] (9) in, This represents the final metadata tag (i.e., the tag after updating the third tag). Indicates the third tag, This indicates an adjustment of the label factor, which can represent the spatiotemporal correlation strength of video frames. It indicates position shift information, reflecting the position shift of video frames.

[0069] It should be noted that in the embodiments of this application, within each video playback cycle, video frames may sequentially correspond to different metadata blocks of multiple metadata streams. At this time, the corresponding metadata block can be selected for enhancement processing based on spatiotemporal speculation and dynamic adjustment factors. By spatiotemporal mapping between the video stream and N metadata streams, it is ensured that the processing capacity of each metadata stream is fully utilized and that no offset or loss occurs. The multi-data stream synchronization formula is shown in the following formula (10).

[0070] (10) in, This represents the synchronized metadata. This represents the final label of the i-th metadata block. This represents the spatiotemporal correlation strength between the i-th metadata block and the video frame.

[0071] Optionally, in the embodiments of this application, when a frame order error or delay is found during playback, the previous state can be traced back to recalculate the spatiotemporal alignment between the video frame and the metadata stream (i.e., metadata block). For example, if there is a deviation between the predicted position and the actual position of the current frame, the spatiotemporal alignment can be traced back and corrected using the following formula (11).

[0072] (11) in, This represents the final metadata tag of the video frame after backtracking and correction. This indicates the final metadata tag. This represents the backtracking correction factor, used to control the magnitude of the backtracking correction. This indicates the relative position of the predicted video frame (i.e., the first position information). This indicates the actual position observed during playback.

[0073] Optionally, in the embodiments of this application, the video enhancement device can also perform real-time monitoring and processing of the playback status of the second video stream. When it is detected that the first video frame and the corresponding metadata block are out of sync, as shown in the following formula (12), the synchronization information between the first video frame and the corresponding metadata block can be corrected by the second adjustment factor so that the first video frame and the corresponding metadata block can be aligned.

[0074] (12) in, This represents the spatiotemporal adaptive correction factor (i.e., the second adjustment factor), used to correct the alignment between the video frame (i.e., the first video frame) and the metadata stream (i.e., the metadata block). This represents the final tag after spatiotemporal adaptive correction, used to align video frames and metadata streams, ensuring synchronized playback. This indicates the second tag.

[0075] This application provides a video enhancement method, which includes: acquiring a first video stream and a first data stream; wherein the first data stream includes transcoding operation information and data enhancement information; the transcoding operation information is used to mark the type of a first video frame in a second video stream, and the data enhancement information is used to enhance the first video frame; the second video stream includes a video stream after transcoding the first video stream, and the first video frame includes newly added video frames in the second video stream; assigning a first tag to the first video frame based on the transcoding operation information, and determining a first position information of the first video frame based on the first tag; wherein the first position information includes the predicted temporal position of the first video frame in the second video stream, and the first tag is used to characterize the feature information corresponding to the first video frame; establishing a mapping relationship between the first video frame and the data enhancement information based on the first position information, and enhancing the first video frame based on the mapping relationship. Therefore, in the embodiment of this application, during the stage of transcoding the first video stream to generate the second video stream, the first data stream synchronously carries transcoding operation information, directly marking the type of the newly added first video frame in advance. Then, a first tag can be assigned to the first video frame based on the transcoding operation information. The first tag represents the feature information corresponding to the first video frame, and the first position information of the first video frame can be determined based on the first tag. That is, after assigning the first tag to the first video frame, this embodiment of the application directly calculates the predicted temporal position of this frame in the final transcoded stream, using dynamic position instead of traditional sequential index. Even if the original sequence of the original frame is disrupted due to B-frame rearrangement or frame insertion, the position information remains accurate. Indeed; finally, based on the first location information, a mapping relationship between the first video frame and the data enhancement information can be established, and the first video frame can be enhanced based on the mapping relationship. That is, the embodiments of this application can directly establish a mapping relationship between enhancement information and video frames based on dynamically determined location information, thereby getting rid of the dependence on traditional SEI sequential indexing. No matter how many interpolated frames are added after transcoding or whether the original frame order is disrupted, it can ensure that each video frame and the corresponding enhancement information are accurately aligned, thus improving the playback quality of the video stream; and there is no need to store additional index tag information in the SEI area of ​​the video frame, avoiding increasing the redundant volume of the video bitstream and not occupying additional bandwidth and storage space.

[0076] Based on the above embodiments, another embodiment of this application provides a video enhancement method. This method removes traditional SEI tags and tag indexes dependent on video frames, and employs a new dynamic matching algorithm to achieve precise synchronization between video frames and metadata streams. This method ensures real-time alignment of video content and enhanced metadata streams during video playback, improving the effect and stability of video playback and avoiding inconsistencies between video frames and metadata streams caused by transcoding or frame interpolation. The video enhancement method can be implemented based on a terminal video enhancement playback system consisting of a mobile video app, a CDN server, a video source (i.e., the first video stream), and a metadata source (i.e., the first data stream). Figure 2 This is a schematic diagram of the terminal video enhancement playback system proposed in the embodiments of this application, such as... Figure 2 As shown, the system can consist of a mobile video app and a CDN server. The mobile video app includes a playback network module and a metadata synchronization module. The video source (video content) is responsible for pushing video stream resources to the CDN server. The metadata source (video enhancement information) is contained within each metadata block, corresponding to each frame of the video source, and is responsible for pushing metadata resources to the CDN server. The CDN server, located in a data center, is responsible for forwarding and distributing video streams and metadata stream resources. The playback network module, located in the user's mobile video app, downloads live video streams and metadata stream resources from the CDN server and sends them to the metadata synchronization module. The metadata synchronization module, also located in the user's mobile video app, receives the video stream and metadata data and searches for corresponding metadata information based on the rendered video frame data, enabling subsequent playback services to complete video enhancement capabilities.

[0077] It should be noted that, in the embodiments of this application, the video enhancement method may include the following steps: Step S1: Combine each frame of the video source to create a corresponding video enhancement metadata file. Each metadata file contains the video enhancement data of the frame. When creating the corresponding video enhancement metadata file, SEI insertion is not required, and each metadata file does not need to contain a tag index; Step S2: Push the completed video source (i.e., the first video stream) and video enhancement metadata source (i.e., the first data stream) to the CDN server; Step S3: The terminal video APP playback network module obtains the video source data stream and video enhancement metadata stream from the CDN server and sends the two data streams to the metadata synchronization module; Step S4: Metadata synchronization module: Decapsulates and decodes the video stream data and puts it into the video frame queue; Decapsulates the metadata stream and puts it into the metadata queue. The metadata stream decapsulation is preferably parsed using ffmpeg registered metadata format; The synchronization of video frames and metadata is divided into the following four stages: First stage: (1) Obtain video frames from the video frame queue and obtain external metadata streams; (2) Assign temporary tags (i.e., the first tag) to each video frame (i.e., the first video frame) according to the information recorded in the external metadata stream. The temporary tag allocation rules are as follows: Rule 1: Based on the information recorded in the external metadata stream (i.e., transcoding operation information), determine whether the video frame to be assigned a temporary tag (i.e., the first video frame) is simultaneously inserted according to the time sequence during the video transcoding process, and whether complex frame insertion and reconstruction are performed based on the interweaving pattern of spatial information (such as screen content, inter-frame differences, etc.) and temporal information; if not, skip Rule 1 and execute other rule judgments; if yes, assign a temporary spatiotemporal interweaving reconstruction tag (i.e., the fourth tag) to the video frame in the following manner, with the tag format including: {Frame_Type, GOP_Index, Temporal_Position, S The table lists the following tags: Spatial_Index, Time_Space_Interaction, and Drift_Factor. Spatial_Index represents the spatial index of the frame (i.e., the first video frame). Time_Space_Interaction represents the degree of interaction between the frame and the time and space dimensions, reflecting the spatiotemporal interleaving complexity during insertion. Drift_Factor represents the time drift factor of the frame, i.e., the time offset of the frame relative to the normal insertion position. Frame_Type represents the frame type. GOP_Index represents the Group of Pictures (GOP) number to which the frame belongs. Temporal_Position represents the temporal position of the frame in its GOP. In the temporary spatiotemporal interleaving reconstruction tag (i.e., the fourth tag), each inserted frame, in addition to recording its position, also needs to embed spatiotemporal interleaving information. Rule 2: Determine whether the video transcoding process involves multiple recursive operations based on information recorded in the external metadata stream. If not, skip this rule and execute other rule judgments. Conversely, if it is determined that frame insertion during video transcoding does not simply occur in chronological order but requires multiple recursive operations, and each level introduces different temporal, spatial, and state factors, then assign a temporary recursive backtracking frame insertion label (i.e., the fifth label) to the video frame (i.e., the first video frame) as follows: The label format includes: {Frame_Type, GOP_Index, Recursive_Position, Recursive_Level, Temporal_Shift, Layer_Dependency}; where Recursive_Position indicates the final insertion position of the frame (i.e., the first video frame), but it is affected by the recursive level; Recursive_Level indicates the recursive insertion level of the frame. Temporal_Shift: Indicates the time offset of this frame relative to the original frame during recursive insertion. Layer_Dependency: Indicates the dependency of this frame on its upper or lower layer frames, indicating the insertion order between recursive levels.For example, suppose that during the recursive insertion process, the original video frame P1 is delayed to the 4th frame in GOP_2, after two backtrackings. Then the temporary recursive backtracking insertion frame label is: (P, GOP_2, 4, 2, 3, [P2->P1, P3->P2]), indicating the second level of recursive insertion with a time offset of 3. Frame P1 depends on P2 and P3 in the second level of recursion.

[0078] It should be noted that, in the embodiments of this application, the temporary tag allocation rule also includes rule three: Based on the information recorded in the external metadata stream, determine whether the encoder inserts an I-frame or P-frame according to the frame state change during transcoding to restore video quality; if not, skip this rule and execute other rule judgments; if yes, allocate a temporary state-driven frame insertion tag (i.e., the sixth tag) to the video frame (i.e., the first video frame) in the following manner, with the tag format including: {Frame_Type, GOP_Index, Position_in_GOP, State_Change, Change_Threshold, Insert_Strategy}; where State_Change: represents the state change experienced by the original video frame. State changes (such as quality degradation, increased motion intensity, etc.), Change_Threshold: indicates the threshold at which the state change is reached, indicating when insertion should occur, Insert_Strategy: indicates the strategy used when inserting this frame (i.e., the first video frame), such as based on quality changes, noise changes, etc., Position_in_GOP: indicates the position number in the GOP to which the first video frame belongs; for example, assuming it is determined that during the period of image quality degradation, the transcoder inserts an I-frame in GOP_4, then the temporary state-driven insertion label of this I-frame is: (I, GOP_4, 3, Quality_Drop, 0.7, Quality_Recovery_Insert), indicating that the state change is quality degradation and the insertion threshold is 0.7. The insertion strategy is based on quality restoration; Rule 4: Determine whether frame fusion and reverse mirroring operations are involved during transcoding based on information recorded in the external metadata stream; if not, skip this rule and execute other rule judgments; if so, assign a temporary mirror insertion tag (i.e., the seventh tag) to the video frame as follows, with the tag format including: {Frame_Type, GOP_Index, Position_in_GOP, Fusion_Level, Source_Frames, Mirror_Factor}; where Fusion_Level: indicates the fusion level of the frame (i.e., the first video frame), indicating from which level the frame is fused. The frame is obtained by fusing from the original frames. `Mirror_Factor` indicates the degree of mirroring of the frame, indicating the mirroring change of the frame relative to the original frame (e.g., orientation flip, timing reversal, etc.). `Source_Frames` represents the set of identifiers of all original frames on which the newly added frame depends. For example, suppose frames P1 and P3 are fused into a new P-frame, and this frame is inserted into the video stream, but its timing is reversed during insertion. A temporary mirror insertion label can be assigned to this video frame: (P, GOP_5, 2, 2, [P1, P3], -1), indicating that this is a new frame generated by fusing frames P1 and P3, with level 2 fusion, and inserted in reverse mirror order. If a video frame does not conform to any of the above rules, its position in the GOP structure during transcoding is used as a temporary label, in the format (GOP_Index, Frame_Type, Position_in_GOP). For example, (1, I, 1) represents the first I-frame in the first GOP.

[0079] It should be noted that, in the embodiments of this application, the video frame and metadata synchronization process further includes a second stage: predicting the relative position (i.e., first position information) of each video frame by combining the temporarily assigned tag information (i.e., the first tag) mentioned above, including: (1) determining the temporary tag type of each video frame (i.e., the first video frame) according to the temporary tag information (i.e., the first tag); (2) if the temporary tag type of the video frame is a temporary spatiotemporal interleaving reconstruction tag (i.e., the fourth tag), then obtaining information such as the time position in the temporary tag, the spatial characteristics of the video frame, the change in spatial characteristics between the frame and the previous frame, and the time difference from the temporary spatiotemporal interleaving reconstruction tag; (3) predicting the relative position (i.e., the first position information) of the video frame with the tag type of temporary spatiotemporal interleaving reconstruction tag according to the above formula (1) based on the above information; (4) if the temporary tag type of the video frame is a temporary recursive back-off insertion tag (i.e., the fifth tag), then obtaining the time position in the temporary tag, the back-off factor of the recursion level k, the time offset in the kth recursion, and the spatial offset from the temporary recursive back-off insertion tag. (5) According to the information such as the time position in the temporary tag, the backoff factor of the recursion level k, the time offset and spatial offset in the kth recursion, the relative position of the video frame (i.e., the first position information) is predicted according to the above formula (2); (6) If the temporary tag type of the video frame is a temporary state-driven interpolation tag (i.e., the sixth tag), the information such as the time position in the temporary tag, the intensity of the frame state change, and the nonlinear adjustment factor of the state change on time and space is obtained from the tag information; (7) According to the information such as the time position in the temporary tag, the intensity of the frame state change, and the nonlinear adjustment factor of the state change on time and space, the relative position of the video frame with the tag type of temporary state-driven interpolation tag is predicted according to the above formula (3); (8) If the temporary tag type of the video frame is a temporary mirror insertion tag (i.e., the seventh tag), the relative position of the video frame is predicted according to the spatial quality, time offset and mirror reversal effect of the frame, which can be predicted by the above formula (4). The video frame and metadata synchronization process also includes a third stage: (1) The temporary label is mapped to the final label (i.e., the second label) based on the temporary label information (i.e., the first label) and the predicted relative position of the video frame (i.e., the first position information), which can be calculated by the above formula (5); (2) For recursively inserted frames, the final label (i.e., the second label) will be corrected layer by layer until the top-level frame finally determines its label. The correction formula is shown in the above formula (8).The video frame and metadata synchronization process also includes a fourth stage: before playback, establish a mapping relationship between the video frame and the external metadata stream to ensure that the video frame and the metadata stream can be accurately synchronized during playback; (1) construct a spatiotemporal relationship graph to connect each video frame with its corresponding multiple metadata stream nodes (i.e., metadata blocks). Each node in the graph represents a video frame or a metadata block of a metadata stream. There is an edge between each video frame and the metadata stream. The weight of this edge is determined by the spatiotemporal correlation strength, representing the spatiotemporal dependency relationship between the video frame and the metadata stream; (2) based on the predicted relative position of the video frame (i.e., the second position information) and the spatiotemporal relationship graph, for each Each metadata stream is assigned an initial tag (i.e., the third tag). The tag includes: the frame position inferred from the position (i.e., the second position information), the final tag (i.e., the second tag), and the metadata type and metadata block position. Among them, the initial tag of each metadata block is mapped to its corresponding video frame position. The format of the initial tag (i.e., the third tag) assigned to the metadata stream is as follows: Meta_Tag=(Frame_Type, Temporal_Position, Spatial_Position, Metadata_Type, Video_Frame_Association, ...). In this context, Meta_Tag represents the third tag, Frame_Type represents the frame (i.e., the first video frame) type, Temporal_Position represents the temporal position of the frame (i.e., the first video frame) within its GOP, Spatial_Position represents the spatial position of the frame (i.e., the first video frame), Metadata_Type represents the type of metadata block, and Video_Frame_Association represents the association attribute between the metadata block and the corresponding video frame. The second label indicates the second position information, which may include Temporal_Position and Spatial_Position. At this time, the position of the video frame has been initially mapped to the metadata blocks of multiple metadata streams. The playback and enhancement operations of the video stream depend on this mapping relationship. (3) During video playback, the position of the video frame will keep moving, and the content of the metadata stream also needs to be continuously aligned with the video frame. At this time, the mapping relationship between the predicted position of each video frame and the corresponding metadata stream is adjusted in real time according to the spatiotemporal correlation strength, as shown in the above formula (9), to ensure that the video frame and the corresponding metadata are always synchronized during playback. (4) In each playback cycle, the video frame may correspond to different metadata blocks of multiple metadata streams in sequence. At this time, the corresponding metadata block is selected for enhancement processing according to the spatiotemporal prediction and dynamic adjustment factor. By mapping the video stream to N metadata streams in a spatiotemporal manner, the processing capacity of each metadata stream is fully utilized and there is no offset or loss. The multi-data stream synchronization formula is shown in formula (10) above. (5) When a frame order error or delay is found during playback, the previous state will be traced back and the spatiotemporal alignment between the video frame and the metadata stream (i.e., metadata block) will be recalculated. The backtracking correction of spatiotemporal alignment can be completed by formula (11) above. (6) According to the synchronization information of the video frame, each metadata block in the metadata stream will be applied when the video frame is rendered to complete video enhancement processing such as image enhancement, color correction, and motion enhancement. The video frame and the corresponding multiple metadata streams (i.e., metadata blocks) will complete the synchronization enhancement processing when each frame is rendered to ensure that all video enhancement tasks can be completed accurately. (7) The terminal device monitors the playback status through a feedback mechanism and checks the synchronization between the video frame (i.e., the first video frame) and the metadata stream (i.e., metadata block) in real time. If an inconsistency is detected, as shown in formula (12) above, it can be corrected in real time by a spatiotemporal adaptive adjustment factor (second adjustment factor) to ensure the continuous stability of the video enhancement effect.

[0080] This application provides a video enhancement method, which includes: acquiring a first video stream and a first data stream; wherein the first data stream includes transcoding operation information and data enhancement information; the transcoding operation information is used to mark the type of a first video frame in a second video stream, and the data enhancement information is used to enhance the first video frame; the second video stream includes a video stream after transcoding the first video stream, and the first video frame includes newly added video frames in the second video stream; assigning a first tag to the first video frame based on the transcoding operation information, and determining a first position information of the first video frame based on the first tag; wherein the first position information includes the predicted temporal position of the first video frame in the second video stream, and the first tag is used to characterize the feature information corresponding to the first video frame; establishing a mapping relationship between the first video frame and the data enhancement information based on the first position information, and enhancing the first video frame based on the mapping relationship. Therefore, in the embodiment of this application, during the stage of transcoding the first video stream to generate the second video stream, the first data stream synchronously carries transcoding operation information, directly marking the type of the newly added first video frame in advance. Then, a first tag can be assigned to the first video frame based on the transcoding operation information. The first tag represents the feature information corresponding to the first video frame, and the first position information of the first video frame can be determined based on the first tag. That is, after assigning the first tag to the first video frame, this embodiment of the application directly calculates the predicted temporal position of this frame in the final transcoded stream, using dynamic position instead of traditional sequential index. Even if the original sequence of the original frame is disrupted due to B-frame rearrangement or frame insertion, the position information remains accurate. Indeed; finally, based on the first location information, a mapping relationship between the first video frame and the data enhancement information can be established, and the first video frame can be enhanced based on the mapping relationship. That is, the embodiments of this application can directly establish a mapping relationship between enhancement information and video frames based on dynamically determined location information, thereby getting rid of the dependence on traditional SEI sequential indexing. No matter how many interpolated frames are added after transcoding or whether the original frame order is disrupted, it can ensure that each video frame and the corresponding enhancement information are accurately aligned, thus improving the playback quality of the video stream; and there is no need to store additional index tag information in the SEI area of ​​the video frame, avoiding increasing the redundant volume of the video bitstream and not occupying additional bandwidth and storage space.

[0081] Based on the above embodiments, this application provides a video enhancement device. Figure 3 This is a schematic diagram of the composition of a video enhancement device, such as... Figure 3 As shown, the device 10 includes: an acquisition unit 11, an allocation unit 12, a determination unit 13, an establishment unit 14, and an enhancement unit 15; wherein, The acquisition unit 11 is used to acquire a first video stream and a first data stream; wherein, the first data stream includes transcoding operation information and data enhancement information; the transcoding operation information is used to mark the type of the first video frame in the second video stream, the data enhancement information is used to enhance the first video frame, the second video stream includes a video stream after transcoding the first video stream, and the first video frame includes a newly added video frame in the second video stream. The allocation unit 12 is used to allocate a first tag to the first video frame based on the transcoding operation information; The determining unit 13 is used to determine the first position information of the first video frame based on the first tag; wherein, the first position information includes the predicted temporal position of the first video frame in the second video stream, and the first tag is used to characterize the feature information corresponding to the first video frame. The establishment unit 14 is used to establish a mapping relationship between the first video frame and the data enhancement information based on the first location information; The enhancement unit 15 is used to enhance the first video frame based on the mapping relationship.

[0082] In the embodiments of this application, further, Figure 4 This is a schematic diagram of the composition and structure of a video enhancement device, such as... Figure 4 As shown, the video enhancement device 20 proposed in this application embodiment may further include a processor 16, a memory 17 storing instructions executable by the processor 16, and further, the device 20 may also include a communication interface 18 and a bus 19 for connecting the processor 16, the memory 17 and the communication interface 18.

[0083] In the embodiments of this application, the processor 16 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other types, and this application embodiment does not specifically limit this. The device 20 may also include a memory 17, which can be connected to the processor 16. The memory 17 is used to store executable program code, which includes computer operation instructions. The memory 17 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0084] In embodiments of this application, bus 19 is used to connect communication interface 18, processor 16, and memory 17, as well as the mutual communication between these devices.

[0085] In embodiments of this application, memory 17 is used to store instructions and data.

[0086] Further, in an embodiment of this application, the processor 16 is configured to acquire a first video stream and a first data stream; wherein the first data stream includes transcoding operation information and data enhancement information; the transcoding operation information is used to mark the type of a first video frame in a second video stream, the data enhancement information is used to enhance the first video frame, the second video stream includes a video stream after transcoding the first video stream, and the first video frame includes newly added video frames in the second video stream; a first tag is assigned to the first video frame based on the transcoding operation information, and a first position information of the first video frame is determined based on the first tag; wherein the first position information includes the predicted temporal position of the first video frame in the second video stream, and the first tag is used to characterize the feature information corresponding to the first video frame; a mapping relationship is established between the first video frame and the data enhancement information based on the first position information, and the first video frame is enhanced based on the mapping relationship.

[0087] In practical applications, the aforementioned memory 17 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 16.

[0088] This application provides a video enhancement device that acquires a first video stream and a first data stream. The first data stream includes transcoding operation information and data enhancement information. The transcoding operation information is used to mark the type of a first video frame in a second video stream, and the data enhancement information is used to enhance the first video frame. The second video stream includes a video stream after transcoding the first video stream, and the first video frame includes newly added video frames in the second video stream. A first tag is assigned to the first video frame based on the transcoding operation information, and a first position information of the first video frame is determined based on the first tag. The first position information includes the predicted temporal position of the first video frame in the second video stream, and the first tag is used to characterize the feature information corresponding to the first video frame. Based on the first position information, a mapping relationship between the first video frame and the data enhancement information is established, and the first video frame is enhanced based on the mapping relationship. Therefore, in the embodiment of this application, during the stage of transcoding the first video stream to generate the second video stream, the first data stream synchronously carries transcoding operation information, directly marking the type of the newly added first video frame in advance. Then, a first tag can be assigned to the first video frame based on the transcoding operation information. The first tag represents the feature information corresponding to the first video frame, and the first position information of the first video frame can be determined based on the first tag. That is, after assigning the first tag to the first video frame, this embodiment of the application directly calculates the predicted temporal position of this frame in the final transcoded stream, using dynamic position instead of traditional sequential index. Even if the original sequence of the original frame is disrupted due to B-frame rearrangement or frame insertion, the position information remains accurate. Indeed; finally, based on the first location information, a mapping relationship between the first video frame and the data enhancement information can be established, and the first video frame can be enhanced based on the mapping relationship. That is, the embodiments of this application can directly establish a mapping relationship between enhancement information and video frames based on dynamically determined location information, thereby getting rid of the dependence on traditional SEI sequential indexing. No matter how many interpolated frames are added after transcoding or whether the original frame order is disrupted, it can ensure that each video frame and the corresponding enhancement information are accurately aligned, thus improving the playback quality of the video stream; and there is no need to store additional index tag information in the SEI area of ​​the video frame, avoiding increasing the redundant volume of the video bitstream and not occupying additional bandwidth and storage space.

[0089] This application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the video enhancement method described above.

[0090] Specifically, the program instructions corresponding to a video enhancement method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to a video enhancement method in the storage media are read or executed by an electronic device, the following steps are included: Acquire a first video stream and a first data stream; wherein, the first data stream includes transcoding operation information and data enhancement information; the transcoding operation information is used to mark the type of the first video frame in the second video stream, the data enhancement information is used to enhance the first video frame, the second video stream includes a video stream after transcoding the first video stream, and the first video frame includes newly added video frames in the second video stream. A first tag is assigned to the first video frame based on the transcoding operation information, and a first position information of the first video frame is determined based on the first tag; wherein, the first position information includes the predicted temporal position of the first video frame in the second video stream, and the first tag is used to characterize the feature information corresponding to the first video frame. Based on the first location information, a mapping relationship is established between the first video frame and the data enhancement information, and the first video frame is enhanced based on the mapping relationship.

[0091] This application also provides a computer program product, including a computer program that can be executed by the processor 16 of the video enhancement device 20 to perform the steps described in any of the foregoing methods.

[0092] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0093] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A video enhancement method, characterized in that, The method includes: Acquire a first video stream and a first data stream; wherein, the first data stream includes transcoding operation information and data enhancement information; the transcoding operation information is used to mark the type of the first video frame in the second video stream, the data enhancement information is used to enhance the first video frame, the second video stream includes a video stream after transcoding the first video stream, and the first video frame includes newly added video frames in the second video stream. A first tag is assigned to the first video frame based on the transcoding operation information, and a first position information of the first video frame is determined based on the first tag; wherein, the first position information includes the predicted temporal position of the first video frame in the second video stream, and the first tag is used to characterize the feature information corresponding to the first video frame. Based on the first location information, a mapping relationship is established between the first video frame and the data enhancement information, and the first video frame is enhanced based on the mapping relationship.

2. The method according to claim 1, characterized in that, The data enhancement information includes one or more metadata blocks; establishing the mapping relationship between the first video frame and the data enhancement information based on the first location information includes: A second tag is determined based on the first location information and the first tag; wherein, the second tag includes second location information after correcting the first location information; Based on the second location information and the spatiotemporal relationship graph, a corresponding third tag is assigned to each of the metadata blocks; wherein, the third tag is used to characterize the mapping relationship between the second location information of the first video frame and the corresponding metadata block; wherein, the spatiotemporal relationship graph is used to characterize the association between the first video frame and the metadata block.

3. The method according to claim 1, characterized in that, Assigning a first tag to the first video frame based on the transcoding operation information includes: Based on the transcoding operation information, the type information corresponding to the first video frame is determined. If the type information represents a first type, a fourth tag is assigned to the first video frame; wherein, the fourth tag is at least used to represent the position offset information and spatiotemporal interleaving information corresponding to the first video frame, and the first tag includes the fourth tag; or... When the type information represents a second type, a fifth tag is assigned to the first video frame; wherein the fifth tag is at least used to represent the recursive hierarchy information and hierarchical dependency information corresponding to the first video frame, and the first tag includes the fifth tag; or... When the type information represents a third type, a sixth tag is assigned to the first video frame; wherein the sixth tag is at least used to represent the insertion condition corresponding to the first video frame, and the first tag includes the sixth tag; or... When the type information represents a fourth type, a seventh tag is assigned to the first video frame; wherein the seventh tag is used to at least represent the mirror transformation information corresponding to the first video frame, and the first tag includes the seventh tag.

4. The method according to claim 3, characterized in that, The recursive hierarchy information includes the time offset information and spatial offset information of the first video frame recursively at the Kth level, where K is a positive integer; the insertion condition includes one or more of the following: the intensity of the state change corresponding to the second video frame, the time information of the duration of the state change, and the spatial information of the duration of the state change. Determining the first location information of the first video frame based on the first tag includes: The position information of the first video frame is determined based on the position offset information and the spatiotemporal interleaving information; or... The position information of the first video frame is determined based on the time offset information, spatial offset information, and hierarchical dependency information; or... The location information of the first video frame is determined based on one or more of the following: the intensity of the state change, the time information, and the spatial information; or... The position information of the first video frame is determined based on the mirror transformation information.

5. The method according to claim 3, characterized in that, The step of determining the second tag based on the first location information and the first tag includes: The second tag is determined based on one or more of the first location information, the first tag, the first adjustment factor, and the third location information corresponding to the first video frame; The third location information includes the actual temporal position of the first video frame in the second video stream, and the first adjustment factor is used to control the magnitude of the correction to the first tag.

6. The method according to claim 1, characterized in that, The method further includes: During the playback of the second video stream, the mapping relationship between the first video frame and the data enhancement information is updated in real time based on the position shift information and spatiotemporal correlation information of the first video frame to obtain the updated mapping relationship; wherein, the spatiotemporal correlation information is used to characterize the spatiotemporal correlation strength between the first video frame and the corresponding metadata block; The corresponding first video frame is enhanced according to the updated mapping relationship.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: The playback status of the second video stream is monitored in real time. When the first video frame and the corresponding metadata block are found to be out of sync, the synchronization information between the first video frame and the corresponding metadata block is corrected by the second adjustment factor so that the first video frame and the corresponding metadata block are aligned.

8. A video enhancement device, characterized in that, The video enhancement device includes: a processor and a memory; wherein... The memory is used to store computer programs that can run on the processor; The processor is configured to perform the method as described in any one of claims 1-7 when running the computer program.

9. A computer-readable storage medium, characterized in that, The storage medium stores computer program code, which, when executed by a computer, performs the method described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1-7.

Images