Super-resolution method, system, electronic device and storage medium for video

Abstract

The application provides a kind of video super-resolution method, system, electronic equipment and storage medium, it is related to video processing technical field, the method comprises: the target video is separated by text and background, obtains target text area and target background area;According to target text area, it is enhanced respectively by geometric flow and texture flow feature, and text area super-resolution feature is obtained;The feature of target background area is enhanced using the convolutional neural network of optical flow supervision, and the background area super-resolution feature is obtained;According to text area super-resolution feature and background area super-resolution feature, feature fusion is carried out, and super-resolution video is obtained.The application separates the text and background area of target video, fully considers the structure feature and semantic information of text for super-resolution processing, the background area is also processed for super-resolution, and finally generates global clear and text details rich super-resolution view, effectively improves video quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video super-resolution method and system, an electronic device and a storage medium. BACKGROUND

[0002] Under the background of digital transformation, video has become the core medium of information dissemination and is widely used in education and scientific research, film and television production, digital entertainment and many other fields. With the continuous improvement of users' requirements for video quality, subtitle clarity as a key quality indicator is increasingly valued. An ideal video should have clear text and background details on the basis of consistent lighting timing. Video super-resolution technology is an important technical means to reconstruct low-resolution video sequences into high-resolution sequences with rich details and coherent content through algorithm processing.

[0003] However, the current video super-resolution technology still faces significant technical bottlenecks when processing videos containing text content. As a key carrier of information transmission, text has the characteristics of complex structure and fine strokes, and often forms a multi-level spatial interaction and occlusion relationship with the background. Traditional super-resolution methods based on global processing are difficult to achieve precise optimization of the text area, which easily leads to visual defects such as blurring, jaggedness and stroke breakage, seriously affecting the information transmission efficiency and video viewing quality. With the development of deep learning technology, data-driven super-resolution methods have made important progress. Through large-scale data set training, convolutional neural networks, generative adversarial networks and Transformer architectures can effectively learn the mapping relationship from low resolution to high resolution. However, the existing methods generally have the defect of insufficient modeling of text semantic information and contextual relevance, which limits the performance of text detail restoration and semantic consistency maintenance. Therefore, in the context of text-intensive videos, how to solve the problem of insufficient clarity and readability caused by existing super-resolution methods has become a key technical problem that needs to be broken through in the current video processing field. SUMMARY

[0004] In view of the above shortcomings of the prior art, the present application provides a video super-resolution method, system, electronic device and storage medium, which effectively solves the problem of insufficient clarity and readability caused by existing super-resolution methods.

[0005] In a first aspect, the present application provides a video super-resolution method, which comprises:

[0006] Separating the target video into text and background to obtain a target text area and a target background area;

[0007] Respectively enhancing the geometric flow and texture flow features of the target text area to obtain text area super-resolution features;

[0008] The light flow supervised convolutional neural network is used for feature enhancement on the target background region to obtain a background region super-resolution feature;

[0009] Feature fusion is performed according to the text region super-resolution feature and the background region super-resolution feature to obtain a super-resolution video.

[0010] In an optional implementation, the target video is separated into a text region and a background region to obtain a target text region and a target background region, including:

[0011] Different size text structures in each video frame of the target video are extracted to obtain an initial text feature;

[0012] The initial text feature is subjected to a step-by-step upsampling and channel weighting operation to obtain an enhanced text feature;

[0013] The channel weight of the enhanced text feature is adjusted to obtain the target text region and the target background region.

[0014] In an optional implementation, the target text region is subjected to geometric flow and texture flow feature enhancement to obtain a text region super-resolution feature, including:

[0015] The local receptive field shape of a text feature in the target text region is adjusted to obtain a text geometric feature;

[0016] The text feature in the target text region is subjected to adversarial training to constrain the authenticity of high-frequency detail generation to obtain a text texture feature;

[0017] The fusion weight of the text geometric feature and the text texture feature is adjusted to obtain a text fusion feature;

[0018] The semantic reasonableness of the text fusion feature is subjected to inspection processing to obtain the text region super-resolution feature.

[0019] In an optional implementation, the light flow supervised convolutional neural network is used for feature enhancement on the target background region to obtain a background region super-resolution feature, including:

[0020] An initial optical flow field of adjacent video frames in the target background region is calculated, and the initial optical flow field is optimized to obtain a target optical flow field;

[0021] Feature extraction is performed on each video frame in the target background region to obtain a video frame super-resolution result;

[0022] The video frame super-resolution result is subjected to spatio-temporal consistency optimization according to the target optical flow field to obtain the background region super-resolution feature.

[0023] In an optional implementation, the feature fusion according to the text region super-resolution feature and the background region super-resolution feature to obtain the super-resolution video comprises:

[0024] The text region super-resolution feature and the background region super-resolution feature are geometrically aligned and feature-aligned.

[0025] The fusion weights of the text region super-resolution feature and the background region super-resolution feature are dynamically allocated according to a text detection confidence and an optical flow field.

[0026] The text region super-resolution feature and the background region super-resolution feature are feature-fused according to the fusion weights to obtain fused super-resolution features.

[0027] The overlapping region of text and background in the fused super-resolution features is mask-compensated to obtain the super-resolution video.

[0028] In an optional implementation, the geometric alignment and feature alignment of the text region super-resolution feature and the background region super-resolution feature comprise:

[0029] An affine transformation matrix of the target text region and the target background region is predicted.

[0030] The coordinates of the text region super-resolution feature are mapped to a global coordinate system of the background region super-resolution feature according to the affine transformation matrix.

[0031] The edge contour, text stroke and semantic information of the text region super-resolution feature and the background region super-resolution feature are aligned.

[0032] In an optional implementation, the mask compensation of the overlapping region of text and background in the fused super-resolution features to obtain the super-resolution video comprises:

[0033] The fused super-resolution features are semantically segmented to obtain low-level features and high-level features.

[0034] The low-level features are inserted into a dilated convolution to expand the receptive field, and the high-level features are progressively aligned with the low-level features through upsampling to obtain a main mask.

[0035] The optical flow consistency difference of the front and back frames in the fused super-resolution features is calculated, and an error mask is obtained according to the optical flow consistency difference.

[0036] The occlusion compensation of the fused super-resolution features is performed according to the main mask and the error mask to obtain the super-resolution video.

[0037] In a second aspect, the present application provides a super-resolution system of a video, which comprises:

[0038] a text background separation module, configured to separate a target video into a target text region and a target background region;

[0039] a text region super-resolution module, configured to respectively perform geometric flow and texture flow feature enhancement on the target text region according to the target text region, to obtain text region super-resolution features;

[0040] a background region super-resolution module, configured to perform feature enhancement on the target background region by using a convolutional neural network supervised by an optical flow, to obtain background region super-resolution features;

[0041] a super-resolution feature fusion module, configured to perform feature fusion on the text region super-resolution features and the background region super-resolution features, to obtain a super-resolution video.

[0042] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the super-resolution method of the video according to the first aspect of the present application.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the super-resolution method of the video according to the first aspect of the present application.

[0044] The super-resolution method of the video, system, electronic device and storage medium provided by the present application can accurately locate the position and boundary of the text in the video frame by designing a neural network specially recognizing and separating the text region. For the text region, the super-resolution processing is performed by fully considering the structural features and semantic information of the text, which can effectively improve the clarity and readability of the text, so that the audience can clearly see the text content in the video, and the information acquisition is not affected by the blurred text. For the background region, the super-resolution processing is also performed, which focuses on improving the quality of the text while not ignoring the processing of the background picture, ensuring that the quality of the background picture is not affected and maintaining the overall coordination and aesthetics of the video picture. The processed text region and background region are effectively fused to generate the final high-definition video frame, realizing the differential processing of the text and the background in the video frame. Finally, the super-resolution video with clear overall and rich text details is generated, which improves the quality of the video as a whole, brings better visual experience to the user, meets the user's demand for high-quality video, and plays a positive role in video playing, information display and other application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those of ordinary skill in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0046] Figure 1 is a first schematic diagram of the video super-resolution method process provided by the embodiments of the present application;

[0047] Figure 2 is a second schematic diagram of the video super-resolution method process provided by the embodiments of the present application;

[0048] Figure 3 is a third schematic diagram of the video super-resolution method process provided by the embodiments of the present application;

[0049] Figure 4 is a fourth schematic diagram of the video super-resolution method process provided by the embodiments of the present application;

[0050] Figure 5 is a fifth schematic diagram of the video super-resolution method process provided by the embodiments of the present application;

[0051] Figure 6 is a sixth schematic diagram of the video super-resolution method process provided by the embodiments of the present application;

[0052] Figure 7 is a seventh schematic diagram of the video super-resolution method process provided by the embodiments of the present application;

[0053] Figure 8 is a schematic diagram of the structure of the video super-resolution system provided by the embodiments of the present application;

[0054] Figure 9 is a schematic diagram of the structure of the electronic device provided by the embodiments of the present application.

[0055] Main element symbol explanation:

[0056] 200, video super-resolution system; 210, text background separation module; 220, text area super-resolution module; 230, background area super-resolution module; 240, super-resolution feature fusion module; 300, electronic device; 310, processor; 320, communication interface; 330, memory; 340, communication bus. DETAILED DESCRIPTION

[0057] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further clearly and completely describe the technical solutions of the present application with reference to the accompanying drawings of the embodiments of the present application. It should be noted that the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0058] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly specified.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0060] Existing video super-resolution methods face many challenges when processing pictures containing text. As an important element for conveying key information in videos, the clarity and readability of text are crucial. However, text often has a complex structure and small strokes, and may have complex interweaving and occlusion relationships with the background in videos. Traditional video super-resolution methods mostly use global processing strategies, which are difficult to accurately process text areas, resulting in problems such as blurred, jagged, and stroke loss of text, which seriously affects the transmission of text information and the overall quality of the video. After the rise of deep learning, a model is generally trained with a large amount of data to learn the mapping relationship between low-resolution and high-resolution images, such as automatically extracting image features through convolutional neural networks. The Transformer architecture has also been applied in video super-resolution due to its powerful parallel computing capability and long sequence modeling capability. However, existing methods mostly do not fully consider the semantic information and contextual relationships of text, and cannot effectively restore the details and semantic consistency of text. Therefore, in the context of text-intensive videos, how to solve the problem of insufficient clarity and readability caused by existing super-resolution methods has become a key technical problem that needs to be broken through in the current video processing field.

[0061] Embodiment 1

[0062] The embodiments of the present application provide a video super-resolution method, which effectively solves the problem of insufficient clarity and readability caused by existing super-resolution methods. Figure 1is a first schematic diagram of a video super-resolution method provided by an embodiment of the present application, as shown in the figure, the method comprises the following steps: Figure 1

[0063] S100, separate the target video into text and background to obtain a target text region and a target background region.

[0064] In the embodiment of the present application, for the target video, a convolutional neural network integrating a dilated convolution layer, a pyramid feature fusion strategy and a channel attention mechanism can be used to separate the text and background regions, thereby improving the detection accuracy of the text region in the video picture. Figure 2 is a second schematic diagram of a video super-resolution method provided by an embodiment of the present application, as shown in the figure, the text and background region separation specifically comprises the following steps: Figure 2

[0065] S110, extract text structures of different sizes in each video frame of the target video to obtain initial text features.

[0066] In the embodiment of the present application, a dilated convolution is used to extract low-resolution text structures of different sizes in each video frame of the target video through a multi-scale receptive field and sparse sampling to obtain initial text features. The multi-scale receptive field uses convolution kernels of different dilation rates to extract text structure features in parallel. When a convolution kernel of a small dilation rate is used, it can focus on the fine strokes of the text, and fine structures such as strokes and dots can be accurately captured, which is crucial for preserving the details of the text. When a convolution kernel of a large dilation rate is used, it can cover the context information of large fonts. In text recognition, understanding the overall environment of the text helps to more accurately determine the text content. The design of the multi-scale receptive field allows the dilated convolution to consider both the local details and the overall structure of the text, and fully extracts the text features to provide rich and accurate information for subsequent processing.

[0067] Sparse sampling inserts holes in the convolution kernel, i.e., does not perform convolution, which effectively expands the field of view without increasing the number of parameters. This allows the dilated convolution to detect distant text edges. For low-resolution text, edge information is often not clear enough, but sparse sampling allows the dilated convolution to obtain edge information across a certain distance, enhancing the detection capability of the text edge, thereby better extracting the text structure and improving the effect of low-resolution text structure extraction.

[0068] S120, perform level-by-level upsampling and channel weighting operations on the initial text features to obtain enhanced text features.

[0069] ​​In the embodiment of the present application, the pyramid feature fusion network is adopted to enhance the initial character features by multi-level feature alignment and channel weighting, and to obtain enhanced character features. First, the deep features and shallow features of the initial character features are extracted respectively. The deep features contain global semantic information including character block position, and the shallow features retain high-frequency details such as stroke edges. Then, the deep features are upsampled to the resolution of the shallow features by deconvolution, and are aligned with the shallow features channel by channel. Finally, 1x1 convolution is used to generate the weight coefficients of each channel, and the weight of the smooth region is reduced to suppress the background noise channel, and the weight of the region with large gradient change is increased to enhance the character edge.

[0070] In the embodiment of the present application, the pyramid feature fusion network is adopted to enhance the initial character features by multi-level feature alignment and channel weighting, and to obtain enhanced character features. First, the deep features and shallow features of the initial character features are extracted respectively. The deep features contain global semantic information including character block position, and the shallow features retain high-frequency details such as stroke edges. Then, the deep features are upsampled to the resolution of the shallow features by deconvolution, and are aligned with the shallow features channel by channel. Finally, 1x1 convolution is used to generate the weight coefficients of each channel, and the weight of the smooth region is reduced to suppress the background noise channel, and the weight of the region with large gradient change is increased to enhance the character edge.

[0071] In the embodiment of the present application, the pyramid feature fusion network is adopted to enhance the initial character features by multi-level feature alignment and channel weighting, and to obtain enhanced character features. First, the deep features and shallow features of the initial character features are extracted respectively. The deep features contain global semantic information including character block position, and the shallow features retain high-frequency details such as stroke edges. Then, the deep features are upsampled to the resolution of the shallow features by deconvolution, and are aligned with the shallow features channel by channel. Finally, 1x1 convolution is used to generate the weight coefficients of each channel, and the weight of the smooth region is reduced to suppress the background noise channel, and the weight of the region with large gradient change is increased to enhance the character edge. H W C H W C

[0072]

[0073] ​​​​​​​Finally, the obtained weight coefficient is multiplied with the original channel feature before the feature map is reused. For the channel related to the text, such as the channel containing the high-frequency component of the horizontal and vertical strokes, since the weight coefficient is large, the feature of these channels will be enhanced after multiplication, and the key information of the text can be captured more easily. For example, when detecting Chinese characters, horizontal and vertical strokes are the basic elements of characters. By enhancing the channel features corresponding to these strokes, the shape and structure of the characters can be more accurately recognized. For the channel related to the background or irrelevant noise, since the weight coefficient is small, the feature of these channels will be suppressed after multiplication. For example, for the background channel such as a uniform texture area, the weight coefficient tends to be 0, and the feature of these channels is almost ignored after multiplication, thereby reducing noise interference.

[0074] In the embodiment of the present application, the hole convolution layer adopts a multi-scale convolution kernel with multiple expansion rates to improve the receptive field of the convolutional neural network to capture text structures of different sizes. The pyramid feature fusion strategy integrates the multi-level feature maps of the convolutional neural network, and through the operations of progressive upsampling and channel weighting, the response ability to small characters is enhanced. The convolutional neural network introduces a channel attention mechanism to adaptively adjust the weight of the channel, strengthen the edge contour features of the characters, and enhance the features related to the characters. At the same time, since the irrelevant noise channels are suppressed, the difference between the background region and the text region is more obvious, thereby obtaining the target text region and the target background region.

[0075] S200, according to the target text region, respectively, the geometric flow and the texture flow feature enhancement are carried out, and the text region super-resolution feature is obtained.

[0076] In the embodiment of the present application, the text super-resolution processing is carried out for the target text region, the double-flow feature enhancement architecture is used to process the structure and detail information of the text in parallel with the geometric flow and the texture flow, and then the fusion weight of the geometric feature and the texture feature is dynamically adjusted based on the adaptive feature fusion strategy through the gating mechanism. Considering that the semantics of the blurred text may change after super-resolution, finally, the pre-trained language model is introduced to constrain the semantic rationality of the generated text, and the text region super-resolution feature is obtained. Figure 3 is the third schematic diagram of the video super-resolution method provided by the embodiment of the present application, as shown in Figure 3 The super-resolution processing of the text specifically includes the following steps:

[0077] S210, adjusting the local receptive field shape of the text feature in the target text region, obtaining the text geometric feature.

[0078] In the embodiments of the present application, the deformed convolutional network is used to adjust the shape of the local receptive field of the character features in the target character region, accurately capture the geometric features of horizontal and vertical strokes, and obtain character geometric features. The deformed convolutional network introduces learnable offsets, which can be adjusted for each sampling point of the convolution kernel. Specifically, before the convolution operation, an additional network branch can be used to learn the offset of each sampling point. Then, the sampling position of the convolution kernel is adjusted according to the offset, so as to change the shape of the local receptive field.

[0079] For example, for horizontal strokes, the deformed convolutional network can adjust the offset to expand the receptive field in the horizontal direction and shrink it in the vertical direction, so as to better fit the shape of the horizontal stroke and more accurately extract the edge and length of the horizontal stroke. Similarly, for vertical strokes, the deformed convolutional network can expand the receptive field in the vertical direction and shrink it in the horizontal direction to accurately capture the features of the vertical stroke. For the stroke, the deformed convolutional network can tilt the receptive field to the lower left to better capture the starting point and curvature of the stroke. For the stroke, the deformed convolutional network can tilt the receptive field to the lower right to accurately extract the starting point and curvature of the stroke.

[0080] By adjusting the shape of the local receptive field, the deformed convolutional network can more accurately capture the geometric features of the character strokes, which helps to improve the accuracy of subsequent character recognition and detection tasks. The learnable offset of the deformed convolutional network enables it to automatically adapt to different fonts and styles of characters, enhancing the universality and adaptability of character recognition.

[0081] S220, performing adversarial training on the character features in the target character region to constrain the authenticity of high-frequency details, and obtaining character texture features.

[0082] In the embodiments of the present application, the adversarial generative network is used to perform adversarial training on the character features in the target character region to constrain the authenticity of high-frequency details, and obtain character texture features. The generator of the adversarial generative network is responsible for generating character texture features in the target character region, while the discriminator judges the character texture features generated by the generator and distinguishes whether they are real data from real character images or generated data. Through this adversarial training, the generator will continuously adjust its parameters to generate more realistic character texture features, making it difficult for the discriminator to distinguish. The existence of the discriminator prompts the generator to generate character texture features with higher authenticity. It can capture subtle differences and patterns in real character textures, such as the connection method of strokes in different fonts and the curvature of strokes.

[0083] For example, when generating the texture of handwritten characters, the generator may initially generate some blurred and unnatural stroke details that the discriminator can easily identify. However, during the adversarial training process, the generator gradually learns the high-frequency detail features of real handwritten characters, such as stroke thickness variation and ink distribution, and generates texture closer to real handwritten characters. When generating the texture of Songti characters, the generator learns the characteristics of Songti characters, such as horizontal thinness and vertical thickness, and clear edges, so that the generated texture is more realistic.

[0084] In an embodiment of the present application, the loss function of the adversarial generation network includes the weighted sum of the adversarial loss, the content loss, and the perceptual loss. The weighted sum of the adversarial loss, the content loss, and the perceptual loss can comprehensively consider different requirements. The adversarial loss is the core loss function of the adversarial generation network, which is used to measure the difference between the character texture features generated by the generator and the real data. It encourages the generator to generate data that can deceive the discriminator, while also prompting the discriminator to continuously improve its judgment ability. For example, when the character texture features generated by the generator become closer and closer to the real data, the adversarial loss gradually decreases, indicating that the performance of the generator is improving. The content loss usually measures the similarity between the character texture features generated by the generator and the target content based on pixel-level differences. It ensures that the texture generated by the generator is consistent with the target in terms of content and does not have content missing or errors. For example, when generating the texture of a specific character, the content loss ensures that the shape and structure of the character in the generated texture are consistent with the target character. The perceptual loss extracts high-level features of the character texture features generated by the generator and the real data, and then calculates the difference between these high-level features. It focuses on the semantic information and overall perceptual quality of the texture, and can capture features that the human visual system pays more attention to. For example, the perceptual loss can ensure that the generated character texture is similar to the real data in terms of color, texture, and other aspects, making the generated texture look more natural and beautiful.

[0085] By adjusting the weights of each loss, the performance of the generator in terms of authenticity, content accuracy, and perceptual quality can be balanced. For example, if more attention is paid to the authenticity of the generated character texture, the weight of the adversarial loss can be appropriately increased, and if more attention is paid to the consistency of the generated content with the target, the weight of the content loss can be increased. This weighted sum approach can be flexibly adjusted according to specific task requirements to generate more required character texture features.

[0086] S230, adjusting the fusion weight of the character geometric features and the character texture features to obtain character fusion features.

[0087] In the embodiments of the present application, an adaptive feature fusion strategy is adopted, and the fusion weight of the character geometric features and the character texture features is dynamically adjusted according to the characteristics of the character stroke structure, so as to realize the character feature fusion. The adaptive feature fusion strategy is realized through a gating mechanism, and the gating mechanism uses a learnable dynamic weight parameter to adaptively control the feature fusion process, so that the optimal feature representation can be generated in different cases.

[0088] Specifically, a 1x1 convolution layer is introduced to process the input character geometric features and the character texture features, and a gating signal is generated. The value range of the gating signal is limited to [0, 1], which represents the relative importance of different feature streams in the fusion process. For example, when the gating signal is close to 1, it means that the character geometric features play a dominant role in the fusion, and when the gating signal is close to 0, the character texture features are more important. Based on the generated gating signal, the character geometric features and the character texture features are linearly weighted and fused. The linear weighting method is simple and effective, and can flexibly adjust the contribution ratio of the two features according to the value of the gating signal.

[0089] Optionally, the basis for dynamically adjusting the weight parameter is the stroke result of the character stroke. When the character stroke structure is simple, the gating signal tends to be 1, and the structure fidelity of the character geometric features is emphasized. When the character stroke is complex, the gating signal tends to be 0.5, and the geometric and texture features are balanced.

[0090] Illustratively, the stroke structure of printed characters is usually regular and simple, with clear geometric shapes and structures. In this case, the structural information provided by the character geometric features is crucial for accurately restoring the shape and structure of the characters. When the gating signal tends to be 1, the weight of the character geometric features in the fusion is larger, and the weight of the character texture features is smaller. The fused features retain more structural information of the character geometric features, and can better maintain the shape and structure of the characters, avoiding deformation or distortion of the characters due to the interference of the texture features. The stroke structure of handwritten characters has great variability and diversity, and different people have different writing styles. The strokes may be curved and connected, etc. At this time, only relying on the structural information of the character geometric features may not be able to completely and accurately express the features of the handwritten characters, and the stroke details and style information contained in the character texture features are particularly important. When the gating signal tends to be 0.5, the weights of the character geometric features and the character texture features are comparable, and both play a balancing role in the fusion process. Therefore, the fused features not only retain the structural information of the character geometric features, but also incorporate the details and style information of the character texture features, and can better restore the true form and style of the handwritten characters.

[0091] In this embodiment of the invention, the adaptive feature fusion strategy dynamically adjusts the fusion weights of text geometric features and text texture features through a gating mechanism, and flexibly fuses them according to the characteristics of the text stroke structure, effectively improving the quality of text super-resolution reconstruction and enhancing adaptability and generalization ability.

[0092] S240. The semantic rationality of the text fusion features is verified and processed to obtain the super-resolution features of the text region.

[0093] In this embodiment of the invention, a pre-trained language model is used to verify the semantic rationality of the text fusion features. Through a dual mechanism of semantic penalty and post-processing verification, it is ensured that the generated text conforms to linguistic norms while achieving visual enhancement, thus obtaining super-resolution features of the text region. The pre-trained language model is integrated into the loss function to calculate the semantic similarity between the text sequence of the text fusion features and the original low-resolution text. This pre-trained language model has powerful language understanding and semantic analysis capabilities, enabling accurate evaluation of the semantics of the text. For example, if the original low-resolution text is "HD," and the generated text is "Shangqing," the pre-trained language model can identify the semantic difference between the two. If the generated text contains typos or semantic contradictions, the loss value will increase. During backpropagation, the pre-trained language model will adjust the generator's parameters based on the increased loss value, making the generator more focused on semantic accuracy in subsequent generation processes. For example, when generating "Shangqing" leads to an increased loss value, the generator will adjust its parameters to avoid generating similar erroneous text again.

[0094] Post-processing verification detects grammatical errors in the generated text by comparing it with the trained language model. This analyzes the grammatical structure of the text and determines whether grammatical errors exist, such as subject-verb disagreement or inappropriate collocation. If verification fails, i.e., a grammatical error is detected, a rollback mechanism is triggered. The rollback mechanism re-processes the target text region using super-resolution to regenerate the text, aiming to obtain a result that is both semantically and grammatically correct, thus acquiring the super-resolution features of the text region.

[0095] In this embodiment of the invention, the pre-trained language model effectively solves the problem of semantic distortion of text through a dual mechanism of semantic penalty term and post-processing verification, thereby improving the quality and reliability of text generation.

[0096] S300: An optically flow-supervised convolutional neural network is used to enhance the features of the target background region, thereby obtaining super-resolution features of the background region.

[0097] In this embodiment of the invention, efficient super-resolution enhancement of the target background region is achieved through optical flow iterative optimization and perception quality enhancement techniques. Figure 4 This is the fourth schematic diagram of the video super-resolution method flow provided in this embodiment of the invention, as shown below. Figure 4As shown, the super-resolution rate enhancement of the target background region specifically includes the following steps:

[0098] S310, calculate the initial optical flow field of adjacent video frames in the target background region, and optimize the initial optical flow field to obtain the target optical flow field.

[0099] In the embodiment of the present application, for the continuous multi-frame low-resolution video sequence in the target background region, a pre-trained optical flow network model such as RAFT, a deep neural network for optical flow estimation, can be used to calculate the initial optical flow field between adjacent video frames. At the same time, local feature matching constraints are introduced to extract local features between adjacent video frames, and the optical flow field is optimized by minimizing the local feature difference between adjacent frames. Specifically, for each pixel point, its corresponding position in the adjacent frame is found through feature matching, and the optical flow vector is adjusted according to the feature difference. The local feature matching constraint can correct the local error in the initial optical flow field, improve the accuracy of the optical flow field, especially in areas with complex texture or intense motion.

[0100] Optionally, a spatiotemporal consistency regularization term and a Laplacian matrix regularization term can also be introduced to ensure the long-term consistency and spatial smoothness of the optical flow field in the video sequence, and to avoid discontinuous or jumping optical flow vectors. The spatiotemporal consistency regularization term is achieved by limiting the variation amplitude of the optical flow field between adjacent frames, and the Laplacian matrix regularization term is achieved by calculating the second derivative of the optical flow field to punish the discontinuity and discontinuity in the optical flow field. Not only rely on adjacent frames, but also introduce the optical flow field of multiple frames before and after to jointly optimize, enhance the global consistency of motion estimation.

[0101] In the embodiment of the present application, by introducing various constraints and regularization terms to optimize the initial optical flow field, higher quality reconstruction results can be achieved in the super-resolution task of the target background region.

[0102] S320, perform feature extraction on each video frame in the target background region to obtain a video frame super-resolution result.

[0103] In the embodiment of the present application, a convolutional neural network can be used to extract deep features and shallow features of the low-resolution background region. Deep features focus on high semantic information, such as identifying whether the object in the image is a person, a landscape, or other objects. Shallow features focus on high-resolution detail features, such as object edge contours and texture details, etc. Then the deep features and shallow features are spliced in the channel dimension to generate fused features, so that different levels of information can be integrated, and the fused features contain rich semantic information and enough detail information, and a video frame super-resolution result is obtained.

[0104] In the training process of the convolutional neural network, the structural similarity loss is taken as one of the optimization objectives, and the parameters of the generator are adjusted during back propagation, so that the structure of the generated frame is closer to the original frame. The structural similarity loss is an index for measuring the structural similarity of two images, which considers the brightness, contrast and structural information of the image. By calculating the structural similarity loss between the generated frame and the original frame, the generated frame can be constrained to be similar to the original frame in structure.

[0105] A band-pass filter is added to the generator of the convolutional neural network, which allows signals of a specific frequency range to pass through while suppressing signals of other frequency ranges, so that only specific frequency details in the video frame image can be retained and high-frequency noise can be removed. A high-frequency consistency constraint is introduced in the discriminator of the convolutional neural network, which forces the generator to avoid generating non-real high-frequency noise.

[0106] S330, according to the target optical flow field, the spatio-temporal consistency of the video frame super-resolution result is optimized, and the background area super-resolution feature is obtained.

[0107] In the embodiment of the application, the adjacent frame is deformed to the coordinate system of the current frame by calculating the target optical flow field, and the adjacent frame deformation result is obtained. The feature difference between the video frame super-resolution result of the current frame and the adjacent frame deformation result is calculated, and the feature difference is added to the loss function as a regularization term. By minimizing the feature difference, the spatio-temporal consistency between frames is maintained in the super-resolution reconstruction process, thereby effectively avoiding the discontinuity of the super-resolution result in the time dimension, such as frame flicker or jumping, thereby generating content coherent and stable background area super-resolution feature.

[0108] S400, according to the text area super-resolution feature and the background area super-resolution feature, the feature fusion is performed to obtain a super-resolution video.

[0109] In the embodiment of the application, through the strategies such as multi-modal feature alignment, dynamic weight distribution and overlapping region processing, seamless fusion of the text area super-resolution feature and the background area super-resolution feature is realized. Figure 5 is the fifth schematic diagram of the super-resolution method process of the video provided by the embodiment of the application, as shown in Figure 5 the feature fusion specifically includes the following steps:

[0110] S410, the text area super-resolution feature and the background area super-resolution feature are geometrically aligned and feature-aligned.

[0111] Figure 6 is the sixth schematic diagram of the super-resolution method process of the video provided by the embodiment of the application, as shown in Figure 6 the geometric alignment and the feature alignment specifically include the following steps:

[0112] S411. Predict an affine transformation matrix of the target text region and the target background region.

[0113] In the embodiments of the present application, a spatial transformation network is used to predict the affine transformation matrix of the target text region and the target background region, which is used to describe the linear transformation of the target text region to the target background region, including translation, rotation and scaling operations. By minimizing the semantic feature difference, the spatial transformation network can learn the optimal transformation parameters.

[0114] S412. Map the coordinates of the text region super-resolution features to the global coordinate system of the background region super-resolution features according to the affine transformation matrix.

[0115] According to the affine transformation matrix, a sampling grid covering the entire video frame is generated, which defines the new position of each pixel in the video frame after transformation. Optionally, using bilinear interpolation or nearest neighbor interpolation method, the text region super-resolution features are pixel-level transformed according to the generated sampling grid, and these interpolation methods can smoothly align the text region super-resolution features to the global coordinate system of the background region super-resolution features. After transformation, the text region super-resolution features and the background region super-resolution features are spatially aligned, realizing pixel-level geometric alignment.

[0116] S413. Align the edge contours, text strokes and semantic information of the text region super-resolution features and the background region super-resolution features.

[0117] In the embodiments of the present application, a feature pyramid fusion architecture is used to align the text region super-resolution features and the background region super-resolution features. The bottom layer of the feature pyramid fusion architecture focuses on the fusion of edge contours, and the bottom layer features usually contain rich detail information, such as edges and textures, which are suitable for maintaining the clear outline of the text. The middle layer of the feature pyramid fusion architecture strengthens the stroke features of the text, and the middle layer features can capture the structural information of the text, such as the direction and connection of strokes, which helps to maintain the readability of the text. The high layer of the feature pyramid fusion architecture constrains the semantic consistency of the fused features with the original background, and the high layer features contain rich semantic information, which can ensure that the fused background features are semantically consistent with the original background.

[0118] S420. Dynamically assign the fusion weights of the text region super-resolution features and the background region super-resolution features according to the text detection confidence and the optical flow field.

[0119] In the embodiment of the present application, the space-time joint optimization strategy is adopted to solve the weight distribution of the text region super-resolution feature and the background region super-resolution feature when the text and the background region are fused. The fusion weight is distributed according to the text detection confidence in the horizontal dimension, and the weight transmission of adjacent frames is transmitted in the vertical dimension, so as to ensure the inter-frame coherence of the generated result. The weight distribution is based on the text detection confidence. When the text detection is sufficiently confident, the text feature weight is increased, and the background feature weight is reduced, so as to preferentially retain the text details. At the same time, the weight transmission constraint of adjacent frames can be guided by the optical flow field. The optical flow change amplitude is constrained by calculating the optical flow field difference of adjacent frames, so as to avoid the flicker and jumping of the text in the video.

[0120] S430, according to the fusion weight, the text region super-resolution feature and the background region super-resolution feature are fused to obtain a fused super-resolution feature.

[0121] In the embodiment of the present application, according to the fusion weight obtained by calculation, the text region super-resolution feature and the background region super-resolution feature are weighted and summed to obtain a fused super-resolution feature.

[0122] S440, the overlapping area of the text and the background in the fused super-resolution feature is compensated by a mask to obtain a super-resolution video.

[0123] In the embodiment of the present application, for the overlapping area of the text and the background in the fused super-resolution feature, a double-flow mask generation network is designed. The main mask network marks the invisible area based on semantic segmentation, the auxiliary mask network filters the dynamic occlusion error through optical flow consistency test, and finally the fused mask is normalized to realize pixel-level occlusion compensation. Figure 7 is the seventh schematic diagram of the super-resolution method process of the video provided by the embodiment of the present application, as shown in Figure 7 The mask compensation specifically includes the following steps:

[0124] S441, the fused super-resolution feature is subjected to semantic segmentation to obtain low-level features and high-level features.

[0125] In the embodiment of the present application, the pre-trained semantic segmentation model can be used to extract the multi-scale features of the fused super-resolution feature to obtain low-level features and high-level features.

[0126] S442, the low-level features are inserted into the dilated convolution to expand the receptive field, and the high-level features are aligned with the low-level features through upsampling to obtain a main mask.

[0127] A hole convolution is inserted in the low-level feature to expand the receptive field to capture the global structure information of the text. The high-level feature is aligned with the low-level feature by a upsampling operation, enhances the recognition ability of the small text, and obtains a semantic segmentation result. The upsampling can use deconvolution or interpolation method, so that the spatial resolution of the high-level feature matches the low-level feature. The occluded or invisible text area is marked by the semantic segmentation result to generate a main mask.

[0128] S443, calculate the optical flow consistency difference between the front and rear frames in the fused super-resolution feature, and obtain an error mask according to the optical flow consistency difference.

[0129] In the embodiment of the present application, the optical flow field between the front and rear frames in the fused super-resolution feature is calculated using an optical flow estimation network, and the optical flow consistency of the front and rear frames is analyzed. If the optical flow difference of a certain area is too large, it may indicate that the area has occlusion and motion blur or other abnormal conditions, and is therefore determined as a potential error area. A suitable difference threshold is set, and the area whose optical flow difference exceeds the difference threshold is marked as an error area to generate an error mask, which is a binary image, wherein the error area is 1 and the non-error area is 0.

[0130] S444, according to the main mask and the error mask, the fused super-resolution feature is compensated for occlusion to obtain a super-resolution video.

[0131] In the embodiment of the present application, the main mask and the error mask are combined to suppress the missegmentation of the text in the occlusion area through logical operation. Specifically, if a certain area is marked as an error area in the error mask, the pixel value corresponding to the area in the main mask is set to 0 (i.e. background), thereby avoiding missegmentation. The fused mask is normalized to make the pixel value range between [0, 1], and the normalized mask can be used to guide the super-resolution reconstruction, realize the pixel-level occlusion compensation, and obtain a super-resolution video.

[0132] As a further embodiment of the present application, a super-resolution model of the video can be constructed based on the above-mentioned embodiments. The super-resolution model of the video needs to be trained by multiple sets of low-resolution video and high-resolution video pairs. The learning goal is to minimize the difference between the model super-resolution video and the real high-resolution video, which can be measured from the high-fidelity reconstruction of the text and background area and the global consistency.

[0133] Optionally, the collection of low-resolution videos can be through real data and degradation data collection, directly using the low-resolution materials shot, such as mobile phone recorded videos, to retain real noise and compression artifacts. It is also possible to perform bicubic interpolation degradation sampling on original high-definition videos of 4K resolution or the like to generate low-resolution videos. It is also possible to add scene data of motion blur such as racing and dancing videos in the training data to improve the processing capability of the model for complex motion. The text region labeling data can adopt manual labeling or a semantic segmentation model to generate a text mask, and meanwhile, data enhancement methods such as random cropping and color jittering are introduced to improve the generalization of the training data. Then, the training data is divided into a training set, a validation set and a test set according to a ratio of 7:2:1.

[0134] In the embodiment of the application, the loss function of the video super-resolution model can adopt a multi-task joint optimization loss function, including an adversarial loss, a content loss, a perception loss, a structural similarity loss and an inter-frame consistency loss, and multi-objective collaborative optimization is realized through dynamic weight distribution. In the initial training, the adversarial loss and the content loss are focused on, and in the later period, the perception loss, the structural similarity loss and the inter-frame consistency loss are increased to improve the detail authenticity.

[0135] Optionally, in the training process of the video super-resolution model, the optimizer adopts an Adam optimizer, and gradient clipping is adopted to limit the gradient norm to not more than 1 to prevent unstable training. In the training process, a cosine annealing strategy is adopted, the learning rate is gradually reduced in the later training period, and L2 regularization is added to prevent overfitting of the model training.

[0136] The video super-resolution method provided by the embodiment of the application can accurately locate the position and boundary of the text in the video frame through the design of the neural network specially recognizing and separating the text region. For the text region, the structural features and semantic information of the text are fully considered for super-resolution processing, which can effectively improve the clarity and readability of the text, so that the audience can clearly see the text content in the video, and the information acquisition is not affected by the blurred text. For the background region, super-resolution processing is also performed, which focuses on improving the quality of the text while not ignoring the processing of the background picture, ensuring that the quality of the background picture is not affected and maintaining the overall coordination and aesthetic degree of the video picture.

[0137] Embodiment 2

[0138] Based on the same technical concept as the above embodiment, the embodiment of the application provides a video super-resolution system. Figure 8 is a structural schematic diagram of the video super-resolution system provided by the embodiment of the application, as Figure 8 shown, the video super-resolution system 200 includes:

[0139] The character background separation module 210 is configured to separate a target video into a target character region and a target background region.

[0140] The character region super-resolution module 220 is configured to perform geometric flow and texture flow feature enhancement on the target character region respectively to obtain character region super-resolution features.

[0141] The background region super-resolution module 230 is configured to perform feature enhancement on the target background region by using a convolutional neural network supervised by an optical flow to obtain background region super-resolution features.

[0142] The super-resolution feature fusion module 240 is configured to perform feature fusion on the character region super-resolution features and the background region super-resolution features to obtain a super-resolution video.

[0143] The super-resolution system for videos provided in the embodiments of the present application can generate a super-resolution video that is clear in the whole and rich in character details, thereby improving the quality of the video as a whole, bringing better visual experience to users, meeting the demand of users for high-quality videos, and playing a positive role in multiple application scenarios such as video playing and information display.

[0144] It can be understood that the implementation manners in the super-resolution method for videos described in Embodiment 1 are also applicable to the present embodiment and can achieve the same technical effects, and thus will not be described herein again.

[0145] Embodiment 3

[0146] Based on the same idea, the embodiments of the present application further provide an electronic device, Figure 9 is a structural schematic diagram of an electronic device provided in the embodiments of the present application, as Figure 9 indicated, the electronic device 300 can include a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 complete mutual communication through the communication bus 340. The processor 310 can invoke logical instructions in the memory 330 to execute the steps of the super-resolution method for videos as described in the above embodiments. For example, it includes:

[0147] S100, separating a target video into a target character region and a target background region;

[0148] S200, performing geometric flow and texture flow feature enhancement on the target character region respectively to obtain character region super-resolution features.

[0149] S300, performing feature enhancement on the target background region by using a convolutional neural network supervised by an optical flow to obtain background region super-resolution features.

[0150] S400, performing feature fusion according to the text region super-resolution feature and the background region super-resolution feature to obtain a super-resolution video.

[0151] The processor 310 can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, or combinations thereof.

[0152] In addition, the logic instructions in the memory 330 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0153] The memory 330 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required by a function, and the like. The data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0154] Example 4

[0155] Based on the same concept, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program containing at least one code, and the at least one code is executable by a host device to control the host device to implement steps of the video super-resolution method as described in the above embodiments. For example, the computer readable storage medium includes:

[0156] S100, performing text and background separation on the target video to obtain a target text region and a target background region;

[0157] S200, performing geometric flow and texture flow feature enhancement on the target text region respectively to obtain text region super-resolution features;

[0158] S300, performing feature enhancement on the target background region by using a convolutional neural network supervised by an optical flow to obtain background region super-resolution features;

[0159] S400, performing feature fusion according to the text region super-resolution features and the background region super-resolution features to obtain a super-resolution video.

[0160] Based on the same technical concept, the embodiments of the present application also provide a computer program, which is used to implement the above method embodiments when the computer program is executed by a host device.

[0161] The computer program can be stored in a computer readable storage medium packaged with the processor in whole or in part, or stored in a memory not packaged with the processor in whole or in part.

[0162] Based on the same technical concept, the embodiments of the present application also provide a processor, which is used to implement the above method embodiments. The processor can be a chip.

[0163] In summary, the video super-resolution method, system, electronic device and storage medium provided by the present application can accurately locate the position and boundary of the text in the video frame by designing a neural network specially recognizing and separating the text area. For the text area, the super-resolution processing is performed by fully considering the structural features and semantic information of the text, which can effectively improve the clarity and readability of the text, so that the audience can clearly see the text content in the video, and avoid affecting information acquisition due to blurred text. For the background area, super-resolution processing is also performed, which focuses on improving the quality of the text while not ignoring the processing of the background picture, ensuring that the quality of the background picture is not affected, maintaining the overall coordination and aesthetics of the video picture. The processed text area and background area are effectively fused to generate the final high-definition video frame, realizing the differential processing of the text and the background in the video frame. The finally generated super-resolution video is globally clear and has rich text details, which improves the quality of the video as a whole, brings better visual experience to the user, meets the user's demand for high-quality video, and plays a positive role in video playing, information display and other application scenarios.

[0164] Reference to "an embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in another embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive.

[0165] The above-described embodiments are merely illustrative of the present application and are not to be taken in a limiting sense. It is to be understood that various modifications and improvements can be made to the embodiments without departing from the spirit and scope of the present application. Accordingly, the scope of the present application should be determined by the appended claims.

[0166] Finally, it should be noted that the above embodiments are merely used to illustrate the technical solutions of the present application, and are not intended to limit the same; even though the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features; and such modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A super-resolution method of a video, characterized by, The method comprises: text and background separation of the target video to obtain a target text region and a target background region; geometric flow and texture flow feature enhancement is performed on the target text region respectively to obtain text region super-resolution features; a convolutional neural network supervised by optical flow is used to perform feature enhancement on the target background region to obtain background region super-resolution features; feature fusion is performed according to the text region super-resolution features and the background region super-resolution features to obtain a super-resolution video, comprising: geometric alignment and feature alignment are performed on the text region super-resolution features and the background region super-resolution features; the fusion weights of the text region super-resolution features and the background region super-resolution features are dynamically allocated according to text detection confidence and optical flow field; feature fusion is performed on the text region super-resolution features and the background region super-resolution features according to the fusion weights to obtain fused super-resolution features; mask compensation is performed on the overlapping region of text and background in the fused super-resolution features to obtain the super-resolution video, comprising: semantic segmentation is performed on the fused super-resolution features to obtain low-level features and high-level features; the low-level features are inserted into a dilated convolution to expand the receptive field, and the high-level features are progressively aligned with the low-level features through upsampling to obtain a main mask; the optical flow consistency difference between the previous and subsequent frames in the fused super-resolution features is calculated to obtain an error mask according to the optical flow consistency difference; occlusion compensation is performed on the fused super-resolution features according to the main mask and the error mask to obtain the super-resolution video.

2. The super-resolution method of video according to claim 1, wherein, The text and background separation of the target video to obtain a target text region and a target background region comprises: extracting text structures of different sizes in each video frame of the target video to obtain initial text features; performing progressive upsampling and channel weighting operations on the initial text features to obtain enhanced text features; adjusting the channel weights of the enhanced text features to obtain the target text region and the target background region.

3. The super-resolution method of video according to claim 1, wherein, The geometric flow and texture flow feature enhancement performed on the target text region respectively to obtain text region super-resolution features comprises: adjusting the local receptive field shape of the text features in the target text region to obtain text geometric features; performing adversarial training on the text features in the target text region to constrain the authenticity of high-frequency detail generation to obtain text texture features; adjusting the fusion weights of the text geometric features and the text texture features to obtain text fusion features; performing semantic rationality inspection processing on the text fusion features to obtain the text region super-resolution features.

4. The super-resolution method of video according to claim 1, wherein, The convolutional neural network supervised by optical flow is used to perform feature enhancement on the target background region to obtain background region super-resolution features, comprising: calculating the initial optical flow field of adjacent video frames in the target background region, and optimizing the initial optical flow field to obtain a target optical flow field; performing feature extraction on each video frame in the target background region to obtain video frame super-resolution results; performing spatio-temporal consistency optimization on the video frame super-resolution results according to the target optical flow field to obtain the background region super-resolution features.

5. The super-resolution method of video according to claim 1, wherein, The geometric alignment and feature alignment of the text region super-resolution feature and the background region super-resolution feature comprises: predicting an affine transformation matrix of the target text region and the target background region; mapping coordinates of the text region super-resolution feature to a global coordinate system of the background region super-resolution feature according to the affine transformation matrix; aligning edge contours, text strokes and semantic information of the text region super-resolution feature and the background region super-resolution feature.

6. A super-resolution system of a video, characterized by, The system comprises: a text-background separation module for separating a target video into text and background to obtain a target text region and a target background region; a text region super-resolution module for respectively performing geometric flow and texture flow feature enhancement on the target text region to obtain text region super-resolution features; a background region super-resolution module for performing feature enhancement on the target background region by using an optical flow supervised convolutional neural network to obtain background region super-resolution features; a super-resolution feature fusion module for performing feature fusion on the text region super-resolution features and the background region super-resolution features to obtain a super-resolution video, comprising: geometric alignment and feature alignment of the text region super-resolution features and the background region super-resolution features; dynamically allocating fusion weights of the text region super-resolution features and the background region super-resolution features according to text detection confidence and optical flow field; performing feature fusion on the text region super-resolution features and the background region super-resolution features according to the fusion weights to obtain fused super-resolution features; mask compensation on overlapping regions of text and background in the fused super-resolution features to obtain the super-resolution video, comprising: semantic segmentation of the fused super-resolution features to obtain low-level features and high-level features; inserting the low-level features into an empty convolution to expand the receptive field, and aligning the high-level features with the low-level features step by step through upsampling to obtain a main mask; calculating optical flow consistency differences between previous and subsequent frames in the fused super-resolution features, and obtaining an error mask according to the optical flow consistency differences; occlusion compensation on the fused super-resolution features according to the main mask and the error mask to obtain the super-resolution video.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the super-resolution method of the video according to any one of claims 1-5.

8. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the super-resolution method of the video according to any one of claims 1-5.

Images