Image enhancement method, device and server
By performing algorithm processing on text detection and area matching of video frames, the problem of oversharpening of text areas in the prior art is solved, and the overall visual effect of video images is improved.
Patent Information
- Application Number
- CN201911057130.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-10-31
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-10-31
AI Technical Summary
Existing image enhancement methods provide the same degree of enhancement to all areas of the image, resulting in excessive sharpening of the edges of the text area, affecting the overall visual effect of the image.
By performing text detection on video frames, the text area is determined, and the image enhancement process is performed using the first preset algorithm that matches the text area. The second preset algorithm is used for processing areas other than the text area to ensure images in different areas Appropriate degree of enhancement.
It effectively avoids excessive enhancement of text areas, improves the overall visual effect of video images, and improves the clarity and readability of the image.
Smart Images

Figure CN112749599B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image enhancement method, device and server. Background Art
[0002] The image enhancement method in the related art usually enhances all areas of the image to the same degree. Compared with the image background, the edge gradient of the text in the image changes more dramatically. Therefore, this method of enhancing all areas of the image to the same degree can easily cause the edges of the text area to be too sharp, affecting the overall visual effect of the image. Summary of the invention
[0003] The object of the present invention is to provide an image enhancement method, device and server to improve the overall visual effect of video images.
[0004] In a first aspect, an embodiment of the present invention provides an image enhancement method, which includes: obtaining a video to be processed; the video to be processed includes multiple video frames; performing text detection on the multiple video frames to obtain detection results; determining text areas of the multiple video frames based on the degree of overlap of the detection results between adjacent video frames in the multiple video frames; using a first preset algorithm to perform image enhancement processing on the text areas in the multiple video frames; using a second preset algorithm to perform image enhancement processing on areas other than the text areas in the multiple video frames; wherein the first preset algorithm matches the text area.
[0005] In a preferred embodiment of the present invention, the above-mentioned step of performing text detection on multiple video frames to obtain detection results includes: for each video frame in the multiple video frames, performing text region segmentation processing on the current video frame to obtain multiple candidate regions of the text region in the video frame; wherein the scales of the multiple candidate regions are multiple; performing morphological processing on the multiple candidate regions to obtain detection results of the video frames.
[0006] In a preferred embodiment of the present invention, the above-mentioned step of performing text area segmentation processing on the current video frame includes: inputting the current video frame into a pre-trained text area segmentation model to obtain multiple candidate areas of the text area in the current video frame; wherein the text area segmentation model is trained in the following manner: determining a target training image based on a preset training set; wherein the target training image carries text area identifiers of multiple scales; inputting the target training image into an initial network model, and outputting pixel feature data of the target training image; the pixel feature data includes a feature value of each pixel in the target training image; calculating a model loss value based on the pixel feature data and the text area identifier; adjusting the network parameters in the initial network model according to the model loss value, and continuing to perform the step of determining the target training image based on the preset training set until the model loss value meets the preset specified value or the number of target training images input into the initial network model meets the preset number, thereby obtaining a text area segmentation model.
[0007] In a preferred embodiment of the present invention, the above-mentioned text area identification includes: a mask corresponding to the text area in the target training image; wherein the scale of the maximum-scale mask is the same as the scale of the text area in the target training image; the scales of the masks other than the maximum-scale mask are obtained by reducing the scale of the previous-scale mask by a preset percentage.
[0008] In a preferred embodiment of the present invention, among the above-mentioned multiple candidate areas, candidate areas of the same scale are identified in the same mask; the step of performing morphological processing on the multiple candidate areas includes: extracting a mask corresponding to the candidate area of the smallest scale from the multiple candidate areas; determining a connected domain from the mask corresponding to the candidate area of the smallest scale; using a preset template to perform a morphological dilation operation on the connected domain; and determining the detection result of the video frame according to the dilated connected domain.
[0009] In a preferred embodiment of the present invention, the step of determining the detection result of the video frame according to the expanded connected domain includes: determining the previous scale of the minimum scale as the current scale; wherein the masks corresponding to the candidate regions of different scales are arranged in descending order according to the scales of the candidate regions; performing a bit operation on the pixel values of the corresponding pixels in the expanded connected domain and the mask of the current scale to obtain a calculation result; continuing to perform a morphological expansion operation on the expanded connected domain until the calculation result converges; determining the previous scale of the current scale as the new current scale, and continuing to perform a bit operation on the pixel values of the corresponding pixels in the expanded connected domain and the mask of the current scale to obtain a calculation result until the current scale is the maximum scale and the calculation result converges; determining the detection result of the video frame according to the connected domain after the last expansion.
[0010] In a preferred embodiment of the present invention, the above-mentioned detection results are identified by a text box, which is used to identify the image area containing text in the video frame; after performing text detection on multiple video frames to obtain the detection results, the method also includes: calculating the number of pixels in the image area in each text box in the detection results to obtain the calculation results; based on the calculation results, filtering the text boxes and using the filtered results as the final detection results.
[0011] In a preferred embodiment of the present invention, the above-mentioned step of filtering text boxes according to the calculation results and using the filtering results as the final detection results includes: for each text box, determining whether the number of pixels in the image area within the current text box is less than a preset number threshold; if it is less than the number threshold, deleting the current text box; and using the remaining text boxes as the final detection results.
[0012] In a preferred embodiment of the present invention, after the above step of taking the remaining text boxes as the final detection result, the method further comprises: expanding the scale of the remaining text boxes according to a preset percentage; and taking the expanded text boxes as the final detection result.
[0013] In a preferred embodiment of the present invention, the above detection result is identified by a text box, and the text box is used to identify the image area containing text in the video frame; the step of determining the text area of the multiple video frames according to the overlap degree of the detection results between adjacent video frames in the multiple video frames includes: taking the first video frame of the multiple video frames as the current video frame, calculating the IOU value of the text box between the current video frame and the next video frame of the current video frame; determining the text area of the next video frame of the current video frame according to the calculated IOU value;
[0014] The next video frame of the current video frame is used as a new current video frame, and the step of calculating the IOU value of the text box between the current video frame and the next video frame of the current video frame is continued until the text area of the last video frame among the multiple video frames is determined.
[0015] In a preferred embodiment of the present invention, the step of determining the text area of the next video frame of the current video frame based on the calculated IOU value includes: judging whether the IOU value of the text box between the current video frame and the next video frame of the current video frame is greater than or equal to a preset overlap threshold; if it is greater than or equal to the overlap threshold, taking the image area corresponding to the text box of the next video frame as the text area of the next video frame; if it is less than the overlap threshold, judging whether the detection result of the current video frame contains the text box, and determining the text area of the next video frame of the current video frame according to the judgment result.
[0016] In a preferred embodiment of the present invention, the step of determining the text area of the next video frame of the current video frame based on the judgment result includes: if the detection result of the current video frame includes a text box, predicting the position of the text box in the next video frame based on the text box in the current video frame, and using the prediction result as the text area of the next video frame; if the detection result of the current video frame does not include a text box, using the image area corresponding to the text box of the next video frame as the text area of the next video frame.
[0017] In a preferred embodiment of the present invention, before the step of using the first preset algorithm to perform image enhancement processing on the text area in multiple video frames, the method also includes: adjusting the algorithm parameters in the second preset algorithm to match the algorithm parameters with the text area; wherein the algorithm parameters include filtering parameters and / or sharpening parameters; and determining the second preset algorithm after parameter adjustment as the first preset algorithm.
[0018] In a second aspect, an embodiment of the present invention provides an image enhancement device, which includes: a video acquisition module, used to acquire a video to be processed; the video to be processed includes multiple video frames; a text detection module, used to perform text detection on the multiple video frames to obtain detection results; a text determination module, used to determine the text area of the multiple video frames according to the degree of overlap of the detection results between adjacent video frames in the multiple video frames; an image enhancement module, used to use a first preset algorithm to perform image enhancement processing on the text area in the multiple video frames; and use a second preset algorithm to perform image enhancement processing on the area other than the text area in the multiple video frames; wherein the first preset algorithm matches the text area.
[0019] In a third aspect, an embodiment of the present invention provides a server, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned image enhancement method.
[0020] In a fourth aspect, an embodiment of the present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned image enhancement method.
[0021] The embodiments of the present invention bring the following beneficial effects:
[0022] The present invention provides an image enhancement method, device and server, firstly obtaining a video to be processed; then performing text detection on multiple video frames in the video to be processed to obtain detection results; then determining text regions of multiple video frames according to the degree of overlap of detection results between adjacent video frames in the multiple video frames; finally performing image enhancement processing on the text regions in the multiple video frames using a first preset algorithm matching the text regions, and performing image enhancement processing on regions other than the text regions in the multiple video frames using a second preset algorithm. The present invention uses different algorithms to perform different degrees of image enhancement on text regions and non-text regions in video frames, thereby avoiding excessive enhancement of text regions and helping to improve the overall visual effect of video images.
[0023] Other features and advantages of the present invention will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present invention.
[0024] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are specifically listed below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0026] Figure 1 A flowchart of an image enhancement method provided by an embodiment of the present invention;
[0027] Figure 2 A flowchart of another image enhancement method provided by an embodiment of the present invention;
[0028] Figure 3 A flowchart of another image enhancement method provided by an embodiment of the present invention;
[0029] Figure 4 A schematic diagram of masks of various scales in a target training image provided by an embodiment of the present invention;
[0030] Figure 5 A structural schematic diagram of a text region segmentation model provided by an embodiment of the present invention;
[0031] Figure 6 A schematic diagram of a processing flow of a morphological dilation operation provided by an embodiment of the present invention;
[0032] Figure 7 A flowchart of another image enhancement method provided by an embodiment of the present invention;
[0033] Figure 8 A flowchart of another image enhancement method provided by an embodiment of the present invention;
[0034] Fig. 9 A schematic diagram of the structure of an image enhancement device provided by an embodiment of the present invention;
[0035] Fig.10 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0036] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] With the development of intelligent high-definition technology, video processing technology based on image analysis is increasingly widely used in actual video processing tasks. For example, in a live show task scenario, analyzing the anchor character in the video and analyzing the text area in the video are all conducive to video compression, image quality improvement, and image quality restoration. Image enhancement in related technologies usually uses a global image enhancement method, that is, all areas of the image are enhanced to the same degree; but the text in the image usually has more dramatic edge gradient changes than the image background. The global image enhancement method can achieve a good enhancement effect in the background area, but it is easy to cause excessive sharpening of the edges of the text area, affecting the overall visual effect of the image.
[0038] Based on this, the embodiment of the present invention provides an image enhancement method, device and server. The technology can be applied to image enhancement in various scenarios, especially in complex scenarios such as network live broadcast, cable TV live broadcast, games, videos, etc. To facilitate understanding of this embodiment, firstly, an image enhancement method disclosed in the embodiment of the present invention is introduced in detail. Figure 1 As shown, the method comprises the following steps:
[0039] Step S102, obtaining a video to be processed; the video to be processed includes a plurality of video frames.
[0040] The video to be processed may be a video file or a live video, etc. The video to be processed usually includes multiple video frames, which usually include characters, backgrounds, and CG (Computer generated) texts, etc. Among them, CG texts are usually computer-generated texts, which can be applied to image or video post-processing. For example, subtitles or bullet comments are added to the video in post-processing. The subtitles and bullet comments can be called CG texts, and the image area where the CG texts are located can be called text areas.
[0041] Step S104: Perform text detection on the multiple video frames to obtain detection results.
[0042] In specific implementation, for each video frame, a specific area in the video frame can be segmented by an image segmentation algorithm, a neural network model for recognizing text, etc. The specific area can be a person area, a background area, or a text area, etc. The detection result of the text area in the video frame can be obtained according to the image segmentation result. The detection result can be marked on the video frame, for example, by marking the image area containing text in the video frame by an identification frame, and the identification frame can be a rectangular frame, a polygonal frame, or an elliptical frame, etc.
[0043] Step S106: determining text regions of the multiple video frames according to the overlapping degree of detection results between adjacent video frames in the multiple video frames.
[0044] In actual implementation, there may be false detections in the above detection results, for example, a non-text area is detected as a text area. In order to more accurately identify the text area in the video frame, the above detection results need to be processed in one step, that is, the text area is further determined by the degree of overlap of the detection results between adjacent video frames.
[0045] Usually, due to the continuity of video frames, the detection results between adjacent video frames are usually not much different, which can also be understood as the text identifiers in the detection results of two adjacent video frames should overlap, or the overlapping area is large. Therefore, in a specific implementation, the text area of the video frame can be obtained by calculating the overlapping area of the text identifiers in the detection results of every two adjacent video frames in multiple video frames. For example, the overlapping area of the text identifiers in the detection results of two adjacent video frames is calculated. If the overlapping area is less than a preset value, it is determined that the detection of the text identifier in the subsequent video frame in the adjacent video frames is incorrect. At this time, the text identifier position of the subsequent video frame is generally predicted for the text identifier of the previous video frame in the adjacent video frames, and the predicted text identifier position is marked in the subsequent video frame to determine the text area of the subsequent video frame; if the overlapping area is greater than or equal to the preset value, the detection result of the subsequent video frame will be retained.
[0046] Step S108, using a first preset algorithm to perform image enhancement processing on the text area in multiple video frames; using a second preset algorithm to perform image enhancement processing on the area other than the text area in the multiple video frames; wherein the above-mentioned first preset algorithm matches the text area.
[0047] The above-mentioned first preset algorithm and second preset algorithm can be an image filtering algorithm, an image sharpening algorithm, an image color enhancement algorithm, etc., to perform image enhancement processing on the video frames; in one embodiment, the first preset algorithm can be different from the image enhancement principle of the second preset algorithm, such as the first preset algorithm is an image filtering algorithm, and the second preset algorithm is an image sharpening algorithm; in another embodiment, the first preset algorithm can also be the same as the image enhancement principle of the second preset algorithm, such as both are image filtering algorithms, but the algorithm parameters or algorithm models are different, so as to ensure that the first preset algorithm matches the characteristics of the text area itself.
[0048] In specific implementation, due to the strong change in the edge gradient of the text area itself, the algorithm parameters of the first preset algorithm or the sharpening degree in the algorithm model are set to be smaller, and the filtering degree is usually set to be larger than that of the second preset algorithm. For example, in a live show video, a low noise model can be used to filter the non-text area of the video frame, and a medium noise model can be used to process the text area of the video.
[0049] The present invention provides an image enhancement method, firstly obtaining a video to be processed; then performing text detection on multiple video frames in the video to be processed to obtain detection results; then determining text regions of multiple video frames according to the degree of overlap of detection results between adjacent video frames in the multiple video frames; finally performing image enhancement processing on the text regions in the multiple video frames using a first preset algorithm matching the text regions, and performing image enhancement processing on regions other than the text regions in the multiple video frames using a second preset algorithm. The present invention uses different algorithms to perform different degrees of image enhancement on text regions and non-text regions in video frames, thereby avoiding excessive enhancement of the text regions, and is conducive to improving the overall visual effect of the video image.
[0050] The embodiment of the present invention also provides another image enhancement method, which is implemented on the basis of the method described in the above embodiment; the method focuses on describing the specific process of performing text detection on multiple video frames (see steps S204 to S206 for details); Figure 2 As shown, the image enhancement method comprises the following steps:
[0051] Step S202, obtaining a video to be processed; the video to be processed includes a plurality of video frames.
[0052] Step S204, for each of the multiple video frames, performing text region segmentation processing on the current video frame to obtain multiple candidate regions of the text region in the video frame; wherein the multiple candidate regions have multiple scales.
[0053] When performing text region segmentation processing on the current video frame, a label can be assigned to each pixel in the video frame, and each pixel can be classified according to the label. Finally, the segmentation result of the text region in the video frame can be obtained based on the characteristics of the pixels in the text region. The segmentation result can be further segmented at multiple levels to obtain multiple candidate regions corresponding to the text region.
[0054] There are multiple scales of the above-mentioned candidate regions, including a maximum-scale candidate region and multiple small-scale candidate regions. Among them, the small-scale candidate region can be obtained by reducing the large-scale candidate region by a certain proportion, or it can be a preset fixed size; the maximum-scale candidate region may include one or more lines of text, and the small-scale candidate region may only include part of a line of text. Therefore, the large-scale candidate region can usually obtain the outline of the text area, but cannot separate dense text, while the small-scale candidate region can separate dense text, but cannot obtain the outline of the text area, so it is necessary to combine candidate regions of different scales to obtain a more accurate text area.
[0055] Step S206: Perform morphological processing on the above-mentioned multiple candidate regions to obtain detection results of the video frame.
[0056] This morphology can also be called mathematical morphology. This morphology can extract image components that are meaningful for expressing and depicting the shape of the region from an image or video frame, so that subsequent image recognition and image detection work can capture the most essential or distinguishing shape features of the target object. The shape features can be the boundaries of a region or connected regions, etc. Morphological processing is performed on candidate regions of different scales to obtain text regions under the constraints of the largest-scale candidate region. The obtained text region usually includes the segmentation results of each line of text, or the minimum bounding rectangle of each line of text, etc.
[0057] Step S208: determining text regions of the multiple video frames according to the overlapping degree of detection results between adjacent video frames in the multiple video frames.
[0058] Step S210, using a first preset algorithm to perform image enhancement processing on text areas in multiple video frames; using a second preset algorithm to perform image enhancement processing on areas other than text areas in multiple video frames; wherein the first preset algorithm matches the text area.
[0059] The above-mentioned image enhancement method performs text region segmentation processing on each of the multiple video frames in the acquired video to obtain multiple candidate regions of the text region in the video frame; then performs morphological processing on the multiple candidate regions to obtain the detection results of the video frame; then determines the text regions of the multiple video frames according to the degree of overlap of the detection results between adjacent video frames in the multiple video frames; finally, uses the first preset algorithm to perform image enhancement processing on the text regions in the multiple video frames; and uses the second preset algorithm to perform image enhancement processing on the regions other than the text regions in the multiple video frames. This method processes the candidate regions obtained after the text region segmentation processing by morphology, and can quickly detect the text region in the video frame, thereby improving the efficiency of text region detection. At the same time, image enhancement is performed separately on the text region and the non-text region, thereby improving the visual effect of the image in the video.
[0060] The embodiment of the present invention also provides another image enhancement method, which is implemented on the basis of the method described in the above embodiment; the method focuses on describing the specific process of performing text region segmentation on a video frame by using a text region segmentation model (see step S304 for details), and the specific process of performing morphological processing on multiple candidate regions (see steps S306 to S312 for details); Figure 3 As shown, the image enhancement method comprises the following steps:
[0061] Step S302, obtaining a video to be processed; the video to be processed includes a plurality of video frames.
[0062] Step S304: for each of the above-mentioned multiple video frames, input the current video frame into a pre-trained text region segmentation model to obtain multiple candidate regions of the text region in the current video frame.
[0063] The above-mentioned text region segmentation model can be a neural network, other artificial intelligence networks or models, and can be trained through the following steps 01-04:
[0064] Step 01, determining a target training image based on a preset training set; wherein the target training image carries text region identifiers of multiple scales.
[0065] The preset training set usually includes multiple images (for example, 3,000, 10,000 or even more), and may also include images of various scenes, such as live scene images, game scene images, outdoor scene images, indoor scene images, etc.; the images in the training set may also include text lines of various colors, font sizes, shapes, fonts, and languages, so that the trained model can segment various types of text lines.
[0066] Each image in the training set contains a manually annotated text region, and the target training images determined from the preset training set carry text region identifiers of multiple scales, which can be a rectangular border of the text region or other borders that match the text.
[0067] The above-mentioned text area mark can be a rectangular frame marked on the image, or a mask corresponding to the text area in the target training image; the mask usually covers the selected area in the image in whole or in part to control the area of image processing; the area covered by the mask is usually the area within the rectangular frame or other frame matching the text in the target training image. There are masks of multiple scales corresponding to the text area in the target training image, wherein the scale of the mask with the largest scale is the same as the scale of the text area in the target training image; the scales of the masks other than the mask with the largest scale are obtained by reducing the scale of the mask of the previous scale by a preset percentage.
[0068] In the specific implementation, the text area in the target training image F can be filled with 1, and other areas can be filled with 0 to form the maximum scale mask S n ,like Figure 4 As shown, the maximum scale mask S is then n The corresponding text area is reduced by a preset percentage (e.g., 10%) to obtain the mask S of the next scale. n-1 , and then the mask S of the next scale n-1 The corresponding text area is reduced according to a preset percentage to obtain the mask S of the next scale n-2 , continue to shrink according to the preset ratio until a mask S1 of the minimum scale is obtained. The scale of the mask of the minimum scale is usually set according to demand, that is, when the mask is shrunk to the minimum scale, the shrinking will stop.
[0069] Step 02: input the target training image into the initial network model, and output pixel feature data of the target training image; the pixel feature data includes the feature value of each pixel in the target training image.
[0070] The initial network model is usually a convolutional neural network model or other types of neural networks, wherein the convolutional neural network is usually a neural network designed specifically for image recognition problems, which imitates the multi-layer process of human image recognition, such as pupil intake of pixels, preliminary processing of certain cells in the cerebral cortex (e.g., discovering shape edges and directions), abstract judgment of shapes (e.g., circles, squares), and further abstract judgment (e.g., judging that the object is a balloon). Before inputting the target training image into the initial network model, the target training model can be adjusted to an RGB (Red Green Blue) image of a preset size. For example, the target training image is adjusted to a standard 320*320 RGB image.
[0071] Step 03, calculate the model loss value based on the above pixel feature data and text area identification.
[0072] Based on the pixel feature data and the text area identifier of each scale, the loss value of the corresponding scale can be calculated, and the weighted sum of the loss values of each scale can be used to obtain the model loss value. Since the text area only accounts for a small proportion of the image, when calculating the model loss value, only the non-text area that is very similar to the text area needs to be selected to test the model output performance.
[0073] Step 04, adjust the network parameters in the initial network model according to the above model loss value, and continue to perform the step of determining the target training image based on the preset training set until the model loss value meets the preset specified value or the number of target training images input into the initial network model meets the preset number, thereby obtaining the text area segmentation model.
[0074] Figure 5 The invention shows a structural schematic diagram of a text region segmentation model; the target training image is input into a convolutional neural network, pixel feature data can be output, the model loss value can be obtained according to the pixel feature data, the pixel feature data can be used to obtain a pixel score map of the target training image through a sigmoid function, and the pixel feature data can be used to obtain multiple candidate regions of the target training image through a sign function. When the model loss value is greater than a preset specified value, or the number of target training images input into the initial network model is less than a preset number (equivalent to the training of the network model not meeting the number of iterations), the target training image will continue to be input into the initial network model, and the model loss value will be output.
[0075] Step S306, extracting a mask corresponding to a candidate region with the smallest scale from the multiple candidate regions; wherein, among the multiple candidate regions, candidate regions with the same scale are identified in the same mask.
[0076] The above-mentioned multiple candidate regions include multiple scales. The scale of the candidate region is the same as the scale of the mask. It can also be understood that the candidate regions of the same scale are marked in the same mask. For example, for a candidate region of a certain scale, the region in the candidate region of the scale is marked as 1, and the region outside the candidate region is marked as 0, so that the mask at the scale can be obtained.
[0077] Step S308, determining a connected domain from the mask corresponding to the candidate region of the minimum scale; the connected domain is usually a mask region that is horizontally connected in the mask corresponding to the candidate region, wherein one connected domain usually corresponds to one text line in the video frame.
[0078] The mask corresponding to the candidate region of the smallest scale may include one or more connected domains, such as Figure 3 As shown in Figure 1, the mask corresponding to the candidate region of the smallest scale is equivalent to S1. Figure 4 There are four connected domains in S1.
[0079] Step S310: Using a preset template, perform a morphological dilation operation on the connected domain.
[0080] The above preset template is usually a conventional morphological template. In specific implementation, an image matrix can be established according to the structural element, and then the origin of the structural element can be determined from the image matrix to obtain the morphological template. The morphological template can be in various shapes, such as square, rectangle or other irregular shapes; Figure 6 The matrix composed of four pixels in is the morphological template, which is a square.
[0081] A morphological dilation operation is performed on each connected region corresponding to the candidate region of the minimum scale. The processing process of the morphological dilation operation is as follows: Figure 6 As shown, first obtain the video frame corresponding to the connected domain with the text area identifier, wherein the text area in the connected domain is filled with 1, and the area outside the connected area is filled with 0; then use the morphological template to slide from left to right and from top to bottom in the video frame corresponding to the connected domain with the text area identifier (equivalent to the original image in the figure), so that the morphological template can traverse each pixel in the video frame, when the pixel in the video frame to which the morphological template slides is multiplied by the filling number of the corresponding position of the pixel in the morphological template and the sum is greater than or equal to 1, the position of the pixel in the slid image corresponding to the origin pixel in the morphological model is filled with 1, and the surroundings of the pixels filled with 1 in the image are also filled with 1 to obtain the expanded connected domain.
[0082] Figure 6 The first image in the upper left corner is the original image without morphological dilation operation, and the image in the lower right corner is the image after morphological dilation operation. Figure 6It can be seen that the area of the text area filled with 1 in the expanded image is larger than that in the original image, that is, the expanded image is equivalent to filling some of the surrounding pixels with 1 in the pixels filled with 1 in the original image.
[0083] Step S312: determining the detection result of the video frame according to the expanded connected domain.
[0084] After dilating each minimum-scale connected domain, each dilated connected domain obtained corresponds to a text line in the video frame, and the dilated connected domains obtained are combined to obtain the text region of the video frame. Then, according to the dilated connected domains, the step of determining the text region of the image to be processed can be implemented by the following steps 10-14:
[0085] Step 10, determining the previous scale of the minimum scale as the current scale; wherein the masks corresponding to the candidate regions of different scales are arranged in descending order according to the scales of the candidate regions, thereby obtaining masks whose scales are arranged in descending order.
[0086] Step 11, bitwise operation is performed on the expanded connected domain and the pixel value of the corresponding pixel in the mask of the current scale to obtain the operation result; the pixel value is usually the filling value of the pixel at the position, usually 1 or 0. In specific implementation, the pixel value in the video frame corresponding to the expanded connected domain is bitwise operated (for example, AND) with the pixel value at the corresponding position in the video frame corresponding to the mask of the current scale to obtain the operation result.
[0087] Step 12, continue to perform morphological dilation operation on the expanded connected domain until the operation result converges; if the operation result obtained this time is different from the operation result obtained last time, then continue to perform morphological dilation operation on the expanded connected domain until the operation result converges, which can also be understood as the operation result obtained this time is the same as the operation result obtained last time.
[0088] Step 13: determine the previous scale of the current scale as the new current scale, and continue to execute the above step 11 until the current scale is the maximum scale and the calculation result converges.
[0089] For example, the previous scale of the previous scale of the minimum scale is determined as the current scale, and a bit operation is performed on the expanded connected domain and the pixel value at the corresponding position in the mask of the current scale; based on the bit operation result obtained this time and the bit operation result obtained last time, it is determined whether to continue the expansion operation on the expanded connected domain; if the expansion is not to continue, the previous scale of the current scale is determined as the new current scale and the bit operation is performed on the expanded connected domain until the maximum scale is determined as the current scale.
[0090] Step 14: Determine the detection result of the video frame according to the connected domain after the last dilation.
[0091] The above steps 10-14 are performed for each connected domain corresponding to the candidate region of the minimum scale to obtain the connected domain after the last expansion. In order to determine whether the connected domain after the last expansion is a text region, it is necessary to extract the pixel feature values corresponding to the connected domain obtained by the last expansion operation from the pixel feature data of the image to be processed output by the pre-trained text region segmentation model; calculate the average value of the extracted pixel feature values; if the average value is greater than the preset average value threshold, determine that the connected domain obtained by the last expansion operation is the final detection result.
[0092] Step S314: determining text regions of the multiple video frames according to the overlapping degree of detection results between adjacent video frames in the multiple video frames.
[0093] Step S316, using a first preset algorithm to perform image enhancement processing on the text areas in the multiple video frames; using a second preset algorithm to perform image enhancement processing on the areas other than the text areas in the multiple video frames.
[0094] The above-mentioned image enhancement method first obtains multiple video frames of the video to be processed; for each video frame in the multiple video frames, the current video frame is input into a pre-trained text region segmentation model to obtain multiple candidate regions of the text region in the current video frame; then the mask corresponding to the candidate region with the minimum scale is extracted from the multiple candidate regions; the connected domain is determined from the mask corresponding to the candidate region with the minimum scale, and the connected domain is morphologically expanded; then the detection result of the video frame is determined according to the expanded connected domain, and the text region of the multiple video frames is determined according to the overlap degree of the detection results between adjacent video frames in the multiple video frames; finally, the first preset algorithm is used to perform image enhancement processing on the text region in the multiple video frames; and the second preset algorithm is used to perform image enhancement processing on the regions other than the text region in the multiple video frames. This method can quickly obtain the detection result of the video frame text, and at the same time, different algorithms are used to perform image enhancement on the text region and non-text region in the video, which can improve the visual effect of the image.
[0095] The present invention also provides another image enhancement method, which is implemented on the basis of the method described in the above embodiment; the method focuses on the process of further processing the detection result of the video frame (see step S706 to step S708 for details); Figure 7 As shown, the image enhancement method comprises the following steps:
[0096] Step S702, obtaining a video to be processed; the video to be processed includes multiple video frames.
[0097] Step S704, performing text detection on the above-mentioned multiple video frames to obtain detection results; the detection results are marked by text boxes, and the text boxes are used to mark the image area containing text in the video frames.
[0098] The detection result of each video frame usually includes one or more text boxes, which can have various shapes, such as rectangle, square, oval or irregular shape. The area within the text box is usually the image area containing text, but in the process of text detection, some background areas may be misjudged as text areas, and the background areas are marked with text boxes, so it is necessary to further determine whether the area within the text box contains text.
[0099] Step S706, calculating the number of pixels in the image area in each text box in the above detection result to obtain a calculation result.
[0100] Considering that the detection results may include the results of false detection, the image area in the text box may be a text area or a background area. Usually, the text in the text area in the video frame needs to have a certain readability, so the area of the text area is usually not too small, and the number of pixels in the text area is usually large. Based on this, if the number of pixels in the text box is small, it can be determined that the image area in the text box is not a text area. In specific implementation, the number of pixels can be obtained by multiplying the horizontal pixel points and the vertical pixel points of the image area in the text box, and whether the image area is a text area can be determined based on the number of pixels.
[0101] Step S708, based on the above calculation results, filter the text boxes and use the filter results as the final detection results.
[0102] Generally, it is possible to determine whether to delete the text box based on whether the number of pixels in the text box in the calculation result meets the preset conditions to obtain the final detection result. The final detection result can also be determined based on the length and width of the text box. For example, if the width of the text box is less than a preset width threshold, or the length of the text box is less than a preset length threshold, the text box can be deleted. In one implementation, the final detection result can be obtained specifically through the following steps 20-21:
[0103] Step 20: for each text box, determine whether the number of pixels in the image area of the current text box is less than a preset number threshold; if less than the number threshold, delete the current text box. The number threshold is usually an empirical value (eg, 20).
[0104] Step 21, taking the remaining text boxes as the final detection results.
[0105] For example, the quantity threshold may be 20, and text boxes whose number of pixels in the image area within the text box is less than 20 are deleted; and text boxes whose number of pixels is greater than or equal to 20 are retained as the final detection result.
[0106] Furthermore, after the text boxes are screened, in order to ensure that the text boxes in the detection result are not cut off from the real text area, the scale of the remaining text boxes can be expanded according to a preset percentage; the expanded boxes are used as the final detection result. For example, the scale of the remaining text boxes can be expanded outward by 5% to obtain the final detection result.
[0107] Step S710: determining text regions of the multiple video frames according to the overlapping degree of final detection results between adjacent video frames in the multiple video frames.
[0108] Step S712, using a first preset algorithm to perform image enhancement processing on the text areas in the multiple video frames; using a second preset algorithm to perform image enhancement processing on the areas other than the text areas in the multiple video frames.
[0109] The above-mentioned image enhancement method performs text detection on multiple video frames in the processed video to obtain a detection result, which is marked by a text box; then calculates the number of pixels in the image area of each text box in the detection result, and screens the text box according to the number of pixels to determine the final detection result, and then determines the text area of the multiple video frames according to the overlap degree of the final detection results between adjacent video frames in the multiple video frames, and finally uses the first preset algorithm to perform image enhancement processing on the text area in the multiple video frames; and uses the second preset algorithm to perform image enhancement processing on the area other than the text area in the multiple video frames. This method can obtain more accurate video frame detection results, which is conducive to the subsequent determination of the text area in the video frame.
[0110] The present invention also provides another image enhancement method, which is implemented on the basis of the method described in the above embodiment; the method focuses on describing the specific process of screening the detection results of the video frame (see steps S806 to S814 for details), and the way of image enhancement for the text area and the non-text area (see steps S816 to S818 for details); Figure 8 As shown, the image enhancement method comprises the following steps:
[0111] Step S802, obtaining a video to be processed; the video to be processed includes a plurality of video frames.
[0112] Step S804, performing text detection on the above-mentioned multiple video frames to obtain detection results; the detection results are marked by text boxes, and the text boxes are used to mark the image areas containing texts in the video frames.
[0113] Step S806: taking the first video frame among the multiple video frames as the current video frame.
[0114] Step S808 , calculating an IOU (Intersection over Union) value of a text box between the current video frame and a video frame next to the current video frame.
[0115] The IOU is usually a standard for accurately detecting corresponding objects in a specific data set. The IOU value can measure the correlation between two data (which can also be understood as overlap). The higher the correlation, the higher the value. The calculation formula of the IOU value can be Among them, A represents the text box of the current video frame, B represents the text box of the next video frame, and S A∩B represents the area where A and B intersect, S A∪B represents the area of the union of A and B.
[0116] In specific implementation, the detection result of the current video frame usually includes one or more text boxes. When the current video frame includes multiple text boxes, the first text box of the current video frame can be determined as the current text box, and the IOU value of the current text box and the text box at the corresponding position in the next video frame can be calculated. Then, the second text box in the video frame is determined as the current text box, and the step of calculating the IOU is continued until the current text box is the last text box of the current video, and finally the IOU values corresponding to all text boxes between the current video frame and the next video frame are obtained.
[0117] Step S810: determining a text area of a next video frame of the current video frame according to the calculated IOU value.
[0118] Calculate the IOU values corresponding to all text boxes between the current video frame and the next video frame. If the IOU value is greater than or equal to the specified threshold, the area in the text box in the next video frame is determined as the text area; if it is less than the specified threshold, the position of the text box corresponding to the text box in the next video frame is predicted by the text box in the current video frame, and the area in the text box in the prediction result is determined as the text area. Specifically, the text area of the next video frame of the current video frame can be determined through steps 30-32:
[0119] Step 30, determine whether the IOU value of the text box between the current video frame and the next video frame of the current video frame is greater than or equal to a preset overlap threshold; if it is greater than or equal to the overlap threshold, execute step 31; if it is less than the overlap threshold, execute step 32. The overlap threshold can be set to an empirical value or according to the detection requirements, for example, the overlap threshold is set to 0.5.
[0120] Step 31: Use the image area corresponding to the text box of the next video frame as the text area of the next video frame.
[0121] When the IOU value is greater than or equal to a preset overlap threshold, it can be determined that the position of the text box of the next video frame of the current video frame is correct, the text box is retained, and the picture area within the text box is determined as the text area.
[0122] Step 32, determining whether the detection result of the current video frame contains a text box, and determining the text area of the next video frame of the current video frame according to the determination result.
[0123] When the IOU value is less than the overlap threshold, there are usually three possible situations: one is that the current video frame contains a text box, but the next video frame does not contain a text box; the second is that the current video frame contains a text box, but the next video frame does not detect the text box (also known as missed detection); the third is that the current video frame does not contain a text box, but the next video frame contains a text box. Based on the above three situations, the text area of the next video frame of the current video frame can be determined by one or two of the following methods:
[0124] Method 1: If the detection result of the current video frame contains a text box, the position of the text box in the next video frame is predicted based on the text box in the current video frame, and the prediction result is used as the text area of the next video frame.
[0125] In specific implementation, a tracking algorithm can be used to predict the position of the text box in the next video frame. The tracking algorithm will predict the position of the current text box in the next video frame. If it is the first case, the tracking algorithm cannot obtain a prediction result, which can also be understood as predicting that there is no text box in the next video frame; if it is the second case, the tracking algorithm has a prediction result, and the position of the text box in the next video frame can be predicted, and the image area in the text box is used as the text area of the next video frame.
[0126] Method 2: If the detection result of the current video frame does not contain a text box, the image area corresponding to the text box of the next video frame is used as the text area of the next video frame. It can also be understood that for the third case, no processing is done on the border in the next video frame.
[0127] Step S812, determining whether the current video frame is the last video frame among the multiple video frames; if yes, executing step S816; if no, executing step S814.
[0128] Step S814: Use the next video frame of the current video frame as a new current video frame, and continue to execute step S808.
[0129] For example, the second video frame among multiple video frames is determined as the current video frame, the IOU value of the text box between the current video frame and the next video frame of the current video frame is calculated, and the text area of the next video frame is determined according to the IOU value, and then the third video frame is determined as the current video frame until the last video frame is determined as the current scale, thereby obtaining the text areas of multiple video frames.
[0130] Step S816, adjusting the algorithm parameters in the second preset algorithm to match the algorithm parameters with the text area; wherein the algorithm parameters include filtering parameters and / or sharpening parameters.
[0131] The algorithm parameters in the second preset algorithm may include only one of the filtering parameters or the sharpening parameters, or may include both the filtering parameters and the sharpening parameters. Normally, the image can be filtered by the filtering parameters, that is, the noise of the target image is suppressed while retaining the image detail features. The effect of the filtering process will also affect the effectiveness and reliability of subsequent image processing and analysis. The image can be sharpened by the sharpening parameters, that is, the contour of the image is compensated, the edge of the image and the grayscale jump part are enhanced, so that the image becomes clear. Image sharpening is also to highlight the edges, contours of objects on the image, or the features of certain linear target elements.
[0132] Step S818: determine the second preset algorithm after parameter adjustment as the first preset algorithm.
[0133] Step S820, using a first preset algorithm to perform image enhancement processing on the text areas in the multiple video frames; using a second preset algorithm to perform image enhancement processing on the areas other than the text areas in the multiple video frames.
[0134] In this method, the first preset algorithm and the second preset algorithm are only different in algorithm parameters, but the principle of image enhancement is the same, so that the text area and the non-text area can be filtered or sharpened to different degrees to enhance the subjective effect of the image. At the same time, since the gradient at the junction of the text edge and the background in the image is large (a large gradient usually refers to high-frequency information), when filtering the text area separately, the gradient can be appropriately reduced to reduce the high-frequency information in the image, which is beneficial to the transmission and storage of the image.
[0135] The above-mentioned image enhancement method first performs text detection on multiple video frames in the processed video to obtain the detection results; then determines the text area of the video frame according to the IOU value of the text box between adjacent video frames, and then adjusts the algorithm parameters in the second preset algorithm to match the algorithm parameters with the text area, and determines the second preset algorithm after the parameter adjustment as the first preset algorithm, and finally uses the first preset algorithm to perform image enhancement processing on the text area in multiple video frames; and uses the second preset algorithm to perform image enhancement processing on the areas other than the text area in multiple video frames. This method can enhance the text area and non-text area in the video to different degrees to improve the user's subjective experience.
[0136] Corresponding to the above method embodiment, the embodiment of the present invention provides an image enhancement device, such as Fig. 9 As shown, the device comprises:
[0137] The video acquisition module 90 is used to acquire a video to be processed; the video to be processed includes a plurality of video frames;
[0138] A text detection module 91 is used to perform text detection on the above-mentioned multiple video frames to obtain a detection result;
[0139] A text determination module 92, configured to determine text regions of the plurality of video frames according to the degree of overlap of detection results between adjacent video frames in the plurality of video frames;
[0140] The image enhancement module 93 is used to use a first preset algorithm to perform image enhancement processing on the text area in multiple video frames; and use a second preset algorithm to perform image enhancement processing on the area other than the text area in multiple video frames; wherein the first preset algorithm matches the text area.
[0141] The above-mentioned image enhancement device first obtains the video to be processed; then performs text detection on multiple video frames in the video to be processed to obtain detection results; then determines the text areas of the multiple video frames according to the overlap degree of the detection results between adjacent video frames in the multiple video frames; finally, uses a first preset algorithm matching the text area to perform image enhancement processing on the text area in the multiple video frames, and uses a second preset algorithm to perform image enhancement processing on the areas other than the text area in the multiple video frames. This method uses different algorithms to perform different degrees of image enhancement on the text area and non-text area in the video frame, avoids excessive enhancement of the text area, and is conducive to improving the overall visual effect of the video image.
[0142] Furthermore, the above-mentioned text detection module 91 is used to: for each video frame in a plurality of video frames, perform text region segmentation processing on the current video frame to obtain a plurality of candidate regions of the text region in the video frame; wherein the scales of the plurality of candidate regions are various; perform morphological processing on the plurality of candidate regions to obtain detection results of the video frame.
[0143] Furthermore, the above-mentioned text detection module 91 is also used to: input the current video frame into a pre-trained text area segmentation model to obtain multiple candidate areas of the text area in the current video frame; wherein the text area segmentation model is trained in the following manner: determine the target training image based on a preset training set; wherein the target training image carries text area identifiers of multiple scales; input the target training image into the initial network model, and output pixel feature data of the target training image; the pixel feature data contains the feature value of each pixel in the target training image; based on the pixel feature data and the text area identifier, calculate the model loss value; adjust the network parameters in the initial network model according to the model loss value, and continue to execute the step of determining the target training image based on the preset training set, until the model loss value meets the preset specified value or the number of target training images input into the initial network model meets the preset number, and obtain the text area segmentation model.
[0144] The above-mentioned text area identification includes: a mask corresponding to the text area in the target training image; wherein the scale of the maximum-scale mask is the same as the scale of the text area in the target training image; the scales of the masks other than the maximum-scale mask are obtained by reducing the scale of the previous-scale mask by a preset percentage.
[0145] Furthermore, among the above-mentioned multiple candidate areas, candidate areas of the same scale are identified in the same mask; the above-mentioned text detection module 91 is also used to: extract the mask corresponding to the candidate area of the smallest scale from the multiple candidate areas; determine the connected domain from the mask corresponding to the candidate area of the smallest scale; use a preset template to perform a morphological dilation operation on the connected domain; and determine the detection result of the video frame based on the dilated connected domain.
[0146] Furthermore, the above-mentioned text detection module 91 is also used to: determine the previous scale of the minimum scale as the current scale; wherein the masks corresponding to the candidate areas of different scales are arranged in order from large to small according to the scales of the candidate areas; perform bitwise operation on the expanded connected domain and the pixel values of the corresponding pixels in the mask of the current scale to obtain the operation result; continue to perform morphological expansion operation on the expanded connected domain until the operation result converges; determine the previous scale of the current scale as the new current scale, and continue to perform bitwise operation on the pixel values of the corresponding pixels in the mask of the expanded connected domain and the current scale to obtain the operation result, until the current scale is the maximum scale and the operation result converges; determine the detection result of the video frame according to the connected domain after the last expansion.
[0147] The above detection results are marked by a text box, which is used to identify the image area containing text in the video frame; after the above text detection module 91, the device also includes a text processing module, which is used to: calculate the number of pixels in the image area in each text box in the detection result to obtain the calculation result; according to the calculation result, filter the text box and use the filtered result as the final detection result.
[0148] Furthermore, the above-mentioned text processing module is also used to: for each text box, determine whether the number of pixels in the image area within the current text box is less than a preset number threshold; if it is less than the number threshold, delete the current text box; and use the remaining text boxes as the final detection results.
[0149] Furthermore, the text processing module is further configured to: expand the scale of the remaining text boxes according to a preset percentage; and use the expanded text boxes as the final detection result.
[0150] Furthermore, the above detection result is identified by a text box, which is used to identify the image area containing text in the video frame; the above text determination module 92 is used to: take the first video frame among the multiple video frames as the current video frame, and calculate the IOU value of the text box between the current video frame and the next video frame of the current video frame; determine the text area of the next video frame of the current video frame according to the calculated IOU value; take the next video frame of the current video frame as the new current video frame, and continue to execute the step of calculating the IOU value of the text box between the current video frame and the next video frame of the current video frame, until the text area of the last video frame among the multiple video frames is determined.
[0151] Furthermore, the above-mentioned text determination module 92 is also used to: determine whether the IOU value of the text box between the current video frame and the next video frame of the current video frame is greater than or equal to a preset overlap threshold; if it is greater than or equal to the overlap threshold, use the image area corresponding to the text box of the next video frame as the text area of the next video frame; if it is less than the overlap threshold, determine whether the detection result of the current video frame contains a text box, and determine the text area of the next video frame of the current video frame according to the judgment result.
[0152] Furthermore, the above-mentioned text determination module 92 is also used to: if the detection result of the current video frame contains a text box, predict the position of the text box in the next video frame according to the text box in the current video frame, and use the predicted result as the text area of the next video frame; if the detection result of the current video frame does not contain a text box, use the image area corresponding to the text box of the next video frame as the text area of the next video frame.
[0153] Furthermore, the device also includes a parameter adjustment module, which is used to: adjust the algorithm parameters in the second preset algorithm to match the algorithm parameters with the text area; determine the second preset algorithm after parameter adjustment as the first preset algorithm; the algorithm parameters include filtering parameters and / or sharpening parameters.
[0154] The image enhancement device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0155] The embodiment of the present invention also provides a server for running the image enhancement method; see Fig.10 As shown, the server includes a processor 101 and a memory 100. The memory 100 stores machine executable instructions that can be executed by the processor 101. The processor 101 executes the machine executable instructions to implement the above-mentioned image enhancement method.
[0156] Further, Fig.10 The server shown also includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .
[0157] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0158] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 101 or the instruction in the form of software. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and completes the steps of the method of the above embodiment in combination with its hardware.
[0159] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned image enhancement method. The specific implementation can be found in the method embodiment, which will not be repeated here.
[0160] The computer program products of the image enhancement method, device and server provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.
[0161] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. An image enhancement method, characterized in that: The method comprises: Acquire a video to be processed; the video to be processed includes multiple video frames; Performing text detection on the multiple video frames to obtain detection results; Determining text regions of the plurality of video frames according to an overlap degree of detection results between adjacent video frames in the plurality of video frames; Using a first preset algorithm, performing image enhancement processing on the text area in the multiple video frames; using a second preset algorithm, performing image enhancement processing on the area other than the text area in the multiple video frames; wherein the first preset algorithm matches the text area; Before the step of using the first preset algorithm to perform image enhancement processing on the text area in the multiple video frames, the method further includes: adjusting the algorithm parameters in the second preset algorithm to match the algorithm parameters with the text area; wherein the algorithm parameters include filtering parameters and / or sharpening parameters; determining the second preset algorithm after parameter adjustment as the first preset algorithm; wherein the sharpening degree of the first preset algorithm is less than that of the second preset algorithm, and the filtering degree of the first preset algorithm is greater than that of the second preset algorithm; The step of performing text detection on the multiple video frames to obtain detection results includes: For each video frame in the multiple video frames, perform text region segmentation processing on the current video frame to obtain multiple candidate regions of the text region in the video frame; wherein the multiple candidate regions have multiple scales; Performing morphological processing on the multiple candidate regions to obtain a detection result of the video frame; Among the multiple candidate regions, candidate regions of the same scale are identified in the same mask; The step of performing morphological processing on the multiple candidate regions comprises: Extracting a mask corresponding to a candidate region of minimum scale from the multiple candidate regions; Determine a connected domain from the mask corresponding to the candidate region of the minimum scale; Using a preset template, performing a morphological dilation operation on the connected domain; The detection result of the video frame is determined according to the expanded connected domain.
2. The method according to claim 1, characterized in that The step of performing text region segmentation processing on the current video frame includes: inputting the current video frame into a pre-trained text region segmentation model to obtain multiple candidate regions of the text region in the current video frame; The text region segmentation model is trained in the following way: Determining a target training image based on a preset training set; wherein the target training image carries text region identifiers of multiple scales; Inputting the target training image into the initial network model, and outputting pixel feature data of the target training image; the pixel feature data includes a feature value of each pixel in the target training image; Calculating a model loss value based on the pixel feature data and the text area identifier; The network parameters in the initial network model are adjusted according to the model loss value, and the step of determining the target training image based on the preset training set is continued until the model loss value meets the preset specified value or the number of target training images input into the initial network model meets the preset number, thereby obtaining a text area segmentation model.
3. The method according to claim 2, characterized in that The text region identification includes: a mask corresponding to the text region in the target training image; The scale of the maximum-scale mask is the same as the scale of the text area in the target training image; the scales of the masks other than the maximum-scale mask are obtained by reducing the scale of the previous-scale mask by a preset percentage.
4. The method according to claim 1, characterized in that: The step of determining the detection result of the video frame according to the expanded connected domain includes: The previous scale of the minimum scale is determined as the current scale; wherein the masks corresponding to the candidate regions of different scales are arranged in descending order according to the scales of the candidate regions; Performing a bitwise operation on the expanded connected domain and the pixel values of corresponding pixels in the mask of the current scale to obtain an operation result; Continue to perform morphological dilation operation on the dilated connected domain until the operation result converges; Determine the previous scale of the current scale as the new current scale, and continue to perform bitwise operation on the expanded connected domain and the pixel values of the corresponding pixels in the mask of the current scale to obtain the operation result, until the current scale is the maximum scale and the operation result converges; The detection result of the video frame is determined according to the connected domain after the last dilation.
5. The method according to claim 1, characterized in that The detection result is marked by a text box, and the text box is used to mark the image area containing text in the video frame; After the step of performing text detection on the multiple video frames to obtain detection results, the method further includes: Calculate the number of pixels in the image area of each text box in the detection result to obtain a calculation result; According to the calculation result, the text box is filtered and the filtering result is used as the final detection result.
6. The method according to claim 5, characterized in that The step of filtering the text box according to the calculation result and using the filtering result as the final detection result includes: For each of the text boxes, determining whether the number of pixels in the image area within the current text box is less than a preset number threshold; if it is less than the number threshold, deleting the current text box; The remaining text boxes are taken as the final detection results.
7. The method according to claim 6, characterized in that After the step of taking the remaining text boxes as the final detection result, the method further includes: Expand the size of the remaining text box according to a preset percentage; The expanded text box is used as the final detection result.
8. The method according to claim 1, characterized in that The detection result is marked by a text box, and the text box is used to mark the image area containing text in the video frame; The step of determining text areas of the plurality of video frames according to the overlapping degree of detection results between adjacent video frames in the plurality of video frames comprises: Taking a first video frame among the multiple video frames as a current video frame, and calculating an IOU value of a text box between the current video frame and a video frame next to the current video frame; Determine a text area of a next video frame of the current video frame according to the calculated IOU value; The next video frame of the current video frame is used as a new current video frame, and the step of calculating the IOU value of the text box between the current video frame and the next video frame of the current video frame is continued until the text area of the last video frame among the multiple video frames is determined.
9. The method according to claim 8, characterized in that The step of determining the text area of the next video frame of the current video frame according to the calculated IOU value includes: Determine whether an IOU value of a text box between the current video frame and a next video frame of the current video frame is greater than or equal to a preset overlap threshold; If it is greater than or equal to the overlap threshold, taking the image area corresponding to the text box of the next video frame as the text area of the next video frame; If it is less than the overlap threshold, it is determined whether the detection result of the current video frame contains a text box, and the text area of the next video frame of the current video frame is determined according to the determination result.
10. The method according to claim 9, characterized in that The step of determining the text area of the next video frame of the current video frame according to the judgment result includes: If the detection result of the current video frame includes a text box, predicting the position of the text box in the next video frame according to the text box in the current video frame, and using the prediction result as the text area of the next video frame; If the detection result of the current video frame does not include a text box, the image area corresponding to the text box of the next video frame is used as the text area of the next video frame.
11. An image enhancement device, characterized in that: The device comprises: A video acquisition module, used to acquire a video to be processed; the video to be processed includes a plurality of video frames; A text detection module, used to perform text detection on the multiple video frames to obtain a detection result; A text determination module, configured to determine text regions of the plurality of video frames according to an overlap degree of detection results between adjacent video frames in the plurality of video frames; An image enhancement module, configured to perform image enhancement processing on the text areas in the plurality of video frames using a first preset algorithm; Using a second preset algorithm, performing image enhancement processing on the regions other than the text region in the plurality of video frames; wherein the first preset algorithm matches the text region; The device further comprises a parameter adjustment module, configured to adjust algorithm parameters in the second preset algorithm so that the algorithm parameters match the text area; wherein the algorithm parameters include filtering parameters and / or sharpening parameters; and determining the second preset algorithm after parameter adjustment as the first preset algorithm; wherein the sharpening degree of the first preset algorithm is less than that of the second preset algorithm, and the filtering degree of the first preset algorithm is greater than that of the second preset algorithm; A text detection module is used to perform text region segmentation processing on a current video frame for each of the multiple video frames to obtain multiple candidate regions of the text region in the video frame; wherein the multiple candidate regions have multiple scales; and perform morphological processing on the multiple candidate regions to obtain a detection result of the video frame; Among the multiple candidate areas, candidate areas of the same scale are identified in the same mask; a text detection module is used to extract a mask corresponding to a candidate area of the smallest scale from the multiple candidate areas; determine a connected domain from the mask corresponding to the candidate area of the smallest scale; use a preset template to perform a morphological dilation operation on the connected domain; and determine a detection result of the video frame based on the dilated connected domain.
12. A server, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the image enhancement method according to any one of claims 1 to 10.
13. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the image enhancement method according to any one of claims 1 to 10.
