Text detection method, brightness adjustment method, and device

Abstract

The present disclosure relates to the technical field of computer vision. Provided are a text detection method, a brightness adjustment method, and a device, which are used for enabling real-time detection of text. The method comprises: determining respective text candidate regions in a previous image frame and a current image frame, wherein the position of the text candidate region in the previous image frame and the position of the text candidate region in the current image frame are in one-to-one correspondence; determining similarity between corresponding text candidate regions among the respective text candidate regions of the previous image frame and the current image frame; and, according to the similarity between the corresponding text candidate regions, determining whether or not to trigger text detection for the text candidate region of the current image frame.

Description

Text detection method, brightness adjustment method and device Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a text detection method, a brightness adjustment method, and a device. Background Art

[0002] Since videos are played in real time on terminal devices such as TVs and tablets, the speed of text detection in videos must be relatively high. Existing technologies usually perform text detection directly on videos frame by frame, but this detection method is time-consuming and requires a high level of computing power.

[0003] Summary of the Invention

[0004] The present disclosure provides a text detection method, a brightness adjustment method and a device, which are used to cooperate with the real-time playback characteristics of videos, improve the speed of text detection in a manner with low computing power requirements, and realize real-time text detection.

[0005] In a first aspect, an embodiment of the present disclosure provides a text detection method, the method comprising:

[0006] Determine the text to-be-detected regions of the previous image frame and the current image frame, wherein the positions of the text to-be-detected regions between the previous image frame and the current image frame are in one-to-one correspondence;

[0007] Determining the similarity between the text to-be-inspected regions having corresponding relationships in the previous image frame and the current image frame;

[0008] Whether to trigger text detection on the text-to-be-detected area of ​​the current image frame is determined according to the similarity between the corresponding text-to-be-detected areas.

[0009] In a second aspect, an embodiment of the present disclosure provides a brightness adjustment method, the method comprising:

[0010] Performing text detection according to the text detection method provided in the first aspect to determine the text area of ​​the image frame;

[0011] The brightness of the text area of ​​the image frame is adjusted according to the brightness of the non-text area of ​​the image frame.

[0012] In a third aspect, an embodiment of the present disclosure further provides a display device, the display device comprising a display screen and a control circuit;

[0013] The display screen is used to display content;

[0014] The control circuit includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0015] Determine the text to-be-detected regions of the previous image frame and the current image frame, wherein the positions of the text to-be-detected regions between the previous image frame and the current image frame are in one-to-one correspondence;

[0016] Determining the similarity between the text to-be-inspected regions having corresponding relationships in the previous image frame and the current image frame;

[0017] Whether to trigger text detection on the text-to-be-detected area of ​​the current image frame is determined according to the similarity between the corresponding text-to-be-detected areas.

[0018] In a fourth aspect, an embodiment of the present disclosure further provides a display device, the display device comprising a display screen and a control circuit;

[0019] The display screen is used to display content;

[0020] The control circuit includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0021] Performing text detection according to the text detection method provided in the first aspect to determine the text area of ​​the image frame;

[0022] The brightness of the text area of ​​the image frame is adjusted according to the brightness of the non-text area of ​​the image frame.

[0023] In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, comprising a processor and a memory, wherein the memory is configured to store a program executable by the processor, and the processor is configured to read the program in the memory and perform the following steps:

[0024] Determine the text to-be-detected regions of the previous image frame and the current image frame, wherein the positions of the text to-be-detected regions between the previous image frame and the current image frame are in one-to-one correspondence;

[0025] Determining the similarity between the text to-be-inspected regions having corresponding relationships in the previous image frame and the current image frame;

[0026] Whether to trigger text detection on the text-to-be-detected area of ​​the current image frame is determined according to the similarity between the corresponding text-to-be-detected areas.

[0027] In a sixth aspect, an embodiment of the present disclosure further provides an electronic device, comprising a processor and a memory, wherein the memory is configured to store a program executable by the processor, and the processor is configured to read the program in the memory and perform the following steps:

[0028] Performing text detection according to the text detection method provided in the first aspect to determine the text area of ​​the image frame;

[0029] The brightness of the text area of ​​the image frame is adjusted according to the brightness of the non-text area of ​​the image frame.

[0030] In a seventh aspect, an embodiment of the present disclosure further provides a subtitle detection device, the device comprising:

[0031] An area determination module is used to determine the text to-be-detected areas of the previous image frame and the current image frame, wherein the positions of the text to-be-detected areas between the previous image frame and the current image frame are in one-to-one correspondence;

[0032] A region comparison module is used to determine the similarity between the text to be inspected regions having corresponding relationships in the previous image frame and the current image frame;

[0033] The judgment trigger module is used to determine whether to trigger text detection on the text to be detected area of ​​the current image frame according to the similarity between the text to be detected areas with corresponding relationships.

[0034] In an eighth aspect, an embodiment of the present disclosure further provides a brightness adjustment device, the device comprising:

[0035] a text determination module, configured to perform text detection according to the text detection method provided in the first aspect, and determine the text area of ​​the image frame;

[0036] The brightness adjustment module is used to adjust the brightness of the text area of ​​the image frame according to the brightness of the non-text area of ​​the image frame.

[0037] In a ninth aspect, an embodiment of the present disclosure further provides a computer storage medium on which a computer program is stored, which, when executed by a processor, is used to implement the steps of the method described in any one of the first or second aspects above.

[0038] In a tenth aspect, the present disclosure provides a computer program product, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method described in any one of the first aspect or the second aspect.

[0039] These and other aspects of the present disclosure will become more readily apparent from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] FIG1 is a flowchart of a specific implementation of a text detection method provided by an embodiment of the present disclosure;

[0042] 2A-2B are schematic diagrams of determining text detection areas in adjacent image frames according to an embodiment of the present disclosure;

[0043] FIG3 is a schematic diagram of a preset area position of a display screen provided by an embodiment of the present disclosure;

[0044] FIG4 is a flowchart of text detection using a frame skipping strategy according to an embodiment of the present disclosure;

[0045] FIG5 is a schematic diagram of a display screen partition provided by an embodiment of the present disclosure;

[0046] FIG6 is a schematic diagram of a first text detection multi-task network structure provided by an embodiment of the present disclosure;

[0047] FIG7 is a schematic diagram of a second text detection multi-task network structure provided by an embodiment of the present disclosure;

[0048] FIG8 is a schematic diagram of a third text detection multi-task network structure provided by an embodiment of the present disclosure;

[0049] FIG9 is a flowchart illustrating a specific implementation of a brightness adjustment method provided by an embodiment of the present disclosure;

[0050] FIG10 is a schematic diagram of a display device provided by an embodiment of the present disclosure;

[0051] FIG11 is a schematic diagram of another display device provided by an embodiment of the present disclosure;

[0052] FIG12 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure;

[0053] FIG13 is a schematic diagram of another electronic device provided by an embodiment of the present disclosure;

[0054] FIG14 is a schematic diagram of a text detection device provided by an embodiment of the present disclosure;

[0055] FIG15 is a schematic diagram of a brightness adjustment device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the present disclosure will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only a portion of the embodiments of the present disclosure, rather than all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure without creative effort are intended to fall within the scope of protection of the present disclosure.

[0057] In the embodiments of the present disclosure, the term "and / or" describes the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0058] The application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Persons skilled in the art will appreciate that, as new application scenarios emerge, the technical solutions provided by the embodiments of the present disclosure will also be applicable to similar technical problems. In the description of the present disclosure, unless otherwise specified, "multiple" means two or more.

[0059] Before introducing the text detection method and the brightness adjustment method provided by the embodiments of the present disclosure, for ease of understanding, the technical background of the embodiments of the present disclosure is first introduced in detail below.

[0060] Since videos are played in real time on terminal devices such as TVs and tablets, the speed of text detection in videos must be relatively high. Existing technologies usually perform text detection on videos frame by frame, but this detection method is time-consuming and requires a high level of computing power.

[0061] Taking the text in the video, including subtitles, as an example, subtitles refer to non-visual content such as dialogues in film and television works displayed in text form, and also generally refer to text processed in the post-production of film and television works. Commentary text and other types of text appear on terminal devices such as movie screens, televisions, and tablets. With the rapid development of digital media technology, subtitles, as an important means of information transmission, are widely used in scenes such as movies, TV series, and online videos. Since videos are played in real time on terminal devices such as televisions and tablets, the speed of subtitle detection is required to be high. In order to ensure the real-time performance of subtitle detection, the existing technology usually performs subtitle detection directly frame by frame or combines it with audio for detection. However, this detection method is time-consuming and requires a high level of computing power.

[0062] In order to solve the above technical problems, the embodiments of the present disclosure provide a text detection method. The core idea is to use the similarity comparison of the text to be detected areas in corresponding positions in adjacent image frames to determine whether to trigger text detection of the text to be detected area of ​​the current image frame. The similarity comparison result can be used to avoid triggering text detection of the text to be detected area of ​​the current image frame. Through frame skipping detection, multiple text detections of video pictures with the same text or no text can be effectively avoided, thereby improving the efficiency and speed of text detection. Combined with the real-time playback characteristics of the video, the effect of real-time text detection can be achieved.

[0063] It should be noted that the text detection method provided in this embodiment can be applied to smart display devices, such as televisions, tablets, mobile phones, computers, all-in-one conference machines, smart home display devices and other terminal playback devices, to achieve real-time detection of text in video images.

[0064] As shown in FIG1 , the specific implementation process of a text detection method provided in this embodiment is as follows:

[0065] Step 100: Determine the text to-be-detected regions of the previous image frame and the current image frame, wherein the positions of the text to-be-detected regions of the previous image frame and the current image frame are in one-to-one correspondence;

[0066] It should be noted that, in this embodiment, there may be one or more text areas to be inspected. When there is only one text area to be inspected, the position of the text area to be inspected in the previous image frame is the same as the position of the text area to be inspected in the current image frame, that is, the position of the text area to be inspected in the previous image frame corresponds to the position of the text area to be inspected in the current image frame. When there are multiple text areas to be inspected, the positions of the text areas to be inspected in the previous image frame and the current image frame are one-to-one corresponding. For example, if the text area to be inspected includes a first text area to be inspected and a second text area to be inspected, then the position of the first text area to be inspected in the previous image frame is the same as the position of the first text area to be inspected in the current image frame, and the position of the second text area to be inspected in the previous image frame is the same as the position of the second text area to be inspected in the current image frame.

[0067] During implementation, if the current image frame is an initial image frame and there is no previous image frame, text detection is performed directly on the initial image frame.

[0068] Before determining the text detection areas of the previous image frame and the current image frame, this embodiment obtains the text detection result of the previous image frame, where the text detection result includes whether text is detected and the text area is determined, or whether text is not detected. The text detection area of ​​the current image frame is determined based on the text detection result of the previous image frame.

[0069] Optionally, the text detection areas of the previous image frame and the current image frame are determined by the following steps:

[0070] According to the text detection result of the previous image frame, the text detection areas of the previous image frame and the current image frame are determined.

[0071] In implementation, based on the text detection result of the previous image frame, any one or more of the following determination situations may be included:

[0072] Case 1) In response to the text detection result of the previous image frame being that text is detected, determining the text detection regions of the previous image frame and the current image frame according to the text region detected in the previous image frame;

[0073] In implementation, the text region detected in the previous image frame is used as the text detection region. Simultaneously, based on the position of the text region in the previous image frame, the region in the current image frame at that position is used as the text detection region for the current image frame. The text detection result refers to the position of the detected text region, which can be represented by the coordinates of the four corners of the output detection box.

[0074] As shown in Figures 2A-2B, this embodiment provides a schematic diagram for determining the text to be inspected areas of each adjacent image frame. If the previous image frame detects one text area, the detected text area is used as the text to be inspected area of ​​the previous image frame, and the text to be inspected area at that position in the current image frame is determined based on the position of the detected text area. If the previous image frame detects two text areas, the first text area and the second text area detected in the previous image frame are used as the text to be inspected areas of the previous image frame, and based on the positions of the first text area and the second text area, the area at the same position in the current image frame is used as the text to be inspected area of ​​the current image frame.

[0075] Case 2) In response to the text detection result of the previous image frame being that no text is detected, the text detection areas of the previous image frame and the current image frame are determined according to preset areas.

[0076] During implementation, the position of the preset area can be determined based on the frequency of text appearance in the image frame. For example, text usually appears in the four directions of up, down, left, and right of the image frame and is close to the boundary. The preset area can be one or more. The initial preset area can also be determined based on the resolution of the video, where the video includes multiple image frames, and the multiple image frames include the previous image frame and the current image frame.

[0077] Optionally, after determining the initial preset area, this embodiment may also update the preset area according to the text area detected by the text detection result.

[0078] During implementation, the initial preset area can be updated. During the file detection process, the updated preset area can be dynamically updated again according to the text detection results to adapt to the size of the text area detected by the latest text detection results, that is, the size of the detection box.

[0079] In some embodiments, this embodiment may also update the preset area in any of the following ways:

[0080] Method a) in response to performing text detection on a preset area of ​​a previous image frame and detecting text in the preset area for the first time, updating the preset area according to the size of the first detected text area and a preset magnification;

[0081] During implementation, when text is detected in a preset area for the first time, the adjustment accuracy of the preset area can be set by setting a preset magnification, and the size of the preset area can be adjusted by multiplying the size of the actually detected text area by the preset magnification.

[0082] Mode b) In response to the text detection result of the previous image frame, in which the size of the text area detected is larger than the preset area, the preset area is updated.

[0083] Optionally, if the height of the text area detected by the text detection result is greater than the preset area, the height of the preset area is updated; if the width of the text area detected by the text detection result is greater than the preset area, the width of the preset area is updated. The purpose of the update is to update the current preset area size to a size closer to the size of the detected actual text.

[0084] During implementation, the preset area may be updated according to the size of the text area detected in the current image frame, and the position of the preset area may be updated to the position of the text area detected in the current detection result within a certain error range.

[0085] For example, the resolution of the displayed video is frame_h×frame_w, where frame_h represents the height of the image frame and frame_w represents the width of the image frame. The video includes multiple image frames. As shown in FIG3 , a schematic diagram of the position of the preset area of ​​the display screen is provided. The preset area is initialized to the top 1 / 5 of the display screen, the bottom 1 / 5 of the display screen, the left 1 / 8 of the display screen, and the right 1 / 8 of the display screen. The area of ​​the display screen can be evenly divided in the X-axis and Y-axis directions. For example, the X-axis direction is divided into 8 equal sub-areas from left to right. In this case, the leftmost sub-area and the rightmost sub-area are used as the preset areas. Similarly, the Y-axis direction is divided into 5 equal sub-areas from top to bottom. In this case, the top and bottom sub-areas are used as the preset areas. The preset areas at different positions may or may not overlap, and this embodiment does not impose any restrictions on this.

[0086] Optionally, the preset area may be determined based on the probability of text appearance in the image and the text form, which is not limited in this embodiment.

[0087] In one example, when text is first detected in a preset area, the preset area is updated by multiplying the size of the detected text area by a preset magnification. The updating formula is as follows: area_h = min(0.2×frame_h, merge(det_box_h)×s1); area_w = min(0.8×frame_w, merge(det_box_w)×s2); Formula (1);

[0088] In formula (1), s1 and s2 represent preset magnifications, area_h represents the height of the updated text area, area_w represents the width of the updated text area, frame_h represents the height of the image frame, frame_w represents the width of the image frame, det_box_h represents the height of the detection frame, which also represents the height of the detected text area, det_box_w represents the width of the detection frame, which also represents the width of the detected text area. Usually, there are multiple detection frames for the same text area, and the height and width of the final text area are determined by merging multiple detection frames in terms of area. That is, merge(det_box_h) represents the height of the merged text area, and merge(det_box_w) represents the width of the merged text area.

[0089] It should be noted that, when performing text detection on an input image frame, this embodiment determines the position and size information of the detected text area by outputting a detection frame.

[0090] After the initial preset area is updated once, when text detection is performed on the current image frame and the width and / or height of the detected text area is larger than the current preset area, the current preset area is updated.

[0091] In some embodiments, the frame rate of most videos is within 30 fps, but some high-frame-rate movies can reach 120 fps. To enhance the real-time performance of text detection at high frame rates (frame rates greater than 30 fps), this embodiment further provides a text detection solution that enhances the frame skipping strategy. The specific implementation process is as follows:

[0092] Solution 1) In response to a frame rate of a video frame being greater than a first frame rate threshold, a preset area to be inspected is determined from the text area to be inspected, and in two image frames separated by one or more frames, the similarity between the preset areas to be inspected with a corresponding relationship is determined, and based on the similarity between the preset areas to be inspected with a corresponding relationship, it is determined whether to trigger text detection in the preset areas to be inspected.

[0093] It should be noted that the positions of the preset inspection areas in the two image frames are one-to-one corresponding. The position of the preset inspection area in one image frame is the same as the position of the preset inspection area in the other image frame. The preset inspection area can be one or more, and this embodiment does not impose too many restrictions on this. The position and size of the preset inspection area in different image frames are the same.

[0094] In implementation, when the frame rate of the video frame is greater than the first frame rate threshold, such as the frame rate of the video frame is greater than 30fps (frame per second), if the text to be inspected area includes multiple areas, such as one area above, below, left and right of the display screen, a total of 4 text to be inspected areas, then the preset areas to be inspected are screened out from the text to be inspected areas, and the probability (or frequency) of the text appearing in each text to be inspected area can be screened out, and areas with a low probability of text appearance can be screened out as preset areas to be inspected. For example, if the probability of text appearing on the left and right sides of the display screen is low, the two areas on the left and right sides are used as preset areas to be inspected. Since the probability of text appearing in the preset areas to be inspected is low, the detection speed can be improved by performing text detection on alternate frames to meet the real-time requirements.

[0095] Solution 2) In response to the frame rate of the video frame being greater than a second frame rate threshold, the similarity between the corresponding text areas to be inspected in two image frames separated by one or more frames is determined, and based on the similarity between the corresponding text areas to be inspected, it is determined whether to trigger text detection of the text area to be inspected in the current image frame.

[0096] It should be noted that the positions and sizes of the corresponding text regions to be inspected in different image frames are the same. There may be one or more text regions to be inspected in an image frame.

[0097] In this embodiment, the second frame rate threshold is greater than the first frame rate threshold.

[0098] During implementation, if the frame rate of the video frame is greater than the second frame rate threshold, such as greater than 60fps, text detection can be performed by frame skipping. That is, when the frame rate is greater than the second frame rate threshold, frame-by-frame judgment is no longer performed, and a similarity comparison is performed on the text-to-be-detected areas of two image frames separated by one or more frames, thereby triggering text detection of the current image frame, thereby improving the detection speed and meeting the real-time performance of text detection at high frame rates.

[0099] This embodiment is used as an example to ensure the real-time performance of subtitle detection and improve the detection speed. When the frame rate is greater than 30fps, since the probability of the display image appearing in the preset area to be detected is lower than the probability threshold, the preset area to be detected can be calculated every other frame, that is, the similarity of the preset area to be detected is calculated every other frame. Other strategies are the same as above. When the frame rate is greater than 60fps, one or more frames are skipped directly without frame-by-frame judgment. Other strategies are the same as above.

[0100] Step 101: Determine the similarity between the corresponding text to-be-inspected regions in the previous image frame and the current image frame;

[0101] Optionally, the similarity between the corresponding text regions to be inspected includes the similarity of the content between the corresponding text regions to be inspected. The similarity between the corresponding text regions to be inspected can be determined by taking screenshots of the text regions to be inspected and comparing image features between the two screenshots.

[0102] In some embodiments, this embodiment determines the similarity between the corresponding text regions to be inspected through the following steps:

[0103] Screenshot processing is performed on the text to-be-checked regions of the previous image frame and the current image frame to obtain corresponding screenshots, and the similarity between the corresponding screenshots is determined.

[0104] In practice, the text inspection regions between adjacent image frames are in one-to-one correspondence. The corresponding text inspection regions are located in the same position in each image frame, and are the same size. It should be noted that the location and size of the corresponding text inspection regions are identical, and this "identical" does not mean absolutely identical, but rather identical within a certain error range.

[0105] After obtaining the previous image frame and the current image frame, since it takes a long time to directly perform a similarity comparison on the entire frame, the image feature comparison time is reduced by taking screenshots. By taking screenshots of the text to be inspected areas of each of the previous image frame and the current image frame, the detection time is reduced by comparing the screenshots. If the text to be inspected areas include multiple areas, such as one above and one below the display screen, screenshots of the two text to be inspected areas in the previous image frame and the two text to be inspected areas in the current image frame are taken, and the similarity calculation is performed on the screenshot located above in the previous image frame and the screenshot located above in the current image frame, and the similarity calculation is performed on the screenshot located below in the previous image frame and the screenshot located below in the current image frame.

[0106] During implementation, in order to prevent the screenshot size from being too large, after taking screenshots of the text to-be-inspected areas of the previous image frame and the current image frame, this embodiment can also scale the screenshots proportionally to adjust the screenshot size within a preset range.

[0107] In some embodiments, you can scale the screenshot proportionally to adjust the size of the screenshot within a preset range by following the steps below:

[0108] When scaling a screenshot, ensure that the shortest dimension of the screenshot is greater than or equal to a first threshold;

[0109] In response to the longest dimension of the screenshot being greater than a second threshold after the screenshot is scaled, a middle area whose longest dimension is within the second threshold is captured and determined as the adjusted screenshot.

[0110] In implementation, the longest side is limited to a second threshold value, L, and the shortest side is limited to a first threshold value, S. The screenshot is scaled proportionally to ensure that the shortest side is no less than the first threshold value, S, and the longest side is no greater than the second threshold value, L. Furthermore, while maintaining the shortest side, scaling is performed proportionally. If the longest side of the scaled screenshot exceeds the second threshold value, L, the center area of ​​the scaled screenshot is captured, and the length of the captured center area is L.

[0111] It should be noted that larger images take longer to calculate similarity. To keep this time manageable, we took screenshots of the images and limited their size to avoid overly large screenshots. Furthermore, prioritizing the short side ensures text clarity and prevents excessive compression. Proportional scaling maintains the image's aspect ratio and prevents excessive text distortion.

[0112] Step 102: Determine whether to trigger text detection on the text-to-be-detected area of ​​the current image frame based on the similarity between the corresponding text-to-be-detected areas.

[0113] Optionally, this embodiment calculates the similarity between the text regions to be inspected by any one or more of the following methods:

[0114] Calculate the structural similarity (SSIM) of the text region to be inspected;

[0115] Calculate the Peak Signal to Noise Ratio (PSNR) of the text area to be inspected.

[0116] The above-mentioned method of calculating similarity is only an example. Other methods of calculating image-text similarity also fall within the scope of protection of the present disclosure, and this embodiment does not impose any additional limitations on this.

[0117] In some embodiments, whether to trigger text detection in the text-to-be-detected area in the current image frame is determined by any of the following conditions:

[0118] Case a) in response to the similarity between the corresponding text regions to be detected being less than a similarity threshold, triggering text detection on the text region to be detected in the current image frame to determine the text region of the current image frame;

[0119] In implementation, when the similarity between the corresponding text detection regions is less than the similarity threshold, it indicates that the text detection regions of the previous image frame and the current image frame are not similar, and text detection needs to be performed on the current image frame.

[0120] Case b) In response to the similarity between the corresponding text detection regions being greater than or equal to a similarity threshold, the text region of the current image frame is determined based on the text detection result of the previous image frame.

[0121] During implementation, text detection of the text to be detected area of ​​the current image frame is triggered only when the similarity between the corresponding text to be detected areas is greater than or equal to the similarity threshold. When the similarity between the corresponding text to be detected areas is less than the similarity threshold, it means that the text to be detected areas of the previous image frame and the current image frame are similar. The text detection result of the previous image frame can be retained as the text detection result of the current image frame, realizing the frame skipping detection strategy, which improves the text detection speed while ensuring the detection accuracy, and meets the real-time nature of text detection.

[0122] As shown in FIG4 , this embodiment also provides a text detection process using a frame skipping strategy, which is specifically as follows:

[0123] Step 400: Obtain video stream;

[0124] Step 401: Determine if a sudden change occurs in the brightness of the video image;

[0125] Step 402: triggering the text detection process;

[0126] Step 403: Determine whether text is detected in the previous image frame. If yes, proceed to step 404; otherwise, proceed to step 405.

[0127] Step 404: Determine the text detection regions of the previous image frame and the current image frame based on the text region detected in the previous image frame, and then execute step 406;

[0128] Step 405: Determine the text detection area of ​​each of the previous image frame and the current image frame according to the preset area, and execute step 406;

[0129] Step 406: Screenshot the text-to-be-checked regions of the previous image frame and the current image frame to obtain corresponding screenshots, and determine the similarity between the corresponding screenshots.

[0130] Step 407: Determine whether the similarity is less than a similarity threshold. If yes, proceed to step 408; otherwise, proceed to step 410.

[0131] Step 408: triggering text detection on the text-to-be-detected area of ​​the current image frame, and determining the text area of ​​the current image frame according to the text detection result;

[0132] Step 409: Filter the text area of ​​the image frame according to the depth of each pixel in the text area of ​​the image frame, determine the text area of ​​the image frame after filtering, and execute step 411;

[0133] Step 410: Determine the text area of ​​the current image frame based on the text detection result of the previous image frame, and execute step 411;

[0134] Step 411: output the position of the text area in the current image frame;

[0135] Step 412: Determine whether the brightness of the video image is restored.

[0136] Step 413: End the text detection process.

[0137] In implementation, this embodiment provides a frame skipping strategy. If the text detection result of the previous image frame is not empty, that is, text is detected, each detection frame of the previous image frame is traversed, and the text area to be detected is determined according to the position of each detection frame. Screenshots are taken of the text area to be detected of the previous image frame and the current image frame, and the longest side size of the screenshot is limited to no more than the second threshold L, and the shortest side size is no less than the first threshold S. The screenshot is scaled proportionally on the basis of ensuring the shortest side size. If the long side exceeds the limit, the middle area is captured, and the structural similarity of the scaled screenshot is calculated. If the structural similarity is less than the similarity threshold (such as 0.85), text detection of the current image frame is triggered. Otherwise, the text detection result of the previous image frame is returned, where the text detection result includes no text detected or text detected and the position of the text area is output;

[0138] If the text detection result of the previous image frame is empty, that is, no text is detected, the text detection areas of the previous image frame and the current image frame are determined according to the preset areas, and screenshots of the text detection areas of the previous image frame and the current image frame are taken. After the screenshots are scaled, the structural similarity between the scaled screenshots is calculated. If the structural similarity is less than the similarity threshold (such as 0.85), the text detection of the current image frame is triggered, otherwise the text detection result of the previous image frame is returned, where the text detection result includes no text detected or text detected and the position of the text area output.

[0139] In order to avoid frame-by-frame detection, this embodiment adopts a frame skipping method to ensure the accuracy of text detection while ensuring the real-time performance of text detection. Generally speaking, the area where text appears in the video screen is relatively fixed, mainly located at the bottom of the video screen, and prompt text occasionally appears above, left and right of the screen. Based on this prior information, the position of the text in the preset area of ​​the display screen is determined; after the text detection function is triggered, if there is no text in the previous image frame, the inter-frame similarity comparison of the preset area is performed. If the similarity is lower than the similarity threshold, it means that there is a sudden change in the content of the preset area, and it is believed that text may begin to appear, triggering text detection of the current image frame; if there is text in the previous image frame, the inter-frame similarity calculation of the detected text area is performed. If the similarity is lower than the similarity threshold, it is believed that there may be text updates or no text, triggering text detection of the current image frame. The frame skipping text detection method of this embodiment avoids multiple detections of the same text / no-text video screen, thereby improving the detection speed and achieving real-time detection.

[0140] In some embodiments, text detection is performed on the text detection area of ​​the image frame in the following manner:

[0141] Performing text detection on the text-to-be-detected area of ​​the image frame and outputting a mask image, wherein the mask image includes a mask area related to the text;

[0142] The text area is determined according to the mask area in the mask image, and the size of the mask area is obtained by expanding the minimum circumscribed rectangle of the mask area.

[0143] In implementation, the mask map includes but is not limited to a binary map, a probability map, etc., and the size and position of the detected text area are determined by the mask area related to the text in the mask map.

[0144] This embodiment reduces the influence of mask corners by expanding the mask area and improves the accuracy of detection. Optionally, the size of the mask area can be determined by:

[0145] Determine each coordinate pair in the shortest side direction of the minimum circumscribed rectangle of the mask area, wherein the coordinate pair includes two coordinate points of the boundary of the mask area in the shortest side direction; determine the distance between the two coordinate points in each coordinate pair, and determine the size of the mask area based on the average value of each distance.

[0146] During implementation, due to the interference of complex background, the mask of the text area predicted by the text detection network is prone to produce sharp corners. To this end, the mask area in the mask image output by the text detection network is expanded, and the average height of the short side of the minimum circumscribed rectangle of the mask area is taken as the height of the text area, and the position and size of the text area are recalculated.

[0147] It should be noted that the mask area is a polygonal closed area. First, the minimum enclosing rectangle of the mask area is determined, and the short side direction of the minimum enclosing rectangle is determined. Optionally, the short side direction of the minimum enclosing rectangle is usually the X direction or the Y direction (the X direction and the Y direction are determined according to the coordinate system of the display screen). When the short side direction is the Y direction, the Y-axis coordinate point of the boundary of the mask area in the Y direction is determined. According to the difference between the two Y-axis coordinate points corresponding to the same X coordinate in the Y direction, the height corresponding to the X coordinate is determined. The difference between the two Y-axis coordinate points corresponding to each X coordinate on the boundary of the mask area is calculated. According to the average value of the difference corresponding to each X coordinate, the average height of the mask area is determined. The average height of the mask area is used as the height of the text area, and the size of the text area output by the text detection network is updated based on the height of the text area.

[0148] Similarly, when the short side direction is the X direction, determine the X-axis coordinate point of the boundary of the mask area in the X direction, and determine the width corresponding to the Y coordinate according to the difference between the two X-axis coordinate points corresponding to the same Y coordinate in the X direction. Calculate the difference between the two X-axis coordinate points corresponding to each Y coordinate on the boundary of the mask area, and determine the average width of the mask area according to the average value of the difference corresponding to each Y coordinate. The average width of the mask area is used as the width of the text area, and the size of the text area output by the text detection network is updated based on the width of the text area.

[0149] In some embodiments, this embodiment specifically performs text detection on the text-to-be-detected area of ​​the image frame in the following manner:

[0150] Use the text detection network to perform text detection on the text detection area of ​​the image frame;

[0151] The loss function of the text detection network is determined based on the text area marked in the input sample and the mask area related to the text in the output mask map.

[0152] Optionally, when calculating the loss function based on the annotated text area and the mask area, the size of the mask area is obtained by expanding the minimum bounding rectangle of the mask area.

[0153] In practice, in conjunction with the frame skipping scheme of this embodiment, each text is detected in only one image frame. In order to improve the accuracy of text detection and reduce the interference of complex background textures in the image frame, this embodiment optimizes the post-processing and loss function of text detection. The loss function of the text detection network in this embodiment is as follows: L det =L s +αL b +βL t Formula (2);

[0154] In formula (2), L det Represents the loss function of the text detection network, L s Represents the loss value of the probability map output by the text detection network, L b Represents the loss value of the binary image output by the text detection network, L t Represents the loss value of the threshold map output by the text detection network, and α and β are balance coefficients.

[0155] During implementation, the text detection network predicts and outputs a probability map and a threshold map of the text area. The probability represents the probability that the corresponding pixel position in the original image is a text area, and the threshold represents the threshold for binarizing the corresponding pixel position in the probability map.

[0156] Optional, L s BCE loss (Binary Cross-Entropy Loss) can be used, L t L1 loss (absolute error loss, also known as Mean Absolute Error, MAE) can be used, L b Dice loss (named after the Dice coefficient, which is a metric function used to evaluate the similarity between two samples) can be used.

[0157] In the implementation, in order to enhance the robustness to complex backgrounds, the low-confidence false detection areas are penalized. Specifically, in L s , L bA mask is used in the calculation to limit the area involved in the calculation. The mask area used in this embodiment is not the true value mask after the text area is shrunk, but an expanded mask is used to penalize the background noise close to the text. In order to avoid too many negative samples, the true value points of the negative samples can also be weighted by using focal loss. Focal loss adjusts the sample weights based on the prediction accuracy, so that the model pays more attention to samples that are misclassified and difficult to classify. The idea behind focal loss is to add a coefficient factor on the basis of the standard cross entropy loss. It is a non-fixed value and is calculated based on the prediction results. This weakens the learning of easy samples and strengthens the learning of difficult samples, thereby improving the classification ability of the model. The weight factor is used to improve the problem of loss being dominated by the majority of samples caused by sample imbalance.

[0158] Optionally, in order to reduce the inference time of the text detection network, this embodiment scales the input screenshot, for example, to 384×640 size, and uses a lightweight DBNet (the backbone network is LCNetV3) as the text detection network; here, the lightweight DBNet is an example of a basic model, and in actual applications, it can be pruned and quantized to improve the model inference speed. DBNet is a text detection algorithm based on segmentation. The algorithm introduces a differentiable binarization module (DB) into the segmentation model, so that the model can be binarized through an adaptive threshold map, and the adaptive threshold map can calculate the loss, which can play an auxiliary effect in the model training process. The core of DBNet uses segmentation-based text detection, performs semantic segmentation on each text block, and then simply binarizes the segmentation probability map, and finally converts it into a text detection result in the detection box or poly format.

[0159] In some embodiments, after triggering text detection on the text area to be detected in the image frame, the method screens the text area by depth estimation. The specific process is as follows:

[0160] Determine the text area of ​​the image frame according to the text detection result;

[0161] The text area of ​​the image frame is filtered according to the depth of each pixel in the text area of ​​the image frame.

[0162] Optionally, filter the text area of ​​the image frame by any of the following methods:

[0163] Method a: Use the depth of pixels in the depth map to filter the text area.

[0164] During implementation, depth estimation is performed on the image frame to obtain a depth map, and a depth region corresponding to the text region is determined from the depth map; and the text region is screened according to the depth of each pixel in the depth region.

[0165] The encoder of the depth estimation network in this embodiment is implemented by multi-layer convolution. The decoder of the depth estimation network can share the same decoder parameters as the text detection network, thereby saving time in extracting features from the image. The depth estimation network can predict the entire image, that is, the depth of each pixel in the entire image frame. The loss function of the depth estimation network is as follows:

[0166] In formula (3), represents the absolute error loss, Represents the relative error loss, l d Indicates the depth of the annotation, p d Represents the depth of the depth estimation network output (prediction).

[0167] Using depth estimation for text screening can support full-image text detection, including but not limited to hanging subtitles such as character introductions at random locations on the display screen.

[0168] The specific steps are: traverse the detection frame (detected text area) output by the text detection network, intercept the depth area corresponding to the detection frame in the depth map, intercept the original text area corresponding to the detection frame in the current image frame, perform Gaussian blur and edge detection (such as Canny edge detection) on the original text area, and obtain a closed area, where the closed area refers to the area containing specific glyphs in the text. If the closed area is greater than 50% of the depth area, it means that the closed area is more accurate at this time. According to the average value of the depth of each pixel in the closed area, the text area is screened. If the closed area is less than or equal to 50% of the depth area, it means that the depth area is more accurate at this time. According to the average value of the depth of each pixel in the depth area, the text area is screened.

[0169] Optionally, in response to the average value of the depth of each pixel in the depth area being greater than a depth threshold, the text area corresponding to the depth area is deleted from the text detection result; or, in response to the average value of the depth of each pixel in the depth area being less than or equal to the depth threshold, the text area in the text detection result is retained.

[0170] Method b: Use the depth of pixels in the original image to filter the text area.

[0171] During implementation, the original text area corresponding to the text area is determined from the current image frame, Gaussian blurring and edge detection are performed on the original text area to obtain a closed area; and the text area is screened according to the depth of each pixel in the closed area.

[0172] Optionally, in response to the average value of the depth of each pixel in the closed area being greater than a depth threshold, the text area corresponding to the closed area is deleted from the text detection result; in response to the average value of the depth of each pixel in the closed area being less than or equal to the depth threshold, the text area in the text detection result is retained.

[0173] Method c: By comparing the depth map and the original image, the depth of the pixels in the depth map or the original image is selected to filter the text area.

[0174] During implementation, depth estimation is performed on the image frame to obtain a depth map, and a depth region corresponding to the text region is determined from the depth map; and an original text region corresponding to the text region is determined from the image frame, and Gaussian blurring and edge detection are performed on the original text region to obtain a closed region;

[0175] In response to the closed area being larger than the product of the depth area and the adjustment factor, screening the text area according to the depth of each pixel in the closed area; or,

[0176] In response to the closed area being less than or equal to the product of the depth area and the adjustment factor, the text area is filtered according to the depth of each pixel in the depth area.

[0177] Optionally, the text area is filtered according to the depth of each pixel in the closed area, as follows:

[0178] In response to the average value of the depth of each pixel in the closed area being greater than the depth threshold, the text area corresponding to the closed area is deleted from the text detection result; or, in response to the average value of the depth of each pixel in the closed area being less than or equal to the depth threshold, the text area in the text detection result is retained.

[0179] Optionally, the text area is filtered according to the depth of each pixel in the depth area, as follows:

[0180] In response to the average value of the depth of each pixel in the depth area being greater than the depth threshold, the text area corresponding to the depth area is deleted from the text detection result; or, in response to the average value of the depth of each pixel in the depth area being less than or equal to the depth threshold, the text area in the text detection result is retained.

[0181] In implementation, the decoder of the depth estimation network is implemented by multi-layer convolution, which can predict the depth of each pixel in the entire image. The depth estimation network and the text detection network share the same encoder parameters.

[0182] It should be noted that the pixel depth in this embodiment refers to the distance information from the pixel to the camera, and depth estimation refers to obtaining the distance information from each pixel in the scene in the image frame to the camera. The camera here is not a real camera, but a concept of a virtual camera proposed to explain depth.

[0183] Text detection is to locate the text area in the image. The basic function it relies on is text detection. At the same time, it is necessary to exclude the interference of the text in the film and television screen itself (such as the text on the billboards in the movie, the text on the characters' clothing, etc.). There are certain limitations in distinguishing based solely on the text color, font or texture. For example, it cannot guarantee that the text in the screen is not a square font. Based on this, this embodiment uses depth estimation to distinguish between text and non-text. Taking the text as subtitles as an example, since subtitles are text that is pasted into the image later, the pixel depth in the subtitles is relatively small. The text area in the text detection result can be filtered by a threshold and the non-subtitle area can be deleted.

[0184] In some embodiments, this embodiment also provides another method for filtering text detection results, which filters the position of the text area in the text detection result based on the probability of the text appearing at different positions in the image frame and the appearance characteristics of the text in the image frame. As shown in Figure 5, this embodiment provides a schematic diagram of display screen partitioning, and the text detection results can be filtered by any one or more of the following methods, as shown below:

[0185] 1) Delete the text area whose center point is located in the center area of ​​the display screen in the text detection results;

[0186] 2) retaining the text area in the text detection result, where the center point of the text area is located in the preset area;

[0187] 3) For the text detection results, the center point of the text area is located between the center area of ​​the display screen and the preset area, and the non-horizontal text area in the text detection results and the text area whose height difference with the height of the preset area is greater than the height threshold (such as 0.1) are deleted.

[0188] 4) Since the preset area is usually larger than the actual text area, the text area in the text detection result can be screened again. For the text area close to the edge of the preset area (this edge is relative to the center area), the three main colors in the text area are extracted, and the text area with main color outliers is deleted.

[0189] For movies and TV shows in different regions, there are subtitles in multiple languages. This embodiment can also support text detection in multiple languages.

[0190] In some embodiments, text detection is triggered in the text detection area of ​​the current image frame, and language classification of the text can also be performed. The specific implementation steps are as follows:

[0191] Use the text detection network to perform text detection on the image frame and obtain the text detection result;

[0192] The language classification network is used to classify the language of the text in the text area in the text detection result, and the language category of the text in the text area is output.

[0193] The language classification head uses a fully convolutional network. Its input features are the multiplication of the multi-scale features output by the DBNet encoding stage and the binary image predicted by the DBNet (to guide the location of the text area). The output is a map the size of the original image with N+1 channels (N is the number of languages, and 1 is non-language). The loss function uses CE loss (cross entropy loss), and positive and negative samples are weighted to limit the proportion of negative samples.

[0194] Optionally, the language classification network and the text detection network reuse the same encoder parameters; the text features of the text area extracted by the encoder of the text detection network are input into the decoder of the language classification network, and the language category of the text in the text area is output.

[0195] To maintain the real-time performance of text detection, the head (decoder) of the language classification network is removed during the deployment phase, which does not affect the overall speed of text detection. The detected text features are then temporarily stored. When the language classification network is triggered, the temporarily stored text features are used to input the head (decoder) of the language classification network for inference. Specifically, the text detection results output by DBNet are post-processed (the minimum bounding rectangle of the mask area of ​​the output mask image is expanded) and mapped back to features, that is, the text features of the text area are extracted. Horizontal and vertical text do not need to be inferred separately. The text features of different texts are spliced ​​in the width and height dimensions, and zeros are added to the insufficient size. The text features are then input into the decoder of the language classification network for inference (based on the number of text features in the temporary queue, multiple inferences are performed until the language categories corresponding to all the temporarily stored text features are obtained). In implementation, the text detection network is trained first. After the text detection network training is completed, the language classification network is trained.

[0196] Since the number of network parameters supported for deployment on the terminal is small and the feature fitting capability is low, directly using a text detection model that supports multiple languages ​​will reduce the detection accuracy. Therefore, in order to maintain the accuracy of multilingual text detection, this embodiment provides a language switching process to reduce the impact of the decline in detection accuracy of a single multilingual text detection model.

[0197] In some embodiments, texts in different language categories are classified as follows:

[0198] The text detection network outputs multiple text regions, where the language categories of the text in the multiple text regions are different;

[0199] By switching decoder parameters of a text detection network, text detection is performed on text regions of different language categories; and / or, by switching decoder parameters of a language classification network, language classification is performed on text regions of different language categories.

[0200] During implementation, for the input video stream, a text detection model that supports all languages ​​is first used. When S texts are obtained, language classification is triggered (wherein, temporary text features are temporarily stored. When the number of temporary text feature queues reaches S / 2, the text length mean m is calculated, and temporary text features with a text length less than 0.6×m are dynamically screened out to ensure the accuracy of language judgment). The language classification results of the S texts are integrated and switched to the text detection network of the corresponding specific language. The text detection networks of different languages ​​reuse the backbone network (encoder) parameters, and only the decoder parameters are updated when switching between different languages ​​(thereby reducing the time consumed when switching between the text detection network and the language classification network. Since the video memory and internal memory on the terminal are limited, it is not suitable to start multiple network models at the same time. Therefore, this embodiment saves time by starting a network and replacing the network weights. By fixing the encoder parameters and replacing the decoder parameters with fewer parameters, the switching between text detection networks and language classification networks of different languages ​​is realized.

[0201] During model training, the full dataset is used to train the text detection network for all languages. The backbone network (encoder parameters) is then frozen, and the decoder parameters are fine-tuned using language-specific data and a small amount of data from other languages. For bilingual subtitles, if a language-specific text detection network is available, such as a Chinese-English text detection network, it is used. Otherwise, the full language detection model is retained without switching.

[0202] Among them, the language-specific text detection network means that the same network model is trained for similar or commonly used bilingual subtitle languages, and different network models are used for texts with large differences in font size. For example, the same network model can be used for Chinese, Japanese, Korean, and English; another network model is used for German, French, Italian, Russian, Portuguese, English, and Chinese; and another network model is used for Latin and Arabic languages.

[0203] To ensure the real-time performance of text detection, this embodiment also provides a multi-task network that performs a feature extraction on the original image (this image frame) and simultaneously implements multiple tasks including text detection, depth estimation, and language classification. Based on DBNet as the basic structure, the backbone network (encoder) adopts LCNetV3, adding depth estimation and language classification branches (the lightweight DBNet here is used as an example of the basic model. In actual applications, pruning and quantization can be considered to improve the model inference speed). To reduce the model inference time, the input image can be scaled to 384×640.

[0204] As shown in Figure 6, this embodiment provides a schematic diagram of the first multi-task network structure for text detection. Based on the DBNet architecture, the text detection network, depth estimation network, and language classification network share the same encoder and its parameters. When performing language classification, the language category of the detected text region is determined based on the probability map output by the text detection network and the image features output by the encoder. The probability map output by the text detection network and the image features output by the encoder of the text detection network are used as inputs to the language classification network decoder. The language classification network decoder then classifies the text features associated with the text region in the image features and outputs probability values ​​for N+1 language categories, where N represents the number of languages ​​and 1 represents non-language. This reduces the time required to perform feature extraction on the text region again, effectively reducing model inference time. Similarly, when performing depth estimation, the image features output by the encoder of the text detection network are used as inputs to the depth estimation network decoder, directly performing depth judgment on the image features and reducing model inference time.

[0205] As shown in Figure 7, this embodiment provides a schematic diagram of the second text detection multi-task network structure. With DBNet as the basic structure, the text detection network, depth estimation network, and language classification network share the same encoder and its parameters. When performing language classification, based on the probability map output by the text detection network, the text features corresponding to the text area are intercepted from the image features output by the encoder of the text detection network, and the intercepted text features are input into the language classification network decoder for classification. This network structure is for language prediction of text features. According to the text detection results, the text features of the corresponding area are intercepted from the multi-scale features output in the DBNet encoding stage, and input into the language classification head (decoder) for classification (which first undergoes two layers of convolution, then obtains features of the same size through ROI Align (Region of Interest Alignment), and then inputs into the FC (Full Connection) layer). When performing depth estimation, the image features output by the encoder of the text detection network are used as the input of the depth estimation network decoder, and the depth of the image features is directly judged, thereby reducing the model inference time.

[0206] As shown in Figure 8, this embodiment provides a schematic diagram of the third text detection multi-task network structure, which is based on DBNet. The text detection network, depth estimation network and language classification network share the same encoder and its parameters. When performing language classification, based on the binary image output by the text detection network, the text features corresponding to the text area are intercepted from the image features output by the encoder of the text detection network (image features after dimensionality reduction processing), and the intercepted text features are input into the language classification network decoder for classification; when performing depth estimation, the image features output by the encoder of the text detection network are used as the input of the depth estimation network decoder, and the depth of the image features is directly judged, thereby reducing the model inference time.

[0207] The feature extraction capability of lightweight models is limited by the number of model parameters, and the model accuracy can be improved through training methods.

[0208] This embodiment also provides a method for multi-scale training and dynamic adjustment of the shrinkage ratio. To improve the robustness of the model, the input image size is randomly scaled during training. In the early stages of training, the input resolution is maintained at 960×960. Subsequently, three sizes (384×640, 640×640, and 960×960) are randomly selected for scaling within each epoch. An epoch is defined as training with all samples in the training set. Simultaneously, the shrinkage ratio is adjusted from a fixed value to a dynamic one. As the number of training epochs increases, the shrinkage ratio increases linearly from 0.4 to 0.6 and then decreases to 0.4. The shrinkage ratio changes dynamically based on the linear increase in epochs.

[0209] This embodiment also provides a model distillation method for improving the accuracy of the deployed lightweight model. Model distillation is a model compression technology that aims to transfer the knowledge of a complex, large model (usually called a teacher model) to another smaller, simpler model (usually called a student model). In model distillation, a trained teacher model that performs well on large data sets is usually used. Then, the knowledge of the teacher model is transferred to a smaller student model, so that the student model can perform reasoning on a smaller scale and maintain similar performance to the teacher model. This embodiment distills the lightweight model by training a high-precision large model (for example, the backbone network is ResNet50) as a teacher model. The large model in this embodiment is only trained for text detection. That is, a large model is first trained for text detection, and it is used as a teacher model. The lightweight model is distilled to obtain a lightweight text detection network.

[0210] This embodiment designs a frame skipping scheme based on feature comparison, which, in conjunction with a lightweight text detection network, can meet the real-time requirements of text detection and, in conjunction with the real-time playback characteristics of the video, achieve fast subtitle detection. Since the text detection accuracy of a multilingual text detection model is lower than that of a single language, a language switching process is designed to address the problem of multilingual detection, thereby reducing the impact of the decline in detection accuracy of a single multilingual text detection model. Depth detection can also be performed, using the depth information of pixels to distinguish between text and non-text areas in the picture, which can be used to distinguish between subtitles and non-subtitle areas in the display picture.

[0211] This embodiment can be applied to terminal playback devices such as TVs and tablets. It can dynamically adjust the brightness of subtitles according to the color and brightness of the video being played, presenting a softer viewing effect (for example, if a dark scene uses the same subtitle brightness as a bright scene, the subtitles will be too prominent). To achieve the above functions, this embodiment provides a text detection method that can be used to detect subtitles and obtain the position of subtitles. Since the video is played in real time on terminal devices such as TVs, the speed of subtitle detection is high. Direct subtitle detection frame by frame or detection combined with audio is time-consuming and requires high computing power. Therefore, the frame-skipping text detection solution provided by this embodiment, combined with a lightweight text detection model, can achieve real-time subtitle detection.

[0212] Based on the same inventive concept, the embodiment of the present disclosure further provides a brightness adjustment method, as shown in FIG9 . The specific implementation process of the method is as follows:

[0213] Step 900: Perform text detection according to a text detection method to determine the text area of ​​the image frame;

[0214] Step 901: Adjust the brightness of the text area of ​​the current image frame according to the brightness of the non-text area of ​​the current image frame.

[0215] Optionally, the brightness and contrast of the text area of ​​the image frame may be adjusted according to the average brightness of the non-text area of ​​the image frame, so that the brightness of the text area and the non-text area are kept consistent.

[0216] Optionally, this embodiment can adjust the brightness of the text area by adjusting the grayscale of the text area, or by adjusting the display screen parameters. This embodiment does not impose too many restrictions on the specific method of adjusting the brightness of the text area.

[0217] Optionally, the above text detection method specifically includes the following steps:

[0218] Determine the text to-be-inspected regions of the previous image frame and the current image frame, wherein the text to-be-inspected regions of the previous image frame and the current image frame are in one-to-one correspondence;

[0219] Determining the similarity between the corresponding text to-be-detected regions based on the text to-be-detected regions of the previous image frame and the current image frame;

[0220] Whether to trigger text detection on the text-to-be-detected area of ​​the current image frame is determined based on the similarity between the corresponding text-to-be-detected areas.

[0221] Based on the same inventive concept, the embodiment of the present disclosure also provides a display device. Since the display device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0222] As shown in FIG10 , the display device includes a display screen 1000 and a control circuit 1001 ;

[0223] The display screen 1000 is used to display content;

[0224] The control circuit 1001 includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0225] Determine the text to-be-detected regions of the previous image frame and the current image frame, wherein the positions of the text to-be-detected regions between the previous image frame and the current image frame are in one-to-one correspondence;

[0226] Determining the similarity between the text to-be-inspected regions having corresponding relationships in the previous image frame and the current image frame;

[0227] Whether to trigger text detection on the text-to-be-detected area of ​​the current image frame is determined according to the similarity between the corresponding text-to-be-detected areas.

[0228] Based on the same inventive concept, the embodiment of the present disclosure also provides another display device. Since the display device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0229] As shown in FIG11 , the display device includes a display screen 1100 and a control circuit 1101 ;

[0230] The display screen 1100 is used to display content;

[0231] The control circuit 1101 includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0232] Perform text detection according to a text detection method to determine the text area of ​​the image frame;

[0233] The brightness of the text area of ​​the image frame is adjusted according to the brightness of the non-text area of ​​the image frame.

[0234] Based on the same inventive concept, the embodiment of the present disclosure also provides an electronic device. Since the electronic device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0235] As shown in FIG12 , the electronic device includes a processor 1200 and a memory 1201 . The memory is used to store a program executable by the processor. The processor is used to read the program in the memory and perform the following steps:

[0236] Determine the text to-be-detected regions of the previous image frame and the current image frame, wherein the positions of the text to-be-detected regions between the previous image frame and the current image frame are in one-to-one correspondence;

[0237] Determining the similarity between the text to-be-inspected regions having corresponding relationships in the previous image frame and the current image frame;

[0238] Whether to trigger text detection on the text-to-be-detected area of ​​the current image frame is determined according to the similarity between the corresponding text-to-be-detected areas.

[0239] As an optional implementation manner, the processor 1200 is specifically configured to execute:

[0240] In response to the text detection result of the previous image frame being that text is detected, determining the text detection regions of the previous image frame and the current image frame according to the text region detected in the previous image frame; or

[0241] In response to the text detection result of the previous image frame being that no text is detected, text detection areas of the previous image frame and the current image frame are determined according to preset areas.

[0242] As an optional implementation manner, the processor 1200 is further configured to execute:

[0243] In response to performing text detection on a preset area of ​​a previous image frame and detecting text in the preset area for the first time, updating the preset area according to the size of the text area detected for the first time and a preset magnification; or,

[0244] In response to the text detection result of the previous image frame indicating that the size of the text area detected is larger than the preset area, the preset area is updated.

[0245] As an optional implementation manner, the processor 1200 is specifically configured to execute:

[0246] Screenshot processing is performed on the text to-be-checked regions of the previous image frame and the current image frame to obtain corresponding screenshots, and the similarity between the corresponding screenshots is determined.

[0247] As an optional implementation manner, after taking screenshots of the text-to-be-checked areas of the previous image frame and the current image frame, the processor 1200 is further configured to execute:

[0248] Scale the screenshot proportionally and adjust the screenshot size within the preset range;

[0249] When scaling a screenshot, ensure that the shortest dimension of the screenshot is greater than or equal to a first threshold;

[0250] In response to the longest dimension of the screenshot being greater than a second threshold after the screenshot is scaled, a middle area with the longest dimension within the second threshold is captured and determined as the adjusted screenshot.

[0251] As an optional implementation manner, the processor 1200 is further configured to execute:

[0252] In response to a frame rate of a video frame being greater than a first frame rate threshold, determining a preset area to be detected from the text area to be detected, determining similarity between the preset areas to be detected having a corresponding relationship in two image frames separated by one or more frames, and determining whether to trigger text detection in the preset areas to be detected based on the similarity between the preset areas to be detected having a corresponding relationship; or

[0253] In response to the frame rate of the video frame being greater than a second frame rate threshold, the similarity between the text-to-be-detected areas with corresponding relationships is determined in two image frames separated by one or more frames, and it is determined whether to trigger text detection of the text-to-be-detected areas in the current image frame based on the similarity between the text-to-be-detected areas with corresponding relationships.

[0254] As an optional implementation manner, the processor 1200 is specifically configured to execute:

[0255] In response to the similarity between the corresponding text regions to be detected being less than a similarity threshold, triggering text detection on the text region to be detected in the current image frame to determine the text region of the current image frame; or

[0256] In response to the similarity between the corresponding text regions to be detected being greater than or equal to a similarity threshold, the text region of the current image frame is determined according to the text detection result of the previous image frame.

[0257] As an optional implementation manner, the processor 1200 is specifically configured to perform text detection on the text detection area of ​​the current image frame in the following manner:

[0258] Performing text detection on the text-to-be-detected area of ​​the image frame and outputting a mask image, wherein the mask image includes a mask area related to the text;

[0259] The text area is determined according to the mask area in the mask image, and the size of the mask area is obtained by expanding the minimum circumscribed rectangle of the mask area.

[0260] As an optional implementation manner, the processor 1200 is specifically configured to determine the size of the mask area in the following manner:

[0261] Determine each coordinate pair in the shortest side direction of the minimum circumscribed rectangle of the mask area, wherein the coordinate pair includes two coordinate points of the boundary of the mask area in the shortest side direction;

[0262] The distance between two coordinate points in each coordinate pair is determined, and the size of the mask area is determined according to an average value of each distance.

[0263] As an optional implementation manner, the processor 1200 is specifically configured to execute:

[0264] Use the text detection network to perform text detection on the text detection area of ​​the image frame;

[0265] Among them, the loss function of the text detection network is determined based on the text area marked in the input sample and the mask area related to the text in the output mask map, and the size of the mask area is obtained by expanding the minimum circumscribed rectangle of the mask area.

[0266] As an optional implementation, after triggering text detection on the text detection area of ​​the current image frame, the processor 1200 is further configured to execute:

[0267] Determine the text area of ​​the image frame according to the text detection result;

[0268] The text area of ​​the image frame is filtered according to the depth of each pixel in the text area of ​​the image frame.

[0269] As an optional implementation manner, the processor 1200 is specifically configured to execute:

[0270] Performing depth estimation on the image frame to obtain a depth map, determining a depth region corresponding to the text region from the depth map; screening the text region according to the depth of each pixel in the depth region; or,

[0271] An original text region corresponding to the text region is determined from the current image frame, Gaussian blurring and edge detection are performed on the original text region to obtain a closed region; and the text region is screened according to the depth of each pixel in the closed region.

[0272] As an optional implementation manner, the processor 1200 is specifically configured to execute:

[0273] performing depth estimation on the current image frame to obtain a depth map, and determining a depth region corresponding to the text region from the depth map; and determining an original text region corresponding to the text region from the current image frame, and performing Gaussian blurring and edge detection on the original text region to obtain a closed region;

[0274] In response to the closed area being larger than the product of the depth area and the adjustment factor, screening the text area according to the depth of each pixel in the closed area; or,

[0275] In response to the closed area being less than or equal to the product of the depth area and the adjustment factor, the text area is filtered according to the depth of each pixel in the depth area.

[0276] As an optional implementation manner, the processor 1200 is specifically configured to execute:

[0277] If the average value of the depths of the pixels in the text area is greater than the depth threshold, the text area is deleted from the text detection result.

[0278] As an optional implementation manner, to trigger text detection on the text detection area of ​​the current image frame, the processor 1200 is further configured to execute:

[0279] Use the text detection network to perform text detection on the image frame and obtain the text detection result;

[0280] The language classification network is used to classify the text in the text area in the text detection result into different languages, and the language category of the text in the text area is output.

[0281] As an optional implementation, the language classification network and the text detection network reuse the same encoder parameters;

[0282] The text features of the text region extracted by the encoder of the text detection network are input into the decoder of the language classification network, and the language category of the text in the text region is output.

[0283] As an optional implementation manner, the processor 1200 is specifically configured to execute:

[0284] The text detection network outputs a plurality of text regions, wherein the language categories of the texts in the plurality of text regions are different;

[0285] By switching decoder parameters of a text detection network, text detection is performed on text regions of different language categories; and / or, by switching decoder parameters of a language classification network, language classification is performed on text regions of different language categories.

[0286] Based on the same inventive concept, the embodiment of the present disclosure also provides another electronic device. Since the electronic device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0287] As shown in FIG13 , the electronic device includes a processor 1300 and a memory 1301 . The memory is used to store a program executable by the processor. The processor is used to read the program in the memory and perform the following steps:

[0288] Perform text detection according to a text detection method to determine the text area of ​​the image frame;

[0289] The brightness of the text area of ​​the image frame is adjusted according to the brightness of the non-text area of ​​the image frame.

[0290] Based on the same inventive concept, the embodiment of the present disclosure also provides a text detection device. Since the device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0291] As shown in FIG14 , the device includes:

[0292] An area determination module 1400 is configured to determine the text-to-be-detected areas of the previous image frame and the current image frame, wherein the positions of the text-to-be-detected areas of the previous image frame and the current image frame are in one-to-one correspondence;

[0293] The region comparison module 1401 is used to determine the similarity between the corresponding text regions to be inspected in the previous image frame and the current image frame;

[0294] The judgment trigger module 1402 is configured to determine whether to trigger text detection on the text to be detected area of ​​the current image frame according to the similarity between the corresponding text to be detected areas.

[0295] Based on the same inventive concept, the embodiment of the present disclosure also provides a brightness adjustment device. Since the device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0296] As shown in FIG15 , the device includes:

[0297] A text determination module 1500 is configured to perform text detection according to a text detection method to determine a text area of ​​the image frame;

[0298] The brightness adjustment module 1501 is configured to adjust the brightness of the text area of ​​the current image frame according to the brightness of the non-text area of ​​the current image frame.

[0299] Based on the same inventive concept, embodiments of the present disclosure provide a computer storage medium comprising computer program code. When the computer program code is executed on a computer, the computer executes any of the text detection methods and brightness adjustment methods discussed above. Because the principles underlying the problems solved by the computer storage medium are similar to those of the text detection method and brightness adjustment method, the implementation of the computer storage medium can be referenced to the implementation of the method, and any repetitions will not be repeated.

[0300] In a specific implementation process, computer storage media may include: Universal Serial Bus Flash Drive (USB), mobile hard disk, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0301] Based on the same inventive concept, embodiments of the present disclosure further provide a computer program product, comprising: computer program code, which, when executed on a computer, causes the computer to execute any of the text detection methods and brightness adjustment methods discussed above. Because the principles underlying the problems solved by the aforementioned computer program products are similar to those of the text detection methods and brightness adjustment methods, the implementation of the aforementioned computer program products can be referenced to the implementation of the methods, and any repetitions will not be repeated.

[0302] The computer program product can employ any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0303] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0304] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0305] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0306] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0307] Obviously, those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A text detection method, wherein: The method includes: Determine the text to-be-detected regions of the previous image frame and the current image frame, wherein the positions of the text to-be-detected regions between the previous image frame and the current image frame are in one-to-one correspondence; Determining the similarity between the text to-be-inspected regions having corresponding relationships in the previous image frame and the current image frame; Whether to trigger text detection on the text-to-be-detected area of ​​the current image frame is determined according to the similarity between the corresponding text-to-be-detected areas.

2. The method according to claim 1, wherein Determining the text to-be-detected areas of the previous image frame and the current image frame includes: In response to the text detection result of the previous image frame being that text is detected, determining the text detection regions of the previous image frame and the current image frame according to the text region detected in the previous image frame; or In response to the text detection result of the previous image frame being that no text is detected, text detection areas of the previous image frame and the current image frame are determined according to preset areas.

3. The method according to claim 2, wherein: The method further includes: In response to performing text detection on a preset area of ​​a previous image frame and detecting text in the preset area for the first time, updating the preset area according to the size of the text area detected for the first time and a preset magnification; or, In response to the text detection result of the previous image frame indicating that the size of the text area detected is larger than the preset area, the preset area is updated.

4. The method according to claim 1, wherein Determining the similarity between the corresponding text areas to be inspected in the previous image frame and the current image frame includes: Screenshot processing is performed on the text to-be-checked regions of the previous image frame and the current image frame to obtain corresponding screenshots, and the similarity between the corresponding screenshots is determined.

5. The method according to claim 4, wherein After taking screenshots of the text-to-be-checked areas of the previous image frame and the current image frame, the method further includes: Scale the screenshot proportionally and adjust the screenshot size within the preset range; When scaling a screenshot, ensure that the shortest dimension of the screenshot is greater than or equal to a first threshold; In response to the longest dimension of the screenshot being greater than a second threshold after the screenshot is scaled, a middle area with the longest dimension within the second threshold is captured and determined as the adjusted screenshot.

6. The method according to claim 1, wherein The method further includes: In response to a frame rate of a video frame being greater than a first frame rate threshold, determining a preset area to be detected from the text area to be detected, determining similarity between the preset areas to be detected having a corresponding relationship in two image frames separated by one or more frames, and determining whether to trigger text detection in the preset areas to be detected based on the similarity between the preset areas to be detected having a corresponding relationship; or In response to the frame rate of the video frame being greater than a second frame rate threshold, the similarity between the text-to-be-detected areas with corresponding relationships is determined in two image frames separated by one or more frames, and it is determined whether to trigger text detection of the text-to-be-detected areas in the current image frame based on the similarity between the text-to-be-detected areas with corresponding relationships.

7. The method according to claim 1, wherein The determining whether to trigger text detection on the text to be detected area in the current image frame according to the similarity between the corresponding text to be detected areas includes: In response to the similarity between the corresponding text regions to be detected being less than a similarity threshold, triggering text detection on the text region to be detected in the current image frame to determine the text region of the current image frame; or In response to the similarity between the corresponding text regions to be detected being greater than or equal to a similarity threshold, the text region of the current image frame is determined according to the text detection result of the previous image frame.

8. The method according to claim 1, wherein Perform text detection on the text-to-be-detected area of ​​this image frame in the following manner: Performing text detection on the text-to-be-detected area of ​​the image frame and outputting a mask image, wherein the mask image includes a mask area related to the text; The text area is determined according to the mask area in the mask image, and the size of the mask area is obtained by expanding the minimum circumscribed rectangle of the mask area.

9. The method according to claim 8, wherein The size of the mask area is determined as follows: Determine each coordinate pair in the shortest side direction of the minimum circumscribed rectangle of the mask area, wherein the coordinate pair includes two coordinate points of the boundary of the mask area in the shortest side direction; The distance between two coordinate points in each coordinate pair is determined, and the size of the mask area is determined according to an average value of each distance.

10. The method according to claim 8, wherein The performing text detection on the text-to-be-detected area of ​​the image frame includes: Use the text detection network to perform text detection on the text detection area of ​​the image frame; Among them, the loss function of the text detection network is determined based on the text area marked in the input sample and the mask area related to the text in the output mask map, and the size of the mask area is obtained by expanding the minimum circumscribed rectangle of the mask area.

11. The method according to claim 1, wherein After triggering text detection on the text-to-be-detected area of ​​the current image frame, the method further includes: Determine the text area of ​​the image frame according to the text detection result; The text area of ​​the image frame is filtered according to the depth of each pixel in the text area of ​​the image frame.

12. The method according to claim 11, wherein The filtering of the text area of ​​the image frame according to the depth of each pixel in the text area of ​​the image frame includes: Performing depth estimation on the image frame to obtain a depth map, determining a depth region corresponding to the text region from the depth map; screening the text region according to the depth of each pixel in the depth region; or, An original text region corresponding to the text region is determined from the current image frame, Gaussian blurring and edge detection are performed on the original text region to obtain a closed region; and the text region is screened according to the depth of each pixel in the closed region.

13. The method according to claim 11, wherein The filtering of the text area of ​​the image frame according to the depth of each pixel in the text area of ​​the image frame includes: Depth estimation is performed on the image frame to obtain a depth map, and the depth map is used to determine the depth of the text. and determining an original text area corresponding to the text area from the current image frame, performing Gaussian blur and edge detection on the original text area to obtain a closed area; In response to the closed area being larger than the product of the depth area and the adjustment factor, screening the text area according to the depth of each pixel in the closed area; or, In response to the closed area being less than or equal to the product of the depth area and the adjustment factor, the text area is filtered according to the depth of each pixel in the depth area.

14. The method according to claim 11, wherein The filtering of the text area of ​​the image frame according to the depth of each pixel in the text area of ​​the image frame includes: If the average value of the depths of the pixels in the text area is greater than the depth threshold, the text area is deleted from the text detection result.

15. The method according to claim 1, wherein Triggering text detection on the text-to-be-detected area of ​​the current image frame, the method further includes: Use the text detection network to perform text detection on the image frame and obtain the text detection result; The language classification network is used to classify the text in the text area in the text detection result into different languages, and the language category of the text in the text area is output.

16. The method according to claim 15, wherein The language classification network and the text detection network reuse the same encoder parameters; The text features of the text region extracted by the encoder of the text detection network are input into the decoder of the language classification network, and the language category of the text in the text region is output.

17. The method according to claim 15, wherein: The method of utilizing a language classification network to classify the language of the text in the text region in the text detection result and outputting the language category of the text in the text region includes: The text detection network outputs a plurality of text regions, wherein the language categories of the texts in the plurality of text regions are different; By switching decoder parameters of a text detection network, text detection is performed on text regions of different language categories; and / or, by switching decoder parameters of a language classification network, language classification is performed on text regions of different language categories.

18. An electronic device, wherein: The device includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and execute the steps of any one of the methods described in claims 1 to 17.

19. A computer storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

20. A computer program product, comprising: Computer program code, when the computer program code is run on a computer, causes the computer to execute the method according to any one of claims 1 to 17.

Images