Mobile application barrier-free subtitle recognition method and device based on double recognition positioning enhancement

Abstract

The invention discloses a mobile application barrier-free subtitle recognition method and device based on double recognition positioning enhancement, and the method comprises the steps: automatically collecting a mobile application barrier-free subtitle data set through an automatic traversal script, constructing a training and testing data set, and recording an error case in a training and testing process; carrying out image preprocessing optimization; on the basis of an error case generated in a training test of the optimized model, adopting a strategy of removing an underline and applying a segmentation algorithm based on pixel column continuity, modifying a processing flow according to the condition that characters are not recognized in a test result, and only calling a text content recognition module, so that the character content is recognized; and training a mobile application barrier-free subtitle text position target detection model based on YOLO, training a subtitle message box position detection model again based on YOLO, and carrying out a secondary identification method of positioning the position of a message box first and then positioning the position of a text. The method shows high robustness and accuracy under the conditions of background interference, overlong subtitles and multiple rows of subtitles.

Description

Technical field:

[0001] The invention relates to a mobile application barrier-free subtitle recognition method and device based on double recognition positioning enhancement, belonging to the field of computer vision. Background technology:

[0002] "Accessibility" comes from the English word Accessibility, which originally referred to providing accessible products, equipment, services and environments for people with all kinds of disabilities. In 2000, the Okinawa Charter for the Realization of a Global Information Society formally proposed the concept of "information accessibility" for the first time, defining it as anyone (whether able-bodied or not, young or old) being able to understand, interact and use information equally and conveniently under any circumstances. With the rapid development of mobile Internet, most network services have now shifted to mobile applications, and the focus of research and development on information accessibility has also shifted to mobile application accessibility. Statistics show that the total number of smartphone users worldwide has reached 3.9 billion in 2021, a year-on-year increase of 6.1%, indicating that mobile Internet has become an indispensable part of people's daily lives.

[0003] Although existing mobile application accessibility assistive technologies, such as the TalkBack function built into the Android operating system (Android) and the VoiceOver function built into the Apple operating system (iOS), have achieved certain results in improving the accessibility and comprehensibility of mobile application content, their use depends on developers' accurate acquisition of component functions and displayed text within the mobile application page. Developers need to implement functional expansion and accessibility detection based on the acquired text information to ensure that users can use mobile application content more accurately and conveniently. However, in practical applications, existing optical character recognition (OCR) technologies, such as PaddleOCR, have a high accuracy rate when recognizing general text, but still have many problems when processing accessible subtitles. Accessible subtitles usually have special display modes, such as fixed positions, specific fonts and colors, and possible underlines or background interference. These characteristics cause traditional OCR technologies to easily have problems such as inaccurate recognition, missed recognition, or misrecognition when recognizing accessible subtitles. In addition, the existence of long subtitles and multi-line subtitles also increases the difficulty of recognition. Traditional text position detection models often find it difficult to accurately locate the subtitle position, which in turn affects the effectiveness of subsequent text recognition modules. Summary of the invention:

[0004] In view of the above problems and difficulties, the present invention discloses a method and device for mobile application barrier-free subtitle recognition based on double recognition and positioning enhancement. The present invention improves the text recognition effect of barrier-free subtitles through image preprocessing optimization (including filling color blocks, binarization, image magnification), underline removal and image segmentation. At the same time, the YOLO model is used to train the barrier-free subtitle text position detection model and the barrier-free subtitle message box position detection model, and the subtitle position detection is further optimized through secondary recognition, thereby significantly improving the accuracy and robustness of barrier-free subtitle recognition for mobile applications.

[0005] A mobile application accessible subtitle recognition method based on double recognition positioning enhancement, the specific steps are as follows:

[0006] S101: Collect and generate mobile application accessible subtitles datasets through automated traversal scripts for model training and testing;

[0007] S102: performing image preprocessing on the collected accessible subtitle dataset, including filling color blocks, image binarization processing and image magnification, so as to improve the recognition accuracy of subtitle content;

[0008] S103: further processing the preprocessed data set, including removing the underline part in the image, and applying a segmentation algorithm based on pixel column continuity to reduce the phenomenon of missed recognition or misrecognition of long subtitles and multi-line subtitles;

[0009] S104: Modify the pipeline in the PaddleOCR source code to only call the text recognition module and remove other unnecessary processing steps to ensure the efficiency and pertinence of the recognition process;

[0010] S105: Use the YOLO model to train the accessible subtitle text position detection model to replace the text position detection module of PaddleOCR;

[0011] S106: according to the characteristics of barrier-free subtitles background interference, YOLO is used to train a barrier-free subtitles message box position detection model, and a double target recognition method is used to optimize the barrier-free subtitles position detection model;

[0012] S107: Input the accessible subtitle image into the subtitle message box position detection model, and after obtaining the subtitle message box position information, input it into the accessible subtitle text position detection model, and after obtaining the subtitle text position information, input it into the text recognition module of PaddleIOCR to obtain the final recognition result of accessible subtitle recognition.

[0013] The specific method of obtaining the accessible subtitles dataset described in step S101 is as follows: enable the voice-over function on an iOS device, or enable the TalkBack function on an Android device, so that the focus box focuses on the control and generates accessible subtitles at the bottom of the screen. Then, take a screenshot of the screen and capture the accessible subtitles part to form a dataset.

[0014] Furthermore, the image format is png format.

[0015] The specific implementation of the color block filling, image binarization and image magnification described in step S102 is as follows:

[0016] S1021: Fill the accessible subtitle image data with black blocks on the top, bottom, left and right sides;

[0017] S1022: converting pixel values ​​in the image to only two colors, black and white, wherein the subtitle text is white and the rest is black blocks;

[0018] S1023: Enlarge the width and height of the image by four times.

[0019] The specific method of removing the underline and applying the segmentation algorithm based on the continuity of pixel columns in step S103 is as follows:

[0020] The specific operation is to remove the underlined part contained in the image to avoid the interference of underline on text recognition; split the image into two parts, which is equivalent to splitting a long subtitle into two parts, and then splicing the recognition results after recognizing them separately. The split position is to detect the columns where the pixel values ​​change from non-zero to zero to the left of the middle of the image, calculate the column spacing, and divide according to the column with the largest spacing. At the same time, set a limit on the shortest spacing to avoid recognition errors caused by excessive segmentation.

[0021] Among them, step S104 locates the Pipeline part in the PaddleOCR source code, adjusts the model structure and configuration files, and only retains the text recognition module; modifies the training and prediction scripts to only call the text recognition model and remove the text detection module to ensure the efficiency and pertinence of the recognition process.

[0022] The specific method of training the accessible subtitle text position detection model by YOLO in step S105 is:

[0023] Analyze the width, spacing and centering information of the subtitle text in the picture, calculate the regional position of each character according to the subtitle text label and measurement data, and convert the text area data into the label format required by YOLO (category number, center point X, center point Y, width W, height H); organize the file structure and label file of the dataset according to the requirements of the YOLO official document, and use the organized dataset to train the YOLO model to improve the accuracy of subtitle position detection.

[0024] Among them, step S106 describes optimizing the accessible subtitle position detection model by using two target recognition methods according to the characteristics of accessible subtitle background interference. The specific steps include: measuring the width, height, text interval and pop-up box position of the text in the subtitle box, converting the message box area position into the YOLO label format for model training, thereby improving the recognition accuracy of the subtitle position and ensuring the accuracy and continuity of the recognition.

[0025] The second aspect of the present invention relates to a mobile application accessible subtitle recognition device based on double recognition and positioning enhancement, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the mobile application accessible subtitle recognition method based on double recognition and positioning enhancement of the present invention.

[0026] The third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the mobile application accessible subtitle recognition method based on double recognition and positioning enhancement of the present invention.

[0027] The present invention adopts a multi-step optimization process to improve the two sub-models of text content recognition and text position detection in mobile application barrier-free subtitle recognition. First, the mobile application barrier-free subtitle data set is automatically collected through an automated traversal script, training and test data sets are constructed, and error cases are recorded during the training and testing process. For the problems in text content recognition, image preprocessing optimization is performed, including color block filling, binarization processing and image magnification, to improve the recognition accuracy of subtitle text content. Subsequently, based on the error cases generated by the optimized model in the training test, the underline is further removed and the segmentation algorithm strategy based on pixel column continuity is applied to solve the problems of missed recognition and misrecognition of long subtitles and multi-line subtitles. Finally, according to the situation of unrecognized characters appearing in the test results, the processing flow is modified to call only the text content recognition module, and a mobile application barrier-free subtitle text position target detection model is trained based on YOLO to replace the text position detection submodule in the traditional OCR method to obtain the subtitle text position, thereby further improving the accuracy and stability of subtitle recognition. In addition, in view of the characteristics of barrier-free subtitles, a subtitle message box position detection model is trained again based on YOLO. The accuracy and stability of subtitle position recognition are improved by first locating the message box position and then locating the text position. The experimental results show that after multiple optimizations and model adjustments, the full-text matching accuracy of the method of the present invention on the test set is improved from the initial 25.61% to 99.22%. This technical solution is suitable for subtitle recognition in barrier-free mode of iOS and Android mobile applications, especially in the case of background interference, too long subtitles and multi-line subtitles, showing high robustness and accuracy.

[0028] In summary, the present invention has the following beneficial effects: (1) It realizes high-precision recognition of accessible subtitle text content of mobile applications, thereby obtaining accurate application control text information; (2) It optimizes the recognition effect of long subtitles and multi-line subtitles, and reduces the influence of background interference on accessible subtitle recognition; (3) It is universal and applicable to mobile applications of different platforms and models. Description of the drawings:

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 It is the overall flow chart of the method of the present invention;

[0031] Figure 2It is a schematic diagram of an image after color block filling, binarization, and image enlargement processing in the method of the present invention;

[0032] Figure 3 is a schematic diagram of an image after the underline is removed in the method of the present invention;

[0033] Figure 4 It is a schematic diagram of the device of the method of the present invention; Specific implementation method:

[0035] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0036] Example 1

[0037] This embodiment relates to a mobile application barrier-free subtitle recognition method based on double recognition positioning enhancement. This example takes a mobile application (APP) as an example. Figure 1-Figure 3 The specific steps include:

[0038] S101: Enable the voice-over function on the iOS device, or enable the TalkBack function on the Android device, so that the focus box is focused on the control and accessible subtitles are generated at the bottom of the screen. Take a screenshot of the screen and capture the accessible subtitle part to form an accessible subtitle dataset;

[0039] S102: preprocessing the collected accessible subtitle dataset, including filling black blocks in the four directions of the subtitle image, converting the image pixel values ​​into black and white (the subtitle text is white and the rest is black) to highlight the text information and reduce background interference; enlarging the width and height of the image by four times to improve the image resolution. The effect after preprocessing is as follows: Figure 2 As shown;

[0040] S103: Remove the underlined part contained in the image, such as Figure 3 As shown in the figure, to avoid the interference of underline on text recognition; from the middle of the image to the left, detect multiple columns where the pixel columns change from non-zero to zero, and calculate the distance between adjacent non-zero columns. Split according to the column with the largest spacing, split the long subtitle into left and right parts, recognize them separately, and then splice the recognition results. Set a limit on the shortest spacing to avoid recognition errors caused by excessive segmentation;

[0041] S104: Locate the Pipeline part in the PaddleOCR source code, adjust the model structure so that it only contains the text recognition module; modify the configuration file to ensure that only necessary models and parameters are loaded; adjust the training and prediction scripts to only call the text recognition model and avoid loading the text detection module, thereby ensuring the efficiency and pertinence of the recognition process;

[0042] S105: using the YOLO model to train an accessible subtitle text position detection model, the specific operations include: measuring the width and spacing of the characters in the accessible subtitle picture and the centering information of the subtitles in the picture; calculating the area position of each character according to the subtitle text label and the measurement data, and converting the text area data into the label format required by YOLO (category number, center point X, center point Y, width W, height H); organizing the file structure and label file of the data set according to the requirements of the YOLO official document, and using the sorted data set to train the YOLO model to improve the accuracy of subtitle position detection;

[0043] S106: using the YOLO model to train an accessible subtitle message box position detection model, specifically, measuring information such as the width, height, text interval, and pop-up box position of the text in the subtitle box, converting the message box area position into a YOLO label format for model training, thereby improving the recognition accuracy of the subtitle position and ensuring the accuracy and continuity of recognition;

[0044] S107: Input the preprocessed accessible subtitle image into the optimized subtitle position detection model, and process it using a double recognition method. In the first recognition, the preprocessed subtitle image is input into the accessible subtitle message box position detection model to identify and locate the position of the message box, thereby determining the position of the entire subtitle area; in the second recognition, within the located subtitle area, it is further input into the accessible subtitle text position detection model to identify the specific text position and obtain the text position information within the subtitle area; the text block position information obtained above is input into the text recognition module of PaddleOCR to recognize the text content, and finally obtain accurate accessible subtitle recognition results.

[0045] Example 2

[0046] like Figure 4 This embodiment relates to a mobile application accessible subtitle recognition device based on double recognition and positioning enhancement, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the mobile application accessible subtitle recognition method based on double recognition and positioning enhancement of Example 1.

[0047] Example 3

[0048] The present embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method for mobile application accessible subtitle recognition based on double recognition and positioning enhancement of embodiment 1 is implemented.

[0049] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0050] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A mobile application accessible subtitle recognition method based on double recognition and positioning enhancement, characterized in that: The following steps are involved: S101: Obtain mobile application accessible subtitles dataset through automated traversal scripts for training and testing; S102: performing image preprocessing on the collected data set, including color block filling, image binarization processing and image magnification, so as to improve the recognition accuracy of the subtitle content; S103: further performing image preprocessing on the data set, including cutting off underlines in the image and applying a segmentation algorithm based on pixel column continuity to reduce the phenomenon of long subtitles and multi-line subtitles not being recognized or being misrecognized; S104: Modify the pipeline in the PaddleOCR source code to only call the text recognition module; S105: Use YOLO to train the accessible subtitle text position detection model to replace the text position detection module of PaddleOCR; S106: according to the characteristics of barrier-free subtitles background interference, YOLO is used to train a barrier-free subtitles message box position detection model, and a double target recognition method is used to optimize the barrier-free subtitles position detection model; S107: Input the accessible subtitle image into the subtitle message box position detection model, and after obtaining the subtitle message box position information, input it into the accessible subtitle text position detection model, and after obtaining the subtitle text position information, input it into the text recognition module of PaddleIOCR to obtain the final recognition result of accessible subtitle recognition.

2. A mobile application accessible subtitle recognition method based on double recognition and positioning enhancement according to claim 1, characterized in that: The specific method of obtaining the accessible subtitle dataset in step S101 is as follows: When the accessible narration function is enabled on an iOS device, or the TalkBack function is enabled in an Android mobile application, the focus box is focused on the control and accessible subtitles are generated at the bottom of the screen. At this time, a screenshot is taken of the screen and the accessible subtitle part in the image is captured to form an accessible subtitle dataset.

3. A mobile application accessible subtitle recognition method based on double recognition and positioning enhancement according to claim 1, characterized in that: The specific implementation of the color block filling, image binarization and image enlargement described in step S102 is as follows: S1021: Fill the accessible subtitle image data with black blocks on the top, bottom, left and right sides; S1022: converting pixel values ​​in the image to only two colors, black and white, wherein the subtitle text is white and the rest is black blocks; S1023: Enlarge the width and height of the image by four times.

4. A mobile application accessible subtitle recognition method based on double recognition and positioning enhancement according to claim 1, characterized in that: The specific method of removing the underline and applying the segmentation algorithm based on the continuity of pixel columns in step S103 is as follows: S1032: split the image into two parts, which is equivalent to splitting the long subtitle into two parts, and then splicing the recognition results together after recognizing them separately. The specific segmentation position is from the middle to the left, detecting the transition columns where multiple pixel columns change from non-zero values ​​to zero values, calculating the distance between adjacent columns, and dividing according to the column with the largest spacing, while limiting the shortest spacing.

5. A mobile application accessible subtitle recognition method based on double recognition and positioning enhancement according to claim 1, characterized in that: Step S104 specifically includes: locating the Pipeline part in the PaddleOCR source code, adjusting the model structure and configuration files, and retaining only the text recognition module; modifying the training and prediction scripts so that they only call the text recognition model and remove the text detection module to ensure the efficiency and pertinence of the recognition process.

6. A mobile application accessible subtitle recognition method based on double recognition and positioning enhancement according to claim 1, characterized in that: The specific method of training the accessible subtitle text position detection model through YOLO in step S105 is: S1051: Analyze the accessible subtitle image, determine the width and spacing of the text, and ensure that the text is centered in the image; S1052: Obtain the number of characters according to the text label, and calculate the character area in combination with step S1051; S1053: Convert the text area data into YOLO format, which is the category number, center point X coordinate, center point Y coordinate, width W, and height H in order according to the official document requirements; S1054: Organize the dataset structure according to the YOLO documentation requirements; S1055: Train the YOLO model.

7. A mobile application accessible subtitle recognition method based on double recognition and positioning enhancement according to claim 1, characterized in that: According to the characteristics of barrier-free subtitles background interference in step S106, the barrier-free subtitles position detection model is optimized by using a two-target recognition method. The specific method is: S1061: measuring the width, height, text interval and starting height of the subtitle pop-up box, calculating the position of the message box area based on the above information and the subtitle text, and converting the four-point position information into a YOLO label format for subsequent YOLO target detection model training; S1062: Using a double target recognition method, first recognize the position of the message box, then recognize the position of the text block in the message box, use the text recognition module of PaddleOCR to recognize the text in the text block, and splice the recognition results in height order.

8. A mobile application accessible subtitle recognition method based on double recognition and positioning enhancement according to claim 2, characterized in that: The specific requirements for the screenshots are: The image format is png format.

9. A mobile application barrier-free subtitle recognition device based on double recognition and positioning enhancement, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the mobile application accessible subtitle recognition method based on double recognition and positioning enhancement as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the method for mobile application accessible subtitle recognition based on double recognition and positioning enhancement as described in any one of claims 1-8 is implemented.

Images