A mobile application accessibility subtitle recognition method and device based on double-recognition positioning enhancement

Abstract

A mobile application accessibility subtitle recognition method and device based on double identification positioning enhancement, the method comprising: automatically collecting mobile application accessibility subtitle data sets by automatically traversing scripts, constructing training and test data sets, and recording error cases during the training and test process; image preprocessing optimization; based on the error cases produced in the training and test of the optimized model, removing underlines and applying a cutting algorithm strategy based on pixel column continuity; according to the unrecognized character situation in the test result, modifying the processing flow, calling only the text content recognition module, and based on YOLO, training a mobile application accessibility subtitle text position target detection model based on YOLO, and training a subtitle message box position detection model again, through the double identification method of positioning the message box position first and then positioning the text position. The present application shows high robustness and accuracy under the conditions of background interference, long subtitles and multiple line subtitles.

Description

Technical fields:

[0001] This invention relates to a method and apparatus for mobile application accessibility caption recognition based on dual-recognition positioning enhancement, belonging to the field of computer vision. Background technology:

[0002] The term "accessibility" originates from the English word "accessibility," initially referring to providing accessible products, equipment, services, and environments for people with disabilities. In 2000, the Okinawa Charter for a Global Information Society formally introduced the concept of "information accessibility," defining it as the ability of anyone (regardless of ability, age) to understand, interact with, and utilize information equally and conveniently in all circumstances. With the rapid development of the mobile internet, most online services have now fully shifted to mobile applications, and the focus of research and development in information accessibility has also shifted to mobile application accessibility. Statistics show that in 2021, the total number of smartphone users worldwide reached 3.9 billion, a year-on-year increase of 6.1%, indicating that the mobile internet has become an indispensable part of people's daily lives.

[0003] While existing mobile application accessibility technologies, such as Android's TalkBack and iOS's VoiceOver, have achieved some success in improving the accessibility and comprehensibility of mobile application content, their use relies on developers' accurate acquisition of the functionality of components and the displayed text within the mobile application page. Developers need to implement functional extensions and accessibility detection based on the acquired text information to ensure that users can use mobile application content more accurately and conveniently. However, in practical applications, existing optical character recognition (OCR) technologies, such as PaddleOCR, while having high accuracy in recognizing general text, still have many problems when processing accessible captions. Accessible captions often have special display methods, such as fixed positions, specific fonts and colors, and the possibility of underlines or background interference. These characteristics make traditional OCR technologies prone to inaccurate recognition, missed recognition, or misrecognition when recognizing accessible captions. In addition, the existence of long captions and multi-line captions also increases the difficulty of recognition; traditional text position detection models often struggle to accurately locate the caption position, thus affecting the effectiveness of subsequent text recognition modules. Summary of the Invention:

[0004] To address the aforementioned problems and challenges, this invention discloses a method and apparatus for mobile application accessibility caption recognition based on dual-recognition localization enhancement. This invention improves the text recognition performance of accessibility captions through image preprocessing optimization (including color block filling, binarization, and image magnification), underline removal, and image segmentation. Simultaneously, it trains an accessibility caption text position detection model and an accessibility caption message box position detection model using a YOLO model, and further optimizes caption position detection through secondary recognition, thereby significantly improving the accuracy and robustness of mobile application accessibility caption recognition.

[0005] A mobile application accessibility caption recognition method based on dual-recognition positioning enhancement, the specific steps of which are as follows:

[0006] S101: Collect and generate mobile application accessibility caption datasets through automated traversal scripts for model training and testing;

[0007] S102: Perform image preprocessing on the collected accessible caption dataset, including filling color blocks, image binarization, and image magnification, to improve the recognition accuracy of caption content;

[0008] S103: Further process the preprocessed dataset, including removing the underlined parts in the images and applying a segmentation algorithm based on pixel column continuity to reduce the omission or misidentification of long and multi-line subtitles.

[0009] S104: Modify the pipeline in the PaddleOCR source code to only call the text recognition module and remove other unnecessary processing steps to ensure the efficiency and relevance of the recognition process;

[0010] S105: The YOLO model is used to train the accessible caption text position detection model, replacing the text position detection module of PaddleOCR;

[0011] S106: Based on the characteristics of background interference in accessible subtitles, a YOLO-trained accessible subtitle message box position detection model is used, and a dual-target recognition method is adopted to optimize the accessible subtitle position detection model.

[0012] S107: Input the accessible caption image into the caption message box position detection model to obtain the caption message box position information, then input it into the accessible caption text position detection model to obtain the caption text position information, and then input it into the text recognition module of PaddleIOCR to obtain the final recognition result of accessible caption recognition.

[0013] The specific method for obtaining the accessible caption dataset in step S101 is as follows: Enable the narration function on an iOS device or the TalkBack function on an Android device, causing the focus box to focus on the control and generate accessible captions at the bottom of the screen. Then, take a screenshot of the screen and extract the accessible caption portion to form the dataset.

[0014] Furthermore, the image format is PNG.

[0015] The specific implementation methods for color block filling, image binarization, and image magnification in step S102 are as follows:

[0016] S1021: Fill the accessible caption image data with black blocks in all directions;

[0017] S1022: Convert the pixel values ​​in the image to only black and white, where the subtitle text is white and the rest is black;

[0018] S1023: Enlarge the width and height of the image by a factor of four.

[0019] The specific method for removing underlines and applying a pixel column continuity-based segmentation algorithm in step S103 is as follows:

[0020] The specific steps involve removing underlined portions from the image to avoid interference with text recognition; splitting the image into left and right parts, essentially dividing the long subtitle into two parts, recognizing each part separately, and then stitching the recognition results together. The segmentation location is determined by detecting columns where multiple pixel values ​​change from non-zero to zero from the center of the image to the left, calculating the column spacing, and dividing the image based on the column with the largest spacing. A limit is also set on the shortest spacing to avoid over-segmentation that could lead to recognition errors.

[0021] In step S104, the Pipeline part of the PaddleOCR source code is located, the model structure and configuration file are adjusted, and only the text recognition module is retained; the training and prediction scripts are modified so that they only call the text recognition model and the text detection module is removed to ensure the efficiency and specificity of the recognition process.

[0022] The specific method for training the accessible caption text location detection model using YOLO in step S105 is as follows:

[0023] This study analyzes the width, spacing, and centering information of the subtitle text in the image. Based on the subtitle text labels and measurement data, it calculates the region position of each character and converts the text region data into the label format required by YOLO (category number, center point X, center point Y, width W, height H). Following the requirements of the official YOLO documentation, it organizes the file structure and label files of the dataset and uses the organized dataset to train the YOLO model to improve the accuracy of subtitle position detection.

[0024] In step S106, the accessible subtitle position detection model is optimized by using two target recognition methods based on the characteristics of background interference in accessible subtitles. The specific steps include: measuring the width, height, text spacing, and pop-up position of the text in the subtitle box, converting the message box area position into YOLO label format for model training, thereby improving the recognition accuracy of the subtitle position and ensuring the accuracy and continuity of the recognition.

[0025] A second aspect of the present invention relates to a mobile application accessibility caption recognition device based on dual-recognition positioning enhancement, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the mobile application accessibility caption recognition method based on dual-recognition positioning enhancement of the present invention.

[0026] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the mobile application accessibility caption recognition method based on dual-recognition positioning enhancement of the present invention.

[0027] This invention employs a multi-step optimization process to improve two sub-models in mobile application accessibility caption recognition: text content recognition and text position detection. First, an automated traversal script automatically collects a mobile application accessibility caption dataset, constructing training and testing datasets, and recording error cases during the training and testing process. To address issues in text content recognition, image preprocessing optimization is performed, including color block filling, binarization, and image magnification to improve the recognition accuracy of caption text content. Subsequently, based on error cases generated during training and testing of the optimized model, further strategies are adopted, such as underline removal and the application of a pixel column continuity-based segmentation algorithm, to solve the problems of missed and false recognition of long and multi-line captions. Finally, based on instances of unrecognized characters in the test results, the processing flow is modified to only call the text content recognition module, and a mobile application accessibility caption text position target detection model is trained based on YOLO to replace the text position detection sub-module in traditional OCR methods, thereby further improving the accuracy and stability of caption recognition. Furthermore, considering the characteristics of accessible subtitles, a subtitle message box position detection model was retrained based on YOLO. This secondary recognition method, which first locates the message box position and then the text position, improves the accuracy and stability of subtitle position recognition. Experimental results show that, after multiple optimizations and model adjustments, the full-text matching accuracy of the proposed method on the test set increased from the initial 25.61% to 99.22%. This technical solution is suitable for subtitle recognition in accessible modes of iOS and Android mobile applications, exhibiting high robustness and accuracy, especially in cases of background interference, excessively long subtitles, and multi-line subtitles.

[0028] In summary, the present invention has the following beneficial effects: (1) it achieves high-precision recognition of the text content of accessible subtitles in mobile applications, thereby obtaining accurate application control text information; (2) it optimizes the recognition effect of long subtitles and multi-line subtitles, and reduces the impact of background interference on accessible subtitle recognition; (3) it has universality. It is applicable to mobile applications on different platforms and different models. Attached image description:

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is the overall flowchart of the method of the present invention;

[0031] Figure 2This is a schematic diagram of the image after color patch filling, binarization, and image magnification in the method of this invention;

[0032] Figure 3 This is a schematic diagram of the image after the underline removal process in the method of the present invention;

[0033] Figure 4 This is a schematic diagram of the apparatus used in the method of the present invention; Specific implementation methods:

[0034] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0035] Example 1

[0036] This embodiment relates to a mobile application accessibility caption recognition method based on dual-recognition positioning enhancement. This example uses a specific mobile application (APP) as an example. Figures 1-3 The specific steps include the following:

[0037] S101: Enable the VoiceOver feature on iOS devices or the TalkBack feature on Android devices, causing the focus box to focus on the control and generating accessible captions at the bottom of the screen. Take a screenshot of the screen, extract the accessible caption portion, and create an accessible caption dataset;

[0038] S102: Preprocess the collected accessible caption dataset, including filling the caption image with black blocks in the top, bottom, left, and right directions; converting the image pixel values ​​to black and white (with caption text in white and the rest in black) to highlight the text information and reduce background interference; and enlarging the image width and height by a factor of four to improve the image resolution. The preprocessed effect is as follows: Figure 2 As shown;

[0039] S103: Remove the underlined parts from the image, such as... Figure 3 As shown, to avoid underline interference with text recognition, multiple pixel columns that change from non-zero to zero are detected from the center of the image to the left, and the distance between adjacent non-zero columns is calculated. Long subtitles are split into left and right parts according to the column with the largest spacing, recognized separately, and then the recognition results are stitched together. A limit is set on the shortest spacing to avoid over-segmentation leading to recognition errors.

[0040] S104: Locate the Pipeline section in the PaddleOCR source code, adjust the model structure to include only the text recognition module; modify the configuration file to ensure that only the necessary models and parameters are loaded; adjust the training and prediction scripts to only call the text recognition model and avoid loading the text detection module, thereby ensuring the efficiency and specificity of the recognition process.

[0041] S105: Train an accessible caption text location detection model using the YOLO model. Specific operations include: measuring the width, spacing, and centering information of the text in the accessible caption image; calculating the region location of each character based on the caption text labels and measurement data, and converting the text region data into the label format required by YOLO (category number, center point X, center point Y, width W, height H); organizing the dataset file structure and label files according to the requirements of the official YOLO documentation, and using the organized dataset to train the YOLO model to improve the accuracy of caption location detection.

[0042] S106: The YOLO model is used to train the accessibility caption message box position detection model. The specific operations include: measuring the width, height, text spacing and pop-up position of the text in the caption box, converting the message box area position into YOLO label format for model training, thereby improving the recognition accuracy of the caption position and ensuring the accuracy and continuity of the recognition.

[0043] S107: The preprocessed accessible subtitle image is input into the optimized subtitle position detection model. A dual recognition method is used for processing. In the first recognition, the preprocessed subtitle image is input into the accessible subtitle message box position detection model to identify and locate the position of the message box, thereby determining the position of the entire subtitle area. In the second recognition, within the located subtitle area, it is further input into the accessible subtitle text position detection model to identify the specific text position and obtain the text position information within the subtitle area. The obtained text block position information is then input into the text recognition module of PaddleOCR to recognize the text content, ultimately obtaining an accurate accessible subtitle recognition result.

[0044] Example 2

[0045] like Figure 4 This embodiment relates to a mobile application accessibility caption recognition device based on dual-recognition positioning enhancement, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the mobile application accessibility caption recognition method based on dual-recognition positioning enhancement of Embodiment 1.

[0046] Example 3

[0047] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements the mobile application accessibility caption recognition method based on dual-recognition positioning enhancement as described in Embodiment 1.

[0048] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0049] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A mobile application accessibility caption recognition method based on dual-recognition positioning enhancement, characterized in that, Includes the following steps: S101: Obtain the mobile application accessibility caption dataset through an automated traversal script for training and testing; S102: Perform image preprocessing on the collected dataset, including color block filling, image binarization, and image magnification, to improve the recognition accuracy of subtitle content; S103: Further image preprocessing of the dataset, including removing underlines from images and applying a pixel column continuity-based segmentation algorithm to reduce the phenomenon of unrecognized or misrecognized long and multi-line subtitles; specifically including: S1031: Remove the underlined portion from the image; S1032: Split the image into left and right parts, which is equivalent to dividing the long subtitle into two parts, recognizing them separately, and then stitching the recognition results together; The specific segmentation position is from the middle to the left, detecting multiple pixel columns that change from non-zero values ​​to zero values, calculating the distance between adjacent columns, and dividing according to the column with the largest distance, while restricting the shortest distance. S104: Modify the pipeline in the PaddleOCR source code to only call the text recognition module; S105: Use YOLO to train an accessible caption text location detection model to replace the text location detection module of PaddleOCR; S106: Based on the characteristics of background interference in accessible subtitles, a YOLO-based model for detecting the location of accessible subtitle message boxes is trained, and a dual-target recognition method is used to optimize the model; specifically including: S1061: Measure the width, height, text spacing, and starting height of the subtitle pop-up. Based on the above information and the subtitle text, calculate the position of the message box area and convert the four position information into YOLO label format for subsequent YOLO object detection model training. S1062: A dual target recognition method is adopted. First, the position of the message box is recognized, then the position of the text block in the message box is recognized. The text recognition module of PaddleOCR is used to recognize the text in the text block, and the recognition results are spliced ​​together in height order. S107: Input the accessible caption image into the caption message box position detection model to obtain the caption message box position information, then input it into the accessible caption text position detection model to obtain the caption text position information, and then input it into the text recognition module of PaddleOCR to obtain the final recognition result of accessible caption recognition.

2. The mobile application accessibility caption recognition method based on dual-recognition positioning enhancement according to claim 1, characterized in that, The specific method for obtaining the accessible caption dataset mentioned in step S101 is as follows: When the TalkBack feature is enabled on an iOS device or in an Android mobile application, focus the viewfinder on the control and generate accessible captions at the bottom of the screen. Take a screenshot of the screen and capture the accessible caption portion of the image to create an accessible caption dataset.

3. The mobile application accessibility caption recognition method based on dual-recognition positioning enhancement according to claim 1, characterized in that: The specific implementation methods for color block filling, image binarization, and image magnification described in step S102 are as follows: S1021: Fill the accessible caption image data with black blocks in all directions; S1022: Convert the pixel values ​​in the image to only black and white, where the subtitle text is white and the rest is black; S1023: Enlarge the width and height of the image by a factor of four.

4. The mobile application accessibility caption recognition method based on dual-recognition positioning enhancement according to claim 1, characterized in that: Step S104 specifically includes: locating the Pipeline part in the PaddleOCR source code, adjusting the model structure and configuration file, and retaining only the text recognition module; modifying the training and prediction scripts so that they only call the text recognition model, removing the text detection module, and ensuring the efficiency and specificity of the recognition process.

5. The mobile application accessibility caption recognition method based on dual-recognition positioning enhancement according to claim 1, characterized in that: The specific method for training the accessible caption text location detection model using YOLO in step S105 is as follows: S1051: Analyze accessible caption images, determine the character width and spacing, and ensure that the text is centered in the image; S1052: Based on the text label, obtain the number of characters, and combine with step S1051 to calculate the text region; S1053: Convert the text area data to YOLO format, following the official documentation requirements as follows: category number, center point X coordinate, center point Y coordinate, width W, height H; S1054: Organize the dataset structure according to the YOLO documentation requirements; S1055: Train the YOLO model.

6. The mobile application accessibility caption recognition method based on dual-recognition positioning enhancement according to claim 2, characterized in that, The specific requirements for the screenshots are as follows: The image format is PNG.

7. A mobile application accessibility caption recognition device based on dual-recognition positioning enhancement, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the mobile application accessibility caption recognition method based on dual-recognition positioning enhancement as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the mobile application accessibility caption recognition method based on dual-recognition positioning enhancement as described in any one of claims 1-6.

Images