A method and system for monitoring and relocating screen text in three states

Abstract

The application discloses a screen text three-state monitoring and relocation method and system. The method obtains a screen image frame and a target area description, obtains target area coordinates and recognizes text through a visual language positioning model, and generates area state records binding the coordinates, the description, the text, the confidence, the resolution and the version. Subsequent frames continue to be recognized, and according to the text difference, the OCR confidence and the target text validity condition, the state is judged as normal content update, recognition exception or area failure. Only when the area failure occurs, the relocation is triggered, and the new coordinates are written back to the state record after being accepted by the OCR validity.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, screen monitoring and optical character recognition, and in particular to a method and system for three-state monitoring and relocation of screen text. Background Technology

[0002] In software automation testing, point-of-sale (POS) system monitoring, content moderation, and accessibility applications, it is often necessary to continuously read the text content of specific areas on electronic screens. For example, in a POS terminal, it is necessary to monitor the product name, the actual payment amount, or the amount after a coupon is applied; in software testing scenarios, it is necessary to monitor pop-ups, CAPTCHA areas, or status prompt areas. Screen text monitoring typically includes screen image acquisition, area localization, and OCR text recognition.

[0003] Existing text recognition solutions combining RPA and AI allow RPA robots to acquire images and send them to an AI platform, which then detects table regions, cell positions, and cell text. These solutions are suitable for processing text recognition in static or structured images, but their focus is on recognizing text regions or table structures, and they do not establish a runtime feedback mechanism to address the issue of persistently invalid target region coordinates in dynamic screens.

[0004] Existing solutions for RPA element anchor point finding or repair based on multimodal models can identify anchor elements using the coordinates, text, screenshots, and categories of webpage DOM elements, or repair elements based on anchor point relationships when the interface changes. These solutions can improve the stability of webpage element or RPA element positioning, but they typically handle enumerable DOM elements, candidate elements, and anchor point relationships, and do not directly address continuous text monitoring across arbitrary screen areas.

[0005] Another video OCR evaluation scheme can perform cross-frame matching of text recognition targets and manually labeled targets in multiple video frames to evaluate the performance of the OCR system. This type of scheme demonstrates that cross-frame text target matching already has applications, but its purpose is to evaluate the OCR system offline, rather than to control screen area relocation at runtime.

[0006] During dynamic screen text monitoring, screen content may scroll, switch windows, be obscured by pop-ups, have resolution changes, or undergo partial refreshes. At this time, text changes within the target area could be normal business content changes, short-term OCR recognition anomalies, or even invalidation of the original target area coordinates. If the system only triggers relocation based on text changes, it may easily misjudge normal business changes as area failures; if the system only triggers relocation when OCR fails, it may continue erroneous monitoring even when the old coordinates have deviated but non-target text is still recognized. Therefore, a technical solution is needed that can bind the target area coordinates, target area description, and recognized text to the running state, and perform state differentiation and relocation control based on subsequent recognition results. Summary of the Invention

[0007] This invention provides a method and system for three-state monitoring and relocation of screen text, which improves the problem of confusion between normal changes in target area content, short-term OCR recognition anomalies, and target area coordinate failure during dynamic continuous screen text monitoring. This invention binds target area coordinates, target area description information, recognized text, OCR confidence, screen image frame resolution, and time information into a region state record with a region state version. Based on the text differences of subsequent recognized text, OCR confidence, and target text validity conditions, the system performs three-state discrimination, enabling it to perform differentiated processing between normal content updates, recognition anomalies, and region failures, and triggering visual relocation only when a region fails.

[0008] The screen text three-state monitoring and relocation method provided by this invention includes: acquiring a current screen image frame and target region description information, wherein the target region description information is used to describe the semantic target of the screen region to be monitored; inputting the current screen image frame and the target region description information into a visual language localization model to obtain target region coordinates, wherein the visual language localization model is used to output image region coordinates corresponding to the text description based on image content and text description; cropping the target region image from the current screen image frame according to the target region coordinates, and performing OCR text recognition on the target region image to obtain the currently recognized text and the current OCR position. The system determines the validity conditions of the target text based on the target region description information, and generates a region status record with a region status version based on the target region coordinates, the target region description information, the currently recognized text, the current OCR confidence score, the current screen image frame resolution, and time information. It then acquires subsequent screen image frames, performs region cropping and OCR text recognition on the subsequent screen image frames based on the target region coordinates in the region status record, and obtains the subsequent recognized text and subsequent OCR confidence score. Finally, it calculates the text difference between the subsequent recognized text and the currently recognized text in the region status record, and the subsequent OCR confidence score. The system determines whether the subsequently recognized text meets the target text validity condition, and classifies the current monitoring state as a normal content update state, an abnormal recognition state, or a region failure state. When the current monitoring state is a normal content update state, the recognized text, OCR confidence score, and time information in the region status record are updated, while keeping the target region coordinates unchanged. When the current monitoring state is an abnormal recognition state, region cropping and OCR text recognition are re-executed based on the target region coordinates in the region status record to obtain a re-recognition result. Based on the re-recognition result, the current monitoring state is converted to a normal content update state and maintained as an abnormal recognition state. An abnormal state or a transition to a region failure state occurs. When the current monitoring state is a region failure state, a visual relocation process is triggered to clear the positioning context related to the target region coordinates in the region status record. The target region coordinates are then re-determined based on the subsequent screen image frames or re-acquired screen image frames and the target region description information. OCR text recognition is performed based on the re-determined target region coordinates. When the re-recognized text meets the target text validity condition and the corresponding OCR confidence meets the preset confidence condition, the re-determined target region coordinates, the re-recognized text, and the corresponding OCR confidence are written back as a new region status record.

[0009] Furthermore, determining the target text validity condition based on the target region description information includes: determining whether there is a user configuration validity condition corresponding to the target region description information; if the user configuration validity condition exists, then the user configuration validity condition is used as a candidate text validity condition; if the user configuration validity condition does not exist, then semantic type recognition is performed on the target region description information to obtain the target region type, and candidate text validity conditions are called according to the target region type; the currently recognized text is matched with the candidate text validity conditions; when the currently recognized text satisfies one of the candidate text validity conditions and the current OCR confidence satisfies a preset confidence condition, the candidate text validity condition is determined as the target text validity condition; when multiple candidate text validity conditions are satisfied, the candidate text validity condition with the highest matching degree is selected as the target text validity condition based on the matching degree between the target region description information and the region type corresponding to each candidate text validity condition.

[0010] Furthermore, the three-state discrimination includes: normalizing the current identified text and the subsequent identified text according to the text type corresponding to the target text validity condition; calculating the text difference based on the normalized current identified text and the subsequent identified text; determining whether the subsequent identified text meets the target text validity condition; when the subsequent identified text meets the target text validity condition and the subsequent OCR confidence meets the preset confidence condition, marking the text difference as a valid content difference; when the subsequent identified text is not empty and does not meet the target text validity condition, and the subsequent OCR confidence meets the preset confidence condition, marking the text difference as a regional semantic deviation difference; when the subsequent OCR confidence does not meet the preset confidence condition or the subsequent identified text is empty, marking the text difference as an unusable identification difference; and determining the current monitoring state based on the valid content difference, regional semantic deviation difference, or unusable identification difference.

[0011] Furthermore, determining the current monitoring state based on the valid content difference, regional semantic deviation difference, or identification unavailability difference includes: when the text difference is marked as a valid content difference, determining the current monitoring state as a normal content update state; when the text difference is marked as an identification unavailability difference, writing the state of the corresponding subsequent screen image frame into the abnormal frame cache, and counting the number of consecutive abnormal frames recorded under the same regional state; when the number of consecutive abnormal frames does not reach a preset number of abnormal frames, determining the current monitoring state as an identification abnormal state; when the number of consecutive abnormal frames reaches the preset number of abnormal frames, changing the current monitoring state from the identification abnormal state to the regional failure state; when the text difference is marked as a regional semantic deviation difference, determining the current monitoring state as a regional failure state.

[0012] Furthermore, when the current monitoring status is a normal content update status, the subsequent identified text is compared with the current identified text in the area status record; when the subsequent identified text is different from the current identified text, and the subsequent identified text meets the target text validity condition, the subsequent identified text is written into the text update field of the area status record; the subsequent OCR confidence and the time information of the subsequent screen image frame are written into the area status record; the target area coordinates and positioning status in the area status record remain unchanged; and the number of consecutive abnormal frames under the same area status record is cleared.

[0013] Furthermore, when the current monitoring state is an abnormal identification state, without changing the target area coordinates in the area status record, at least one supplementary screen image frame is reacquired according to a preset short-time sampling interval; supplementary target area images are cropped from the at least one supplementary screen image frame based on the target area coordinates; OCR text recognition is performed on each supplementary target area image to obtain supplementary recognized text and supplementary OCR confidence; from the supplementary recognized texts that meet the target text validity condition and whose supplementary OCR confidence meets the preset confidence condition, the supplementary recognized text with the highest supplementary OCR confidence is selected as the recovered recognized text; if the recovered recognized text exists, the current monitoring state is converted to a normal content update state, the area status record is updated based on the recovered recognized text, and the number of consecutive abnormal frames under the same area status record is cleared; if the recovered recognized text does not exist, the number of consecutive abnormal frames under the same area status record is updated; when the number of consecutive abnormal frames does not reach the preset number of abnormal frames, the current monitoring state is maintained as an abnormal identification state; when the number of consecutive abnormal frames reaches the preset number of abnormal frames, the current monitoring state is converted from an abnormal identification state to an area failure state.

[0014] Furthermore, when the current monitoring state is a region failure state, the cause of the region failure is determined, and the cause of the region failure includes at least one of the following: target text validity condition not met, continuous recognition abnormality, resolution mapping failure; the scope of clearing the positioning context is determined according to the cause of the region failure; the positioning context related to the target region coordinates in the failure region status record is cleared; the target region description information and the target text validity condition are retained; a relocation input is constructed based on the retained target region description information and target text validity condition; the relocation input and the subsequent screen image frame or the reacquired screen image frame are input into the visual language positioning model to obtain at least one candidate relocation region coordinate.

[0015] Furthermore, the system crops candidate relocation region images based on the coordinates of at least one candidate relocation region; performs OCR text recognition on each candidate relocation region image to obtain candidate recognized text and candidate OCR confidence; determines whether each candidate recognized text meets the target text validity condition; generates candidate acceptance results for candidate relocation region coordinates that meet the target text validity condition, the candidate acceptance results being determined at least based on the candidate OCR confidence and the result of the candidate recognized text meeting the target text validity condition; selects target acceptance region coordinates from candidate relocation region coordinates whose candidate acceptance results meet preset acceptance conditions; writes the target acceptance region coordinates, corresponding candidate recognized text, corresponding candidate OCR confidence, current time information, and incremented region status version into a new region status record; if there are no candidate relocation region coordinates that meet the preset acceptance conditions, the original region status record is not overwritten, and the system proceeds to the next visual relocation retry or manual takeover process.

[0016] Furthermore, after acquiring subsequent screen image frames, the screen image frame resolution information and target region coordinates in the region status record are read; it is determined whether the resolution of the subsequent screen image frame is consistent with the screen image frame resolution information in the region status record; if they are inconsistent, the target region coordinates in the region status record are mapped according to the resolution change ratio to obtain mapped target region coordinates; based on the mapped target region coordinates, region cropping and OCR text recognition are performed on the subsequent screen image frames to obtain mapped recognized text and mapped OCR confidence; if the mapped recognized text meets the target text validity condition and the mapped OCR confidence meets the preset confidence condition, the region status record is updated based on the mapped target region coordinates; if the mapped recognized text does not meet the target text validity condition, or the mapped OCR confidence does not meet the preset confidence condition, the current monitoring status is determined to be a region failure status.

[0017] This invention also provides a screen text three-state monitoring and relocation system, including a screen capture module, a visual language positioning module, an OCR text recognition module, a status recording module, a three-state discrimination module, and a relocation control module. The screen capture module acquires the current screen image frame and target area description information; the visual language positioning module determines the target area coordinates based on the current screen image frame and the target area description information; the OCR text recognition module crops the target area image based on the target area coordinates and performs OCR text recognition to obtain the currently recognized text and the current OCR confidence level; the status recording module generates an area status record based on the target area coordinates, target area description information, currently recognized text, current OCR confidence level, resolution information, and time information; the screen capture module continues to acquire subsequent screen image frames, and the OCR text... The recognition module performs region cropping and OCR text recognition on subsequent screen image frames based on the target region coordinates in the region status record, obtaining the subsequent recognized text and subsequent OCR confidence. The three-state discrimination module determines the current monitoring state as normal content update state, recognition abnormal state, or region failure state based on the text difference between the subsequent recognized text and the current recognized text, the subsequent OCR confidence, and whether the subsequent recognized text meets the target text validity condition. When the current monitoring state is normal content update state, the status recording module updates the recognized text and OCR confidence while keeping the target region coordinates unchanged. When the current monitoring state is normal content update state, the status recording module updates the recognized text and OCR confidence while keeping the target region coordinates unchanged. When the control state is an abnormal identification state, the OCR text recognition module re-executes region cropping and OCR text recognition based on the target region coordinates to obtain a re-recognition result; when the re-recognition result contains a recoverable recognition text that meets the target text validity condition and the OCR confidence meets the preset confidence condition, the three-state discrimination module converts the current monitoring state to a normal content update state, and the state recording module updates the region state record based on the recoverable recognition text and clears the number of consecutive abnormal frames under the same region state record; when the re-recognition result does not contain the recoverable recognition text, the state... The state recording module updates the number of consecutive abnormal frames. The three-state discrimination module maintains the abnormal identification state when the number of consecutive abnormal frames does not reach the preset number of abnormal frames, and converts the current monitoring state to the area failure state when the number of consecutive abnormal frames reaches the preset number of abnormal frames. When the current monitoring state is the area failure state, the relocation control module triggers the visual language positioning module to redetermine the target area coordinates, and when the re-identified text corresponding to the redetermined target area coordinates meets the target text validity condition and the corresponding OCR confidence meets the preset confidence condition, the state recording module controls the state recording module to write back a new area state record.

[0018] Compared with existing technologies, this invention binds the target region coordinates, target region description, recognized text, OCR confidence, resolution information, and region status version through region status recording, enabling subsequent recognition results to provide feedback on whether the current coordinates are still valid. By using a three-state discrimination mechanism—normal content update, recognition anomaly, and region failure—it avoids directly misjudging normal business text changes as region failure and prevents continued recognition based on old coordinates when the region semantics have deviated. By performing supplementary sampling and re-OCR recognition in recognition anomaly states, it reduces false relocation caused by single-frame recognition fluctuations. By triggering visual relocation only in region failure states and performing OCR validity verification after relocation before writing back a new region status record, it reduces the possibility of erroneous relocation results overwriting the original state. By performing coordinate mapping and recognition verification before resolution changes, it helps distinguish between coordinate offsets caused by resolution changes and true region failure. Attached Figure Description

[0019] Figure 1 The flowchart illustrates the screen text three-state monitoring and relocation method provided in this embodiment of the invention. Detailed Implementation

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings. It should be understood that the following content is for illustrative purposes only and is not intended to limit the scope of protection of the present invention.

[0021] like Figure 1As shown, the screen text three-state monitoring and relocation method provided by the present invention includes: acquiring a current screen image frame and target region description information, wherein the target region description information is used to describe the semantic target of the screen region to be monitored; inputting the current screen image frame and the target region description information into a visual language localization model to obtain target region coordinates, wherein the visual language localization model is used to output image region coordinates corresponding to the text description based on image content and text description; cropping the target region image from the current screen image frame according to the target region coordinates, and performing OCR text recognition on the target region image to obtain the currently recognized text and the current OCR text. R confidence; determine the target text validity conditions based on the target region description information, and generate a region status record with a region status version based on the target region coordinates, the target region description information, the currently recognized text, the current OCR confidence, the current screen image frame resolution, and time information; obtain subsequent screen image frames, perform region cropping and OCR text recognition on the subsequent screen image frames according to the target region coordinates in the region status record, and obtain the subsequent recognized text and subsequent OCR confidence; based on the text difference between the subsequent recognized text and the currently recognized text in the region status record, and the subsequent OCR confidence... The system checks whether the subsequently recognized text meets the target text validity condition, and determines the current monitoring state as a normal content update state, an abnormal recognition state, or a region failure state. When the current monitoring state is a normal content update state, the recognized text, OCR confidence score, and time information in the region status record are updated, while keeping the target region coordinates unchanged. When the current monitoring state is an abnormal recognition state, region cropping and OCR text recognition are re-executed based on the target region coordinates in the region status record to obtain a re-recognition result. Based on the re-recognition result, the current monitoring state is converted to a normal content update state and maintained as a normal content update state. The system detects abnormal states or transitions to a region failure state. When the current monitoring state is a region failure state, a visual relocation process is triggered to clear the positioning context related to the target region coordinates in the region status record. The target region coordinates are then re-determined based on the subsequent screen image frames or re-acquired screen image frames and the target region description information. OCR text recognition is performed based on the re-determined target region coordinates. When the re-recognized text meets the target text validity condition and the corresponding OCR confidence meets the preset confidence condition, the re-determined target region coordinates, the re-recognized text, and the corresponding OCR confidence are written back as a new region status record.

[0022] Furthermore, determining the target text validity condition based on the target region description information includes: determining whether there is a user configuration validity condition corresponding to the target region description information; if the user configuration validity condition exists, then the user configuration validity condition is used as a candidate text validity condition; if the user configuration validity condition does not exist, then semantic type recognition is performed on the target region description information to obtain the target region type, and candidate text validity conditions are called according to the target region type; the currently recognized text is matched with the candidate text validity conditions; when the currently recognized text satisfies one of the candidate text validity conditions and the current OCR confidence satisfies a preset confidence condition, the candidate text validity condition is determined as the target text validity condition; when multiple candidate text validity conditions are satisfied, the candidate text validity condition with the highest matching degree is selected as the target text validity condition based on the matching degree between the target region description information and the region type corresponding to each candidate text validity condition.

[0023] Furthermore, the three-state discrimination includes: normalizing the current identified text and the subsequent identified text according to the text type corresponding to the target text validity condition; calculating the text difference based on the normalized current identified text and the subsequent identified text; determining whether the subsequent identified text meets the target text validity condition; when the subsequent identified text meets the target text validity condition and the subsequent OCR confidence meets the preset confidence condition, marking the text difference as a valid content difference; when the subsequent identified text is not empty and does not meet the target text validity condition, and the subsequent OCR confidence meets the preset confidence condition, marking the text difference as a regional semantic deviation difference; when the subsequent OCR confidence does not meet the preset confidence condition or the subsequent identified text is empty, marking the text difference as an unusable identification difference; and determining the current monitoring state based on the valid content difference, regional semantic deviation difference, or unusable identification difference.

[0024] Furthermore, determining the current monitoring state based on the valid content difference, regional semantic deviation difference, or identification unavailability difference includes: when the text difference is marked as a valid content difference, determining the current monitoring state as a normal content update state; when the text difference is marked as an identification unavailability difference, writing the state of the corresponding subsequent screen image frame into the abnormal frame cache, and counting the number of consecutive abnormal frames recorded under the same regional state; when the number of consecutive abnormal frames does not reach a preset number of abnormal frames, determining the current monitoring state as an identification abnormal state; when the number of consecutive abnormal frames reaches the preset number of abnormal frames, changing the current monitoring state from the identification abnormal state to the regional failure state; when the text difference is marked as a regional semantic deviation difference, determining the current monitoring state as a regional failure state.

[0025] Furthermore, when the current monitoring status is a normal content update status, the subsequent identified text is compared with the current identified text in the area status record; when the subsequent identified text is different from the current identified text, and the subsequent identified text meets the target text validity condition, the subsequent identified text is written into the text update field of the area status record; the subsequent OCR confidence and the time information of the subsequent screen image frame are written into the area status record; the target area coordinates and positioning status in the area status record remain unchanged; and the number of consecutive abnormal frames under the same area status record is cleared.

[0026] Furthermore, when the current monitoring state is an abnormal identification state, without changing the target area coordinates in the area status record, at least one supplementary screen image frame is reacquired according to a preset short-time sampling interval; supplementary target area images are cropped from the at least one supplementary screen image frame based on the target area coordinates; OCR text recognition is performed on each supplementary target area image to obtain supplementary recognized text and supplementary OCR confidence; from the supplementary recognized texts that meet the target text validity condition and whose supplementary OCR confidence meets the preset confidence condition, the supplementary recognized text with the highest supplementary OCR confidence is selected as the recovered recognized text; if the recovered recognized text exists, the current monitoring state is converted to a normal content update state, the area status record is updated based on the recovered recognized text, and the number of consecutive abnormal frames under the same area status record is cleared; if the recovered recognized text does not exist, the number of consecutive abnormal frames under the same area status record is updated; when the number of consecutive abnormal frames does not reach the preset number of abnormal frames, the current monitoring state is maintained as an abnormal identification state; when the number of consecutive abnormal frames reaches the preset number of abnormal frames, the current monitoring state is converted from an abnormal identification state to an area failure state.

[0027] Furthermore, when the current monitoring state is a region failure state, the cause of the region failure is determined, and the cause of the region failure includes at least one of the following: target text validity condition not met, continuous recognition abnormality, resolution mapping failure; the scope of clearing the positioning context is determined according to the cause of the region failure; the positioning context related to the target region coordinates in the failure region status record is cleared; the target region description information and the target text validity condition are retained; a relocation input is constructed based on the retained target region description information and target text validity condition; the relocation input and the subsequent screen image frame or the reacquired screen image frame are input into the visual language positioning model to obtain at least one candidate relocation region coordinate.

[0028] Furthermore, the system crops candidate relocation region images based on the coordinates of at least one candidate relocation region; performs OCR text recognition on each candidate relocation region image to obtain candidate recognized text and candidate OCR confidence; determines whether each candidate recognized text meets the target text validity condition; generates candidate acceptance results for candidate relocation region coordinates that meet the target text validity condition, the candidate acceptance results being determined at least based on the candidate OCR confidence and the result of the candidate recognized text meeting the target text validity condition; selects target acceptance region coordinates from candidate relocation region coordinates whose candidate acceptance results meet preset acceptance conditions; writes the target acceptance region coordinates, corresponding candidate recognized text, corresponding candidate OCR confidence, current time information, and incremented region status version into a new region status record; if there are no candidate relocation region coordinates that meet the preset acceptance conditions, the original region status record is not overwritten, and the system proceeds to the next visual relocation retry or manual takeover process.

[0029] Furthermore, after acquiring subsequent screen image frames, the screen image frame resolution information and target region coordinates in the region status record are read; it is determined whether the resolution of the subsequent screen image frame is consistent with the screen image frame resolution information in the region status record; if they are inconsistent, the target region coordinates in the region status record are mapped according to the resolution change ratio to obtain mapped target region coordinates; based on the mapped target region coordinates, region cropping and OCR text recognition are performed on the subsequent screen image frames to obtain mapped recognized text and mapped OCR confidence; if the mapped recognized text meets the target text validity condition and the mapped OCR confidence meets the preset confidence condition, the region status record is updated based on the mapped target region coordinates; if the mapped recognized text does not meet the target text validity condition, or the mapped OCR confidence does not meet the preset confidence condition, the current monitoring status is determined to be a region failure status.

[0030] This invention also provides a screen text three-state monitoring and relocation system, including a screen capture module, a visual language positioning module, an OCR text recognition module, a status recording module, a three-state discrimination module, and a relocation control module. The screen capture module acquires the current screen image frame and target area description information; the visual language positioning module determines the target area coordinates based on the current screen image frame and the target area description information; the OCR text recognition module crops the target area image based on the target area coordinates and performs OCR text recognition to obtain the currently recognized text and the current OCR confidence level; the status recording module generates an area status record based on the target area coordinates, target area description information, currently recognized text, current OCR confidence level, resolution information, and time information; the screen capture module continues to acquire subsequent screen image frames, and the OCR text... The recognition module performs region cropping and OCR text recognition on subsequent screen image frames based on the target region coordinates in the region status record, obtaining the subsequent recognized text and subsequent OCR confidence. The three-state discrimination module determines the current monitoring state as normal content update state, recognition abnormal state, or region failure state based on the text difference between the subsequent recognized text and the current recognized text, the subsequent OCR confidence, and whether the subsequent recognized text meets the target text validity condition. When the current monitoring state is normal content update state, the status recording module updates the recognized text and OCR confidence while keeping the target region coordinates unchanged. When the current monitoring state is normal content update state, the status recording module updates the recognized text and OCR confidence while keeping the target region coordinates unchanged. When the control state is an abnormal identification state, the OCR text recognition module re-executes region cropping and OCR text recognition based on the target region coordinates to obtain a re-recognition result; when the re-recognition result contains a recoverable recognition text that meets the target text validity condition and the OCR confidence meets the preset confidence condition, the three-state discrimination module converts the current monitoring state to a normal content update state, and the state recording module updates the region state record based on the recoverable recognition text and clears the number of consecutive abnormal frames under the same region state record; when the re-recognition result does not contain the recoverable recognition text, the state... The state recording module updates the number of consecutive abnormal frames. The three-state discrimination module maintains the abnormal identification state when the number of consecutive abnormal frames does not reach the preset number of abnormal frames, and converts the current monitoring state to the area failure state when the number of consecutive abnormal frames reaches the preset number of abnormal frames. When the current monitoring state is the area failure state, the relocation control module triggers the visual language positioning module to redetermine the target area coordinates, and when the re-identified text corresponding to the redetermined target area coordinates meets the target text validity condition and the corresponding OCR confidence meets the preset confidence condition, the state recording module controls the state recording module to write back a new area state record.

[0031] Furthermore, the current screen image frame can originate from a computer monitor, mobile terminal screen, or other electronic screen. The target area description information is used to describe the semantic target of the screen area to be monitored, and can be a natural language description such as "monitoring the actual payment amount area," "identifying the shopping cart list," or "locating the verification code area." The target area description information serves both as visual positioning input and as the basis for subsequently determining the validity conditions of the target text.

[0032] Furthermore, a visual language localization model refers to a multimodal model capable of receiving image and text inputs and outputting the coordinates of the image region corresponding to the object described by the text input. In common terminology, such models are typically called visual language models, or Vision-Language Models (VLMs) for short; in tasks involving locating image regions based on text descriptions, they can also be called visual indexing localization models or visual grounding models. The visual language localization model may include an image encoder, a text encoder, a cross-modal fusion module, and a region coordinate output module. The image encoder extracts visual features from screen image frames, the text encoder extracts semantic features from the target region description information, the cross-modal fusion module establishes the correspondence between visual and semantic features, and the region coordinate output module outputs the target region coordinates corresponding to the target region description information. In one example, the visual language localization model can be a combination of a visual language model (VLM) and a region localization head, or it can be a multimodal visual language model with visual indexing localization capabilities. This invention does not limit the specific product name, training method, or network structure of the visual language localization model, as long as it can output the target region coordinates based on the current screen image frame and the target region description information.

[0033] Furthermore, after inputting the current screen image frame and target region description information into the visual language localization model, the model outputs the target region coordinates. These coordinates can be rectangular bounding box coordinates or other coordinate formats that represent the target region's extent. It is worth noting that this invention does not solely rely on the visual language localization model itself, but rather uses its output as the source of coordinates in the region state record, and subsequently performs state discrimination and relocation verification through OCR text recognition, OCR confidence, and target text validity conditions.

[0034] Furthermore, the target region image is cropped from the current screen image frame based on the target region coordinates, and OCR text recognition is performed on the target region image to obtain the currently recognized text and the current OCR confidence score. The OCR confidence score can be output by the OCR recognition engine, or it can be determined based on character confidence score, text line confidence score, or recognition result stability parameters.

[0035] In one example, when the OCR text recognition module outputs character-level confidence, the current OCR confidence is determined according to the following formula: in, Indicates the current OCR confidence level. This indicates the number of characters in the currently recognized text. Indicates character sequence number. Indicates the first character in the currently recognized text. Confidence level of character recognition, The value range is from 0 to 1. This indicates the first character to the second character. The recognition confidence scores of each character are summed. When the OCR text recognition module directly outputs the confidence score of a text line, that text line confidence score can be used as the current OCR confidence score.

[0036] Furthermore, the validity conditions for the target text are determined based on the target region description information. Specifically, it can be first determined whether there are user-configured validity conditions corresponding to the target region description information; if user-configured validity conditions exist, they are used as candidate text validity conditions; if no user-configured validity conditions exist, semantic type recognition is performed on the target region description information to obtain the target region type, and candidate text validity conditions are invoked based on the target region type. Subsequently, the currently recognized text is matched with the candidate text validity conditions. When the currently recognized text satisfies one of the candidate text validity conditions and the current OCR confidence meets the preset confidence condition, that candidate text validity condition is determined as the target text validity condition.

[0037] In one example, when the system automatically determines the target text validity condition from multiple candidate text validity conditions, a condition matching score is calculated for each candidate text validity condition: in, Indicates the first The condition matching score of the validity conditions of each candidate text. Indicates the sequence number of the candidate text validity condition. Indicates whether the currently recognized text satisfies the first condition. The validity conditions of each candidate text are met. If not satisfied Indicates the current OCR confidence level. Indicates the target region description information and the first The validity criteria for each candidate text correspond to the degree of semantic matching between region types. The value range is from 0 to 1. Because , and All values ​​are dimensionless, so they can be directly multiplied to obtain the conditional matching score.

[0038] When all the validity conditions of multiple candidate texts are met, the system selects... The largest candidate text validity condition is used as the target text validity condition, and this maximum value satisfies a preset confirmation threshold. Through the above processing, the target text validity condition is related not only to the currently recognized OCR text but also to the semantic type of the target region description information, thereby reducing the possibility of mistaking irrelevant text for target text.

[0039] Furthermore, based on the target region coordinates, target region description information, currently recognized text, current OCR confidence level, current screen image frame resolution, and time information, a region status record with a region status version is generated. This region status record binds the target region coordinate status and the recognized text status, and serves as the basis for subsequent monitoring frames to determine whether the current coordinates still correspond to the target semantic region. The region status version increments when new target region coordinates pass OCR validity verification and are written back, thereby avoiding the mixing of old and new coordinates and text.

[0040] In one example, the new region state version is generated according to the following formula: in, This indicates a new region status version. This indicates the current region status version. The region status version is used to distinguish region status records at different positioning stages, avoiding the continued use of coordinates or text from outdated versions during subsequent monitoring.

[0041] Furthermore, subsequent screen image frames are acquired, and based on the target region coordinates in the region status record, region cropping and OCR text recognition are performed on the subsequent screen image frames to obtain the subsequent recognized text and subsequent OCR confidence scores. Before a region is determined to be invalid, the system prioritizes using the target region coordinates in the region status record for continuous monitoring, rather than relocating for each frame, thereby reducing unnecessary relocation operations.

[0042] Furthermore, based on the textual differences between the subsequently recognized text and the currently recognized text in the region status record, the subsequent OCR confidence level, and whether the subsequently recognized text meets the target text validity conditions, the current monitoring status is determined as a normal content update status, an abnormal recognition status, or a region failure status. Specifically, the current and subsequent recognized texts can be normalized according to the text type corresponding to the target text validity conditions. Normalization can include removing irrelevant spaces, standardizing currency symbols, standardizing capitalization, standardizing decimal formats, standardizing line breaks, or standardizing list separators.

[0043] For ease of description, let's denote the currently recognized text as... , record the The subsequent recognized text corresponding to each subsequent screen image frame is Let the text normalization function be denoted as . The normalized current recognized text is then represented as The normalized subsequent recognition text is represented as In one example, text similarity is calculated using the following formula: in, Indicates the first Text similarity corresponding to each subsequent screen image frame Indicates the sequence number of the subsequent screen image frame. This represents the normalized current recognized text. With the normalized subsequent recognition text Edit distance between This indicates the number of characters in the current recognized text after normalization. This indicates the number of characters in the subsequent recognized text after normalization. Indicates in , The maximum value between 1 and 0 is used to avoid division by zero. The closer the value is to 1, the more similar the two texts are.

[0044] In one example, the change in the number of characters is calculated using the following formula: in, Indicates the first The change in the number of characters corresponding to each subsequent screen image frame. This indicates the number of characters in the subsequent recognized text after normalization. This indicates the number of characters in the current recognized text after normalization. This represents the absolute value of the difference in the number of two characters. Indicates in The maximum value between 1 and 0 is used to avoid division by zero.

[0045] Further, it is determined whether the subsequently recognized text meets the target text validity criteria. The target text validity judgment value is determined according to the following formula: ; in, Indicates the first The validity judgment value of the target text corresponding to each subsequent screen image frame; This indicates that the subsequently identified text meets the validity criteria of the target text; This indicates that the subsequently identified text does not meet the validity criteria of the target text; This represents the normalized text for subsequent recognition.

[0046] Furthermore, based on the target text validity judgment value, subsequent OCR confidence, and whether the subsequently recognized text is empty, the text differences are labeled into different difference types. Let the subsequent OCR confidence be denoted as... The preset reliability threshold is Text differences are marked as ,but Determined according to the following formula: ; in, Indicates the first Text difference markers corresponding to each subsequent screen image frame. Indicates differences in effective content. This indicates semantic deviation differences in the region. This indicates that unavailable differences have been identified. Indicates the confidence level of subsequent OCR. This indicates a pre-set confidence threshold.

[0047] Furthermore, the current monitoring status is determined based on valid content differences, regional semantic deviation differences, or unidentifiable differences. When a text difference is marked as a valid content difference, the current monitoring status is determined to be a normal content update status; when a text difference is marked as a regional semantic deviation difference, the current monitoring status is determined to be a regional failure status; when a text difference is marked as an unidentifiable difference, the status of the corresponding subsequent screen image frame is written to the abnormal frame cache, and the number of consecutive abnormal frames recorded under the same regional status is counted.

[0048] In one example, the number of consecutive abnormal frames is updated according to the following formula. ; in, Indicates the first The number of consecutive abnormal frames after processing each subsequent screen image frame. Indicates the first The number of consecutive abnormal frames after processing each subsequent screen image frame. Indicates the first Subsequent screen image frames were marked to identify unavailable differences. Indicates the first Subsequent screen image frames were not marked as identifying unavailable differences. If The preset number of abnormal frames was not reached. If so, the current monitoring status is determined to be an abnormal status; if If this happens, the current monitoring status will be changed from an anomaly detection status to a region failure status. This indicates the preset number of abnormal frames, which can be set according to screen refresh rate, OCR stability, or business scenario.

[0049] In one example, if the target area description is "actual payment amount area", the current recognized text is "¥158.00", and the subsequent recognized text is "¥138.00", and the subsequent recognized text still meets the monetary character structure condition, then this change is considered a normal content update state. If the subsequent recognized text is empty and the OCR confidence is low, and the number of consecutive abnormal frames does not reach the preset number of abnormal frames, then it is considered an abnormal recognition state. When the number of consecutive abnormal frames reaches the preset number of abnormal frames, the current monitoring state is changed to an area failure state. If the subsequent recognized text is "submit order" and the OCR confidence meets the condition, but does not meet the monetary character structure condition, then it is considered an area failure state.

[0050] Furthermore, when the current monitoring status is normal content update, the system updates the recognized text, OCR confidence score, and time information in the regional status record while keeping the target region coordinates unchanged. Specifically, subsequent recognized text is compared with the current recognized text in the regional status record; when the subsequent recognized text differs from the current recognized text and meets the target text validity condition, the subsequent recognized text is written to the text update field of the regional status record; the subsequent OCR confidence score and the time information of the subsequent screen image frame are written to the regional status record; the target region coordinates and positioning status in the regional status record remain unchanged; and the number of consecutive abnormal frames under the same regional status record is cleared. Through this processing, changes in normal business content will not trigger visual relocation. For example, when the amount changes from "¥158.00" to "¥138.00", the system only updates the text in the regional status record without relocating the target region.

[0051] Furthermore, when the current monitoring status is an abnormal identification status, region cropping and OCR text recognition are re-executed based on the target region coordinates in the region status record. Specifically, without changing the target region coordinates in the region status record, at least one supplementary screen image frame is re-acquired according to a preset short sampling interval; supplementary target region images are cropped from each of the at least one supplementary screen image frame based on the target region coordinates; OCR text recognition is performed on each supplementary target region image to obtain supplementary recognized text and supplementary OCR confidence; from the supplementary recognized texts that meet the target text validity condition and whose supplementary OCR confidence meets the preset confidence condition, the supplementary recognized text with the highest supplementary OCR confidence is selected as the recovered recognized text.

[0052] In one example, the recovered recognized text is selected according to the following formula: in, This indicates that the recognized text has been recovered. This indicates a supplementary set of recognized text. Indicates supplementary recognition text set Any supplementary identification text in the text, Indicates supplementary identification text The corresponding supplementary OCR confidence level, Indicates supplementary identification text Does the target text meet the validity criteria? If so, then... This indicates a preset confidence threshold. Indicates that when the condition is met and In the supplementary recognized text, select supplementary OCR confidence. The highest level of supplementary text recognition.

[0053] If it exists If the current monitoring status is not found, the current monitoring status will be changed to a normal content update status. This will be done by restoring the status record of the text update area and clearing the number of consecutive abnormal frames in the same area's status record. If no such record exists... If the number of consecutive abnormal frames recorded in the same area is not reached, the current monitoring state is maintained as an abnormal identification state. If the number of consecutive abnormal frames reaches the preset number, the current monitoring state is changed from an abnormal identification state to an area failure state. This process can reduce false relocation caused by screen refresh, animation transition, or fluctuations in a single OCR frame.

[0054] Furthermore, when the current monitoring status is a region failure state, a visual relocalization process is triggered. Specifically, the cause of the region failure can be determined, including at least one of the following: target text validity conditions not met, continuous recognition anomalies, or resolution mapping failure. The scope of clearing the localization context is determined based on the cause of the region failure. The localization context related to the target region coordinates in the failed region status record is cleared. The target region description information and target text validity conditions are retained. A relocalization input is constructed based on the retained target region description information and target text validity conditions. The relocalization input and subsequent screen image frames or reacquired screen image frames are input into the visual language localization model to obtain at least one candidate relocalization region coordinate. This process ensures that the relocalization input retains the target semantic constraints while preventing the old coordinate context from continuing to interfere with the new localization results.

[0055] Furthermore, OCR text recognition is performed based on the redefined target region coordinates. When the redefined text meets the target text validity condition and the corresponding OCR confidence score meets the preset confidence condition, the redefined target region coordinates, the redefined text, and the corresponding OCR confidence score are written back as a new region status record. Specifically, candidate repositioning region images can be cropped based on at least one candidate repositioning region coordinate; OCR text recognition is performed on each candidate repositioning region image to obtain candidate recognized text and candidate OCR confidence scores; it is determined whether each candidate recognized text meets the target text validity condition; candidate acceptance results are generated for candidate repositioning region coordinates that meet the target text validity condition; and target acceptance region coordinates are selected from the candidate repositioning region coordinates whose candidate acceptance results meet the preset acceptance conditions.

[0056] In one example, the candidate acceptance result is calculated according to the following formula: in, Indicates the first Candidate acceptance results for the coordinates of each candidate relocation region. Indicates the index of the candidate relocation region coordinates. Indicates the first Does the candidate text corresponding to the coordinates of each candidate relocation region meet the target text validity condition? If not satisfied Indicates the first The confidence scores of candidate OCRs corresponding to the coordinates of each candidate relocation region.

[0057] System selects to satisfy and The coordinates of the largest candidate relocation region are used as the coordinates of the target acceptance region. This indicates the preset acceptance threshold. If the target acceptance region coordinates exist, the target acceptance region coordinates, the corresponding candidate recognition text, the corresponding candidate OCR confidence score, the current time information, and the incremented region status version are written into a new region status record. If no candidate relocation region coordinates meet the preset acceptance conditions, the original region status record is not overwritten, and the process proceeds to the next visual relocation retry or manual takeover. It is worth noting that the relocation result will not directly overwrite the original region status record before OCR validity verification.

[0058] Furthermore, after acquiring subsequent screen image frames, the resolution information of the screen image frames and the coordinates of the target area can be read from the area status record, and it can be determined whether the resolution of the subsequent screen image frames is consistent with the resolution information of the screen image frames in the area status record. If they are inconsistent, the coordinates of the target area in the area status record are mapped according to the resolution change ratio to obtain the mapped target area coordinates; based on the mapped target area coordinates, area cropping and OCR text recognition are performed on the subsequent screen image frames to obtain the mapped recognized text and the mapped OCR confidence score; if the mapped recognized text meets the target text validity condition and the mapped OCR confidence score meets the preset confidence condition, the area status record is updated based on the mapped target area coordinates; if the mapped recognized text does not meet the target text validity condition, or the mapped OCR confidence score does not meet the preset confidence condition, the current monitoring status is determined to be an area failure status.

[0059] In one example, if the screen image frame resolution in the region status record is The resolution of subsequent screen image frames is The target area coordinates in the area status record are: Then the coordinates of the target region are mapped. Determined according to the following formula: ; in, Indicates the coordinates of the target region being mapped. This represents the coordinates of the target region before mapping. This indicates the width of the screen image frame in the region status record. This indicates the height of the screen image frame in the region status record. Indicates the width of subsequent screen image frames. Indicates the height of subsequent screen image frames. This represents the x-coordinate of the top-left corner of the target region before mapping. This represents the y-coordinate of the top-left corner of the target region before mapping. This represents the x-coordinate of the lower right corner of the target region before mapping. This represents the y-coordinate of the lower right corner of the target region before mapping. Indicates the scaling ratio in the width direction. This indicates the scaling factor in the height direction.

[0060] Through this coordinate mapping, the system can first attempt to correct the coordinate offset caused by the resolution change, and then determine whether the mapping result is usable by the target text validity condition and OCR confidence, thus avoiding directly misjudging the resolution change as a region failure.

[0061] It should be noted that the specific rules for the screen image frame source, OCR text recognition method, visual language localization model structure, and target text validity conditions in the above-mentioned specific methods can all be configured according to the application scenario. The key to this invention lies in using region state recording, target text validity conditions, and OCR confidence to jointly constitute a three-state discrimination criterion, and solving the problem of the disconnect between region coordinate state and recognized text state in dynamic screen text monitoring through differentiated processing and relocation acceptance write-back.

Claims

1. A method for monitoring and relocating three states of screen text, characterized in that, include: Acquire the current screen image frame and target region description information, wherein the target region description information is used to describe the semantic target of the screen region to be monitored; The current screen image frame and the target region description information are input into the visual language localization model to obtain the target region coordinates; The target region image is cropped from the current screen image frame according to the target region coordinates, and OCR text recognition is performed on the target region image to obtain the current recognized text and the current OCR confidence score; The target text validity conditions are determined based on the target region description information, and a region status record with a region status version is generated based on the target region coordinates, the target region description information, the currently recognized text, the current OCR confidence, the current screen image frame resolution, and time information. Acquire subsequent screen image frames, and perform region cropping and OCR text recognition on the subsequent screen image frames based on the target region coordinates in the region status record to obtain subsequent recognized text and subsequent OCR confidence. Based on the text difference between the subsequent recognized text and the current recognized text in the region status record, the confidence of the subsequent OCR, and whether the subsequent recognized text meets the target text validity condition, the current monitoring status is determined as normal content update status, recognition abnormal status, or region failure status. When the current monitoring status is normal content update status, update the recognized text, OCR confidence and time information in the area status record, and keep the target area coordinates unchanged; When the current monitoring status is an abnormal identification status, the region cropping and OCR text recognition are re-executed based on the target region coordinates in the region status record to obtain the re-identification result. Based on the re-identification result, the current monitoring status is converted to a normal content update status, maintained as an abnormal identification status, or converted to a region failure status. When the current monitoring status is a region failure status, a visual relocation process is triggered to clear the positioning context related to the target region coordinates in the region status record, and to re-determine the target region coordinates based on the subsequent screen image frames or re-acquired screen image frames and the target region description information; OCR text recognition is performed according to the re-determined target region coordinates, and when the re-recognized text meets the target text validity condition and the corresponding OCR confidence meets the preset confidence condition, the re-determined target region coordinates, the re-recognized text and the corresponding OCR confidence are written back as a new region status record.

2. The screen text three-state monitoring and relocation method according to claim 1, characterized in that, The determination of the target text validity conditions based on the target region description information includes: Determine whether there are any user configuration validity conditions corresponding to the target region description information; If the user configuration validity condition exists, then the user configuration validity condition shall be used as a candidate text validity condition. If the user configuration validity condition does not exist, semantic type recognition is performed on the target region description information to obtain the target region type, and candidate text validity conditions are invoked according to the target region type; Match the currently identified text with the validity conditions of the candidate text; When the currently recognized text satisfies one of the candidate text validity conditions and the current OCR confidence satisfies the preset confidence condition, the candidate text validity condition is determined as the target text validity condition; When multiple candidate text validity conditions are met, the candidate text validity condition with the highest matching degree is selected as the target text validity condition based on the matching degree between the target region description information and the region type corresponding to each candidate text validity condition.

3. The screen text three-state monitoring and relocation method according to claim 1, characterized in that, The step of determining the current monitoring status as a normal content update status, an abnormal recognition status, or a region failure status based on the text difference between the subsequently recognized text and the current recognized text in the region status record, the subsequent OCR confidence level, and whether the subsequent recognized text meets the target text validity condition includes: According to the text type corresponding to the target text validity condition, the current identified text and the subsequent identified text are standardized. The text difference is calculated based on the normalized current identified text and the subsequent identified text, and the text difference includes at least one of edit distance, text similarity and character number change; Determine whether the subsequently identified text meets the target text validity condition; When the subsequently recognized text meets the target text validity condition and the subsequent OCR confidence meets the preset confidence condition, the text difference is marked as a valid content difference; When the subsequent recognized text is not empty and does not meet the target text validity condition, and the subsequent OCR confidence meets the preset confidence condition, the text difference is marked as a region semantic deviation difference; When the subsequent OCR confidence does not meet the preset confidence conditions or the subsequent recognized text is empty, the text difference is marked as an unrecognizable difference. The current monitoring status is determined based on the differences in effective content, regional semantic deviation, or identification unavailability.

4. The screen text three-state monitoring and relocation method according to claim 3, characterized in that, Determining the current monitoring status based on the differences in effective content, regional semantic deviation, or unusable identification includes: When the text difference is marked as a valid content difference, the current monitoring status is determined to be a normal content update status; When the text difference is marked as an unidentifiable difference, the state of the corresponding subsequent screen image frame is written into the abnormal frame cache, and the number of consecutive abnormal frames under the state record of the same area is counted. When the number of consecutive abnormal frames does not reach the preset number of abnormal frames, the current monitoring status is determined to be an abnormal identification status. When the number of consecutive abnormal frames reaches the preset number of abnormal frames, the current monitoring status will be changed from the abnormal identification status to the area failure status. When the text difference is marked as a regional semantic deviation difference, the current monitoring status is determined to be a regional failure status.

5. The screen text three-state monitoring and relocation method according to claim 1, characterized in that, When the current monitoring status is normal content update status, update the recognized text, OCR confidence score, and time information in the area status record, while keeping the target area coordinates unchanged, including: The subsequent identified text is compared with the current identified text in the region status record; When the subsequent identified text is different from the current identified text, and the subsequent identified text meets the target text validity condition, the subsequent identified text is written into the text update field of the region status record; Write the subsequent OCR confidence score and the time information of the subsequent screen image frames into the region status record; Keep the target area coordinates and positioning status in the area status record unchanged; Clear the number of consecutive abnormal frames recorded in the same region.

6. The screen text three-state monitoring and relocation method according to claim 1, characterized in that, When the current monitoring status is an abnormal recognition status, the region cropping and OCR text recognition are re-executed based on the target region coordinates in the region status record, including: Without changing the target region coordinates in the region status record, at least one supplementary screen image frame is reacquired according to a preset short-time sampling interval. Based on the target region coordinates, supplementary target region images are cropped from each of the at least one supplementary screen image frames; Perform OCR text recognition on each supplementary target region image to obtain supplementary recognized text and supplementary OCR confidence; From the supplementary recognition texts that meet the target text validity conditions and whose supplementary OCR confidence meets the preset confidence conditions, select the supplementary recognition text with the highest supplementary OCR confidence as the recovered recognition text; If the recovery identification text exists, the current monitoring state is converted to a normal content update state, the regional status record is updated based on the recovery identification text, and the number of consecutive abnormal frames under the same regional status record is cleared; if the recovery identification text does not exist, the number of consecutive abnormal frames under the same regional status record is updated. When the number of consecutive abnormal frames does not reach the preset number of abnormal frames, the current monitoring state is maintained as an abnormal identification state. When the number of consecutive abnormal frames reaches the preset number of abnormal frames, the current monitoring status is changed from an abnormal identification status to a regional failure status.

7. The screen text three-state monitoring and relocation method according to claim 1, characterized in that, When the current monitoring status is a region failure status, a visual relocation process is triggered, including: Determine the cause of region failure, wherein the cause of region failure includes at least one of the following: target text validity conditions not met, continuous recognition anomaly, resolution mapping failure; The scope of the location context to be cleared is determined based on the cause of the area failure; Clear the positioning context associated with the target area coordinates in the failed area status record; The target region description information and the target text validity conditions are retained; The relocation input is constructed based on the retained target region description information and target text validity conditions; The relocation input and the subsequent screen image frame or the reacquired screen image frame are input into the visual language localization model to obtain at least one candidate relocation region coordinate.

8. The screen text three-state monitoring and relocation method according to claim 7, characterized in that, OCR text recognition is performed based on the redefined target region coordinates. When the re-recognized text meets the target text validity condition and the corresponding OCR confidence score meets the preset confidence condition, the redefined target region coordinates, the re-recognized text, and the corresponding OCR confidence score are written back as a new region status record, including: The candidate relocation region image is cropped according to the coordinates of the at least one candidate relocation region; Perform OCR text recognition on each candidate relocalization region image to obtain candidate recognized text and candidate OCR confidence scores; Determine whether each candidate text satisfies the target text validity condition; To generate candidate acceptance results for candidate relocation region coordinates that meet the target text validity conditions, the candidate acceptance results are determined at least based on the candidate OCR confidence and the result of the candidate recognized text meeting the target text validity conditions; Select the coordinates of the target acceptance area from the coordinates of the candidate relocation areas whose candidate acceptance results meet the preset acceptance conditions; Write the target acceptance area coordinates, corresponding candidate recognition text, corresponding candidate OCR confidence, current time information, and incremented area status version into a new area status record; If no candidate relocation area coordinates that meet the preset acceptance conditions are found, the original area status record will not be overwritten, and the process will proceed to the next visual relocation retry or manual takeover.

9. The method for monitoring and relocating three states of screen text according to claim 1, characterized in that, After acquiring subsequent screen image frames, the process also includes: Read the screen image frame resolution information and target area coordinates from the area status record; Determine whether the resolution of the subsequent screen image frame is consistent with the screen image frame resolution information in the region status record; If they are inconsistent, the target region coordinates in the region status record are mapped according to the resolution change ratio to obtain the mapped target region coordinates. Based on the coordinates of the mapped target region, region cropping and OCR text recognition are performed on the subsequent screen image frames to obtain the mapped recognized text and the mapped OCR confidence score. If the mapped and recognized text meets the validity condition of the target text and the mapping OCR confidence meets the preset confidence condition, then the region status record is updated based on the coordinates of the mapped target region. If the mapped and recognized text does not meet the validity conditions of the target text, or the mapping OCR confidence does not meet the preset confidence conditions, then the current monitoring status will be determined as a region failure status.

10. A screen text three-state monitoring and relocation system, used in the screen text three-state monitoring and relocation method as described in any one of claims 1 to 9, characterized in that, The system includes a screen capture module, a visual language localization module, an OCR text recognition module, a status recording module, a three-state discrimination module, and a relocation control module. The system is configured to execute the following process: The screen capture module acquires the current screen image frame and target area description information; The visual language localization module determines the coordinates of the target region based on the current screen image frame and the target region description information; The OCR text recognition module crops the target region image according to the target region coordinates and performs OCR text recognition to obtain the current recognized text and the current OCR confidence level. The status recording module generates a region status record based on the target region coordinates, target region description information, currently recognized text, current OCR confidence, resolution information, and time information. The screen capture module continues to acquire subsequent screen image frames, and the OCR text recognition module performs region cropping and OCR text recognition on the subsequent screen image frames based on the target region coordinates in the region status record to obtain the subsequent recognized text and subsequent OCR confidence. The three-state discrimination module determines the current monitoring state as normal content update state, recognition abnormal state, or region failure state based on the text difference between the subsequent recognized text and the current recognized text, the confidence of the subsequent OCR, and whether the subsequent recognized text meets the target text validity conditions. When the current monitoring status is normal content update status, the status recording module updates the recognized text and OCR confidence while keeping the target area coordinates unchanged; When the current monitoring state is an abnormal recognition state, the OCR text recognition module re-executes region cropping and OCR text recognition based on the target region coordinates to obtain a re-recognition result; when the re-recognition result contains recovered recognition text that meets the target text validity condition and the OCR confidence meets the preset confidence condition, the three-state discrimination module converts the current monitoring state to a normal content update state, and the state recording module updates the region state record based on the recovered recognition text and clears the number of consecutive abnormal frames under the same region state record; when the re-recognition result does not contain the recovered recognition text... The status recording module updates the number of consecutive abnormal frames. The three-state discrimination module maintains the abnormal identification state when the number of consecutive abnormal frames does not reach the preset number of abnormal frames, and converts the current monitoring state to the area failure state when the number of consecutive abnormal frames reaches the preset number of abnormal frames. When the current monitoring state is the area failure state, the relocation control module triggers the visual language positioning module to redetermine the target area coordinates. When the re-identified text corresponding to the redetermined target area coordinates meets the target text validity condition and the corresponding OCR confidence meets the preset confidence condition, the status recording module controls the new area status record to be written back.

Images