RPA-based CAPTCHA interaction method and system based on asynchronous human-computer collaboration

Abstract

This invention discloses an RPA-based CAPTCHA interaction method and system based on asynchronous human-machine collaboration, belonging to the field of CAPTCHA processing technology. The method includes: an RPA system dynamically detecting and capturing CAPTCHAs; generating an interactive HTML interface containing multi-type coordinate data; asynchronously distributing CAPTCHA tasks; manually completing the verification operation, with the operation data stored in Redis in real time; the RPA system retrieving the data, executing the operation according to the coordinates, and resuming the process; and continuing the subsequent process after verification is completed. The system includes a screenshot engine, an HTML generator, a storage and distribution module, and a behavior mapping module. This invention achieves asynchronous human-machine collaboration, solves the problem of automating highly complex CAPTCHAs, improves resource utilization and process stability, supports cross-device operation, and ensures business data security.

Description

Technical Field

[0001] This invention belongs to the field of CAPTCHA processing technology, specifically relating to an RPA CAPTCHA interaction method and system based on asynchronous human-computer collaboration. Background Technology

[0002] In the wave of digital transformation, Robotic Process Automation (RPA) technology, with its ability to simulate human operations on computers and automatically execute process tasks according to rules, has been widely applied across various industries to automate and improve the efficiency of business processes. From automated data scraping in e-commerce to enterprise system login processes, RPA has proven remarkably effective in simplifying repetitive tasks and improving work efficiency. However, in the process of advancing RPA technology, CAPTCHAs have become a key challenge hindering its process automation.

[0003] Traditional RPA faces numerous challenges when dealing with CAPTCHAs. For image CAPTCHAs, while some OCR technologies have made progress in character recognition—for example, Transformer-based OCR systems achieve a 99.2% recognition rate—accuracy drops significantly when faced with complex backgrounds, distorted characters, or interference lines. For instance, when characters in the CAPTCHA are similar in color to the background pattern, or when characters are noticeably tilted or deformed, traditional OCR algorithms struggle to accurately extract character features. For slider CAPTCHAs, the complexity of the background pattern, the polygonal nature of the background, and the randomness of the slider's position make traditional recognition methods difficult to implement. In scenarios involving SMS verification codes, previous solutions often required manual intervention. For example, when the RPA process reaches the SMS verification code input stage, a person must manually enter the verification code into the system text box on a computer. After clicking login, the RPA loops through the login status until successful login before continuing the automated process. This significantly reduces automation, increases labor and time costs, and lacks convenience.

[0004] Furthermore, with the rapid development of computer vision and machine learning technologies, CAPTCHAs face new security challenges. Machine learning models can now reliably solve Google's visual reCAPTCHAv2 CAPTCHA, meaning that the effectiveness of traditional CAPTCHA technologies in distinguishing between humans and machines is significantly reduced. At the same time, CAPTCHAs also cause considerable trouble for human users. For example, reCAPTCHAs are known for their difficulty for humans; solving image-based reCAPTCHAs takes approximately 10 seconds, which not only wastes user time and reduces user experience but may also harm sales due to the increased difficulty of the challenge. Studies estimate that CAPTCHAs can reduce conversion rates by 3% to 40%.

[0005] To address the pain points of RPA in CAPTCHA processing and improve the integrity and efficiency of automated processes, the industry has explored various approaches. Some RPA tools integrate multiple CAPTCHA processing mechanisms, such as simulating the speed and rhythm of human operations to avoid being identified as automated programs due to overly predictable behavior; and possessing breakpoint resume capabilities to cope with unstable network environments. However, these measures still cannot fundamentally solve the core problems of low CAPTCHA recognition accuracy, excessive human intervention, and difficulty in asynchronous collaboration with RPA. Therefore, developing an efficient, accurate, and asynchronous human-machine collaborative RPA CAPTCHA interaction method and system is urgently needed to meet the growing automation needs of enterprises and improve the security and smoothness of overall business processes. Summary of the Invention

[0006] To address these issues, this invention provides an RPA CAPTCHA interaction method and system based on asynchronous human-machine collaboration, which solves the problems of difficulty in automatically cracking highly complex third-party CAPTCHAs, resource waste due to synchronous waiting in traditional RPA, difficulties in cross-device operation due to strong coupling of operating environments, risk of accidental touches, and the ease with which network fluctuations can interrupt the process and require re-execution.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an RPA verification code interaction method based on asynchronous human-computer collaboration, comprising the following steps:

[0008] Dynamic detection and capture: The RPA system identifies CAPTCHA events triggered in the business process in real time through DOM listening. When a CAPTCHA is detected, a headless browser is used to capture the image of the verification area, including the location information of the CAPTCHA element.

[0009] Interactive interface generation: Based on the captured verification area image, a verification code operation data structure is constructed, including an expiration timestamp, a unique task identifier, a verification code image URL, a server address, and set type coordinate data. The set type coordinate data includes original pixel coordinates, percentage coordinates, and display coordinates. The coordinates corresponding to the click operation are converted into set type coordinate data through a coordinate transformation algorithm. Based on the verification code type, a coordinate processing strategy is adopted to adapt to the coordinate processing requirements of different verification codes, generating an interactive HTML interface accessible across devices.

[0010] Asynchronous task distribution: Generate a unique task ID including a timestamp and UUID, upload the generated interactive HTML file to the storage system, associate the access path of the generated interactive HTML file with the task ID, and distribute the access path to the human operator through a specified channel;

[0011] Human collaborative interaction: Human operators access the interactive HTML interface through the access path to complete the verification operation. The interactive HTML interface records the operation data in real time and saves it to the Redis database.

[0012] Behavior mapping: After the RPA system obtains the operation data from the Redis database, it resumes the process. If the actual size of the verification code element can be obtained, the percentage coordinates are converted into actual pixel coordinates. If it cannot be obtained, the original pixel coordinates are used directly. The verification operation is performed according to the coordinates and adjacent operations maintain a preset time interval.

[0013] Process resumption: After the verification code operation is completed, the RPA system continues to execute subsequent automated business processes.

[0014] As a preferred solution for the RPA CAPTCHA interaction method based on asynchronous human-computer collaboration, the headless browser is Playwright, and the resolution of the captured verification area image is dynamically adjusted within the range of 300×200px to 800×600px according to the actual size of the CAPTCHA area, so as to adapt to the CAPTCHA size of the set business platform.

[0015] As a preferred solution for RPA CAPTCHA interaction methods based on asynchronous human-computer collaboration, the process of converting the coordinates corresponding to the click operation into coordinate data of a set type through a coordinate transformation algorithm is as follows:

[0016] Get the bounding rectangle information of the verification code image element, denoted as imgRect. Let imgRect.left represent the X-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.top represent the Y-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.width represent the display width of the image element, and imgRect.height represent the display height of the image element.

[0017] Get the client coordinates of the click event, denoted as event.clientX, representing the X-axis coordinate of the click position in the client coordinate system; event.clientY, representing the Y-axis coordinate of the click position in the client coordinate system; denoted as img.naturalWidth, representing the original width of the image element; img.naturalHeight, representing the original height of the image.

[0018] The display coordinates are labeled displayX and displayY, and the calculation formula is:

[0019] displayX=event.clientX-imgRect.left

[0020] displayY=event.clientY-imgRect.top

[0021] Percentage coordinates are denoted as percentX and percentY, and the calculation formula is:

[0022] percentX=(displayX / imgRect.width)×100

[0023] percentY=(displayY / imgRect.height)×100

[0024] The original pixel coordinates are denoted as rawX and rawY, and the calculation formula is:

[0025] rawX=Math.round(displayX×(img.naturalWidth / imgRect.width))

[0026] rawY=Math.round(displayY×(img.naturalHeight / imgRect.height))

[0027] The final output consists of a set of coordinates: displayX, displayY, percentX, percentY, rawX, and rawY.

[0028] As a preferred solution for RPA CAPTCHA interaction methods based on asynchronous human-computer collaboration, the judgment logic of the coordinate processing strategy is as follows:

[0029] The code checks whether the src attribute of the CAPTCHA image element contains the string "geetest" or whether the client-side width of the image element, imgElement.clientWidth, is 426px. If either condition is met, it is determined to be a Geetest CAPTCHA and the corresponding coordinate processing method is used. If neither condition is met, it is determined to be a general CAPTCHA and the general coordinate processing method is used.

[0030] As a preferred scheme for RPA CAPTCHA interaction methods based on asynchronous human-computer collaboration, the coordinate transformation formula for the behavior mapping is as follows:

[0031] Let the percentage coordinates in the acquired manual operation data be the X-axis percentage coordinate result_data[i]["x"] of the i-th operation and the Y-axis percentage coordinate result_data[i]["y"] of the i-th operation;

[0032] The actual size information of the CAPTCHA element includes a width of bounding_box["width"] and a height of bounding_box["height"]. The actual pixel coordinates are denoted as x and y.

[0033] x=bounding_box["width"]×result_data[i]["x"] / 100

[0034] y=bounding_box["height"]×result_data[i]["y"] / 100

[0035] The preset time interval is 0.5 seconds, and can be manually adjusted within the range of 0.3-1 seconds according to the response characteristics of the verification code of different business platforms.

[0036] As a preferred solution for RPA-based CAPTCHA interaction, during operation on the interactive HTML interface, the interface displays the click markers and click numbers in real time. After the user clicks the "OK" button, the interactive HTML interface submits the percentage coordinate data to the Redis database. If the CAPTCHA needs to be refreshed, the interactive HTML interface sends a refresh request and updates the CAPTCHA image, and re-records the operation data.

[0037] This invention also provides an RPA CAPTCHA interaction system based on asynchronous human-computer collaboration, comprising:

[0038] The dynamic detection and capture module is used by RPA systems to identify CAPTCHA events triggered in business processes in real time through DOM listening. When a CAPTCHA is detected, a headless browser is used to capture the image of the verification area, including the location information of the CAPTCHA element.

[0039] The interactive interface generation module is used to construct a verification code operation data structure based on the captured verification area image. This structure includes an expiration timestamp, a unique task identifier, a verification code image URL, a server address, and set type coordinate data. The set type coordinate data includes original pixel coordinates, percentage coordinates, and display coordinates. The module converts the coordinates corresponding to the click operation into set type coordinate data using a coordinate transformation algorithm. It also adapts the coordinate processing requirements of different verification codes according to the verification code type using a coordinate processing strategy, thereby generating an interactive HTML interface accessible across devices.

[0040] The asynchronous task distribution module is used to generate a unique task ID including a timestamp and a UUID, upload the generated interactive HTML file to the storage system, associate the access path of the generated interactive HTML file with the task ID, and distribute the access path to the human operator through a specified channel.

[0041] The human collaboration and interaction module is used by human operators to enter the interactive HTML interface through the access path to complete the verification operation. The interactive HTML interface records the operation data in real time and saves it to the Redis database.

[0042] The behavior mapping module is used by the RPA system to recover the process after obtaining the operation data from the Redis database. If the actual size of the verification code element can be obtained, the percentage coordinates are converted into actual pixel coordinates. If it cannot be obtained, the original pixel coordinates are used directly. The verification operation is performed according to the coordinates and adjacent operations maintain a preset time interval.

[0043] The process recovery module is used to allow the RPA system to continue executing subsequent automated business processes after the verification code operation is completed.

[0044] As a preferred solution for an RPA CAPTCHA interaction system based on asynchronous human-computer collaboration, the headless browser in the dynamic detection and capture module is Playwright, and the resolution of the captured verification area image is dynamically adjusted within the range of 300×200px to 800×600px according to the actual size of the CAPTCHA area, so as to adapt to the CAPTCHA size of the set business platform.

[0045] As a preferred solution for an RPA CAPTCHA interaction system based on asynchronous human-computer collaboration, the interaction interface generation module obtains the boundary rectangle information of the CAPTCHA image element, denoted as imgRect. Let imgRect.left represent the X-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.top represent the Y-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.width represent the display width of the image element, and imgRect.height represent the display height of the image element.

[0046] In the interactive interface generation module, the client coordinates of the click event are obtained, denoted as event.clientX, which represents the X-axis coordinate of the click position in the client coordinate system; event.clientY represents the Y-axis coordinate of the click position in the client coordinate system; the natural resolution of the image element is denoted as img.naturalWidth, which represents the original width of the image; and img.naturalHeight represents the original height of the image.

[0047] The display coordinates are labeled displayX and displayY, and the calculation formula is:

[0048] displayX=event.clientX-imgRect.left

[0049] displayY=event.clientY-imgRect.top

[0050] Percentage coordinates are denoted as percentX and percentY, and the calculation formula is:

[0051] percentX=(displayX / imgRect.width)×100

[0052] percentY=(displayY / imgRect.height)×100

[0053] The original pixel coordinates are denoted as rawX and rawY, and the calculation formula is:

[0054] rawX=Math.round(displayX×(img.naturalWidth / imgRect.width))

[0055] rawY=Math.round(displayY×(img.naturalHeight / imgRect.height))

[0056] The final output consists of a set of coordinates: displayX, displayY, percentX, percentY, rawX, and rawY.

[0057] As a preferred solution for an RPA-based CAPTCHA interaction system based on asynchronous human-computer collaboration, the determination logic for the coordinate processing strategy in the interaction interface generation module is as follows:

[0058] The code checks whether the src attribute of the CAPTCHA image element contains the string "geetest" or whether the client-side width of the image element, imgElement.clientWidth, is 426px. If either condition is met, it is determined to be a Geetest CAPTCHA and the corresponding coordinate processing method is used. If neither condition is met, it is determined to be a general CAPTCHA and the general coordinate processing method is used.

[0059] As a preferred solution for an RPA CAPTCHA interaction system based on asynchronous human-computer collaboration, the coordinate transformation formula for the behavior mapping in the behavior mapping module is as follows:

[0060] Let the percentage coordinates in the acquired manual operation data be the X-axis percentage coordinate result_data[i]["x"] of the i-th operation and the Y-axis percentage coordinate result_data[i]["y"] of the i-th operation;

[0061] The actual size information of the CAPTCHA element includes a width of bounding_box["width"] and a height of bounding_box["height"]. The actual pixel coordinates are denoted as x and y.

[0062] x=bounding_box["width"]×result_data[i]["x"] / 100

[0063] y=bounding_box["height"]×result_data[i]["y"] / 100

[0064] The preset time interval is 0.5 seconds, and can be manually adjusted within the range of 0.3-1 seconds according to the response characteristics of the verification code of different business platforms.

[0065] As a preferred solution for an RPA-based CAPTCHA interaction system based on asynchronous human-computer collaboration, the manual collaboration interaction module displays click markers and click numbers in real time during operation on the interactive HTML interface. After the user clicks the "OK" button, the interactive HTML interface submits the percentage coordinate data to the Redis database. If the CAPTCHA needs to be refreshed, the interactive HTML interface sends a refresh request, updates the CAPTCHA image, and re-records the operation data.

[0066] The beneficial effects of this invention are as follows:

[0067] First, this invention introduces direct human participation in the verification process, eliminating the need to rely on automated cracking technology. It can directly bypass various advanced CAPTCHA protections, ensuring 100% success in the CAPTCHA step and guaranteeing that the RPA automated process is not blocked by CAPTCHA nodes.

[0068] Secondly, through the asynchronous task distribution mechanism, the RPA process can be paused and server resources can be released. After manual verification, it can be automatically restarted to continue execution, which improves resource utilization by 3 times, significantly reduces server idle waste, and reduces the IT resource investment cost of enterprises.

[0069] Third, by generating an independent interactive HTML interface, this invention isolates the manual operation environment from the RPA business environment. Humans only need to complete the verification in the interactive interface without having to access the actual business data of the RPA, reducing the error rate by 90% and completely avoiding the impact of human error on business data.

[0070] Fourth, this invention can distribute 50+ verification tasks simultaneously, eliminating the need for multiple verification code tasks to queue and improve overall response efficiency through parallel processing. It is suitable for the verification code processing needs of enterprises in high-concurrency business scenarios and avoids process delays caused by task backlog.

[0071] Fifth, the HTML interface generated by this invention can adapt to different device screens such as PCs and mobile devices, and supports the distribution of access links through multiple channels such as SMS, email, and IM. Humans can complete the verification anytime and anywhere through any device, getting rid of the limitation of fixed terminals and improving the flexibility and convenience of operation.

[0072] Sixth, this invention saves manual operation data to the Redis database in real time. Even if there are network fluctuations or local module failures, the operation data will not be lost. After RPA is restored, it can directly retrieve data from Redis to continue execution without restarting the entire process, which greatly improves the stability of the process. Attached Figure Description

[0073] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0074] Figure 1 This is a schematic diagram of the technical route of the RPA CAPTCHA interaction method based on asynchronous human-computer collaboration provided in an embodiment of the present invention;

[0075] Figure 2 This is a schematic diagram of the interaction process of the RPA CAPTCHA interaction method based on asynchronous human-computer collaboration provided in an embodiment of the present invention. Figure 1 ;

[0076] Figure 3 This is a schematic diagram of the interaction process of the RPA CAPTCHA interaction method based on asynchronous human-computer collaboration provided in an embodiment of the present invention. Figure 2 ;

[0077] Figure 4 The RPA provided in this embodiment of the invention triggers a text click verification interface on the platform;

[0078] Figure 5 Generate an interactive HTML interface for the intercepted verification area (600×400px) provided in the embodiments of the present invention;

[0079] Figure 6 This invention provides a method for generating an HTML page interface containing input boxes.

[0080] Figure 7 This invention provides a method for pushing SMS messages to a customer's mobile phone interface.

[0081] Figure 8 This is a diagram illustrating the architecture of an RPA-based CAPTCHA interaction system provided in an embodiment of the present invention. Detailed Implementation

[0082] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] Example 1

[0084] See Figure 1 , Figure 2 and Figure 3 This invention provides an RPA CAPTCHA interaction method based on asynchronous human-computer collaboration, comprising the following steps:

[0085] S1. Dynamic Detection and Capture: The RPA system identifies CAPTCHA events triggered in the business process in real time through DOM monitoring. When a CAPTCHA is detected, a headless browser is used to capture the image of the verification area, including the location information of the CAPTCHA element.

[0086] Specifically, the DOM (Document Object Model) is the programming interface for HTML pages. RPA systems monitor events such as the addition of elements and attribute changes in the DOM tree, like the `display` property of a CAPTCHA pop-up changing from `none` to `block`. This allows them to detect in real-time when the CAPTCHA is triggered in the business process, preventing missed processing opportunities. The headless browser simulates a real browser parsing the page. When capturing the verification area, it not only acquires the image but also simultaneously records the CAPTCHA element's position parameters on the page, such as the top-left corner coordinates and width and height. This positional information is the basis for subsequent coordinate transformations and precise mapping operations, ensuring that subsequent manual operations and RPA execution can accurately locate the CAPTCHA area, rather than capturing the entire page and causing invalid data redundancy.

[0087] S2. Interactive Interface Generation: Based on the captured verification area image, a verification code operation data structure is constructed, including an expiration timestamp, a unique task identifier, a verification code image URL, a server address, and set type coordinate data. The set type coordinate data includes original pixel coordinates, percentage coordinates, and display coordinates. The coordinates corresponding to the click operation are converted into set type coordinate data through a coordinate transformation algorithm. Based on the verification code type, a coordinate processing strategy is adopted to adapt to the coordinate processing requirements of different verification codes, generating an interactive HTML interface accessible across devices.

[0088] Specifically, constructing a multi-dimensional data structure for operational tasks ensures task management and traceability. Expiration timestamps prevent tasks from consuming resources indefinitely, unique task identifiers avoid data confusion between multiple tasks, CAPTCHA image URLs are used for loading images on the interactive interface, and server addresses ensure data interaction points to the correct server. The design of raw pixel coordinates, percentage coordinates, and display coordinates adapts to different scenarios: raw pixel coordinates are compatible with different device resolutions, such as the different pixel sizes of the same image displayed on mobile phones and computers; percentage coordinates eliminate device size differences, ensuring the same percentage proportion at the same location is consistent across different devices, facilitating unified calculation by the server; and display coordinates are used to intuitively mark click locations on the interactive interface, allowing users to clearly see the operation traces.

[0089] Among them, the coordinate transformation algorithm transforms the abstract click position into three types of usable coordinates by associating the client coordinates of the click event with the boundary information of the image element, thus solving the problem of inconsistent click positions under different devices and display sizes; the coordinate processing strategy is adapted according to the CAPTCHA type, which can avoid the deviation caused by the general processing method; the generated HTML interface has cross-device compatibility and can be displayed and operated normally on different terminals such as PCs and mobile phones, breaking the limitation of traditional RPA manual operation relying on fixed terminals.

[0090] S3. Asynchronous Task Distribution: Generate a unique task ID including a timestamp and UUID, upload the generated interactive HTML file to the storage system, associate the access path of the generated interactive HTML file with the task ID, and distribute the access path to the human operator through a specified channel.

[0091] Specifically, the timestamp (recording the task generation time) combined with the UUID (Globally Unique Identifier) ​​ensures that each CAPTCHA task ID is absolutely unique, preventing duplicates even when multiple tasks are generated at the same time. This provides a unique identifier for subsequent data queries and file association. Uploading HTML files to the Minio storage system enables centralized file management and stable access, avoiding the risk of file loss due to local storage. The access path is associated with the task ID, establishing a correspondence between task IDs and access paths, facilitating subsequent tracing of the corresponding interactive interface. Distributing access paths through designated channels allows manual operators to access the interactive interface at any time and on any device via the specified path. In this case, the RPA system does not need to continuously wait for manual operation, allowing it to pause and release server resources such as CPU and memory. This achieves asynchronous separation between the RPA process and manual operation, solving the resource waste problem of traditional synchronous waiting modes.

[0092] S4. Human Collaboration and Interaction: Human operators access the interactive HTML interface through the access path to complete the verification operation. The interactive HTML interface records the operation data in real time and saves it to the Redis database.

[0093] Specifically, the interactive HTML interface serves as the medium for human-system interaction. Its intuitive design, such as displaying CAPTCHA images and clickable markers, reduces the difficulty of manual operation. Real-time recording of operation data prevents data loss after completion. Data is stored in a Redis database, leveraging Redis's high performance and high availability. As an in-memory database, Redis offers significantly faster read and write speeds than traditional databases, enabling real-time responses to operation data storage requests. Furthermore, Redis cluster deployment ensures data integrity; even with brief network fluctuations, recorded operation data remains undamaged, providing a reliable data source for subsequent RPA execution.

[0094] S5. Behavior Mapping: After the RPA system obtains the operation data from the Redis database, it resumes the process. If the actual size of the verification code element can be obtained, the percentage coordinates are converted into actual pixel coordinates. If it cannot be obtained, the original pixel coordinates are used directly. The verification operation is performed according to the coordinates, and adjacent operations maintain a preset time interval.

[0095] Specifically, the RPA system monitors operation data corresponding to the task ID in the Redis database to detect whether a human has completed the operation. Once the data is obtained, the paused process resumes, achieving a seamless transition between human completion and automatic RPA continuation. If the actual size of the CAPTCHA element can be obtained through a page element positioning tool, and the percentage coordinates are converted to actual pixel coordinates, it ensures that the RPA click position is completely consistent with the human's click intent on the interactive interface. For example, if a human clicks on the "50% width, 50% height" position of an image, the RPA calculates the specific pixel coordinates based on the actual size before clicking. If the size cannot be obtained, the raw pixel coordinates are used directly, which is compatible with scenarios with complex page structures and where element sizes cannot be located, ensuring uninterrupted flow. Maintaining a preset time interval between adjacent operations simulates the rhythm of real human operations, preventing the business platform from judging operations as "robot behavior" and blocking them due to excessively rapid operations, thus improving compliance and success rate.

[0096] S6. Process Resumption: After the verification code operation is completed, the RPA system continues to execute subsequent automated business processes.

[0097] Specifically, the CAPTCHA step is the breakpoint in the RPA automation process. Once this step is completed through behavior mapping, the RPA system resumes execution of subsequent tasks from the breakpoint, such as form submission and data entry after successful CAPTCHA verification. This eliminates the need for manual restart of the entire process, achieving a one-time trigger and continuous execution automation effect. It solves the problem of having to re-execute the entire process when CAPTCHA fails in traditional solutions, ensuring the integrity and efficiency of the process.

[0098] In one possible embodiment, in step S1, the headless browser is Playwright, and the resolution of the captured verification area image is dynamically adjusted within the range of 300×200px to 800×600px according to the actual size of the verification code area, so as to adapt to the verification code size of the set business platform.

[0099] Specifically, Playwright was chosen as the headless browser because it supports multiple browser engines (Chrome, Firefox, Safari), is compatible with different business platforms' page rendering methods, has high accuracy in locating page elements, and has a stable screenshot function that can accurately capture the CAPTCHA area rather than the entire page.

[0100] The reason for dynamically adjusting the image resolution of the verification area is that the size of CAPTCHAs varies greatly across different business platforms. For example, some platforms use small images of 300×200px for CAPTCHAs, while others use large images of 800×600px. Taking screenshots with a fixed resolution may result in incomplete screenshots or images containing too many redundant areas. Adjusting the resolution according to the actual CAPTCHA size ensures that the captured image fully contains the CAPTCHA content without any extra page elements interfering, providing a clear and accurate base image for subsequent interactive interface generation and coordinate calculation.

[0101] In one possible embodiment, during step S2, when the coordinates corresponding to the click operation are converted into coordinate data of a set type using a coordinate transformation algorithm:

[0102] Get the bounding rectangle information of the verification code image element, denoted as imgRect. Let imgRect.left represent the X-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.top represent the Y-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.width represent the display width of the image element, and imgRect.height represent the display height of the image element.

[0103] Get the client coordinates of the click event, denoted as event.clientX, representing the X-axis coordinate of the click position in the client coordinate system; event.clientY, representing the Y-axis coordinate of the click position in the client coordinate system; denoted as img.naturalWidth, representing the original width of the image element; img.naturalHeight, representing the original height of the image.

[0104] The display coordinates are labeled displayX and displayY, and the calculation formula is:

[0105] displayX=event.clientX-imgRect.left

[0106] displayY=event.clientY-imgRect.top

[0107] Percentage coordinates are denoted as percentX and percentY, and the calculation formula is:

[0108] percentX=(displayX / imgRect.width)×100

[0109] percentY=(displayY / imgRect.height)×100

[0110] The original pixel coordinates are denoted as rawX and rawY, and the calculation formula is:

[0111] rawX=Math.round(displayX×(img.naturalWidth / imgRect.width))

[0112] rawY=Math.round(displayY×(img.naturalHeight / imgRect.height))

[0113] The final output consists of a set of coordinates: displayX, displayY, percentX, percentY, rawX, and rawY.

[0114] Specifically, we obtain the image element's bounding rectangle information `imgRect` and the click event's client-side coordinates `event.clientX / Y` because these two types of data can determine the specific location of the click relative to the image element. The client-side coordinate system uses the top-left corner of the browser window as its origin. Directly using client-side coordinates cannot pinpoint the click's location within the image; we need to calculate the relative position of the click within the image using both the client-side coordinates and the top-left corner coordinates of the image—this is the display coordinate. These coordinates are used to mark the click trace in the interactive interface, allowing users to see the specific location of the click on the image.

[0115] The percentage coordinates are calculated using "display coordinates / image display width and height × 100". This converts the click position into a percentage of the image's width and height, eliminating differences in image display size. For example, if the same image displays as 200px wide on a mobile phone and 400px wide on a computer, clicking the "100px width" position will be 50% on a mobile phone and 25% on a computer. The percentage coordinates can be standardized to the corresponding proportions, facilitating unified processing across devices by the server and RPA system.

[0116] The original pixel coordinates are calculated using the formula "display coordinates × (original image width and height / displayed image width and height)". The original image width and height are the pixel dimensions of the image itself, while the displayed width and height are the rendered dimensions of the image on the current device. The ratio of the two is the "scaling ratio". By scaling the display coordinates back to the original pixel coordinates, it is ensured that the click position corresponds to the original pixel position of the image on devices with different scaling ratios, avoiding RPA operation deviations caused by scaling. The Math.round function is used to round the calculation result, because the pixel coordinates must be integers.

[0117] In one possible embodiment, in step S2, the determination logic of the coordinate processing strategy is as follows: detect whether the src attribute of the verification code image element contains the string "geetest" or whether the client width of the image element imgElement.clientWidth is 426px. If either condition is met, it is determined to be a Geetest verification code, and the corresponding coordinate processing method for the Geetest verification code is adopted; if not, it is determined to be a general verification code, and the general coordinate processing method is adopted.

[0118] Specifically, Geetest CAPTCHAs, as a common third-party CAPTCHA, have their own specific technical characteristics. The src attribute (image source link) of the image element usually contains the "geetest" identifier string, and the standard display width of some Geetest CAPTCHAs is 426px. By detecting these two characteristics, Geetest CAPTCHAs can be quickly and accurately identified.

[0119] The reason for using a different coordinate processing method for GeeTest CAPTCHAs is that the interaction logic of GeeTest CAPTCHAs differs from that of general CAPTCHAs. The general processing method may lead to coordinate deviations. General CAPTCHAs, on the other hand, do not have special interaction rules, and the general processing method can meet the requirements. This differentiated processing can improve the coordinate processing accuracy of different types of CAPTCHAs and avoid operation failures caused by improper CAPTCHA type adaptation.

[0120] In one possible embodiment, in step S4, when operating on the interactive HTML interface, the interface displays the click marker and click number in real time. After manually clicking the OK button, the interactive HTML interface submits the percentage coordinate data to the Redis database. If it is necessary to refresh the verification code, the interactive HTML interface sends a refresh request and updates the verification code image, and re-records the operation data.

[0121] Specifically, real-time display of click markers (such as blue circles) and click numbers allows users to clearly see their operation trajectory, facilitating verification of correctness (e.g., when clicking specific text in sequence, the click order can be checked via the sequence number), reducing the probability of human error. Only percentage coordinate data is submitted to Redis, rather than all three types of coordinates, because percentage coordinates are sufficient for subsequent behavior mapping, reducing data transmission and storage volume and improving efficiency. Submitting data only after clicking the "OK" button allows users to confirm all operations, preventing irreversible errors. When the CAPTCHA expires (e.g., due to a click error requiring re-verification), the interface supports refresh requests. By sending a refresh command to the server to obtain a new CAPTCHA image and re-recording the operation data, the interface avoids the need for users to re-enter the interface due to expired CAPTCHAs, improving operational convenience.

[0122] In one possible embodiment, in step S5, the coordinate transformation formula for the behavior mapping is:

[0123] Let the percentage coordinates in the acquired manual operation data be the X-axis percentage coordinate result_data[i]["x"] of the i-th operation and the Y-axis percentage coordinate result_data[i]["y"] of the i-th operation;

[0124] The actual size information of the CAPTCHA element includes a width of bounding_box["width"] and a height of bounding_box["height"]. The actual pixel coordinates are denoted as x and y.

[0125] x=bounding_box["width"]×result_data[i]["x"] / 100

[0126] y=bounding_box["height"]×result_data[i]["y"] / 100

[0127] The preset time interval is 0.5 seconds, and can be manually adjusted within the range of 0.3-1 seconds according to the response characteristics of the verification code of different business platforms.

[0128] Specifically, the behavior mapping phase uses percentage coordinates to convert actual pixel coordinates because percentage coordinates are device-independent. The percentage coordinates submitted manually during the interaction can be used to calculate the actual pixel coordinates required for the RPA operation by analyzing the actual size of the CAPTCHA element obtained by the RPA. For example, if the manually submitted percentage coordinates are "50%X, 50%Y", and the actual width of the CAPTCHA is 400px and the height is 300px, then the actual pixel coordinates are "200pxX, 150pxY", ensuring that the RPA click position matches the intention of the manual operation. Figure 1 To.

[0129] The preset time interval of 0.5 seconds is based on research into human operating habits. The natural interval for continuous human clicks is usually 0.3-1 seconds. This interval can avoid being identified as a bot due to excessively fast operation, while ensuring process efficiency. The interval can be adjusted according to the characteristics of the business platform because different platforms have different anti-bot strategies. For example, some platforms have stricter requirements for the operation interval, which need to be extended to 1 second, while some platforms have no strict restrictions and can be shortened to 0.3 seconds to improve the compatibility of the solution.

[0130] Table 1 Application Deployment Scheme

[0131] Module technology stack Deployment Requirements Screenshot Engine Playwright Install client HTML generator Jinja2 Template Engine Stateless microservices Storage distribution Minio + Redis Cluster deployment Behavior mapping Playwright Install client

[0132] The application and deployment of the method in this embodiment are shown in Table 1. The screenshot engine uses the Playwright technology stack and requires client installation. Playwright can accurately simulate browser behavior to dynamically capture the CAPTCHA area, providing an image containing information such as the location of CAPTCHA elements for subsequent interactive interface generation.

[0133] The HTML generator utilizes the Jinja2 template engine and is deployed as a stateless microservice. Jinja2 can flexibly build interactive HTML interfaces that include CAPTCHA operation data structures and are compatible with cross-device access. Its stateless design facilitates rapid response to different interface generation requests. Storage and distribution combine the Minio and Redis technology stacks, employing a cluster deployment. Minio stores the generated interactive HTML files, ensuring reliable storage and fast access; Redis records human operation data in real time, supporting high-concurrency data read and write operations. Cluster deployment improves the stability and scalability of storage and data processing. Behavior mapping also uses the Playwright technology stack, requiring client installation. With Playwright, the human operation data recorded in Redis can be accurately mapped to CAPTCHA operations executable by the RPA system, ensuring the accuracy and consistency of operations.

[0134] In one application scenario of this invention, see... Figure 4 RPA triggers text click verification on the platform, see [link / reference]. Figure 5 The verification area (600×400px) is extracted to generate interactive HTML; the link is pushed to the customer service system's task queue; the customer service representative clicks the three characters [Tang], [Tai], and [Zong] in sequence; the click sequence is recorded.

[0135] `[{"x":76.76470588235294,"y":34.366925064599485},{"x":49.705882352941174,"y":55.81395348837209},{"x":27.35294117647059,"y":56.84754521963824}]`; RPA resumes execution, precisely clicking the corresponding coordinates.

[0136] In one application scenario of this invention, RPA triggers SMS verification in an OA system, see [link to relevant documentation]. Figure 6 Generate an HTML page containing input fields:

[0137] “ <input type＝"text"id＝"sms-code"placeholder＝"输入6位验证码">

[0138] <button onclick＝"submitcode()"> submit< / button> ".

[0139] See Figure 7 A text message will be sent to the customer's mobile phone:

[0140] "

Xiaodi Digital Employee

xxx

[0141] After the customer enters the verification code, RPA automatically fills it in and submits it.

[0142] Example 2

[0143] See Figure 8 Embodiment 2 of the present invention also provides an RPA CAPTCHA interaction system based on asynchronous human-computer collaboration, comprising:

[0144] The dynamic detection and capture module 100 is used by the RPA system to identify CAPTCHA events triggered in the business process in real time through DOM listening. When the CAPTCHA is detected, the headless browser is used to capture the image of the verification area, including the location information of the CAPTCHA element.

[0145] The interactive interface generation module 200 is used to construct a verification code operation data structure based on the captured verification area image, including an expiration timestamp, a unique task identifier, a verification code image URL, a server address, and set type coordinate data. The set type coordinate data includes original pixel coordinates, percentage coordinates, and display coordinates. The coordinates corresponding to the click operation are converted into set type coordinate data through a coordinate transformation algorithm, and a coordinate processing strategy is adopted according to the verification code type to adapt to the coordinate processing requirements of different verification codes, thereby generating an interactive HTML interface accessible across devices.

[0146] The asynchronous task distribution module 300 is used to generate a unique task ID including a timestamp and a UUID, upload the generated interactive HTML file to the storage system, associate the access path of the generated interactive HTML file with the task ID, and distribute the access path to the human operator through a specified channel.

[0147] The 400 manual collaboration interaction module is used by human operators to enter the interactive HTML interface through the access path to complete the verification operation. The interactive HTML interface records the operation data in real time and saves it to the Redis database.

[0148] The behavior mapping module 500 is used by the RPA system to recover the process after obtaining the operation data in the Redis database. If the actual size of the verification code element can be obtained, the percentage coordinates are converted into actual pixel coordinates. If it cannot be obtained, the original pixel coordinates are used directly. The verification operation is performed according to the coordinates and adjacent operations maintain a preset time interval.

[0149] The process recovery module 600 is used to allow the RPA system to continue executing subsequent automated business processes after the verification code operation is completed.

[0150] In this embodiment, in the dynamic detection and capture module 100, the headless browser is Playwright, and the resolution of the captured verification area image is dynamically adjusted within the range of 300×200px to 800×600px according to the actual size of the verification code area, so as to adapt to the verification code size of the set business platform.

[0151] In this embodiment, the interactive interface generation module obtains the boundary rectangle information of the verification code image element, denoted as imgRect. Let imgRect.left represent the X-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.top represent the Y-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.width represent the display width of the image element, and imgRect.height represent the display height of the image element.

[0152] In the interactive interface generation module 200, the client coordinates of the click event are obtained, denoted as event.clientX, which represents the X-axis coordinate of the click position in the client coordinate system; event.clientY, which represents the Y-axis coordinate of the click position in the client coordinate system; the natural resolution of the image element is denoted as img.naturalWidth, which represents the original width of the image; and img.naturalHeight, which represents the original height of the image.

[0153] The display coordinates are labeled displayX and displayY, and the calculation formula is:

[0154] displayX=event.clientX-imgRect.left

[0155] displayY=event.clientY-imgRect.top

[0156] Percentage coordinates are denoted as percentX and percentY, and the calculation formula is:

[0157] percentX=(displayX / imgRect.width)×100

[0158] percentY=(displayY / imgRect.height)×100

[0159] The original pixel coordinates are denoted as rawX and rawY, and the calculation formula is:

[0160] rawX=Math.round(displayX×(img.naturalWidth / imgRect.width))

[0161] rawY=Math.round(displayY×(img.naturalHeight / imgRect.height))

[0162] The final output consists of a set of coordinates: displayX, displayY, percentX, percentY, rawX, and rawY.

[0163] In this embodiment, the determination logic of the coordinate processing strategy in the interactive interface generation module 200 is as follows:

[0164] The code checks whether the src attribute of the CAPTCHA image element contains the string "geetest" or whether the client-side width of the image element, imgElement.clientWidth, is 426px. If either condition is met, it is determined to be a Geetest CAPTCHA and the corresponding coordinate processing method is used. If neither condition is met, it is determined to be a general CAPTCHA and the general coordinate processing method is used.

[0165] In this embodiment, in the human collaboration interaction module 400, when operating on the interactive HTML interface, the interface displays the click marker and click number in real time. After the user clicks the OK button, the interactive HTML interface submits the percentage coordinate data to the Redis database. If it is necessary to refresh the verification code, the interactive HTML interface sends a refresh request and updates the verification code image, and re-records the operation data.

[0166] In this embodiment, the coordinate transformation formula for behavior mapping in the behavior mapping module 500 is as follows:

[0167] Let the percentage coordinates in the acquired manual operation data be the X-axis percentage coordinate result_data[i]["x"] of the i-th operation and the Y-axis percentage coordinate result_data[i]["y"] of the i-th operation;

[0168] The actual size information of the CAPTCHA element includes a width of bounding_box["width"] and a height of bounding_box["height"]. The actual pixel coordinates are denoted as x and y.

[0169] x=bounding_box["width"]×result_data[i]["x"] / 100

[0170] y=bounding_box["height"]×result_data[i]["y"] / 100

[0171] The preset time interval is 0.5 seconds, and can be manually adjusted within the range of 0.3-1 seconds according to the response characteristics of the verification code of different business platforms.

[0172] It should be noted that the information interaction and execution process between the modules of the above system are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0173] Example 3

[0174] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code of an RPA CAPTCHA interaction method based on asynchronous human-computer collaboration. The program code includes instructions for executing an RPA CAPTCHA interaction method based on asynchronous human-computer collaboration as described in Embodiment 1 or any possible implementation thereof.

[0175] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives, SSDs).

[0176] Example 4

[0177] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0178] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can execute the asynchronous human-machine collaboration-based RPA verification code interaction method of Embodiment 1 or any possible implementation thereof by calling the program instructions.

[0179] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0180] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0181] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0182] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. An RPA-based CAPTCHA interaction method based on asynchronous human-computer collaboration, characterized in that, Includes the following steps: Dynamic detection and capture: The RPA system identifies CAPTCHA events triggered in the business process in real time through DOM listening. When a CAPTCHA is detected, a headless browser is used to capture the image of the verification area, including the location information of the CAPTCHA element. Interactive interface generation: Based on the captured verification area image, a verification code operation data structure is constructed, including an expiration timestamp, a unique task identifier, a verification code image URL, a server address, and set type coordinate data. The set type coordinate data includes original pixel coordinates, percentage coordinates, and display coordinates. The coordinates corresponding to the click operation are converted into set type coordinate data through a coordinate transformation algorithm. Based on the verification code type, a coordinate processing strategy is adopted to adapt to the coordinate processing requirements of different verification codes, generating an interactive HTML interface accessible across devices. Asynchronous task distribution: Generate a unique task ID including a timestamp and UUID, upload the generated interactive HTML file to the storage system, associate the access path of the generated interactive HTML file with the task ID, and distribute the access path to the human operator through a specified channel; Human collaborative interaction: Human operators access the interactive HTML interface through the access path to complete the verification operation. The interactive HTML interface records the operation data in real time and saves it to the Redis database. Behavior mapping: After the RPA system obtains the operation data from the Redis database, it resumes the process. If the actual size of the verification code element can be obtained, the percentage coordinates are converted into actual pixel coordinates. If it cannot be obtained, the original pixel coordinates are used directly. The verification operation is performed according to the coordinates and adjacent operations maintain a preset time interval. Process resumption: After the verification code operation is completed, the RPA system continues to execute subsequent automated business processes; The decision logic for the coordinate processing strategy is as follows: The code checks whether the src attribute of the CAPTCHA image element contains the string "geetest" or whether the client-side width of the image element, imgElement.clientWidth, is 426px. If either condition is met, it is determined to be a Geetest CAPTCHA and the corresponding coordinate processing method for Geetest CAPTCHAs is used. If neither condition is met, it is determined to be a general CAPTCHA and the general coordinate processing method is used.

2. The RPA CAPTCHA interaction method based on asynchronous human-computer collaboration according to claim 1, characterized in that, The headless browser is Playwright, and the resolution of the captured verification area image is dynamically adjusted within the range of 300×200px to 800×600px according to the actual size of the verification code area, in order to adapt to the verification code size of the set business platform.

3. The RPA CAPTCHA interaction method based on asynchronous human-computer collaboration according to claim 1, characterized in that, During the process of converting the coordinates corresponding to a click operation into coordinate data of a specified type using a coordinate transformation algorithm: Get the bounding rectangle information of the verification code image element, denoted as imgRect. Let imgRect.left represent the X-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.top represent the Y-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.width represent the display width of the image element, and imgRect.height represent the display height of the image element. Get the client coordinates of the click event, denoted as event.clientX, representing the X-axis coordinate of the click position in the client coordinate system; event.clientY, representing the Y-axis coordinate of the click position in the client coordinate system; denoted as img.naturalWidth, representing the original width of the image element; img.naturalHeight, representing the original height of the image. The display coordinates are labeled displayX and displayY, and the calculation formula is: displayX=event.clientX-imgRect.left displayY=event.clientY-imgRect.top Percentage coordinates are denoted as percentX and percentY, and the calculation formula is: percentX=(displayX / imgRect.width)×100 percentY=(displayY / imgRect.height)×100 The original pixel coordinates are denoted as rawX and rawY, and the calculation formula is: rawX=Math.round(displayX×(img.naturalWidth / imgRect.width)) rawY=Math.round(displayY×(img.naturalHeight / imgRect.height)) The final output consists of a coordinate set composed of displayX, displayY, percentX, percentY, rawX, and rawY.

4. The RPA CAPTCHA interaction method based on asynchronous human-computer collaboration according to claim 1, characterized in that, The coordinate transformation formula for the behavior mapping is as follows: Let the percentage coordinates in the acquired manual operation data be the X-axis percentage coordinate result_data[i]["x"] of the i-th operation and the Y-axis percentage coordinate result_data[i]["y"] of the i-th operation; The actual size information of the CAPTCHA element includes a width of bounding_box["width"] and a height of bounding_box["height"]. The actual pixel coordinates are denoted as x and y. x=bounding_box["width"]×result_data[i]["x"] / 100 y=bounding_box["height"]×result_data[i]["y"] / 100 The preset time interval is 0.5 seconds, and can be manually adjusted within the range of 0.3-1 seconds according to the response characteristics of the verification code of different business platforms.

5. The RPA CAPTCHA interaction method based on asynchronous human-computer collaboration according to claim 1, characterized in that, When operating on the interactive HTML interface, the interface displays the click marker and click number in real time. After the user clicks the OK button, the interactive HTML interface submits the percentage coordinate data to the Redis database. If the CAPTCHA needs to be refreshed, the interactive HTML interface sends a refresh request and updates the CAPTCHA image, and re-records the operation data.

6. An RPA-based CAPTCHA interaction system based on asynchronous human-computer collaboration, characterized in that, include: The dynamic detection and capture module is used by RPA systems to identify CAPTCHA events triggered in business processes in real time through DOM listening. When a CAPTCHA is detected, a headless browser is used to capture the image of the verification area, including the location information of the CAPTCHA element. The interactive interface generation module is used to construct a verification code operation data structure based on the captured verification area image. This structure includes an expiration timestamp, a unique task identifier, a verification code image URL, a server address, and set type coordinate data. The set type coordinate data includes original pixel coordinates, percentage coordinates, and display coordinates. The module converts the coordinates corresponding to the click operation into set type coordinate data using a coordinate transformation algorithm. It also adapts the coordinate processing requirements of different verification codes according to the verification code type using a coordinate processing strategy, thereby generating an interactive HTML interface accessible across devices. The asynchronous task distribution module is used to generate a unique task ID including a timestamp and a UUID, upload the generated interactive HTML file to the storage system, associate the access path of the generated interactive HTML file with the task ID, and distribute the access path to the human operator through a specified channel. The human collaboration and interaction module is used by human operators to access the interactive HTML interface through the access path to complete the verification operation. The interactive HTML interface records the operation data in real time and saves it to the Redis database. The behavior mapping module is used by the RPA system to recover the process after obtaining the operation data from the Redis database. If the actual size of the verification code element can be obtained, the percentage coordinates are converted into actual pixel coordinates. If it cannot be obtained, the original pixel coordinates are used directly. The verification operation is performed according to the coordinates and adjacent operations maintain a preset time interval. The process recovery module is used to allow the RPA system to continue executing subsequent automated business processes after the verification code operation is completed. In the interactive interface generation module, the determination logic for the coordinate processing strategy is as follows: The code checks whether the src attribute of the CAPTCHA image element contains the string "geetest" or whether the client-side width of the image element, imgElement.clientWidth, is 426px. If either condition is met, it is determined to be a Geetest CAPTCHA and the corresponding coordinate processing method for Geetest CAPTCHAs is used. If neither condition is met, it is determined to be a general CAPTCHA and the general coordinate processing method is used.

7. The RPA CAPTCHA interaction system based on asynchronous human-computer collaboration according to claim 6, characterized in that, In the dynamic detection and capture module, the headless browser is Playwright, and the resolution of the captured verification area image is dynamically adjusted within the range of 300×200px to 800×600px according to the actual size of the verification code area, so as to adapt to the verification code size of the set business platform.

8. The RPA CAPTCHA interaction system based on asynchronous human-computer collaboration according to claim 6, characterized in that, In the interactive interface generation module, the boundary rectangle information of the verification code image element is obtained, denoted as imgRect. Let imgRect.left represent the X-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.top represent the Y-axis coordinate of the top left corner of the image element in the client coordinate system, imgRect.width represent the display width of the image element, and imgRect.height represent the display height of the image element. In the interactive interface generation module, the client coordinates of the click event are obtained, denoted as event.clientX, which represents the X-axis coordinate of the click position in the client coordinate system; `event.clientY` represents the Y-coordinate of the click location in the client coordinate system. `img.naturalWidth` represents the original width of the image element, and `img.naturalHeight` represents the original height of the image. The display coordinates are labeled displayX and displayY, and the calculation formula is: displayX=event.clientX-imgRect.left displayY=event.clientY-imgRect.top Percentage coordinates are denoted as percentX and percentY, and the calculation formula is: percentX=(displayX / imgRect.width)×100 percentY=(displayY / imgRect.height)×100 The original pixel coordinates are denoted as rawX and rawY, and the calculation formula is: rawX=Math.round(displayX×(img.naturalWidth / imgRect.width)) rawY=Math.round(displayY×(img.naturalHeight / imgRect.height)) The final output consists of a coordinate set composed of displayX, displayY, percentX, percentY, rawX, and rawY.

9. The RPA CAPTCHA interaction system based on asynchronous human-computer collaboration according to claim 6, characterized in that, In the behavior mapping module, the coordinate transformation formula for behavior mapping is as follows: Let the percentage coordinates in the acquired manual operation data be the X-axis percentage coordinate result_data[i]["x"] of the i-th operation and the Y-axis percentage coordinate result_data[i]["y"] of the i-th operation; The actual size information of the CAPTCHA element includes a width of bounding_box["width"] and a height of bounding_box["height"]. The actual pixel coordinates are denoted as x and y. x=bounding_box["width"]×result_data[i]["x"] / 100 y=bounding_box["height"]×result_data[i]["y"] / 100 The preset time interval is 0.5 seconds, and can be manually adjusted within the range of 0.3-1 seconds according to the response characteristics of the verification code of different business platforms; In the aforementioned human-assisted interactive module, during operation on the interactive HTML interface, the interface displays the click markers and click numbers in real time. After the user clicks the "OK" button, the interactive HTML interface submits the percentage coordinate data to the Redis database. If the verification code needs to be refreshed, the interactive HTML interface sends a refresh request and updates the verification code image, and re-records the operation data.

Images